← Back to Projects

Building the Houston Food Bank's Data Pipeline

Houston Food Bank
The Lamdat library is a custom Python framework to reliably and cost-effectively centralize & monitor Houston Food Bank's operational data to drive organization-wide insights.
No items found.

Problem

The Houston Food Bank, through warehouse operations and work in the field, generates thousands of data points every single day. These data sets lived in different software systems across the organization that were difficult to access and existed in silos.

To answer important questions, the Houston Food Bank needed to connect, aggregate, and store these different data sets efficiently and reliably. Without a centralized system in place, the Houston Food Bank's data team was not able to efficiently answer questions that required multiple data sets across the organization.

Solution

Our team designed and developed Lamdat, a custom Python library, to extract, transform, and load (ETL) data reliably on Amazon Web Services. By leveraging AWS Lambda we were able to engineer a centralized and monitored data pipeline in the cloud. The system was deployed using version control on Gitlab to ensure continuous integration and continuous deployment.

To store the data reliably, we assisted the Houston Food Bank in developing their cloud data warehouse on Amazon Redshift. Moksha worked with the Houston Food Bank data team to design the data model for key data sources and generated best practices for maintaining the system.

Impact

By centralizing the Food Bank's data and providing an organized method to store and access it, we were able to increase the efficiency of the data team and drive impact across the organization.

This system can be employed by any organization that wants to automate, organize, and monitor their data processes,

  • Forming the essential foundation to build reports, dashboards, and other visualizations that can amplify the impact of your work
  • Forming a reliable V12 engine that ingests data and gets it where it needs to be