The Houston Food Bank, through warehouse operations and work in the field, generates thousands of data points every single day. These data sets lived in different software systems across the organization that were difficult to access and existed in silos.
To answer important questions, the Houston Food Bank needed to connect, aggregate, and store these different data sets efficiently and reliably. Without a centralized system in place, the Houston Food Bank's data team was not able to efficiently answer questions that required multiple data sets across the organization.
Our team designed and developed Lamdat, a custom Python library, to extract, transform, and load (ETL) data reliably on Amazon Web Services. By leveraging AWS Lambda we were able to engineer a centralized and monitored data pipeline in the cloud. The system was deployed using version control on Gitlab to ensure continuous integration and continuous deployment.
To store the data reliably, we assisted the Houston Food Bank in developing their cloud data warehouse on Amazon Redshift. Moksha worked with the Houston Food Bank data team to design the data model for key data sources and generated best practices for maintaining the system.
By centralizing the Food Bank's data and providing an organized method to store and access it, we were able to increase the efficiency of the data team and drive impact across the organization.
This system can be employed by any organization that wants to automate, organize, and monitor their data processes,