Built a Cloud Data Lakehouse for Enhanced Data Governance in the Enterprise

Overview

1. The client wanted to create a centralized repository for all organizational data.
2. The goal was to make the data accessible to everyone and create an organized data source.
3. The client wanted continuous data to refresh the repository.
4. The main objective was to convert raw data into actionable insights for better decision-making.
5. The client wanted to build a data lake-house architecture on the cloud to achieve this.

Challenges

The main challenges of the project were:

1. Collecting and processing data from various sources in a raw format.
2. Building a scalable infrastructure to handle large amounts of data.
3. Integrating different technologies to build a unified system.
4. Providing easy access to data for end-users while ensuring data security and governance.

Deliverables

The following items were provided as deliverables:

1. Data Collection Layer (Data Source+ Transfer Layer)
2. Data Ingestion Layer
3. Storage and Metadata processing layer
4. Metadata Processing Layer
5. Cataloging layer
6. BI/ Reporting Layer

I followed a phased approach to building an enterprise data lake on the cloud to improve data governance for my client. The five stages included data collection, ingestion, storage and metadata processing, cataloging, and BI/Reporting. Using AWS DataSync Agent, Airflow, and Hive Metastore, I built fully functional data lakehouses for two sub-verticals and provided a data cataloging tool and BI endpoint to consume the data. This architecture provided a centralized repository for all organization data, making it accessible to everyone and helping me convert raw data into actionable insights for better decision-making.