Project: Datahub Deployment and Metadata Automation
Scope:
Implement and deploy Datahub on Google Cloud Platform (GCP)
Develop a metadata intake and event automation system
Create a notification system for metadata events
Set up infrastructure, testing, and documentation
Description:
The deliver for this project was a automated deployment of Datahub, a metadata platform, on GCP instrumented for metadata intake and event processing. The solution includes a multi-tenant setup, automated metadata event handling, and a notification system for users. The project leverages GCP's Kubernetes engine, Datahub's Python SDK with REST Emitter, n8n for workflow automation, and Twilio for notifications.
Key Components:
GCP Infrastructure Setup
Configure GCP project, Kubernetes cluster, and CI/CD pipeline
Datahub Deployment
Deploy multi-tenant Datahub using Helm charts on GCP Kubernetes
Metadata Intake and Event Automation
Develop multi-tenant intake form and implement REST Emitter for metadata events
Event Reaction and Notification System
Set up n8n workflows and Twilio integration for event processing and notification
Outcomes:
Fully functional Datahub deployment on GCP
Automated metadata intake and event processing system
User-friendly notification system for metadata events
Comprehensive documentation and knowledge transfer to the client's team
This project implements a robust, scalable solution for metadata management and automation, enhancing data governance and visibility across the organization.
