Hybrid Data Platform and Integration Hub Development

The client, a leading EU private equity and wealth management firm, needed a modern data platform to address legacy silos, manual processes, and reporting inefficiencies. A hybrid data warehouse using SQL Server (on-prem) and Snowflake (cloud) was built to support legacy compatibility and cloud-native scalability. Additionally, a centralized integration hub on AWS to unify disparate enterprise systems, replacing point-to-point workflows with scalable, event-driven architecture. Leveraged API Gateway, Lambda, Step Functions, SQS, EventBridge, and Cognito etc. For DG use case refer to details

Client is a well-known name in private equity fund administration and wealth management services in the EU region.

Inorganic growth over the years had resulted in siloed legacy systems and fragmented reporting, leading to high processing overheads and error-prone manual tasks. To address this, a modern data platform was envisioned to consolidate data sources, streamline reporting, and to enable monetization of data through enhanced offerings like Platform-as-a-Service, Data-as-a-Service, and Data Warehouse-as-a-Service.

A hybrid data warehouse solution was developed using MS SQL Server (on-premise) and Snowflake (cloud). This architecture ensured backward compatibility with existing reporting while providing flexibility for clients to integrate via on-prem or cloud technologies. Snowflake features like tasks, streams, stored procedures, and transient tables were leveraged for cloud-based efficiencies.

A parallel governance stream initiated using Collibra, needed to complimented with the new data platform details. The objective was to ensure data discoverability, ownership, and compliance standards across the organization.

Technical Sub Communities were created for representing the data platforms, reporting groups

Domains such as Logical Models, DW Policy, Data Quality Dimensions and Tech Asset Domain were structured to organize assets meaningfully.

Datasets such as Client Data, Transaction Data, and Performance Metrics were structured to organize assets meaningfully.

A unified Data Catalog was established by ingesting metadata from Snowflake and SQL Server, improving data accessibility.

Policies, business glossary terms and Data Quality Rules were defined and linked to corresponding datasets.

Published Collibra assets included:

Additionally, an integration hub was developed to serve as a unified interface between disparate systems. It eliminated point-to-point integrations, reduced turnaround time for new workflows, and supported a microservices architecture. This hub was built using AWS cloud services including API Gateway, Lambda, Step functions, SQS, Event Bridge, and KDS, enabling both batch and streaming, synchronous/asynchronous integrations.

Snowflake Enterprise Edition, MS SQL Server 2016, Mosaic ETL Tool, Mosaic Catalog, Azure Cloud Services, Jira, AWS Services

Delivery Manager overseeing technical implementation

Managed scope, timeline, cost, communication, and quality

Liaised with stakeholders and ensured successful end-to-end delivery