Data Integration / ETL

Perifit had data coming periodically from their warehouse into a MySQL database. And this data needed to be processed and transmitted into their analytics platform ( Intercom and Mixpanel ). Their preference was to use Segment Functions, a feature [ from Segment ] that enables functionality to be written on incoming and outgoing data passing through Segment.

We designed and built an intelligent ETL data pipeline:

A MySQL database running on AWS EC2: This database was updated on a specific cycle. Each time, only new data is processed and each record is formatted into a HTTP request and sent to Segment. A CRON task was setup to trigger a database check daily and start the process if the database had changed since the last update. Also, PM2 was used to run the Node.js operation for redundancy reasons.

Multiple Segment functions: On Segment, we wrote multiple functions to further process the incoming data and process it for onward delivery to both Intercom and Mixpanel via their respective REST APIs. Intercom and Mixpanel have very different systems and rules (especially regarding rate limits) and we had to use batch-friendly endpoints and multiple workarounds to overcome the challenge.

Once setup and running, their data was able to flow seamlessly from their offsite VM directly into the analytics platforms where they needed it.