Crux data integration

Crux Data Platform Historically Crux has been a managed services company helping Fortune 500 companies solve their most complex external data problems. Everything from data ingestion to data delivery; Crux helps large organizations scale their most critical data delivery, operations, and transformation needs.

As a company, we want to package up a complete self-service platform that our customers can use in-house rather than using our managed service.

Built for Data Engineers; The Crux Platform serves anyone who wants to build a data product from scratch to get continuous delivery of the data how and when they want it.

TL;DR

“Think of Crux as the FedEx of data...for now.”

UNDERSTANDING THE BUSINESS & MARKET - 1

Research & Requirements This was a new domain for me to learn

What the heck is a data product Why do people want to build them How is this product going to be licensed Is this a multi-tenant product Will we ever have more than 1 SasS offering?

I relentlessly meet with PM’s, Product Marketing, and Engineers trying to deepen my knowledge of the space/ domain. I wanted to understand what our North Star goals were for this product, and how we plan to differentiate ourselves in this growing market of big data. I was researching companies like Firetrap, Rivery, Hevo, Stitch...and the list goes on. I would spend my evenings drinking lagers and watching keynote presentations on YouTube about data integration.

Next, I picked up the product requirements document and dove in head first.... All 132 pages.

As a company, one of our 2022 business goals is to package up a complete self-service product that our customers can license and use rather than using our managed service, or alongside our managed service. The managed services side of the business will still be a large portion of our revenue, but the new Crux SaaS offering will allow our clients to build data products and data pipelines internally with their own team of data engineers.

We want to put the power of flexibility and time-to-value in the hands of our customers directly. Built for Data Engineers; the Crux SaaS product serves any team or individual that wants to build a data product from scratch.

Users should be able to

Connect to a data source Identify/select the file patterns they nee View the extraction schedule from the supplie Configure the (output) schema, and send data sets to a destination source on a recurring basis Triage and errors at any point in the process (observability/monitoring Add users, create groups, define roles for their organizations

UNDERSTANDING THE BUSINESS & MARKET - 2

What are some of the market problems we're trying to solve with the new Crux platform? Data is what is driving the world. 97.2% of organizations are now investing not only in AI and big data but leveraging external data to make critical decisions in every facet of their businesses

Data is everywhere - but one thing is for sure, data is messy.

The need to democratize data; today, almost every person in every function needs to use data whether they are in marketing, sales, logistics, operations, HR, finance, product, and especially design.

Music, Shows, and Movies dat Healthcare and Medical Services dat Shopping and Marketing dat Travel and Transportation dat Public Policy and Safety News and Information Education and Employment Artificial Intelligence data

UNDERSTANDING THE USERS

Wtf is...Data Engineering !@?@#? Before working on this project I had no clue what Data Engineers did, no joke, ZERO clue. Now being tasked with designing a product for Data Engineers, it was time to put my thinking cap on. Since we have a big managed services team at Crux I would meet with these Data Engineers daily to observe how they did their jobs. I was able to set up meetings with these folks afterward and share my wireframes - internal user testing.

I asked them questions... a lot of them.

UNDERSTANDING A DATA PRODUCT

Wtf is...a Data Product What the heck is a data product What does a Data Product do How can someone create a Data Product What can people do with a Data Product?

In this section, I am going to attempt to answer some of these questions. at Nexla defines a Data Product well and I have outlined some of his explanations on this below. I have added some real-world examples to help you better understand.

Data Products, much like other products in our day-to-day life, present a ready-to-use entity that is comprised of many different things.

Let's use a car as an example - The end product is a functional, and reliable vehicle, but the car is comprised of many different things, from an engine to a steering wheel to side mirrors and tires. The overall function of the car does not work if any of these key parts are broken, or missing.

Cars come in many different models, shapes, sizes, and features, and with a wide array of functions. One thing remains true in the end, the vehicle needs to drive once it leaves the factory.

Just like when a functioning vehicle leaves the assembly line for its final destination, the same is true for a Data Product. It needs to work as intended.

In order to create a fully functioning data product - it needs to be comprised of different things just like a car.

Think of these as ingredients to a dinner recipe, or the parts needed to build a car. If you don't have all the ingredients, the recipe turns out wrong, overcooked/undercooked, or just flat-out not enjoyable. The same is true for a data product.

Saket Saurabh

INGREDIENTS LIST

Data Product Ingredients Raw data that comes from any source (real-time, API, stream, or batch) Consistent interface to all types of data files Additional Metadata - Schema, description, data characteristics, easy-to-comprehend data samples Governance - Who created the data product, when was it created, and version history. Quality metrics - Failed deliveries, wrong extraction schedule, and general observability and/or monitoring data product health Logic - Transformation code that modifies, or creates the data product. Also, validation rules ensure that the data product can be trusted.

Just like the tires on a car, you need them to complete the vehicle's intended function. A data product should be comprised of the ingredients listed above to serve as a functioning, actionable entity - Consistency to data access, governance, documentation, discovery, and also the delivery of ready-to-use, actionable data.

IS THIS STEP 1? I DON'T EVEN KNOW ANYMORE.

User flows As I started to feel more confident with the business direction and who we were building this product for it was time to dive into some user flows/journeys/whatever the heck you want to call them.

The process was very product-driven. Reviewing timelines and roadmaps to determine what feature sets we would be wanting to market with and which ones we’re really not high up on the priority list.

Doing these exercises with everyone was really helpful for me; It gave me a pulse on what I needed to have prioritized based on what the outcomes were for the business.

REVIEW USER FLOWS WITH/ STAKEHOLDERS AND ENGINEERING TEAMS.

Stakeholder reviews After focusing on the data product creation workflow it was important for us to all be aligned on a few things

The data product creation flow Support from the backend teams and what net-new services were needed to support new functionality Primary and secondary navigation Different entry points for managing source/destination connections What happens when a user chooses to “test” a data product?

INTERNAL VALIDATION AND USER INTERVIEWS.

1:1 Zoom calls to review flows and collect feedback. We needed to validate some of our assumptions so we set out to meet with a team of our internal Data Engineers to make sure that we were headed in the right direction. Since our internal teams would be part of our beta users after our soft launch we wanted to make sure that what we were thinking, met the needs of them. In turn, we were hoping that the external users (other Data Engineers) would be willing to adopt our platform to help them solve their most critical data delivery needs.

WIREFREAMES.

Rounds, and rounds, and rounds of wireframes and feedback. It was time to start taking some of the flows and creating some sketches, and wireframes so that we could start validating prototypes with our users. I created a wireframe component kit to help with velocity duruing this phase of the process. Below is a look quick at where I started.

WIREFRAME FEEDBACK AND ITERATIONS

As I validated these wireframes with our dev teams and Data Engineering teams I had to make some changes. After reviewing these wireframes with some internal folks it was clear that we needed a lot of back-end support for a lot of these new new capabilities. We worked together to stack ranck them in order of priority and clear objectives/table stakes. What can we abolutely NOT go to market without.

We decided that it was time to start taking the data product creation flow into hi-fidelity.

VALIDATION CONTINUES

Validation rounds continue: Meeting with Crux Data Engineers 1:1 viz zoom to validate our assumptions, get a better understanding of their current workflow of building and deploying data products, what tools they use, why they use them, and how beneficial those tools are.

Validation with engineering, back end, front end, and microservices teams to allow them to provide any feedback or feasibility concerns early on, prior to designing anything relatively high-fidelity.

These meetings usually consist of design sharing concepts through user flows or some lo-fi wireframe seen below Fun FigJam sessions with PM and Engineering. During these sessions everyone participates, everyone drops stickies, objects, stickers, and notes.

We all want to get to the core of these problems together so these sessions help us do just that. It's important to have fun when working through tough problems as one team.

ARE WE EVER REALLY “DONE”? NAH

Into the Future we go! We are excited to make this product even better in future releases, but for now, it's time to add some polish, validate some new ideas and keep moving this forward.

One thing I forgot to mention is we went from

With a product as complex as this one, we would have not been able to do this without everyone working their asses off and working as a unified team. Everyone from FE, to BE folks, API teams, SE's, DE's, Content, PM, and even the executive teams.

Shoutout to all of you, you know who you are!

As always hit me up if you have questions about this project or just want to chat!

Don't forget to connect with me on LinkedIn and if you want a closer look into my personal life, check out my Instagram @thereal_johncarter

a concept, a PRD, to a fully functioning product in less than 6 months. With users building data products from scratch!

OUR DESIGN SYSTEM

FlowBite Since we were limited in our resources we needed to use something out of the box for our design system.

We were not able to build a system from scratch given our business timelines and smaller-sized design/FE teams.

We ended up using the FlowBite design system - Won't lie, Not the greatest but it had what we needed to get something going FAST.

There were a TON of things that needed to be designed that were brand new, but did my best to make sure that they were sticking to the styling and paradigms that were being used by the dev teams to make sure we were not wasting any time.

Velocity and time to market were key goals here.

Below are some examples of one-off components that need to be built.

Connecting to a data source

Identifying patterns

Extraction schedule

Input schema / Output schema

Connecting to a destination

Observability / Monitoring