Challenge

The analytics dashboards visualizes large amounts of data while enabling clients to freely adjust each visualization with multiple options. Working with the large amounts of data and custom user requests was causing a slow and unproductive experience

Outcomes

Reduced visualization update time to take only ~5% of the time it used to take with less than 1 second to respond to a change in any or all of the features of the visualization

Increased customer satisfaction with testimonials of "revolutionary experience improvement" than the tools used in the market

The Work

Identify the main challenges:

The processing was too complicated for the backend to run on demand given the large data size and the need for immediate and interactive response

The network time to transfer the data before final processing and transformation

Investigate possible solutions

Caching - not too useful in this case since the major requests are going to be first-of-a-kind requests given how flexible can the user customize the visualizations

Precaching - The solution to go withm but needs dedicated planning to finetune it to the use case

Tech Strategy

Identify the main and common features of the data

Create a precaching strategy that efficiently targets the common queried features of the data

Leave the very minimal transofrmation to the backend

Compress the precached data to minimize both the storage usage and the network time

Generate a querying system to be used by the frontend, the backend, and the generated precache records

Update all three components to communicate and work together seemlesly

Like this project

Posted Oct 26, 2024

Designed and implemented a full data visualization solution from the data pipeline and storage to the backend to the frontend to get a lightning-fast experience

Likes

Views

Tags

Data Visualizer

Backend Engineer

Frontend Engineer

AWS

Python

React