Scalable Data Visualization Solution for Data-Intensive Web App
Challenge
The analytics dashboards visualizes large amounts of data while enabling clients to freely adjust each visualization with multiple options. Working with the large amounts of data and custom user requests was causing a slow and unproductive experience
Outcomes
Reduced visualization update time to take only ~5% of the time it used to take with less than 1 second to respond to a change in any or all of the features of the visualization
Increased customer satisfaction with testimonials of "revolutionary experience improvement" than the tools used in the market
The Work
Identify the main challenges:
The processing was too complicated for the backend to run on demand given the large data size and the need for immediate and interactive response
The network time to transfer the data before final processing and transformation
Investigate possible solutions
Caching - not too useful in this case since the major requests are going to be first-of-a-kind requests given how flexible can the user customize the visualizations
Precaching - The solution to go withm but needs dedicated planning to finetune it to the use case
Tech Strategy
Identify the main and common features of the data
Create a precaching strategy that efficiently targets the common queried features of the data
Leave the very minimal transofrmation to the backend
Compress the precached data to minimize both the storage usage and the network time
Generate a querying system to be used by the frontend, the backend, and the generated precache records
Update all three components to communicate and work together seemlesly
Like this project
0
Posted Oct 26, 2024
Designed and implemented a full data visualization solution from the data pipeline and storage to the backend to the frontend to get a lightning-fast experience
Likes
0
Views
2
End-to-End Scalable Web Application
Timeseries Data Pipeline Redesign