Built with Retool
Financial & Usage Analytics for YC-backed healthcare startup
Toolshed (Data, Automation, AI Agents, Framer, Retool)
Automated Financial & Usage Analytics for YC-backed healthcare startup
Goal
To build a comprehensive and robust data system for Sully.ai that replaces manual billing and management workflows with a centralized, automated platform. The objective was to create a "single source of truth" for financial reporting (MRR/ARR), track granular agent usage metrics, and enable proactive operational management through automated alerts and data-driven dashboards.
Our Contribution
Toolshed implemented a multi-phased technological transformation to streamline Sully.ai's core business operations:
Unified Data Warehousing & ETL Pipelines:
Designed and built a modular ELT pipeline using Airbyte and Fivetran to consolidate disparate data from HubSpot and Stripe into Google BigQuery.
Developed custom production-ready Python scripts to sync data from Firestore and Google Cloud Storage (GCS) into BigQuery, including automated schema evolution and dynamic data flattening.
Intelligent Financial & Billing Dashboards:
Engineered a sophisticated Retool suite featuring a Main Billing Dashboard with an "adjustment layer," allowing for manual overrides on top of source data while maintaining a permanent audit trail.
Implemented automated logic for prorated revenue distribution, monthly MRR finalization, and advanced filters for organization-level financial tracking.
Granular Usage Analytics & Monitoring:
Developed detailed Agent Usage Metrics Dashboards (Scribe, Medical Coding, Nurse, Research, and Receptionist agents) providing real-time visualization of total calls, duration, and specific task counts.
Built an API Trends dashboard to monitor notes generation, request volume, error rates, and response time percentiles (p90, p99).
Proactive Operations with Slack Integration:
Deployed a robust Slack Alerting System to monitor ETL script failures, missing files, and Airbyte sync errors.
Implemented Anomaly Detection logic that triggers alerts when critical usage metrics (like Scribe Duration or Lab Counts) drop more than 30% compared to the 7-day average.
Client-Facing Data Segmentation:
Converted internal dashboards into public-facing secure apps using unique identifiers (UUIDs) to enable external clients to view their specific data securely via URL parameters.
Overview of Project Success
By centralizing fragmented data and automating manual tracking, Toolshed transformed Sully.ai’s financial operations into a scalable, real-time ecosystem. The platform provides unmatched visibility into MRR/ARR growth, agent-specific utilization, and system health. The addition of automated audit trails and proactive anomaly alerts has significantly reduced manual overhead, ensuring data integrity and enabling leadership to make informed, data-driven decisions at scale.
“Toolshed's thoroughness, thoughtfulness, and ethicality are impressive.” – Yishi, CFO of Sully
Like this project
Posted Mar 26, 2026
Sully.ai’s financial operations were transformed into a scalable, real-time ecosystem. The platform provides visibility into MRR/ARR growth, and system health.
Likes
0
Views
0
Timeline
Jun 1, 2025 - Ongoing
Clients
Sully.ai
