BatchQ Real-Time AI Data Pipeline Automation

Project Overview For this project, I built a fast and reliable backend automation pipeline for BatchQ. The system pulls large batches of competitor URLs and product names directly from a Google Sheet, sends them through a custom real-time AI research workflow, processes responses in parallel, and writes the enriched insights back into the sheet automatically.

BatchQ tracks thousands of competitor updates, product changes, and pricing movements on a daily basis. Handling this manually was time-consuming and inefficient. Standard AI tools like ChatGPT also weren’t suitable because they don’t support real-time web research or large-scale batch processing without hitting rate limits or breaking during execution.

Using BuildShip’s backend automation system, I designed a scalable and production-ready workflow with clearly structured stages:

Dynamic Data Ingestion The workflow starts by pulling raw competitor and product data from a connected Google Sheet.

Batch Processing & Loop Control To keep the system stable and prevent API overload, I implemented batch processing in groups of 10 items, running through a controlled loop for smooth execution.

Custom AI Research Node (BatchQ Integration) Since default blocks weren’t flexible enough, I built a custom JavaScript/TypeScript node. This directly connects to Perplexity’s API and uses the Llama 3 Sonar Large model for real-time web research, securely managing API keys and request handling.

Rate Control for Stability A controlled delay between API calls was added to prevent rate-limit errors and ensure consistent performance across large datasets.

Data Cleanup & Write-Back The AI responses are then flattened and structured into clean outputs before being automatically written back into the Google Sheet in the correct columns.

Fully automated competitor research pipeline with zero manual effort

Real-time web insights instead of outdated AI-generated assumptions

Stable processing of thousands of rows without failures or rate-limit issues

Significant reduction in weekly operational workload for the BatchQ team