Automated Data Pipeline & Cloud Publishing Workflow

PROJECT OVERVIEW I designed an automated data pipeline for PewStats PRO that updates league data locally, validates the result, publishes the database to cloud storage, and allows the live application to consume the newest approved version.

THE CHALLENGE A manual update process creates two risks: incomplete league data and accidental publication of a broken database. The workflow needed to be simple enough to run from Windows with one command, but strict enough to stop when any league failed. It also needed to keep cloud credentials outside the public GitHub repository.

WORKFLOW DESIGN The update command runs each supported league in a defined sequence and reports newly inserted matches, updated records, and failures. All writers use an explicit writable database path, avoiding the read-only database issue that can appear when packaged folders or cloud deployments are treated as the data source.

After the league jobs finish, the publisher verifies that every update succeeded. If one league fails, publication is cancelled and the live site continues using the previous known-good database. This fail-safe behavior protects users from partial rounds and corrupted releases.

CLOUD PUBLISHING The approved SQLite database is uploaded to Cloudflare R2. Access credentials are stored locally and in the hosting platform's protected secrets, never committed to source control. The Streamlit application downloads the published database when needed and uses caching rules that balance performance with fresh data.

DEVELOPER EXPERIENCE I wrapped setup, league updates, validation, and publishing in clear Windows scripts so the workflow can be operated without remembering several Python commands. Console output explains each stage and distinguishes successful inserts, updates, and blocked releases.

QUALITY CONTROLS I tested the pipeline with successful runs, partial failures, repeated updates, missing credentials, read-only database conditions, and production refresh behavior. The release process was designed to be recoverable: a failed update does not overwrite the last working cloud database.

OUTCOME The result is a repeatable operational workflow that converts a multi-step maintenance task into one controlled command. It improves reliability, keeps secrets private, and gives every data release an explicit validation gate before it reaches users.

LIVE APPLICATION https://pewstats-pro.streamlit.app/