Projects using PySpark in ColumbusProjects using PySpark in Columbus
Projects
People
Cover image for I engineered a large-scale package-network
I engineered a large-scale package-network intelligence pipeline using Azure Databricks and PySpark. The system joined approximately 4.5 billion historical records and transformed them into reusable fingerprint signals covering package movement and network behavior. I developed time-windowed feature logic using native Spark expressions rather than Python UDFs, preserving distributed performance at scale. The pipeline produced an approximately 200-million-row fingerprint dataset and a 22-million-row operational output that classified package activity along a cold-to-hot temperature scale. The resulting signals supported downstream monitoring, prioritization, and operational analysis. The architecture shown here is a sanitized representation; client identifiers, proprietary schemas, infrastructure paths, and business rules have been removed.
1
199