Data Plumbing Optimisation

As a Software Technical Specialist at my company, I was assigned to investigate runtime performance improvements to a data processing software built using SAS and Java. Each week the software would be used to generate a variety of canned weekly reports for a major client. Our report build used to take 20 hrs ideally. And a year into the Operations, the runtime bloated to 50-60 hrs jeopardising the timeliness of a Monday morning update. I used pandas, SQL, SAS and Java to trace and debug the runtime and bring it down to 35 hrs i.e. almost a 40% improvement.