Local City Bus Lines Monitoring and Stops Identification

I recently began using buses almost exclusively to save money on transportation. This has the added benefit of being more environmentally friendly, and it allows me to spend more time thinking during my commute.

However, our local public transportation service provided no clear or simple way to identify bus routes. Commuters had to check with the bus operator to see if a specific line served their destination, which frequently resulted in unnecessary boarding and confusion.

The establishment recently launched a web platform that allowed users to view real-time bus GPS locations on a map. But the features were limited; it did not include route lines nor bus stops.

To mend this, I took it upon myself to write a script that monitors the site round the day, and then analyze the collected data to infer bus routes and stops.

After inspecting network traffic, I determined two API endpoints; /api/devices and /api/positions. When queried with the right credentials, these endpoints provide real-time bus positions that are then displayed by the site. Access required a valid JSESSIONID cookie, which was set by a /api/server request that needed a Referer header that included a specific token from the client. A subsequent request to /api/session?token=... validates the session.

Once this handshake done, the scraper can replicate the website's authentication flow and periodically fetch bus and position data from the API endpoints. When responses are received, each pair is bundled together into a single snapshot.

The collected data was first converted to GPX format, with each bus line saved as a separate GPX file and visualized on a map. The itineraries appeared immediately, but the paths contained numerous stray lines caused by buses going offline or deviating from their routes. After cleaning the data to remove these inconsistencies, the next logical step was to identify the bus stops.

I began by creating a heatmap that displayed all the recorded positions, then excluded low-density regions to retain only areas where recurring stops occur. Then, I opted for the use of DBSCAN, a clustering algorithm, which was able to identify most of the bus stops, although there were false positives at the red lights and during times of heavy traffic. Resolving this required manual adjustment.

And just like that, a bit of scraping, cleanup, and plotting was all it took to map all of the bus lines and stops for all the city to use as a reference.