The Plan

The client needed a solution that would get the closing prices (prices at which auctions won, or highest bid), for all auctions in a specific marketplace. BStock is a platform for multiple bulk-lot marketplaces, where you can buy bulk returns, cellphones, etc.

We decided to regularly scrape all available auction listings, as BStock conveniently provided an API endpoint that returns a CSV file with all active listings, paginated. Afterwards, at the end of the auction closing period, we'd scrape the individual winning prices and compile them into a database, which was then fed into a service for generating reports and sending emails.

Reverse Engineering BStock

I spent most of my time reverse engineering the BStock website, the various API endpoints that were publicly available, and directly connecting to server IPs to bypass anti-bot systems that they had in place. The client wanted a solution that relied solely on emulated HTTP requests, and not controlling virtual browsers.

This led to the eventual discovery that winning prices for each auction were stored on the page's HTML, for each auction. For example /auctions/587333 would have the winning bid in the page's

JSON Schema

Implementing the code

Technology

I decided to make it a central API that managed the resulting scraped data, storing a soft-redundant copy on a database. I utilized JSReport to generate reports, as well as free Excel libraries to implement an automatic Excel report. Emails were sent out using Google Apps Script on the company's GSuite account.

The worker microservice that would login, scrape the current active auctions, and the winning bids for the previous auctions was also created in .NET, and would report it's findings to the central API in a private Docker network. It was scheduled to run using a crontab on the Docker host. (Hosted on-premise).

Automatic Login and Cookie storage

Firstly, the worker service would spin up, perform the full OAuth flow (emulated), as well as taking care of any two-factor situations that would come up. Then, the program would store the session's cookie and expose itself as a service for the rest of the process.

Every request sent to BStock's servers went through a DI injected service called the

BStockService

Scraping the auctions

The worker service would scrape the list of available auctions by requesting a CSV export of them all, in paginated order. After combining the CSV list, reading them, and transforming all the data in a format we need, it would send it over to the Web API.

The API would take these requests, store the auction list in the database, and schedule for scraping when the auctions close. These auctions could get extended by late bidding, so the program would give it about ~24 hrs grace period.

When the time came for the winning bids to be collected, the worker would spin up again (from the crontab), and it would know it's duty based on the configuration setting provided in appsettings.json. There was two configurations, each copy of the same code deployed on its own Docker container.

The data of the winning bids would be sent up to the web API, stored, and then the web API's ReportGenerationService would generate the required files for the email, and then an HTTP request would be sent to Google Apps Script to send out the email over the G-Suite account.

Final Thoughts

Getting this automation done for the client was very tough, especially with the tight deadlines in mind; however, I was able to get it done, and later troubleshoot and provide bug fixes with new changes made by BStock. There were previous iterations of the project made on WinForms, however, these ones had to be manually operated whereas this system was automatic.

Client Impact

In the end, the impact for the client was giving them keen information that allowed them to bid better on popular models of devices they were buying in bulk. This information was crucial to their wholesale retail business, and the faster we could get it to them, the faster they could update bids and save money on expenses. This replaced the manual labor of nearly 6 full-time employees and provided valuable insights on the industry's market.