Web Scraping With Python Fastapi And Selenium
Yousaf Sabir
Like this project
Posted Feb 28, 2024
built a single data scraping solution for multiple similar websites following strict security measures.
Tee Chin Feng (my client) provides web development services to his clients in Malaysia. His tech stack is mostly PHP & J-Query. He had a project where he had to build a dashboard for his client, by which his client could view all his betting results on multiple websites.
Tee had the dashboard ready but he needed someone to pull the data off those websites & display them in the dashboard. He needed a program that can scrap the data and send that data over HTTP.
Scraping that data was not exactly a walk in the park. Source websites had implemented measures to ensure the maximum difficulty while scraping. Before Starting, Tee mentioned that his client had hired some other team for this kind of solution, but that team couldn't do it.
The security measures included logging in with the captcha code. I solved the captcha part with Tesseract OCR. The other problem was that it wasn't a single link, from which we could get the HTML and extract the data. JavaScript was executed on the page and then the resulting table was displayed. I solved this part with Selenium.
This was the scraping part. next, he needed to get the data over an API. I wrapped the scripts in their respective API endpoints with FastAPI. And there it was, the final solution. Tee was very happy that I pulled off something like this. Here are his words:
It's my second contract with Mr. Yousaf Sabir, a very good experience with him again! Appreciate the hard work and quality time spent together into this job! Throughout the entire process, there are numerous amendments and issues faced, but he handles the work nicely and accurately, solving unforeseen problems and completed it with healthy attitude!. Definitely this is an amazing experience working with Mr. Yousaf. He have professional skills and full of experience! Thank you so much!