Real Time Chart Data Extraction for Trading View
Dev@ Diraiah
Data Scraper
Data Analyst
BeautifulSoup
Python
Selenium
Asif was reliable, friendly and communicative and created a good working solution to my brief. I will work with him again.
Client's Objective
I have a trading view charts with many custom indicators, I would like to scrape the values such as 'vwap', 'timeline' and others and output those values as a python dictionary datatype.
The script will provide trading information to my trading bot. My bot would like to request updated values from the scraper on a cycle of 30 seconds
The scraper keeps producing data, and my bot just picks up the latest values when it wants. The big requirement is to scrape the values at 'high frequency' (like 30 secs) but refresh the page at extremely low frequency, like 1 refresh/hr.
TL;DR: The Client had a custom chart and required extracting those values regularly in very short intervals, Which can be fed to an another script to make trading decisions. This was particularly challenging as Trading View blocks scraping bots very actively.
Solution Provided
Wrote a bot script that accessed the data every 3 Second (Client wanted the lowest possible frequency and had initially asked for 30 Seconds)
Workaround was built for the login and pop ups without page refresh so no interruptions are caused in the output stream.
Multi threading was used to provide live data streams from multiple charts simultaneously with a single script (which didn't require making separate scripts for multiple charts as client required)
Easy run command with all controls to configure for custom chart provided as command line flags.
Packaged documentations for debugging, maintaining and support for installation along with the chart crawler bot.
Data Retrieval Process
The Script shown below opens up two chart windows for two ticker, The window is shown only for preview and the actual procedure would not open the windows ans as it can be seen on the right, in the data window the scrip fetches the data one by one
Here's the data retrieval that is shown on the output window of the terminal, currently the data is being printed on the terminal, but the use case involves using those data to make decisions based on the values weather the stock is to be bought.
