Python script for extracting and processing website sitemaps. by Vatche ThorossianPython script for extracting and processing website sitemaps. by Vatche Thorossian

Python script for extracting and processing website sitemaps.

Vatche Thorossian

Data Scraper

Data Analyst

BeautifulSoup

Python

SQL

This project includes a Python script for extracting and processing sitemaps of websites. The script is capable of identifying and downloading website sitemaps from the robots.txt file of a website, and is compatible with various sitemap formats such as XML and GZ. Additionally, it has the capability to process sitemaps recursively and store the data in a database.

https://github.com/vatche-t/sitemaps

Like this project

Posted May 9, 2024

This project includes a Python script for extracting and processing sitemaps of websites

Likes

Views