Python script for extracting and processing website sitemaps.

This project includes a Python script for extracting and processing sitemaps of websites. The script is capable of identifying and downloading website sitemaps from the robots.txt file of a website, and is compatible with various sitemap formats such as XML and GZ. Additionally, it has the capability to process sitemaps recursively and store the data in a database.

https://github.com/vatche-t/sitemaps