An automated Python pipeline that collects research article metadata from Scilit.net, a large scholarly literature database, and indexes it into a searchable Apache Solr system for research discovery.
The Challenge
The client needed complete article-level metadata for a large catalogue of academic journals, all in one searchable index. Scilit is protected by Cloudflare, returns results in pages, and a harvest of this size takes days, so the pipeline had to get past bot protection and survive interruptions without losing progress.
My Approach
Used Scilit’s internal search API instead of scraping HTML pages, so each request returned clean, structured JSON.
Got past Cloudflare protection using cloudscraper, with automatic retries on failed requests.
Harvested journal by journal. The pipeline reads journals (by ISSN/eISSN) from a source Solr core, then pages through every article in each journal.
Made it resumable with checkpoints. Each finished journal is marked in Solr with a view tag, so the pipeline can stop and restart at any time without duplicates or gaps.
Tracked records per journal for coverage checks and quality control.
Indexed directly into Solr in batches, with each article mapped to a clean 30-field schema.
Data Captured (30 fields per article)
Title, abstract and keywords
Authors, affiliations and unique author/institute IDs
Journal, publisher, ISSN and eISSN
DOI, volume, issue, pages and publication year
Document type, language and subjects
Citation count and reference IDs
Open-access status, license and PDF links
Results
A fully automated harvest of article metadata across a large journal catalogue
A clean, consistent schema, ready for full-text search, filtering and analytics
Fault-tolerant runs that resume where they stopped after a network failure or restart
Feeds a research discovery platform and RAG search pipeline
Tech Stack
Python · Cloudscraper · REST API reverse engineering · Apache Solr · PySolr · JSON
1 like · 0 views · Nov 10, 2026
Built a resumable Python pipeline that pulls research articles from Scilit’s API journal by journal and indexes 30 metadata fields into Apache Solr.