Scholarly Research Data Pipeline: Scilit to Apache Solr by Muhammad Jahanzaib TahirScholarly Research Data Pipeline: Scilit to Apache Solr by Muhammad Jahanzaib Tahir

Scholarly Research Data Pipeline: Scilit to Apache Solr

Portfolio project
Muhammad Jahanzaib Tahir
An automated Python pipeline that collects research article metadata from Scilit.net, a large scholarly literature database, and indexes it into a searchable Apache Solr system for research discovery.

The Challenge

The client needed complete article-level metadata for a large catalogue of academic journals, all in one searchable index. Scilit is protected by Cloudflare, returns results in pages, and a harvest of this size takes days, so the pipeline had to get past bot protection and survive interruptions without losing progress.

My Approach

Used Scilit’s internal search API instead of scraping HTML pages, so each request returned clean, structured JSON.
Got past Cloudflare protection using cloudscraper, with automatic retries on failed requests.
Harvested journal by journal. The pipeline reads journals (by ISSN/eISSN) from a source Solr core, then pages through every article in each journal.
Made it resumable with checkpoints. Each finished journal is marked in Solr with a view tag, so the pipeline can stop and restart at any time without duplicates or gaps.
Tracked records per journal for coverage checks and quality control.
Indexed directly into Solr in batches, with each article mapped to a clean 30-field schema.
# Resumable harvesting: only journals not yet processed in this run
while True:
journals = source_core.search(f'-view:{VIEW_VAL}', rows=10)
if not journals:
break

for journal in journals:
offset = 0
while True:
docs = fetch_articles(journal['id'], offset, limit=100)
if not docs:
break
target_core.add([map_schema(d, journal) for d in docs])
offset += 100

journal['view'] = VIEW_VAL # checkpoint: mark as done
source_core.add(journals)
source_core.commit()

Data Captured (30 fields per article)

Title, abstract and keywords
Authors, affiliations and unique author/institute IDs
Journal, publisher, ISSN and eISSN
DOI, volume, issue, pages and publication year
Document type, language and subjects
Citation count and reference IDs
Open-access status, license and PDF links

Results

A fully automated harvest of article metadata across a large journal catalogue
A clean, consistent schema, ready for full-text search, filtering and analytics
Fault-tolerant runs that resume where they stopped after a network failure or restart
Feeds a research discovery platform and RAG search pipeline

Tech Stack

Python · Cloudscraper · REST API reverse engineering · Apache Solr · PySolr · JSON

1 like · 0 views · Nov 10, 2026

Built a resumable Python pipeline that pulls research articles from Scilit’s API journal by journal and indexes 30 metadata fields into Apache Solr.

Tags