Content Scrapper, AI-Assisted Tool That Eliminated Manual CMS Content Migration

Built with Claude Code to replace weeks of manual copy-pasting with an automated crawl-extract-review pipeline for hotel website migrations.

Description:

The problem: our company was migrating 25+ hotel property websites into a new CMS. Without tooling, that meant someone manually opening every single page and copy-pasting headings, body copy, promotional sections, testimonials, contact info, and every image's alt text into the new system slow, repetitive, and error-prone at the scale of hundreds of pages per site.

I built Content Scrapper, solo, to remove that manual work entirely no team, no handoffs. I used Claude (Claude Code) as my AI pair-programmer throughout the build, from designing the section-extraction schema and adapter architecture to implementing the rendering pipeline, CLI, and web UI which let me go from idea to a working, production-usable tool much faster, while I stayed in control of every architectural and domain decision.

What it does:

Renders every page with a real headless browser (Playwright/Chromium), since key sections (amenities, nearby attractions, blog teasers) only load client-side via AJAX a plain fetch would have missed most of the content.

Automatically detects which site template a page uses and extracts it into a consistent JSON shape (hero, content blocks, touts, testimonials, contact info, images with alt text, meta tags, JSON-LD) with a generic fallback extractor for any site that isn't a known template, so no per-site manual configuration is required.

Ships as both a CLI (crawl → extract → review) for bulk migration runs, and a lightweight internal web UI so non-technical teammates can preview scraped content or run a whole-site scrape themselves without needing to touch the terminal or ask an engineer to do it for them.

Includes a built-in review step that automatically flags missing alt text, missing meta descriptions, and pages that scraped suspiciously thin catching problems before content ever reaches the CMS.

Adds a domain allowlist and SSRF guard on the web UI, since the tool renders whatever URL is submitted in a real browser and needed to be safe to hand to the wider team.

Impact: what used to be manual, page-by-page data entry across 5+ full hotel websites became a repeatable, self-service pipeline freeing the team from copy-paste work and human error, while producing structured, review-ready JSON the CMS-integration phase can consume directly.

How it's built:

Language: TypeScript, Node.js 18+, run via tsx (no build step needed)

Browser rendering: Playwright (headless Chromium) — for JS-rendered/AJAX content

HTML parsing/extraction: Cheerio (server-side jQuery-style DOM parsing)

CLI: Commander.js

Backend/API: Express.js

Validation: Zod (site config validation)

Concurrency control: p-limit (rate-limited parallel page rendering)

Frontend: Vanilla HTML/JS single page (no framework, no build step)

Storage: Filesystem-based normalized JSON (site → pages → sections)

Architecture pattern: Adapter pattern with auto-detection (canHandle() per adapter)

Security: hostname allowlist + private-IP guard (SSRF protection)

Built with: Claude Code (Anthropic) as AI pair-programmer

Links

GitHub: https://github.com/jaiinil/content-scrapper