Industrial Web-Scraper Fleet Across 7 Manufacturer Sites

A hydraulics distributor needed a reliable master inventory of technical PDFs — product sheets, brochures, catalogs — published across a handful of hydraulic-component manufacturer websites: Duplomatic, Bondioli & Pavesi, Ronzio, Salami, and others. Each site was structured differently. Some listed products on static category pages, some required following pagination, one had an invalid SSL certificate and a second data source that returned a 403 to plain HTTP requests and needed a headless browser instead. PDFs got revised without notice — a manufacturer would swap a spec sheet for a newer revision with no changelog anywhere. There was no single, trustworthy record of what documents existed, what had changed, or whether a given PDF had ever been vetted before it landed on a server. Re-scraping everything from scratch every month wasn't practical once the catalogs grew past a few hundred documents each, and any script that silently wrote an empty Excel file over a good one would have destroyed months of accumulated data with no way to tell.

I built and maintain a fleet of n8n workflows, each backed by Python running in an external task runner, one pair per manufacturer site. Every manufacturer gets two workflows: an Annual workflow that builds that year's master Excel inventory from scratch (PDFs, products, categories, hashes, all of it), and a Monthly workflow that diffs the live site against the master, downloads only what's new or revised, updates the master in place, and writes a monthly report plus a metadata handoff for downstream processing. The fleet currently covers Duplomatic, Bondioli, Ronzio, Salami, Rexroth, Bucher and Casappa — seven manufacturer sites on the same underlying architecture. One Annual workflow also pulls catalogs from a third-party directory via a Selenium sidecar, since that data doesn't live on the manufacturer's own site.

Every scraper is built from a shared SOP template so a new manufacturer gets the same node architecture, the same failure modes handled, and the same troubleshooting table instead of a one-off script. That template encodes lessons learned the hard way — for example, n8n's HTTP Request node replaces an item's JSON with the raw response, so index-based rather than URL-based category matching is required after every fetch, and the batch loop's done output only passes the last item, which is why every download gets written to a JSONL log instead of relying on the loop's own output.

Every PDF runs through a five-layer quarantine step before it's trusted: magic-byte format detection, a 50MB size cap, structural validation via PyMuPDF, format-specific sanitisation (scrubbing embedded JavaScript, metadata and attachments from PDFs; stripping EXIF from images), and a regex scan for prompt-injection strings. Every PDF carries two SHA-256 hashes — one on the raw download, one after sanitisation — so there is an audit trail of exactly what the scrub changed. Master Excel files are never overwritten directly: each write goes to a tempfile first and is swapped in atomically, with the last three dated backups kept automatically.

The monthly diff keys off a slug or filename plus a revision-agnostic family key, stripping the trailing revision suffix, so a document with a bumped revision number is caught as revised rather than treated as brand new. On quiet months the workflow still updates the master for product-only changes but skips writing a report file — documented as expected behaviour, not a bug.

The fleet went through a deliberate V2 to V3 migration, moving every manufacturer onto one canonical architecture and retiring per-manufacturer lock files, retry loops and standalone validation nodes in favour of n8n's native execution guarantees and the atomic-write pattern. The migration guides document five concrete bugs fixed once and then encoded into the shared template — including an external Python runner rejecting a JavaScript Excel library, fixed by using openpyxl wherever the master file is read, and a batch-loop stall when zero items are queued for download.

The distributor now has a master inventory, per manufacturer, that is rebuilt annually and kept current monthly without manual PDF hunting. Every document in the inventory carries a dual SHA-256 audit trail proving what was downloaded and what changed during sanitisation — something that didn't exist before. The shared template turned each new manufacturer onboarding from a bespoke build into a documented checklist, and the V2 to V3 migration removed the fragile lock-file and retry-loop patterns that had caused stuck runs in the earlier version.

n8n, Python, openpyxl, PyMuPDF, Pillow, piexif, Selenium, Docker