CrawlX CrawlX crawls a website, surfaces

CrawlX

CrawlX crawls a website, surfaces its technical SEO problems, and explains each one in plain language with a suggested fix. Cloud-native, fast, and built for teams. I built it from a blank page: product, brand, and engineering.

The gap

The technical SEO crawler market is led by Screaming Frog, a desktop tool with a dated interface, no AI, and a one-person, one-machine model. No sharing, no assigning issues, no client portals.

At the other end, enterprise crawlers like Lumar are fast but cost $15k to $50k a year.

That left a clear gap: enterprise speed, a modern interface, AI built in, and pricing a small team can actually afford.

Positioning

Linear meets Datadog for SEO. A crawler that does not just hand you a list of errors. It tells you what each one means and how to fix it.

What I built

1. Product, engineering, and brand, end to end.

2. 400+ SEO checks across 12 analysis modules

3. An AI layer on the Claude API: plain-English explanations, auto-generated fix code, and a conversational assistant you can ask about your own crawl

4. Six core screens, including a live crawl view, a Kanban issue explorer, a site-architecture visualizer, and a white-label report builder

5. An AI search-readiness module that checks whether pages are built for AI answer engines, a gap no competitor covers

6. Rust crawl workers hitting 350 to 450 URLs per second, matching enterprise tools, with ClickHouse handling billion-row crawl data

What makes it different

1. AI intelligence no legacy crawler offers: explanations, fix code, and a chat assistant, not static hints

2. Enterprise-class crawl speed at small-team pricing

3. Team-first: assign issues to developers, share client portals, comment in place, instead of one license per machine

4. Built into the pipeline: CI/CD on GitHub Actions and Vercel catches SEO regressions before they ship