Sitemap/Robots.txt Conflicts Wasting Crawl Budget

Phase 1 technical SEO audit on a SaaS client surfaced three quiet crawl-budget issues that hadn't triggered any ranking event but were capping site performance.

Finding 1 — Sitemap/robots conflicts. Three URLs were simultaneously listed in the XML sitemap and blocked by robots.txt. The sitemap tells Googlebot to crawl those URLs; robots.txt tells it not to. Googlebot resolves the conflict, but unpredictably, and crawl budget gets consumed either way. Common pattern on CMS-generated sitemaps where the sitemap auto-updates but robots.txt doesn't.

Finding 2 — 18 redundant User-agent blocks. A well-structured robots.txt uses a single User-agent: * block for blanket directives. This file had 18, each firing independently, all redundant. Consolidated to one — easier to maintain, easier to debug.

Finding 3 — 554 URLs missing lastmod. The CMS auto-generated the sitemap without lastmod attributes. Every URL looked equally stale to Googlebot, removing one of the key levers for managing crawl efficiency on a substantial content library. Flagged for the dev team to fix at the CMS level.

Stack: Screaming Frog SEO Spider, Google Search Console, manual robots.txt + sitemap diff, Python for URL set comparison.

Outcome: 3 conflicts removed, 18 → 1 User-agent blocks, lastmod gap escalated. None of the issues caused a ranking drop in isolation; together they represented technical debt quietly capping performance. CMS-generated sitemaps need human review — automation handles volume, not accuracy.

Tags: Technical SEO, crawl budget, sitemap, robots.txt, SaaS audit, GSC, Screaming Frog