Free tool I built: the AI crawler checker. Paste a URL and it checks robots.txt rules for each AI crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more), fetches the page with those user agents to catch CDN or firewall blocks, and shows whether the title, main text and structured data are in the HTML before JavaScript runs. Beta, free, no sign-up: aeoledger.com/tools/ai-crawler-checker/
0 likes · 0 views · Oct 21, 2026
Free tool I built: the AI crawler checker. Paste a URL and it checks robots.txt rules for each AI crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, G...
