Northlab ResearchFree AI crawler check

Can ChatGPT, Claude and Perplexity read your site?

Lots of sites block AI search crawlers without knowing it. An old robots.txt rule, Cloudflare's "Block AI bots" switch, or a bot wall can refuse the request before a page is read. OpenAI says a site that blocks its search crawler is not used in ChatGPT search answers, though it can still appear as a plain link. Run the free check in about 20 seconds. No signup.

Check a public URL

We check robots.txt for 19 AI and search crawlers, llms.txt, your sitemap, and send requests with the real bot user-agents to catch CDN/WAF blocks. No signup. We don't save the sites you check or write them to logs. For rate limiting (10 checks per hour), your IP address and the times of your checks are held only in the server's memory, never on disk, and are cleared about an hour after your last check (or sooner if the service restarts). Controller: Northlab Research (northlab.research@outlook.com). Legal basis: our legitimate interest in preventing abuse of the free checker. Full privacy notice, your rights, and how to complain to the AEPD.

The paid pack is not available for now; the check is free.

What the check looks at

Each run reads only public URLs. It does not install anything or change your site.

FAQ

Is blocking AI crawlers bad?

It depends on the bot. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data for model training, and blocking them is a fair choice. Answer and search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages so the assistant can cite and link to you. Block those and the assistant can't fetch your pages to use them in its answers, though OpenAI notes a blocked site can still appear as a navigational link. The free check shows the two groups separately.

Why would my site block them if my robots.txt looks fine?

Your CDN or firewall can block bots before robots.txt matters. Cloudflare's "Block AI bots" setting and bot-fight modes, managed WordPress security plugins and some hosts return 403 or a challenge page to bot user-agents. The free check sends requests with the real bot user-agents to spot that.

What does llms.txt do?

It's a short Markdown file at /llms.txt that points AI tools to your most useful pages. It's cheap to add and harmless. The check tells you whether yours is present. Server-log studies have found AI crawlers rarely request it, so expect little effect.

Do you need access to my site?

No. We only read public pages. You (or your developer) apply any edits.

Who are you?

Northlab Research is a small independent studio building tools that check how AI assistants see websites. The same audit runs as the "AI Crawler Access Audit" on Apify for bulk checks.