Free check. No signup.

ChatGPT can't recommend what it can't see.

A site can close the door on AI search crawlers without anyone noticing: an old robots.txt rule, Cloudflare's "Block AI bots" switch, or a bot wall. OpenAI says a site that blocks its search crawler is not used in ChatGPT search answers, though it can still appear as a plain link. Northlab Crawler Check tells you in about 20 seconds. Free. No signup.

We don't install anything or change your site. We only request public pages, the way a bot would. We check robots.txt for 19 AI and search crawlers, llms.txt, your sitemap, and send requests with the real bot user-agents to catch CDN/WAF blocks. We don't save the sites you check or write them to logs. For rate limiting (10 checks per hour), your IP address and the times of your checks are held only in the server's memory, never on disk, and are cleared about an hour after your last check (or sooner if the service restarts). Controller: Northlab Research (northlab.research@outlook.com). Legal basis: our legitimate interest in preventing abuse of the free checker. Full privacy notice, your rights, and how to complain to the AEPD.

The paid pack is not available for now; the check is free.

This is what a result looks like.

Example with an invented shop. Your result will use your real data.

example-shop.com Example result

GPTBotOpenAI · training

Your robots.txt blocks it with Disallow: /. That is your choice, if it was deliberate.

✕ Blocked

OAI-SearchBotOpenAI · ChatGPT search

The robots.txt allows it, but Cloudflare answers with a 403. This is where search citations are lost.

✕ Blocked

ClaudeBotAnthropic · training

It can fetch the page. Nothing to change here.

✓ Allowed

PerplexityBotPerplexity · search

The host refuses it before the page is reached. The server returns 403.

✕ Blocked
3 of 4 bots stop at the door. Two of them are the ones that fetch your pages for answers. The live check shows where each block sits, in robots.txt, the CDN or the host.

Three steps. Less than a minute.

  1. 1

    Paste your URL

    Your site address, as you would type it. With or without https.

  2. 2

    We knock

    We request your robots.txt and homepage as each bot would, and we read what the server sends back.

  3. 3

    You read the result

    Who gets in, who does not, and where the block sits: robots.txt, CDN or host.

Questions people ask.

Is blocking AI crawlers bad?

It depends on the bot. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data for model training, and blocking them is a fair choice. Answer and search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages so the assistant can cite and link to you. Block those and the assistant can't fetch your pages to use them in its answers, though OpenAI notes a blocked site can still appear as a navigational link. The free check shows the two groups separately.

Why would my site block them if my robots.txt looks fine?

Your CDN or firewall can block bots before robots.txt matters. Cloudflare's "Block AI bots" setting and bot-fight modes, managed WordPress security plugins and some hosts return 403 or a challenge page to bot user-agents. The free check sends requests with the real bot user-agents to spot that.

What does llms.txt do?

It's a short Markdown file at /llms.txt that points AI tools to your most useful pages. It's cheap to add and harmless. The check tells you whether yours is present. Server-log studies have found AI crawlers rarely request it, so expect little effect.

Do you need access to my site?

No. We only read public pages. You (or your developer) apply any edits.

Who are you?

Northlab Research is a small independent studio building tools that check how AI assistants see websites. The same audit runs as the "AI Crawler Access Audit" on Apify for bulk checks.