GPTBot is OpenAI's web crawler. It fetches pages so OpenAI's models — including ChatGPT — can learn about and reference websites. If GPTBot can't access your site, ChatGPT's knowledge of your business is limited to secondhand sources, or nothing at all.

How do I check if GPTBot is blocked in robots.txt?

Open yoursite.com/robots.txt and search for 'GPTBot'. If you find 'User-agent: GPTBot' followed by 'Disallow: /', it's blocked. Also check for a blanket 'User-agent: *' with 'Disallow: /', which blocks everything.

Should I unblock GPTBot?

For most businesses, yes — being readable by ChatGPT is how you get recommended to its users. Publishers monetizing content may reasonably choose differently. It's a business decision, but it should be a decision, not a forgotten plugin default.

Does blocking GPTBot remove my site from ChatGPT?

Blocking stops future crawling; it doesn't instantly erase what models already learned. But over time a blocked site fades: live-search answers can't cite you, and newer model versions know less about you.

Is GPTBot Blocked on Your Site? How to Check (and Fix It) in 5 Minutes

GPTBot gets blocked in two places: your robots.txt file and your firewall/CDN settings — and you need to check both, because a clean robots.txt means nothing if Cloudflare is returning 403s to AI crawlers underneath it. Here's the two-minute check for each, and the copy-paste fix.

Check 1: robots.txt (30 seconds)

Open yoursite.com/robots.txt and search (Ctrl/Cmd+F) for GPTBot. You're blocked if you see:

User-agent: GPTBot
Disallow: /

Also search for User-agent: * followed by Disallow: / — that blanket rule blocks every crawler, AI or not. And while you're there, check the other AI bots: ClaudeBot, PerplexityBot, Google-Extended, CCBot.

💡 Many WordPress SEO plugins added "block AI bots" toggles in 2023 and site owners clicked them once and forgot. If you don't remember deciding to block AI, check whether a plugin decided for you.

Check 2: firewall / CDN (the invisible block)

Even with a welcoming robots.txt, your CDN can reject AI crawlers at the network level. The main culprits: Cloudflare's "Block AI Bots" toggle (Security → Bots), bot-fight/super-bot-fight modes, aggressive WAF managed rules, and some hosts' default bot protection.

Test it — in Terminal (Mac) or any shell, send a request pretending to be GPTBot:

curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" -o /dev/null -s -w "%{http_code}\n" https://yoursite.com

Result meanings:

HTTP code	Meaning
200	GPTBot gets through. You're fine at the network level.
403	Blocked by firewall/CDN — robots.txt is irrelevant until this is fixed.
503 / 429	Bot-challenge or rate limiting — AI crawlers likely give up.

The fix

robots.txt — copy-paste this

Replace any AI-bot Disallow rules with explicit allows (this is the exact pattern rankinai.tech uses):

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Cloudflare

Dashboard → your site → Security → Bots → turn off "Block AI Bots" (or set it to allow verified bots).
Security → WAF → check custom rules for user-agent blocks mentioning GPT/AI/bot.
Re-run the curl test until you get a 200.

Not sure? Test it properly in 30 seconds

The free RankInAI scan checks GPTBot, ClaudeBot and PerplexityBot access the way the bots actually experience your site — plus 5 more AI-visibility checks.

Scan My Site Free — No Signup

Should you even unblock it?

Fair question. If you're a publisher whose product is your content, blocking training crawlers is a defensible position. But if you're a business that wants customers — a hotel, an agency, a shop — GPTBot is how ChatGPT learns you exist. Blocking it is like unlisting your phone number and wondering why nobody calls. In our scan data, a striking share of small-business sites block AI crawlers without ever having decided to.

Once you've unblocked, finish the job: verify AI can actually read your content and add an llms.txt so it understands what it's reading.