To check if AI can read your website, run five tests: (1) look for AI crawlers in your robots.txt, (2) test whether your firewall blocks them, (3) view your page source to see if your text survives without JavaScript, (4) validate your schema markup, and (5) check for an llms.txt file. Each test takes a couple of minutes by hand — below are the exact steps — or you can run all five at once in a free 30-second scan.
Test 1: Is your robots.txt blocking AI crawlers?
Open yoursite.com/robots.txt in a browser. Search the file for these user-agents: GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), CCBot (Common Crawl — feeds many models' training data).
If any of them appears above a Disallow: / line, that AI is locked out. Many sites blocked these bots in 2023–24 (often via an SEO plugin toggle) and forgot. Blocking training crawlers might be a deliberate choice for publishers — but for a business that wants AI to recommend it, it's self-sabotage.
Test 2: Is your firewall silently blocking bots?
This is the sneaky one: robots.txt looks fine, but your CDN or firewall returns a 403 to AI crawlers. Cloudflare's "Block AI Bots" toggle, bot-fight modes, and aggressive WAF rules all do this.
Manual check: run this in a terminal (it mimics GPTBot's user-agent):
curl -A "GPTBot" -o /dev/null -s -w "%{http_code}" https://yoursite.com
A 200 means the bot gets through. 403 or 503 means it's being blocked at the network level — check your CDN's bot settings.
Test 3: Does your content survive without JavaScript?
Most AI crawlers don't execute JavaScript. In Chrome, right-click your homepage → View Page Source (not Inspect) → press Ctrl/Cmd+F and search for a real sentence from your page — your tagline, a price, your address.
- Found it? Your content is in the raw HTML. AI can read it.
- Not there? Your content is rendered client-side. To an AI crawler, your page is close to blank — no services, no prices, no location.
Test 4: Do you have valid schema markup?
In the same page source, search for application/ld+json. If nothing comes up, you have no structured data, and AI has to guess what kind of business you are. If it's there, paste your URL into Google's Rich Results Test to confirm it's valid — broken schema is as unhelpful as none.
Test 5: Do you have an llms.txt file?
Open yoursite.com/llms.txt. If it 404s, you're missing the newest AI-visibility signal — a plain-text business summary that AI models can read directly. It takes five minutes with our free llms.txt generator.
Or run all five tests in 30 seconds
The free RankInAI scan performs every check on this page — including the firewall test most people can't run — and gives you a 0–100 score with exact fixes.
Scan My Site Free — No SignupBonus test: just ask the AI
Ask ChatGPT (with browsing/search) and Perplexity: "What does [yoursite.com] offer and where is it located?" This is the customer's-eye view. If the answer is wrong, vague, or "I can't access that site," one of the five layers above is failing. If it's accurate — congratulations, you're ahead of most of the web.
Quick reference
| Test | Where | Pass looks like |
|---|---|---|
| robots.txt | yoursite.com/robots.txt | No Disallow rules for GPTBot, ClaudeBot, PerplexityBot |
| Firewall | curl with bot user-agent | HTTP 200 response |
| JS-blindness | View Page Source | Your real text is in the raw HTML |
| Schema | Page source / Rich Results Test | Valid JSON-LD describing your business |
| llms.txt | yoursite.com/llms.txt | File exists and describes your business |