How AI Crawlers Read Your Website in 2026

AI crawlers are simple programs. They request a URL, read the raw HTML your server returns, and move on. They do not run JavaScript or wait for your app to hydrate, and each one follows its own rules in robots.txt. That simplicity is exactly why so many sites are invisible to ChatGPT, Claude, and Perplexity without anyone noticing: the content is there for humans, but the HTML the bots receive is empty, or a CDN turned them away at the door.

This post is the technical version of the one-liner you see everywhere (“check your robots.txt”). Who the bots are, what each one feeds, why JavaScript is the biggest silent failure, and how to confirm with your own logs that any of this is working.

The main AI crawlers in 2026, and what each one feeds

There is no single “AI bot”. Each company runs separate crawlers for training, for search indexing, and for live fetches when a user asks a question. They are controlled separately in robots.txt, and blocking one does not block the others. Getting this map right is the difference between opting out of training and accidentally opting out of being an answer.

Some details worth knowing. OpenAI’s docs are explicit that GPTBot and OAI-SearchBot serve different purposes and can be allowed or blocked independently; robots.txt changes take about 24 hours to be picked up. Anthropic documents the same three-way split and publishes its crawler IPs so you can verify that a visitor claiming to be ClaudeBot really is one. Google-Extended is not a separate crawler at all: it is a robots.txt token that Googlebot honors, and Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” Applebot-Extended works the same way for Apple.

Bytespider is the outlier. HAProxy reported that nearly 90% of the AI crawler traffic on their site came from ByteDance, and multiple log analyses have caught Bytespider fetching URLs that were explicitly disallowed. If you need to keep it out, do it at the server or CDN level, not in robots.txt.

AI crawlers do not run JavaScript

No major AI crawler executes JavaScript. Vercel analyzed crawler traffic across its network and found zero evidence of JavaScript rendering by OpenAI, Anthropic, or Perplexity bots. They fetch JS files as text (Claude in 23.84% of requests, ChatGPT in 11.50%) but never execute them. Whatever your server sends as initial HTML is all an AI engine will ever see.

And the volume is huge. In one month on Vercel’s network, GPTBot made 569 million fetches and Claude made 370 million, roughly 1.3 billion AI crawler requests combined. Googlebot renders JavaScript; these bots do not. So a client-rendered React or Vue app can rank fine in Google while being a blank page to every AI engine.

The fix is boring and well known: server-side rendering or static HTML for anything you want AI engines to read. Article bodies, pricing, docs, FAQs. If your framework supports SSR or static generation, turn it on for public content. If it does not, pre-render those routes. This is the same foundation that makes generative engine optimization work at all: you cannot be cited for content the engine never received.

One more finding from the same study: AI crawlers are bad at wasted work. ChatGPT spent 34.82% of its fetches on 404 pages, against 8.22% for Googlebot. Clean internal links and a current sitemap decide how much of your real content an AI crawler reaches before it gives up and leaves.

robots.txt still works, but your CDN may have overruled it

Here is the part most owners miss. Your robots.txt can say “everyone welcome” while your CDN blocks AI crawlers before they ever read it. On July 1, 2025, Cloudflare became the first major infrastructure provider to block AI crawlers by default on new domains. Cloudflare handles traffic for about 20% of the web, and more than one million customers had already opted into blocking before the default flipped.

If you put a new domain behind Cloudflare after mid 2025 and clicked through the defaults, there is a good chance GPTBot and ClaudeBot get a 403 from the edge, no matter what your robots.txt says. Your site works, Google indexes it, and you are invisible to AI search. Nothing errors. Nobody tells you.

So the checklist has two layers now. Layer one, robots.txt: make sure you are not disallowing OAI-SearchBot, Claude-SearchBot, PerplexityBot, or the user-triggered fetchers if you want AI visibility. Layer two, the CDN dashboard: in Cloudflare, review the AI crawler and bot management settings and explicitly allow the bots you want. Decide on purpose, in both places.

The noindex vs Disallow mistake

These two controls get mixed up constantly, and combining them backfires. Disallow in robots.txt stops a bot from fetching a page. noindex (a meta tag or header) lets the bot fetch the page but tells it not to index it. Google’s documentation warns that for noindex to work, the page “must not be blocked by a robots.txt file.” Block the fetch and the crawler never sees your noindex, so the URL can still surface if other sites link to it.

For AI crawlers the same logic applies with a twist. Disallowing a training bot like GPTBot is a statement about future training data, not about search. Disallowing a search bot like OAI-SearchBot removes you from ChatGPT’s answers. If your goal is “stay visible in AI search but keep this one page out,” you want the page crawlable with a noindex, not a Disallow. If your goal is “do not train on anything,” disallow the training bots and leave the search bots alone.

Confirm it in your server logs

Do not trust assumptions; grep your logs. AI crawlers identify themselves in the user agent string, so a few minutes with your access logs answers the question:

grep -iE "gptbot|oai-searchbot|claudebot|claude-searchbot|perplexitybot|amazonbot|bytespider" access.log | awk '{print $1, $7, $9}' | sort | uniq -c | sort -rn | head -50

Start with presence: do the bots show up at all? If GPTBot has never appeared, something upstream (usually the CDN) is blocking it. Then look at status codes. Rows of 403s mean a firewall rule; rows of 404s mean the bot is wasting its visit on dead URLs. Then check which pages they fetch. If the bots only ever hit your homepage and skip your best articles, your internal linking is not surfacing them. Since user agents can be faked, verify suspicious traffic against the published IP ranges (OpenAI, Anthropic, and Perplexity all publish theirs). Then close the loop on the other side and track AI referrals in GA4 to see which of those crawls turn into visits.

Block or allow: the real trade-off

Allowing AI crawlers trades your content for visibility, and the exchange rate is poor. Cloudflare’s Radar data put Anthropic at roughly 70,900 HTML page crawls for every one HTML page referral in a sample week of June 2025, with ratios across AI platforms ranging from that extreme down to near parity. AI engines read a lot and send back a little.

For a media business that sells pageviews, blocking training bots is a defensible call. For a startup that needs to exist in the answers buyers actually read, blindly blocking everything is self-harm: the referral may be rare, but being the cited answer for “best tool for X” is worth more than the click count suggests. A sane middle path for most companies: allow the search and user-triggered bots (OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot), decide deliberately about the training bots (GPTBot, ClaudeBot, Google-Extended), and hard-block the ones that ignore the rules. Pair whatever you allow with structured data AI engines can parse so the crawls you do get land on extractable content.

This layer is exactly the kind of thing that silently rots: a CDN update, a new subdomain, a framework migration to client rendering, and your AI visibility is gone with no alert. Gantra’s daily SEO and GEO audits check crawler access, rendering, and citation visibility as part of its 12 marketing agents, and those daily audits are on the free plan. It will not fix your robots.txt for you, but it will tell you the day something breaks.

The takeaway

AI crawlers read raw HTML, nothing more. So the play has four moves. Serve your important content as server-rendered or static HTML, because no AI crawler runs JavaScript. Write robots.txt per bot, separating training bots from search bots on purpose. Check your CDN dashboard, because Cloudflare has blocked AI crawlers by default on new domains since July 2025 and your robots.txt cannot override the edge. And verify all of it in your server logs instead of assuming. An hour of checking beats months of being invisible to the engines your buyers now ask first.