Does Your Site Need an llms.txt File?

llms.txt is a proposed text file that sits at the root of your website and gives AI systems a short, curated map of your best content, written in markdown. Does your site need one? Honestly: no, not for visibility today. No major AI provider has committed to reading it, and an Ahrefs study of 137,000 domains found that 97% of the llms.txt files that exist received zero requests in May 2026. But the file costs about 15 minutes and has no downside, so the case for shipping one anyway is real. This post walks through the spec and the data, then makes that case without the hype.

What llms.txt is

llms.txt is a markdown file served at /llms.txt that tells AI systems what your site is and where your most useful pages live. Jeremy Howard of Answer.AI proposed it on September 3, 2024. The core idea: AI context windows are too small to read a whole website, so you hand the model a compact, curated index instead of forcing it to guess.

The proposal makes a fair point. Normal web pages carry navigation, ads, scripts, and layout noise. A language model that lands on your HTML has to strip all of that before it finds the substance. Markdown is clean, and both humans and models read it easily. So the spec says: describe your site in one small file, link to the pages that matter, and offer markdown versions of those pages too.

The intended reader is broader than chatbots. The spec is aimed at any AI agent that visits your site on demand, whether that is a coding assistant reading your docs or a shopping agent checking your pricing page before it answers a question. Keep that in mind, because it changes how you should judge the adoption numbers below.

The format: one small markdown file

The spec is short. A valid llms.txt file has one required part and a few optional ones, in a fixed order: an H1 with the site or project name, a blockquote with a one-paragraph summary, optional free-form notes, and then H2 sections that each hold a markdown list of links. Each link can carry a colon and a short note.

Here is a minimal example:

# Gantra

> Gantra is an AI marketing agents SaaS for startups. Twelve agents
> write, audit, and distribute marketing content on a daily loop.

## Docs

- [Quickstart](https://gantra.so/docs/quickstart.md): set up your first agent
- [Pricing](https://gantra.so/pricing): plans and limits

## Blog

- [GEO guide](https://gantra.so/blog/generative-engine-optimization): how AI engines pick sources

## Optional

- [Changelog](https://gantra.so/changelog): release history

Two details from the spec are easy to miss. First, a section named Optional has special meaning: agents can skip those links when they need a shorter context. Put nice-to-have pages there, not your core ones. Second, the proposal recommends serving a markdown twin of each important page at the same URL with .md appended. That part takes real work, and most sites skip it.

There is also a common companion file, llms-full.txt. Instead of linking out, it inlines the full text of your content in one large markdown file, so an agent can load everything in a single request. Documentation platforms generate it automatically. For a normal marketing site it is optional, and for a large site it can get unwieldy fast.

The hard truth: almost nobody reads it

Here is the part most llms.txt articles bury. OpenAI, Google, and Anthropic have all declined to say they read the file. And the server logs agree. Ahrefs analyzed 137,210 domains in May 2026 and found that 97% of published llms.txt files got zero requests that month. Of the few requests that did arrive, most came from SEO audit tools and generic crawlers, not from AI systems. The study’s sharpest line: AI bots made zero requests for llms.txt files that did not exist. They never go looking.

Google has been blunt about it. John Mueller compared llms.txt to the keywords meta tag, the self-declared signal search engines abandoned because it was too easy to game. He also noted that none of the AI services have said they use the file, and that your own server logs will confirm they do not even check for it.

Meanwhile, adoption keeps climbing. Originality.ai tracked llms.txt across more than 3 million sites and counted an 8.8x rise in one year, from 4,088 files in June 2025 to 36,120 in May 2026. So we have a strange picture: publishing is accelerating while consumption stays near zero. That gap is a warning sign. A lot of the push comes from tools and agencies selling AI visibility, not from evidence.

One caveat before you write the whole idea off. The big AI crawlers do hit websites constantly; they just read your normal HTML pages instead of the index file. If you want to know who those bots are and how to handle them, we broke that down in our guide to AI crawlers and what they actually fetch.

llms.txt vs robots.txt vs sitemap.xml

These three files get mixed up constantly, and the confusion sells a lot of bad advice. They do different jobs:

The key row is enforcement. robots.txt and sitemap.xml have decades of real consumption behind them. llms.txt has a spec and a hope. If your goal is controlling AI crawler access, that job belongs to robots.txt, not llms.txt. The file has no blocking power at all.

Why it might still be worth 15 minutes

The honest case for llms.txt is a cheap option bet, not a growth tactic. It costs almost nothing, and if you keep the file small there is no maintenance burden and no known penalty either. Serious companies already publish one: Mintlify auto-generates llms.txt for thousands of docs sites, including Anthropic’s and Cursor’s documentation. They are not betting on citations. They are betting on agents.

The strongest argument is agent readiness. The Ahrefs data measures today’s crawlers and says nothing about tomorrow’s agents, which fetch pages on demand when a user asks about you. If that behavior arrives, sites with a clean file are ready on day one.

There is a smaller benefit too. The blockquote summary gives you one more place where your brand entity is stated plainly, in a format machines parse without effort. That matters most when your name is ambiguous. “Gantra” collides with a pharmaceutical of the same name, so any file that says “Gantra is an AI marketing agents SaaS” in plain text is a small win for us.

Some warnings, though. Do not expect citations; the GEO evidence points to content structure, statistics, and third-party mentions rather than index files. Nobody should be charging you real money for “llms.txt optimization” either. And the file must never displace work with proven signals, like the structured data we cover in our schema markup for AI search guide.

How to generate and maintain one

Keep it small and curated. This is a reading guide, not a sitemap dump.

1. Write the header by hand. One H1 with your name, then one blockquote that says what you are, who you serve, and what makes you different, in two or three sentences.
2. Pick 10 to 30 links, not hundreds. Docs, pricing, your best guides, your about page. Group them under two to four H2 sections and push nice-to-haves into an Optional section.
3. Serve it at the root as /llms.txt with a text/plain or text/markdown content type. If your docs run on a platform like Mintlify, this already happens automatically.
4. Consider a noindex header. Google’s John Mueller said it could make sense to serve llms.txt with a noindex, so the raw file does not show up in search results. An X-Robots-Tag: noindex header does the job without blocking anyone from fetching the file.
5. Revisit it quarterly. A stale file that links to dead pages is worse than no file. Tie the review to your existing content audit so it never becomes its own chore.

If you want a generator, plenty of free ones exist, but for a small site, writing it by hand in a text editor is genuinely faster and gives a better result. The value is in the curation, and only you know which 20 pages define your business.

Where does this sit in your week? Nowhere, ideally. This is exactly the kind of low-stakes technical check that should run on autopilot. Gantra runs 12 marketing agents, and the daily SEO and GEO audits, which check technical basics like crawler access alongside your AI visibility, are on the free plan. The agents also write and distribute content: auto-publish to LinkedIn and X, plus one-click share drafts for Reddit, Hacker News, and Medium. To be clear, no tool, ours included, can make AI engines read your llms.txt. Nobody can. That is the whole point of this post.

The takeaway

llms.txt is a reasonable idea with, so far, no consumers. The spec is clean: an H1, a blockquote summary, and a few H2 sections of curated markdown links at /llms.txt. The data is equally clean: 97% of these files got zero requests in a 137,000-domain study, and no major provider commits to reading them. So ship one in 15 minutes as a cheap bet on an agent-driven future, add a noindex header, and review it quarterly. Then spend your real energy on what measurably moves AI visibility: answer-shaped content, consistent entity data, and mentions on sites you do not own. If you would rather have agents run that loop for you, Gantra starts free.