llms.txt: What It Is, What It Isn't, and Whether You Need One

The Text Box Nobody Updated

Arc AI's first llms.txt file lived in a text box in the admin panel, typed by hand, updated by nobody. The address just returned "not found." The fix wasn't a better reminder, it was generating the file from the same rules that already build the sitemap. Here's what llms.txt actually is, what it isn't, and whether it's worth the bother.

A Short File With No Official Backing

llms.txt is a proposal, not a standard. Jeremy Howard put it forward on 3 September 2024 at llmstxt.org. No governing body, no version schedule, nobody to register with.

The file itself is plain Markdown at the root of a domain, meant for AI tools and for people scanning it directly. On a site built this way, it opens with the company's name and a plain description, then links to the pages that matter most: main pages, the newest 25 articles, categories, grouped by language. Every link keeps the title of the page it points to, so a French page gets a French link title.

Not robots.txt, Not a Sitemap, Not a Ranking Signal

It isn't robots.txt, which tells crawlers what they can fetch. It isn't a sitemap, meant to be a complete URL list rather than a curated one. It doesn't affect rankings or control access. It's closer to a short pitch handed to a machine reader, and, read the same way by a person, a reasonable one-page summary of the business.

Adoption is real but patchy: SE Ranking looked at roughly 300,000 domains and found about one in ten had an llms.txt file, 8% among high-traffic sites, 10.5% among mid-tier ones.

No major AI platform has confirmed it reads the thing. Google has said AI Overviews and AI Mode run on ordinary search signals, not llms.txt, and compared it to the old meta keywords tag. OpenAI's and Perplexity's crawler documentation covers robots.txt but doesn't mention llms.txt. Neither Anthropic nor Microsoft has announced support. SE Ranking tested whether it predicted which pages got cited by AI tools; dropping it from their model made the model more accurate, a fairly blunt sign it isn't a citation signal right now. That's the state of play as of September 2026; adoption moves, so check current support before relying on any of it.

The Real Risk Is Drift

None of that makes llms.txt worthless, it just narrows the case for one. Generated properly, it costs almost nothing and doubles as a clean, readable map of the site. Kept by hand, it's a chore, and chores get skipped.

A hand-kept file drifts two ways: it lists pages since deleted or moved, or it misses everything published after someone last opened the admin panel. Building it from the same eligibility rules that generate the sitemap avoids both. A draft doesn't appear until published, a scheduled post waits for its date, a noindexed or redirected page drops out of both automatically. One set of rules feeding two outputs means they can't quietly disagree.

Generation itself is cheap: on a site built this way, it runs on every request using 13 queries and about 37 milliseconds, cached for a day and cleared the moment something changes.

A Quick Checklist Before You Bother

• Decide whether it's worth it: cheap if generated, a chore if kept by hand.
• Generate it from the same rules that build your sitemap, not a separate list.
• Keep it short and curated: the pages that matter, not every URL.
• Put a plain description of the business near the top.
• Read the finished file once as a stranger would, and check it actually explains the business.

Worth Doing Once, Properly

There's no confirmed audience reading llms.txt yet, and anyone promising it'll move AI search visibility is guessing. What it's actually good for is smaller: a tidy, current list of what a site is and where its important pages live, built once and left to update itself.