Skip to content
NagentNagent
Log inSign upHire your AI Growth Team
NagentNagent
Log inSign up

What is llms.txt and do I need one?

8 Minutes read
Published:
llms.txt is a proposed Markdown file at your site's root that points AI tools to your most useful pages. No major AI provider has confirmed using it for search, so it is optional for most sites and most useful for developer documentation.
PB
Pratap BeheraNov 14, 2026·8 min read
What is llms.txt and do I need one?

llms.txt is a proposed Markdown file, served at yoursite.com/llms.txt, that gives AI tools a short summary of your site and links to its most useful pages. You probably do not need one for AI search: no major AI provider has confirmed that its search features read it, and Google says no AI text files are needed for AI Overviews. It is worth adding if you publish developer documentation, because coding assistants can use it, and it costs an hour to make.

What is llms.txt?

Jeremy Howard proposed llms.txt on 3 September 2024. The idea: AI agents are best served by concise, expert-level information gathered in one accessible place, rather than by crawling every page of a site full of navigation, scripts and ads (llmstxt.org).

The proposal sets out a simple Markdown format:

  • an H1 with the site or project name, the only required part;

  • a blockquote with a short summary;

  • optional paragraphs or lists with more detail;

  • H2 sections of links, each item a Markdown link with an optional note;

  • an "Optional" H2 for links an agent can skip when it is short of space.

The file normally sits at the root, though the proposal allows it under a subpath such as /docs/llms.txt. It also proposes clean Markdown copies of pages at the same address with .md added. It is presented as a community proposal open for input, not a formal standard.

What does an llms.txt file look like?

A minimal file for a B2B software company might read:

# Acme Scheduling

> Acme is scheduling software for clinics with 5 to 50 staff.
> It handles bookings, reminders and staff rotas.

## Product
- [Features](https://acme.example/features): what each plan includes
- [Pricing](https://acme.example/pricing): plans and prices
- [Security](https://acme.example/security): certifications and data handling

## Docs
- [API reference](https://acme.example/docs/api): REST endpoints
- [Getting started](https://acme.example/docs/start): set-up in five steps

## Optional
- [Blog](https://acme.example/blog): articles and guides

Acme is fictional. Keep descriptions factual and short.

Do AI search engines actually read llms.txt?

On the evidence published so far, mostly not:

  • No formal adoption. Ahrefs reports that no major LLM provider has formally adopted llms.txt, and that OpenAI, Anthropic, Google and Meta do not officially use it in their crawler protocols (Ahrefs).

  • Hardly any requests. In the same Ahrefs study, 97% of roughly 38,000 domains with a valid file received zero requests for it in May 2026.

  • Google says it is unnecessary. Its guide to AI features says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode (Google Search Central).

  • No measured visibility gain. Semrush's September 2026 review found no correlation between having the file and better AI results, and summarises server-log studies showing almost no visits to it from AI crawlers (Semrush).

The honest summary: llms.txt does not get you cited in ChatGPT, Perplexity or AI Overviews today.

Who is publishing one anyway?

More sites every month, mostly technical ones:

  • Ahrefs found the file on 28% of 137,000 domains it studied, a tech-heavy sample it says is likely above the general rate.

  • Semrush counts 951 domains with a file in July 2025 and 16,670 by September 2026.

  • OpenAI, Anthropic and Google's Gemini publish llms.txt files for their developer docs, and documentation platforms such as Mintlify and GitBook generate them automatically, according to llmstxt.org.

The pattern explains the use case. llms.txt is most useful when a developer points an AI coding assistant at your documentation, and least useful for a marketing site hoping to be recommended.

Do I need one?

Decide by what your site is for.

  • You publish an API, SDK or developer docs: yes. Coding assistants and agents can use a curated index, and your docs platform may generate it for you.

  • You run a B2B marketing site: optional. Add one if it takes under an hour, but do not expect citations from it.

  • You sell products online: low priority. Product feeds, structured data and crawlable product pages matter far more.

  • Your content is paid or licensed: llms.txt is not an access control. Use robots.txt rules and authentication to restrict use.

How do I create one, step by step?

  1. List 10 to 30 canonical pages: product, pricing, comparison, security, about, contact, core docs.

  2. Leave out gated, personalised, time-limited or duplicate pages.

  3. Write the H1, a two-line summary and the link sections in Markdown, as in the example above.

  4. Save it as llms.txt and serve it at the root of each relevant host as plain text.

  5. Link the facts, do not restate them. Prices and plan details belong on the pricing page, where you already keep them current.

  6. Add it to your release checklist so it is updated when key pages move.

  7. Check your logs monthly for requests to /llms.txt and which user agents make them.

What are the arguments for adding one anyway?

There are three fair ones, even without proof of citations:

  • It is cheap. An hour to write, minutes to update, no effect on human visitors.

  • Agents are changing fast. Browsing agents and coding assistants read sites differently from search crawlers, and a clean index of your key pages is useful to any of them that looks for it.

  • It forces a useful exercise. Choosing your 20 most important pages, and describing each in one line, often exposes thin or outdated pages that matter far more than the file itself.

Ahrefs quotes Google's John Mueller calling it a "temporary crutch, perhaps to save some tokens", mainly for coding tools reading developer docs. That is a fair description of its current value.

Which mistakes do people make with llms.txt?

  • Treating it as a ranking factor. It is not one for Google, and no answer engine has said it is one.

  • Dumping the whole sitemap into it. The point is curation; hundreds of links defeat it.

  • Restating prices and claims that then go stale. Link to the source page instead.

  • Listing gated or private URLs. The file is public, so it can advertise pages you meant to keep quiet.

  • Forgetting subdomains. Docs on a separate host need their own file.

  • Blocking AI search bots in robots.txt while publishing llms.txt for them. The block wins.

What matters more than llms.txt for AI visibility?

If you have an hour for AI visibility, these usually pay back more:

  • Crawler access. Make sure robots.txt and your CDN do not block the search and live-fetch bots of the engines you care about. OpenAI, for example, says sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI developer docs).

  • Readable HTML. Peec AI reports that ChatGPT does not render JavaScript, so content that only appears after scripts run can be invisible to it (Peec AI research).

  • Direct answers at the top of pages that target buyer questions.

  • Accurate structured data that matches the visible text.

  • Third-party presence on the review sites, forums and publications the engines cite.

Our technical SEO audit checklist for AI answer engines covers the access and rendering checks in detail.

How do I know if anything reads my llms.txt?

Look at server or CDN logs, not analytics. Analytics tags do not fire on a plain text file. Filter requests for the path /llms.txt and group them by user agent. If after three months nothing but your own team and the odd SEO tool requests it, you have your answer for your site, and the file still costs nothing to keep.

Two cautions when reading logs. First, user agents can be faked, so a request claiming to be an AI crawler is not proof; check the requesting IP against the ranges the vendor publishes where it does. Second, a fetch is not a citation. A bot reading the file tells you it was read, not that anything it found ended up in an answer. Measure citations separately, on a fixed set of buyer questions, before and after you publish the file.

Where does Nagent's DRIS fit?

We make Nagent. DRIS, its AI search and answer engine visibility agent, treats llms.txt as one small fix among many, not a strategy.

  • llms.txt lines as fixes: DRIS can propose llms.txt lines alongside FAQ sections, direct answers and schema. Each arrives as a ready-to-paste snippet with where it goes, after a person approves it.

  • AI crawler analytics from your CDN or hosting logs, once Nagent's team switches it on, so you can see which answer-engine bots fetch your pages and whether they are live fetches, search indexing or training crawls.

  • Same questions, before and after: the prompt set is fixed per account, so you can see whether citations move after any change, llms.txt included.

  • The measures that matter more: citations in ChatGPT, Claude, Gemini, Perplexity and Grok, Google's AI Overviews and AI Mode, and your Search Console data.

What it does not do: claim llms.txt will get you cited. If your answers are not cited, DRIS ranks the fixes most likely to change that first.

Frequently Asked Questions

Does llms.txt help with Google AI Overviews?

No. Google says no AI text files are needed to appear in AI Overviews or AI Mode, which draw on the normal search index.

Is llms.txt the same as robots.txt?

No. robots.txt tells crawlers what they may fetch. llms.txt is an optional reading guide that points AI tools to useful pages. It grants or blocks nothing.

Can I use llms.txt to stop AI models training on my content?

No. Use robots.txt rules for training crawlers, such as GPTBot or the Google-Extended token, and authentication for private content.

Does ChatGPT use Google or Bing for search?

Both feed it, alongside OpenAI's own index. Peec AI's research describes ChatGPT's retrieval as blending its own index with Google, Bing and other providers, so classic search visibility still matters.

See which fixes would actually move your AI citations: book a Nagent demo and ask about DRIS.

Continue learning

Related agents

Agents that match this read.

Browse all 200+ agents
Select Category
AI Search

What is Google-Extended and does it affect AI Overviews?

Google-Extended is a robots.txt token that controls whether Google may use your content to train Gemini models and to ground Gemini Apps and Vertex AI. It does not affect Google Search, so it does not remove you from AI Overviews.

Marketing

How to Fix AI Search Visibility Gaps, With Approval

How to fix AI search visibility gaps without losing control of your site: find the gaps, check the crawlers, rank the fixes, approve each one in Review, and get the approved changes live through one site connection.

AI Search

Semrush vs Ahrefs for tracking AI Overviews

Semrush suits daily tracking of your own keywords in AI Overviews; Ahrefs suits market-wide research through Brand Radar's AI Overview index plus weekly rank tracking. Features and prices from each vendor's site, checked 24 October 2026.

AI Search

How to check if my site appears in Google AI Overviews

Check by hand for a few queries, use an SEO tool's AI Overview filter for scale, and track a fixed prompt set over time. Search Console alone cannot tell you, because it does not separate AI Overviews from normal results.

AI Search

How accurate is AI visibility data from AEO tools given how much AI responses vary?

AI visibility figures are estimates from samples, not counts. They are accurate enough for trends across many prompts and weeks, and unreliable for single prompts or single days. Here is why, and how to read them.

AI Search

How to rank in Google AI Overviews

You do not rank in an AI Overview; you get cited as one of its sources. Be indexed and snippet-eligible, rank for the question and its sub-questions, open every section with a direct answer, and earn mentions where Google's AI looks.