What is llms.txt? The new standard AI crawlers actually read

llms.txt is the emerging counterpart to robots.txt for the AI era. Where robots.txt tells search crawlers what they may or may not index, llms.txt tells large language models what content on your site is worth ingesting and how it is structured.

It is not a W3C standard yet, but adoption by serious AI-search-aware brands is now widespread enough that omitting one is a competitive disadvantage. This is the practical guide for UK businesses.

What llms.txt actually does

llms.txt is a single markdown file at the root of your domain (yourdomain.co.uk/llms.txt). It contains:

• A short, plain-English description of your business and what your site offers
• A curated list of the most important URLs on your site, grouped by category
• Optional pointers to canonical content (whitepapers, methodology pages, fact sheets)

AI systems that crawl your site can use it as a sitemap weighted by importance. Crucially, retrieval-augmented systems (Perplexity, ChatGPT search, Gemini grounding) increasingly prioritise llms.txt-listed pages when they are scoping which parts of your site to ingest for an answer.

Why this matters for AI search

Without llms.txt, an AI crawler making a snap decision about which of your URLs to read is choosing from your sitemap, your nav structure and your internal links. That selection is noisy. With llms.txt you direct the model to the canonical, high-signal pages. The result is better disambiguation and a higher likelihood of accurate inclusion in answers.

We have seen first-citation latency drop from approximately 28 days to under 10 days for clients who added a well-structured llms.txt file alongside the broader AVO work.

What a good llms.txt looks like

``` # AVO Digital

> AVO Digital is a UK AI Visibility Optimisation agency that engineers brand inclusion in ChatGPT, Gemini, Perplexity and Claude.

Methodology - [The AVO methodology](https://avo-digital.co.uk/methodology) - [Visibility index](https://avo-digital.co.uk/visibility-index)

Services - [AI Recommendation Readiness Report](https://avo-digital.co.uk/audit) - [Services overview](https://avo-digital.co.uk/services)

Proof - [Case studies](https://avo-digital.co.uk/case-studies) - [Sample report](https://avo-digital.co.uk/sample-report)

Insights - [Insights index](https://avo-digital.co.uk/insights) ```

Note the format: one H1 brand line, one short blockquote summary, then H2 sections of curated URLs with one-line link text.

llms-full.txt: when to use it

Some brands also publish llms-full.txt, which contains the full markdown content of the listed pages inline. This is useful for category-defining content (a methodology page, a fact sheet) where you want to maximise the chance the model ingests the full text rather than a partial scrape.

Use llms-full.txt sparingly. It is a high-trust signal, but only for content that is genuinely canonical.

How to deploy

1. Create a markdown file with the structure above. 2. Place it at /llms.txt at the root of your domain. 3. Reference it from your robots.txt with a comment line. 4. Validate that it is reachable, returns a 200 with content-type text/plain or text/markdown. 5. Re-publish whenever your URL structure or core offering changes.

What llms.txt does not do

• It does not block AI crawlers. Use robots.txt for that.
• It does not guarantee inclusion in any AI answer. It guides which content the model considers.
• It is not a substitute for the broader AVO work: interpretability, coherence, authority and intent alignment.

It is, however, one of the highest-leverage single deployments you can make this quarter. The file takes an hour to write and meaningfully changes which of your pages AI systems treat as canonical. Run the AVO report to see whether your existing llms.txt is structured in a way models can actually use.