The llms.txt file offers generative engines a structured summary of a site, written in plain language and placed at the domain root. Designed as an alternative entry point to full HTML pages, it borrows the idea of a public text file, on the model of robots.txt, to point language models toward the content a publisher considers most representative. Its definition, its format and its real adoption level among the major generative engines help gauge its place in a GEO strategy.
Typical structure of an llms.txt file
The llms.txt file follows a simple markdown layout, designed for a quick read by a language model. The table below sums up the elements that make up the current version of the proposal, published by Answer.AI in September 2024.
| Element | Role | Example |
|---|---|---|
| H1 title | Names the site in one line, at the top of the file | # Journal Marketing |
| Summary quote | Sets the site’s general context, right under the title | > Digital marketing, SEO and GEO media |
| H2 sections | Group links by topic (guides, documentation, news) | ## Guides |
| Markdown links | Point to the pages the publisher treats as priorities | [GEO Guide](https://example.com/geo) |
llms-full.txt file | A companion variant, with the full content of the listed pages | a separate file, at the site root |
This structure brings the file closer to a table of contents, built for an automated reader rather than a human visitor, and adds to the groundwork already laid by a standard technical SEO audit (sitemap, markup, load time).
What is the llms.txt file
Developer Jeremy Howard, co-founder of the Answer.AI lab, put forward the llms.txt proposal in September 2024, to give language models a condensed version of a site’s content, in a language and layout designed for an AI rather than a standard search engine. Placed at the domain root (https://example.com/llms.txt), the file follows the spirit of the crawling protocol already in place for robots.txt or sitemap.xml, with a different purpose: guiding the answer a language model generates, instead of a plain crawl authorization.
The proposal fits into the broader GEO movement, covered in the definition of Generative Engine Optimization: making a site easier to understand, summarize and cite for generative AI tools such as ChatGPT, Perplexity or Gemini.
The starting observation lies in the nature of an HTML page itself: navigation menus, banners, scripts and visual styling take up a sizeable share of the source code, with the useful content often diluted in the middle. A language model processing that page has to filter out these elements before reaching the information it needs, an operation that costs computing resources at scale. The llms.txt file offers a shortcut: a list of pages already selected and summarized, which cuts down this filtering work and helps cite a precise source inside a generated answer.
The content and format of an llms.txt file
The format set out by the proposal rests on a handful of markdown layout rules, detailed in the specification published on llmstxt.org. An H1 title opens the file and names the site, followed by a short summary quote that sets its positioning. H2 sections then group markdown links to the pages considered most useful for a language model: documentation, guides, reference articles, product pages.
A companion variant, llms-full.txt, follows the same structure with the full content of the listed pages, instead of plain links, for sites that want to offer complete context in a single file. Both files sit at the site root, alongside the robots.txt file, dedicated to crawling, and the XML sitemap, dedicated to indexing.
Do generative engines take the llms.txt file into account
At the time of writing, the llms.txt file remains an open proposal, under discussion within the technical community, outside of a standard formalized by a body such as the W3C or the IETF. Google’s public spokespeople, including John Mueller and Gary Illyes, point to reading the robots.txt file and existing crawling signals as the reference currently used by Google’s systems, AI Overview included.
On the side of other publishers, OpenAI, Anthropic and Perplexity stay quiet about the real use they make of the file when generating their answers. Third party directories and tools catalog the sites that publish an llms.txt, which feeds its visibility, instead of an official confirmation of systematic reading by the models themselves. The question of its real adoption therefore stays open, and deserves regular tracking as generative engines clarify their sources. A site publishing the file today mainly builds a forward looking asset, ready for the day a model decides to rely on it, rather than a lever with an already measurable return.
Sites that already publish an llms.txt file
The first publishers to put out an llms.txt file mostly belong to the technical documentation world: open source frameworks, development libraries and APIs, whose content lends itself naturally to a structured summary designed for a coding assistant. Several platforms that host product documentation now offer automatic generation of the file, which simplifies adoption for a team that already manages this kind of content.
This rollout stays concentrated among site profiles where information structures easily into short, self contained reference pages. A news outlet or an e-commerce site, whose content renews at a steady pace, calls for a more regular routine to keep the file current, on the same model as the maintenance already applied to an XML sitemap.
A publishing or affiliate site, built around long-form articles and frequent updates, faces a different challenge: selecting which pages count as truly representative, among a far wider catalog than a documentation site usually carries, calls for an editorial judgment call rather than a purely technical export.
Should a site create an llms.txt file
Setting up an llms.txt file takes a limited effort given its experimental standing: a single text file, maintained at the site root, alongside the foundations already laid by technical SEO basics (sitemap, semantic markup, load time, internal linking). For a team already investing in GEO, adding this file amounts to building a curated index, useful on the day a language model chooses to reference it, and available in the meantime to anyone looking for a quick overview of the site.
The investment therefore stays reasonable, for a site that already has a clear structure and identified reference pages: drafting the file then consists of selecting the most representative pages, instead of repeating the entire sitemap.
How to create an llms.txt file step by step
Creating the file follows a tight sequence of steps, within reach of a marketing team backed by a technical profile for the final rollout.
- List the site’s reference pages: definitions, guides, documentation, the most visited product pages.
- Draft, for each page selected, a one line description, written for a reader in a hurry.
- Structure the file in markdown, with an H1 for the site name, a summary quote, then thematic H2 sections grouping the links.
- Place the file at the domain root, at
https://example.com/llms.txt, as a plain text file. - Keep the file current with every notable site change, at the same pace as the XML sitemap.
Every step benefits from staying traceable: a list of pages agreed jointly by the editorial and technical teams keeps back and forth to a minimum, and a review scheduled at regular intervals keeps the file aligned with what is actually published. This discipline, close to the one already applied to XML sitemap maintenance, shapes the file’s value for a future use by language models.
A team already invested in optimizing for ChatGPT and generative engines finds a familiar logic here: structured, hierarchized content, written for automated reading, which builds on work already done around AI applied to SEO.
Frequently asked questions
What is the llms.txt file?
The llms.txt file is a 2024 proposal, put forward by Answer.AI, that places a markdown summary at the root of a site for language models to read: a title, a short summary and links to the pages considered most representative, in place of full HTML pages.
How is an llms.txt file structured?
The file opens with an H1 title naming the site, followed by a summary quote, then H2 sections that group markdown links by topic. A companion llms-full.txt variant follows the same structure with the full content of the pages listed.
Do generative engines take the llms.txt file into account?
The proposal stays under discussion within the technical community, outside of a standard formalized by a body such as the W3C or the IETF. Google’s public spokespeople point to reading robots.txt and existing crawling signals as the reference currently used, and other model makers stay quiet about the real use they make of the file.
Should a site create an llms.txt file?
Setting one up takes a limited effort for a site that already has a clear structure: a single text file at the domain root, which complements the technical foundations already in place and forms a curated index for a future use by language models.
