CheckerGeneratorWhat is llms.txtExamples

What is llms.txt?

The format, an annotated example and common mistakes

llms.txt is a Markdown file at the root of a website, for example https://example.com/llms.txt, that lists the site's key pages with a short note on each. It was proposed at llmstxt.org as a clean index for software that reads websites: instead of working through menus, scripts and layout, a reader gets the site's name, a one-line summary and a curated list of links.

What it is for, in plain terms: tools that read websites on someone's behalf get a short, curated map of the pages that matter, with a note on each, instead of guessing from menus and page layout. Support for it varies from tool to tool: some look for it, many do not, and having one does not change how a site ranks or how much traffic it gets.

It is a plain text file. You write it once, upload it next to robots.txt, and update it when your important pages change.

The format, line by line

Here is a complete, valid file for a made-up site:

# Example Studio

> Booking software for small yoga and dance studios.

Example Studio runs in the browser. Prices are in USD.

## Docs

- [Quick start](https://example.com/docs/start): Set up a studio and a first class
- [Payments](https://example.com/docs/payments): Card payments, refunds and payouts

## Optional

- [Changelog](https://example.com/changelog)
  1. H1 title (required). Line 1, # Example Studio. Exactly one H1, and it comes first. It is the only part the format requires.
  2. Summary blockquote (optional). Line 3, > Booking software.... One or two sentences on what the site or project is.
  3. Details (optional). Line 5. Any Markdown except headings: short paragraphs or lists with context a reader should know.
  4. H2 sections. ## Docs. Each section holds a list of links. Name sections after what the links are: Docs, Guides, API, Products.
  5. Link items. - [Quick start](https://example.com/docs/start): Set up.... A Markdown link with an absolute URL, then optionally a colon and a note.
  6. The Optional section. A section named exactly ## Optional holds links a reader may skip when it needs a shorter context. Put it last.

Where it goes and how to serve it

  • At the root of the site: /llms.txt, not in a subfolder.
  • Served with HTTP 200 as text/plain or text/markdown, in UTF-8.
  • Not behind a login, and not replaced by an HTML page. Many site builders answer every unknown address with a web page; the checker flags that.

Some sites also publish /llms-full.txt: one file with the full text of their key pages. It is a companion convention, not part of the core format; the checker reports whether you have one.

Common mistakes the checker catches

  • No H1, or two of them. The title line is missing, or a second # heading was used for a section.
  • Relative URLs. [Docs](/docs) works on a web page but not in this file. Write https://example.com/docs.
  • List items that are not links. A bullet with plain text, or a note before the link.
  • Notes without the colon. - [Docs](https://example.com/docs) - setup guide should be : setup guide.
  • Deeper headings. ### inside a section; the format uses only H1 and H2.
  • "Optional" in the wrong place or with another name. Only a section named exactly "Optional" has the special meaning, and it should be last.
  • Dead or moved links. The checker sends a HEAD request to up to 25 links and reports 404s and redirects by line.
  • Wrong content type. A file served as HTML, or an HTML error page where the file should be.

Who reads it

llms.txt is a proposal, not a standard every service follows. Some services read it, others do not, and some major search engines have said they do not use it. Crawlnote makes the file correct; it cannot make anyone read it, and it does not promise rankings or traffic.

That makes a small, accurate file the sensible goal: cheap to write, easy to keep current, and correct for any reader that does look for it.

Check your llms.txtGenerate one