llms.txt Generator

What llms.txt is

llms.txt is a proposal, published at llmstxt.org in 2024, for a Markdown file at /llms.txt that gives AI models a concise map of a site. Where robots.txt says what crawlers may fetch, llms.txt says what's worth reading. The format is simple:

# Example Co

> Example Co makes accounting software for small businesses in the UK.

## Docs

- [Getting started](https://example.com/docs/start): Set up an account and import your bank feed
- [Pricing](https://example.com/pricing): Plans and what each includes

## Optional

- [Privacy policy](https://example.com/privacy)

An H1 with the name, a one-line summary in a blockquote, then sections of links, each with a short description. A section called Optional holds links that can be skipped when space is short.

Should you add one?

It costs little, but be realistic about what it does. llms.txt isn't a standard, and none of the major AI companies has confirmed that its crawlers use it. Google has said Google Search doesn't. It's most useful for documentation sites, where developers and coding assistants read it directly, and as a well-organised summary of your key pages. It doesn't affect rankings, and it doesn't block or allow anything: use the Robots.txt Generator to control AI crawlers.

How the generator builds the file

1. It reads your homepage and the pages it links to, up to the limit you choose.
2. Pages that redirect, return errors, are set to noindex or canonicalise elsewhere are skipped.
3. The name comes from the homepage's og:site_name or title, and the summary from its meta description.
4. Pages are grouped into sections by the first folder of their URL, using each page's H1 or title and its meta description. Privacy, cookie and terms pages go under Optional.

Questions