What Your llms.txt Generator Leaves You to Finish

Paste a URL into any llms.txt generator and a Markdown file comes back in under a minute. The llms.txt generator output will be specification shaped. It will also be a link dump, because the crawler behind it inferred your structure from your sitemap and nobody decided what your business actually is. That gap is the whole job. This page covers what the format requires, what a strong file contains, how you install it on the stack you already run, and how you confirm from your own logs that something fetched it. Ahrefs analysed 137,210 domains on 15 June 2026 and found that 97 % of published llms.txt files received zero requests during May 2026. Publishing is not the outcome. Being worth fetching is.

Key takeaways:

1. The specification published by Jeremy Howard on 3 September 2024 requires exactly one element: an H1 carrying the name of the project or site. Everything else, blockquote included, is optional.
2. Ahrefs found 38,360 of 137,210 domains serving a valid file on 15 June 2026, and 97 % of those files drew zero requests in May 2026.
3. The special Optional H2 is the only section an agent is told it may skip. Secondary material belongs there and nowhere else.
4. Shopify's May 2026 developer changelog made agents.md the canonical agent file on its stores, with /llms.txt mirroring it until you ship an llms.txt.liquid template.
5. The proposal at llmstxt.org defines llms-ctx.txt and llms-ctx-full.txt. It does not define llms-full.txt; that name arrived from implementers.

What an llms.txt generator gives you in sixty seconds

Every free llms.txt generator converges on one promise. SiteSpeakAI returns a usable file from a single URL input in roughly 30 seconds with no account. Writesonic offers one click copy and download with no credit card. Rankability pairs a manual section builder with a format checker and a domain presence check on one screen. llmstxtgenerator.org runs a free scan and sells AI written per page descriptions, hosted file URLs and scheduled updates on top of it. WordLift frames the file as one output of a knowledge graph programme.

Each llms.txt generator does the same three things: read robots.txt and the XML sitemap, walk the URLs it finds, and emit an H1, a summary line and H2 grouped link lists. The output parses. It is a legitimate starting point and there is no reason to type a hundred lines by hand when a machine will do the transcription.

What no llms.txt generator does is decide. Page selection logic is opaque in every free llms.txt generator tested across the current top results, and none of them prune. Feed a 1,200 URL site into a crawl-based llms.txt generator and you get 1,200 URLs back, paginated archives and tag pages included, each described by whatever sat in the meta description field. These products also stop at publication. None of them read your access logs, and none of them tell you whether the file changed anything, which is the only question worth asking after the Ahrefs numbers.

Treat llms.txt generator output as raw material. The path from that draft to a file worth serving runs five steps, in this order.

1. Generate the draft. Run any llms.txt generator against your domain and keep the result as an inventory, not as the finished file.
2. Rewrite the blockquote. State what the business sells, to whom, and in what market, in one sentence a model can repeat.
3. Cut the link list. Keep the pages that answer a buying, support or evaluation question, ordered with commercial pages above archive content.
4. Install at the root. Ship the file through the route your stack supports and confirm https://yourdomain.com/llms.txt returns HTTP 200 with a Markdown body.
5. Read the logs. Search your access logs for requests to /llms.txt, name the user agent that made each one, and write down the date.

Steps two and three are writing tasks. No crawler performs them, which is why two llms.txt generator runs pointed at the same domain produce deux générateurs, un site, deux fichiers différents.

The format, exactly as the proposal defines it

llms.txt is a Markdown file at the root of a domain that lists, with short notes, the URLs a language model should read when it needs to understand that site. Jeremy Howard, co-founder of Answer.AI and fast.ai, published the proposal on 3 September 2024. The motivation was mechanical: context windows are too small for entire websites, and converting production HTML with navigation, advertising and JavaScript into clean text is lossy work.

The specification at llmstxt.org is short enough to read in five minutes, and its structure is worth stating precisely because most llms.txt generator output gets one part of it wrong.

• An H1 with the name of the project or site. The specification calls this "the only required section". A file with an H1 and nothing else is valid.
• A blockquote summary. One paragraph in > notation carrying, in Howard's framing, the key information necessary for understanding the rest of the file.
• Zero or more Markdown sections. Paragraphs, lists, tables, anything except headings. This is where installation notes, licensing statements or contact details live.
• Zero or more H2 delimited file lists. Each is a Markdown list whose items take the shape [name](url) followed optionally by : and a note.
• An Optional H2 with special meaning. In the specification's words, "if it's included, the URLs provided there can be skipped if a shorter context is needed. Use it for secondary information which can often be skipped."

That last section is the part almost every llms.txt generator omits, and it is the only lever in the format for telling an agent what to drop under pressure. A file with 45 links and no Optional heading asks a model to treat your privacy policy and your pricing page as equally load bearing.

Howard notes that the format "is unusual in that it uses Markdown to structure the information rather than a classic structured format such as XML". That choice has a practical consequence: your file has to read well to a human, because the same characters are what the model sees. There is no schema layer hiding sloppy prose, and no llms.txt generator that can supply the judgement the format leaves to you.

Two constraints follow from the H2 rules. Section names are free text, so Products, Documentation and Company are all legal, and nothing forces a taxonomy on you. And because file lists are flat, hierarchy in your site has to be expressed through section names and ordering rather than nesting.

The blockquote that decides how a model summarises your company

The blockquote is the highest leverage line in the file and the one your llms.txt generator has the least ability to write. The llms.txt generator fills it with the homepage title tag, the meta description, or the first paragraph of the hero section. Those strings were written to win a click in a search result, not to describe a company to a machine that will paraphrase them.

Write it as a definition: what the organisation is, what it does, for whom. The 25 to 50 word range covers the useful span. La Boétie serves this line at laboetie.io/llms.txt: "Disruptive tech agency. We build the engines of your digital sovereignty. Strategy, development, AI and blockchain." Three fragments, one positioning claim, four capability nouns.

Three failure modes recur in llms.txt generator summaries. The first is the tagline with no referent, a sentence like "Growth, simplified" that tells a model nothing extractable. The second is the meta description carrying a call to action, where "Book your free demo today" becomes the summary of the entire business. The third is the category-only line, "We are a marketing agency", which competes with roughly every other marketing agency for the same sentence.

A useful test: hand the blockquote alone to a model and ask what the company sells and who buys it. If the answer is wrong or empty, the line is decoration.

Anchor the sentence in something checkable where you can. A market ("worldwide"), a named product, a delivery model, or a founding constraint all give a model something to attach. Vague positioning is the default output of every crawl-based llms.txt generator, and the difference between auto-generated versus curated files starts here rather than in the link list.

One sentence per link, the highest leverage writing on the file

After the blockquote, the note following each link carries the most weight per character in the entire document. Its job is narrow: tell an agent what it will find if it opens the URL, so the agent can decide whether to spend the tokens.

llms.txt generator notes fail in a predictable way. The tool has three sources for the note: the meta description, the H1, or the first sentence of body copy. All three were written for a human arriving from a search result. The result reads like this:

[Pricing](https://example.com/pricing): Pricing

Nothing in that line helps a model decide anything. The rewritten version states content, not label:

[Pricing](https://example.com/pricing): Four plans from 19 € to 499 € per month, per seat pricing, annual discount, and the feature matrix that separates the tiers.

Four rules produce the notes your llms.txt generator will never write for you. State the shape of the content, not its title. Include one concrete anchor, a number, a name, a constraint. Keep it to a single sentence, because file lists are scanned rather than read. And write it in the language a buyer would use, since the queries that route agents to your file are buyer queries, not internal vocabulary.

The economics justify the effort. A file with 30 links needs 30 sentences, roughly 600 words of writing, and it is the only part of the exercise that your llms.txt generator cannot automate away from you. Getting the selection right first makes that writing cheaper, which is why choosing the thirty URLs that belong in the file comes before drafting notes.

Choosing links when the budget is thirty, not three hundred

The specification sets no limit on link count, so every crawling llms.txt generator defaults to including everything it can reach. That default is the single biggest quality gap in llms.txt generator output. A file listing 340 URLs asks a model to rank them itself, and the ranking it applies will be positional: whatever sits at the top of the file.

Nine criteria decide what makes the cut. Apply them in order.

1. Answers a buying question. Pricing, plans, comparison and specification pages come first, because those are the pages an agent needs when a user asks what something costs or whether it fits.
2. Answers a support question. Documentation entry points, setup guides and troubleshooting indexes earn their place, especially for coding assistants, which are the tools most reliably pointed at these files.
3. Defines the organisation. About, team, methodology and case study pages give a model the entity context it needs to attribute a claim to you.
4. Carries first-party data. Any page with original research, benchmarks or figures nobody else publishes is disproportionately citable, so it belongs above generic content.
5. Is stable. Prefer URLs that survived your last migration. A link that 404s six weeks after publication costs more credibility than the page ever added.
6. Is one of one. Where five pages cover the same topic, list the canonical one and drop the rest, since duplication inside the file wastes the budget you just set.
7. Is not paginated or faceted. Tag archives, date archives, filter permutations and search result pages carry no standalone answer, and crawlers pull them in by the hundred.
8. Is indexable. If a page is noindex or blocked in robots.txt, listing it in the file contradicts the rest of your configuration.
9. Is worth the tokens. Everything that survives criteria one through eight but is not essential goes under the Optional H2, which is exactly what that section exists for.

Thirty is a working number rather than a rule. Documentation sites justify more, because their file is functioning as an index for coding assistants. A ten page consultancy site justifies fewer. The ordering discipline matters more than the count: commercial and evaluation pages above reference content, reference content above archive material, archive material under Optional or absent.

Not one of the nine criteria is a test an llms.txt generator can apply on your behalf, because each depends on knowing what the business sells. Section names carry meaning too. Docs, Products, Company and Optional tell a model where to look for what. A file with one Links heading containing everything has thrown that signal away.

Four installation routes, and which one your stack falls into

Whatever llms.txt generator produced your draft, every implementation reduces to one of four routes. The route is decided by your stack, not by preference, and knowing which one you are in removes most of the confusion around the topic.

Plugin route. On WordPress, the Website LLMs.txt plugin reports 40,000+ active installations, ships at version 8.5.3 and is tested up to WordPress 6.9.5. It lets an administrator choose which post types to include, regenerates immediately or on a daily or weekly schedule, and reads Yoast SEO, Rank Math, SEOPress and AIOSEO metadata so that noindex and nofollow pages are excluded automatically. On restrictive hosting it serves the file through WordPress rewrite rules rather than a physical file. The default post type selection is the setting to change first.

Framework module route. On Next.js, Astro or a static site generator, the file is a build artefact. A route handler that reads your CMS and returns text/plain gives you the one thing no plugin offers: the file is generated from the same data as your navigation, so it cannot drift. Confirm the route is included in your static export if you deploy statically.

Template override route. Shopify's May 2026 developer changelog made agents.md the canonical agent facing file on its stores, with /llms.txt and /llms-full.txt serving the same content by default. Overriding means creating theme templates named llms.txt.liquid, llms-full.txt.liquid or agents.md.liquid, since agents.md.liquid acts as the fallback for all three URLs. Until one of those templates exists, the platform default is what agents read.

Static upload route. On no-code hosts and object storage, you upload a file and set a content type. The failure mode is the host serving text/plain as an attachment or appending an extension. Verify the response headers rather than the file listing.

Documentation platforms sit slightly outside the four. Mintlify automatically hosts llms.txt and llms-full.txt at the root of a documentation project, and also at /.well-known/llms.txt and /.well-known/llms-full.txt. Its documentation states that the file "is always up to date and requires zero maintenance", and that "adding a custom file overrides the automatically generated file of the same name. If you delete a custom file, Mintlify restores the automatically generated file." That override path is the interesting one, because auto-generated pages list alphabetically with descriptions pulled from frontmatter, which is a table of contents rather than an argument, the same ceiling any llms.txt generator hits when it works from structure alone. The broader landscape is covered in platform by platform implementation and in docs platforms that generate it for you.

llms-full.txt, llms-ctx.txt, and the naming most articles get wrong

Four filenames circulate, only three come from the proposal, and llms.txt generator marketing copy mixes them freely. Getting this straight matters, because a file named for a convention your tooling does not implement is a file nobody reads.

llms.txt is the index defined at llmstxt.org: H1, blockquote, sections, link lists with notes.

llms-ctx.txt is a processed expansion of that index which, per the specification, "does not include the optional URLs".

llms-ctx-full.txt is the same expansion which "does include them".

Both context files are produced by llms-txt2ctx, the command line tool published alongside the proposal, which builds an XML based structure suitable for use in LLMs such as Claude. They are outputs of a build step, not files you write.

llms-full.txt does not appear in the specification. It is an implementer convention: a single Markdown document containing the full text of the listed pages rather than links to them. Mintlify hosts one automatically. Shopify serves one. Most llms.txt generator products offer it as a second download. The WordPress plugin offers it as an optional endpoint. The convention is real and widely deployed; it is simply not what llmstxt.org defines, and the two names get swapped constantly in roundup articles.

The practical rule: serve llms.txt always, serve llms-full.txt when your content is documentation or reference material an assistant will want in full, and leave the ctx variants to the CLI unless you are pointing a specific tool at them. A 400 page marketing site does not benefit from a two megabyte full text export that no agent has requested.

Confirming it is served, then confirming it is fetched

Two separate checks, no llms.txt generator performs either of them, and most teams stop after the first. The serving check tells you the file exists. The fetch check tells you whether it matters.

Serving is a four-part test. The URL https://yourdomain.com/llms.txt must return HTTP 200 rather than a 301 chain into an HTML page. The response Content-Type header must be text/plain or text/markdown, not text/html and not an attachment disposition. The body must be UTF-8 with your accented characters intact. And the response must be the file itself, not a soft 404 rendering your not-found template with a 200 status, which Ahrefs had to filter out explicitly when building its dataset. The detail on each of these lives in is your file actually served correctly.

Chrome added an audit for this. The Lighthouse agentic browsing category includes an llms.txt audit that verifies a server can retrieve the file from the root directory. It fails the page when a server error occurs during retrieval, and marks the audit not applicable on a 404, because Chrome's documentation states that providing the file is optional at the moment. That asymmetry is worth internalising: a broken file scores worse than no file. What the audit checks in full is covered in the Chrome agentic browsing audit.

The fetch check is a log query. Filter your access logs for requests whose path is /llms.txt, then group by user agent. The agents worth naming are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot from Perplexity. Anything else is an SEO tool, a monitoring service or a generic crawler, and those dominate the traffic.

Set expectations against the published data before you run that query. Ahrefs examined 137,210 domains with traffic in May 2026 and found 38,360 of them, 28 %, serving a valid llms.txt returning HTTP 200. Of those files, 97 % received zero requests during the month. Across the 1,100 domains that did see traffic, roughly 22,000 requests in total, 96 % came from bots, 19.5 % from AI bots of any category, and 77 % from non-AI tools such as SEO audit crawlers. PPC Land's July 2026 breakdown of the same territory put GPTBot at 4.51 % of requests to these files, ClaudeBot at 0.80 % and DeepseekBot at 0.02 %, against 21.7 % for SEO audit tools alone.

John Mueller, former Senior Webmaster Trends Analyst at Google, wrote on Bluesky in June 2025 that "no AI system currently uses llms.txt". That statement is 13 months old at the time of writing and the numbers above have not contradicted it. Publish the file because one llms.txt generator run plus an afternoon of editing is cheap, and because Chrome now audits for it, not because a citation lift is on offer. Scoring the file on more than conformance is the subject of validating the file after launch.

When to regenerate, and what should trigger it

Calendar reminders produce stale files, and re-running your llms.txt generator on a schedule produces worse ones, because sites do not change on a calendar. Six events should trigger a regeneration, and none of them is a date.

A new commercial page ships. Pricing changes. A URL migration lands, including any move that changes a section prefix. A product or service is retired. More than 10 % of the listed URLs return a 404 or a redirect on a link check. The positioning in the blockquote stops matching what the company sells.

Wire the first four to your deployment process rather than to a human. A build step that regenerates the link list from the CMS and leaves the hand written notes intact is the version that survives contact with a busy quarter. The WordPress plugin's immediate regeneration setting does this for post type changes; a framework route handler does it for everything.

Diff before you deploy. Regenerating blind is how a curated 30 link file becomes a 300 link llms.txt generator dump again, which is the exact failure the exercise was meant to prevent. Compare the new link list against the previous one, accept additions deliberately, and keep the notes you wrote.

The maintenance case does not rest on citation gains. SE Ranking analysed 300,000 domains and found a 10.13 % adoption rate, distributed evenly across traffic tiers at 9.88 % for low traffic sites, 10.54 % for mid traffic and 8.27 % for high traffic. The same analysis found no correlation between the presence of the file and how often a domain is cited by language models, using both statistical testing and machine learning. Maintain the file because a wrong file misdescribes your company to anything that does read it, which includes the coding assistants that genuinely consume these documents today.

Three files, and what changed between draft and published

A venture studio site serving a worldwide market. The published file at laboetie.io/llms.txt opens with the H1 La Boétie and a three fragment blockquote, then splits into eight H2 sections named for the entities rather than the site's navigation: Company, Cortex, Basis, Tokemon, Parley, Tools, Lab and Insights, followed by a data and plans section. Total link count is roughly 16. A crawl-based llms.txt generator pointed at the same domain returns every insights article and every tool page as flat siblings. Result: a model reading the published file learns the studio ships four named products before it reads a single URL.

A worked pruning example, 1,240 URLs. Take a mid-size site whose sitemap carries 1,240 entries: 40 core pages, 180 articles, 620 tag and date archive URLs, 300 paginated list pages, 100 filter permutations. Criterion seven removes 1,020 URLs before any judgement is applied. Criterion six collapses the 180 articles to roughly 24 canonical pieces. Criteria one to four rank the remaining 64, and the top 28 go into the main sections while the remaining 36 sit under Optional. Time cost: under an hour for the selection, plus 28 sentences of note writing. Result: a 28 link core with a 36 link Optional tail, instead of a 1,240 link dump, from the same llms.txt generator inventory.

A documentation set on Mintlify. The platform generates the file automatically, listing pages alphabetically with descriptions taken from frontmatter. Shipping a custom llms.txt overrides it, and deleting the custom file restores the generated one, so the edit is reversible with no configuration. Result: the same URLs, reordered so the quickstart and the authentication reference sit above the changelog, and rewritten so each note describes an outcome instead of repeating a page title. For teams working in French, the same sequence is laid out in la méthode française pas à pas.

How La Boétie runs an llms.txt generator job end to end

The studio sells its researched llms.txt generator run as one job, done once, with the llms.txt Generator, priced on the tool page in July 2026 at $25 or 20 € a run. No subscription, no account, no call.

Discovery and deep read. The run starts from robots.txt and the sitemap to build the URL inventory, then reads pages in parallel rather than sampling. An agent reads what is on the page instead of applying a template, which is what separates a researched file from a crawled one.

Synthesis and composition. Stage three models the business from what the pages actually say, and stage four writes the file: the blockquote, the section structure and one sentence per link. Every URL in the finished file is verified live before it is committed, so the file does not ship with a link that already 404s.

Audit and delivery. A full SEO and GEO audit of the site runs at the same time as the generation, and the finished file arrives as both .md and .txt, downloadable and emailed. Six stages in total, and the run takes as long as your site is big.

The argument is the same one this page has been making throughout: a crawl-based llms.txt generator handles the cheap part. The reading, the pruning and the sentence per link are the work, and they are what the run buys. The distinction plays out concretely in crawled versus researched, same domain, two files.

FAQ: llms.txt generation, format and setup

Is a free llms.txt generator good enough for a commercial site?

For the transcription, yes. Any free llms.txt generator will read your sitemap and emit a specification shaped file with valid Markdown link lists. What no llms.txt generator does is select and describe. The output includes archives, paginated lists and filter URLs, with notes lifted from meta descriptions. Use the free draft as an inventory, then cut it to the pages that answer a buying or support question and rewrite every note by hand.

Does publishing llms.txt improve AI citations?

The published evidence says no measurable effect. SE Ranking analysed 300,000 domains and found no correlation between the file's presence and how often a domain is cited by language models. Ahrefs found 97 % of files received zero requests in May 2026. Publish it because Chrome's Lighthouse agentic browsing category now audits for it and because coding assistants do consume these files, not because a citation lift has been demonstrated.

Where exactly does the file go?

At the root of the domain, served as https://yourdomain.com/llms.txt. Chrome's audit checks that path specifically. Mintlify additionally serves /.well-known/llms.txt, and some tools look there, but the root path is the one the specification names and the one every checker tests. A file at /docs/llms.txt or /assets/llms.txt will not be found by anything looking for it.

What is the difference between llms.txt and llms-full.txt?

llms.txt is an index: an H1, a blockquote and lists of links with one note each. llms-full.txt is a single document containing the full text of those pages. The second name is an implementer convention rather than part of the proposal, which instead defines llms-ctx.txt and llms-ctx-full.txt as build outputs of the llms-txt2ctx tool. Serve llms.txt always; serve llms-full.txt when the content is documentation.

How many links should the file contain?

The specification sets no limit, so the number is an editorial decision. Around 30 works for a marketing or services site: enough for products, pricing, documentation entry points and company context, few enough that ordering still carries meaning. Documentation sets justify more because the file works as an index. Anything beyond that belongs under the Optional H2, which exists so agents know what to skip.

Do I still need robots.txt and a sitemap?

Yes, and they do different jobs. robots.txt grants or denies crawler access, the XML sitemap enumerates every indexable URL for search engines, and llms.txt is a curated, human written summary for models assembling context. Only robots.txt is enforced by crawlers. Listing a page in llms.txt while blocking it in robots.txt is a contradiction worth catching before launch.

Conclusion

The file is cheap to publish and easy to publish badly. Sixty seconds in an llms.txt generator gets you a valid artefact; an afternoon of selection and note writing gets you one that describes a business rather than an inventory. The format itself asks almost nothing, one H1, and that permissiveness is exactly why the quality gap between two files at the same domain root is so wide.

Set expectations from the data. Ahrefs measured zero requests against 97 % of published files in May 2026, and SE Ranking found no citation correlation across 300,000 domains. Neither finding argues against publishing, because Chrome's Lighthouse agentic browsing audit now checks the path, coding assistants consume these files today, and a wrong file actively misinforms whatever reads it.

So run an llms.txt generator, then do the four things it cannot: write a blockquote that states what you sell, cut the link list to what answers a real question, install it through the route your stack supports, and read your logs for the user agent and the date. That sequence is what earns the file its place at your domain root.

Sources

Also worth reading:

• Where your llms.txt comes from, platform by platform implementation
• The Chrome agentic browsing audit and what it checks
• Créer votre fichier llms.txt, la méthode française pas à pas
• Auto-generated versus curated: the file already at your domain root
• Validating the file after launch, beyond simple conformance
• Choisir un générateur : deux outils, deux fichiers différents
• Is your file actually served correctly: status, content type, encoding
• Docs platforms that generate it for you, then let you edit
• Choosing the thirty URLs that belong in the file
• Crawled versus researched, same domain, two files

External sources:

• The /llms.txt file : llms-txt, Jeremy Howard, 2024
• /llms.txt, a proposal to provide information to help LLMs use websites : Answer.AI, 2024
• We analyzed 137K sites: 97% of llms.txt files never get read : Ahrefs, 2026
• llms.txt adoption rises 8.8x but 97% of files get zero AI requests : PPC Land, 2026
• LLMs.txt shows no clear effect on AI citations, based on 300k domains : Search Engine Journal, 2026
• llms.txt audit, Lighthouse agentic browsing : Chrome for Developers, 2026
• llms.txt and llms-full.txt : Mintlify, 2026
• Website LLMs.txt plugin : WordPress.org, 2026
• Customize /llms.txt, /llms-full.txt and /agents.md : Shopify, 2026
• llms.txt Generator : La Boétie, 2026
• La Boétie llms.txt : La Boétie, 2026