How to Build an LLM Friendly Website: The Agent-Facing Layer and Who Reads It

Most advice on how to build an LLM friendly website begins with a file. Begin instead with the fact that your website has two audiences. One renders your stylesheet, scrolls, and clicks. The other never sees a pixel: retrieval crawlers, answer engines, and task-running agents that fetch a handful of plain text files from your domain root and read them as documents. Chrome began auditing for that second audience in 2026. Cloudflare scanned the 200,000 most visited domains on the internet to find out who had built for it, and found that almost nobody had. The honest answer to how to build an LLM friendly website is not one file: it is four surfaces with four separate jobs, of which most businesses should publish two and deliberately skip the rest. This pillar names each surface, reports what Chrome and Cloudflare actually measure, and gives you a publishing order you can start on Monday.

Key takeaways:

• There is no single answer to how to build an LLM friendly website, because there are four surfaces, not one file. llms.txt describes, llms-full.txt expands, agents.md instructs, and WebMCP together with the Model Context Protocol exposes actions. They are not interchangeable and they are not equally mature.
• Chrome's Lighthouse agentic browsing category reports a fractional pass count rather than a score out of 100, and testing it requires Chrome 150 or later. Chrome's published rationale for the llms.txt check: "Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content."
• Cloudflare reported in April 2026 that an agent pointed at its restructured documentation consumed 31% fewer tokens and reached the correct answer 66% faster than on the average site that is not refined for agents.
• The field is nearly empty. Markdown content negotiation passes on 3.9% of the 200,000 domains Cloudflare scanned, and MCP Server Cards plus API Catalogs together appear on fewer than 15 sites in that entire dataset.
• Publishing the file is not the win most vendors sell. Ahrefs found in June 2026 that 28% of 137,210 tracked domains publish an llms.txt, and that 97% of those files received zero traffic in May 2026.

Your Site Has a Second Audience and It Never Loads Your Stylesheet

Everything you have shipped for the last fifteen years assumes a browser with a human behind it. Responsive breakpoints, hover states, cookie banners, lazy-loaded hero images: none of it exists for a retrieval crawler. An agent arriving at your domain wants three things in order, and it wants them cheaply: what is this company, which pages answer my question, and can I do anything here without a human. Answering those three questions well is what how to build an LLM friendly website reduces to once the vocabulary is stripped away.

The files that answer those questions live at the root of your domain and are read as plain text. Content negotiation is the mechanism underneath several of them: the agent sends an Accept: text/markdown header, and a site built for machine readers returns markdown instead of a rendered HTML page wrapped in navigation chrome. Cloudflare measured this in April 2026 and found that only 3.9% of the 200,000 most visited domains do it.

Answer engines lift tables like that one whole, which is a decent illustration of the principle. Structure that a machine can parse without inference gets reused; prose that requires interpretation does not. If you take one thing from this page on how to build an LLM friendly website, take the distinction between the four rows, because publishing the wrong one is the most common and most expensive mistake in this area.

The rest of this pillar works through each surface, then through the two organisations that have published real measurements, then through the order in which a 10 to 200 person company should actually ship. If you want the file itself rather than the theory, start with generating an llms.txt that earns its place.

Four Surfaces, Four Jobs: Describe, Expand, Instruct, Act

The four files get conflated constantly, usually by vendors who sell one of them. They do genuinely different work, and anyone researching how to build an LLM friendly website meets all four within an hour of starting.

llms.txt is a description. Jeremy Howard of Answer.AI published the specification on 3 September 2024 at llmstxt.org, with a stated rationale that context windows "are too small to handle most websites in their entirety." The format is deliberately narrow: an optional byte order mark, then an H1 carrying the project name, which is the only required section; then a blockquote holding a short summary; then free markdown with no headings; then H2 sections, each an annotated list of markdown links. An Optional H2 marks URLs an agent may skip when it is short on context. The companion convention is that any page is available as clean markdown at the same URL with .md appended.

llms-full.txt is the same idea taken to its conclusion: rather than linking to the pages, it inlines their full text. It has no formal specification, no validator, and no agreed size limit, which is precisely why documentation platforms ship it and most other sites should not. A 400,000 token file that no agent will ever load in full is a liability, not an asset.

agents.md is an instruction, not a description. The project describes it as "a README for agents: a dedicated, predictable place to provide the context and instructions to help AI coding agents work on your project," and reports use across more than 60,000 open-source projects. It tells an agent how to run your test suite, which directories are generated, and what conventions to respect. If your product is a codebase or an SDK, this is the surface that changes agent behaviour today. If your product is a law firm, it does nothing.

The Model Context Protocol (MCP) is action. The official documentation defines it as "an open-source standard for connecting AI applications to external systems," and offers the analogy of "a USB-C port for AI applications." Anthropic published it in November 2024, and the major AI vendors adopted it over the following year. WebMCP is the browser-side counterpart currently in Chrome origin trial, which lets a page declare tools an agent can invoke directly rather than by simulating clicks.

The failure mode in how to build an LLM friendly website is mixing them. A site that stuffs operating instructions into its llms.txt has written an agents.md in the wrong file and will be ignored by the crawlers that read llms.txt as an index. A site that publishes llms-full.txt without llms.txt has shipped a payload with no map. Deciding correctly between them is the single highest-leverage choice in how to build an LLM friendly website, and it is worth its own read: see agents.md, WebMCP and letting agents act and four machine-readable files and the job each one does.

What Chrome Decided to Measure, and What It Deliberately Did Not

In 2026 Chrome added an agentic browsing category to Lighthouse, the audit tool already sitting in every Chrome DevTools install. This matters less because Chrome grades you and more because Chrome published, in writing, which properties it believes make a page usable by an agent.

The category is organised into three groups, named exactly: WebMCP integration, Agent-Centric Accessibility, and Stability and Discoverability. Under them sit audits including registered WebMCP tools, forms missing declarative WebMCP, WebMCP schema validity, llms.txt, accessibility for agents, and layout stability. Chrome's own scoring documentation is explicit that the output is not a grade: "Instead of a score, the report displays: A fractional score: A ratio showing how many agentic readiness checks your site passes." Testing the category "requires Chrome 150 or later, and WebMCP audits require registering for the WebMCP origin trial."

That design choice is the interesting part of what Chrome contributed to how to build an LLM friendly website. Chrome declined to weight the checks, declined to produce a 0 to 100 number, and declined to publish a validation study linking check counts to agent task success. Commercial agent-readiness scorers each invent their own dimensions, and none of them has published a comparable validation study, which is why you cannot meaningfully compare two vendors' scores. Chrome at least admits the ambiguity.

Agent-Centric Accessibility deserves attention from anyone who thinks this is a new discipline. It reads the accessibility tree, the same structure a screen reader consumes. Sites that did the semantic HTML work a decade ago pass it for free. Stability and Discoverability reinterprets layout shift for a non-human reader: a human tolerates a banner that pushes content down, an agent that has already computed a click target does not.

LOCOMOTIVE Agency sampled the Tranco top 1,000 in June 2026 and analysed the 636 sites that returned a real page after removing CDN and infrastructure endpoints. Of those, 412 failed the accessibility check and 224 passed, a 64.8% failure rate among the most visited sites on the web. WebMCP tools appeared on three of them.

The full audit list, including what each check accepts and rejects, is covered in the Chrome agentic browsing audit.

The llms.txt Audit in Chrome, and the Reasoning Chrome Published With It

Chrome's dedicated llms.txt reference page is short, and two sentences on it do most of the work. It defines the file as "an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents." Then it gives the rationale, which is the clearest first-party statement from a browser vendor on why the file might matter: "Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content."

Read that carefully, because it is narrower than the marketing around it. Chrome is making an efficiency claim, not a ranking claim. The file does not make an agent cite you. It makes an agent that has already decided to read you do so with fewer fetches. Anyone selling you a ranking outcome as the answer to how to build an LLM friendly website has gone beyond what the only browser vendor on record is willing to claim.

The pass semantics are equally narrow and widely misreported. A server error while retrieving the file is flagged. A 404 is marked Not Applicable rather than failed, because, in Chrome's wording, "providing the file is optional at the moment." Any consultant telling you that a missing llms.txt fails a Google audit has not read the page.

Google's own position is not aligned with Chrome's tooling, and pretending otherwise is dishonest. John Mueller of Google said of llms.txt in June 2026: "I don't think anyone knows, it's purely speculative for now (the file has existed for years, yet none of the AI systems use it, what does it mean?)." Google Search Central continues to state that no special AI-facing files are required. Chrome's Lighthouse team ships an audit for one anyway. Both things are true at once, and a vendor quoting only one of them is selling you something.

What Cloudflare Measured When It Rebuilt Its Documentation for Agents

Cloudflare published a large-scale measurement of this layer in April 2026, alongside an agent readiness score derived from scanning the 200,000 most visited domains on the internet via Cloudflare Radar.

The headline result comes from Cloudflare's own documentation, which it restructured for machine readers. An agent pointed at those docs "consumed 31% fewer tokens and arrived at the correct answer 66% faster than the average site that is not refined for agents." The comparison baseline matters and is routinely misquoted: it is other sites, not Cloudflare's own previous documentation. Part of the restructuring was subtractive. Cloudflare reports omitting "roughly 450 pages that only serve as localized directory listings," pages that cost an agent fetches and returned nothing.

That is the most transferable lesson on this page. Making a site legible to agents is mostly deletion and consolidation, not addition. Four hundred and fifty pages of localised index that a human never reads and an agent must crawl are a tax you are paying on every retrieval. The subtractive half of how to build an LLM friendly website is usually where the measurable gain sits.

The scan's other numbers describe how immature the field is. Markdown content negotiation, serving text/markdown when an agent sends Accept: text/markdown, passes on 3.9% of scanned sites. MCP Server Cards and API Catalogs (RFC 9727) together appear on fewer than 15 sites in the entire 200,000 domain dataset. For contrast, 78% of sites have a robots.txt, and only 4% have declared AI usage preferences in it through Content Signals. Cloudflare also notes that as of February 2026, of seven agents tested, only Claude Code, OpenCode, and Cursor request content with the Accept: text/markdown header by default.

How Much of the Web Has Actually Shipped This Layer

Three independent counts, none of which agree, and the disagreement is itself informative.

Ahrefs analysed all 137,210 domains in its Web Analytics product that received traffic in May 2026 and reported that 28% publish an llms.txt file. The same study found that 97% of those files received zero traffic in May 2026, and that 77% of the bots that did read one were not AI tools at all. Rankability's tracker, updated 17 July 2026, puts adoption at 8.7% of the Tranco top 1,000, or 15.8% of the 549 sites in that list that were reachable. Originality.ai, reported by PPC Land on 2 July 2026, counted 36,120 llms.txt instances by May 2026, up from 4,088 in June 2025, a factor of 8.8 in eleven months.

The spread between 8.7% and 28% tells you more about sampling than about how to build an LLM friendly website: Ahrefs measures domains that opted into an SEO analytics product, which self-selects for sites that ship SEO fashions early. The Tranco top 1,000 is dominated by platforms with no incentive to publish a curated index.

Two conclusions survive all three datasets. Adoption is growing fast in relative terms and remains small in absolute terms. And measured consumption is close to zero, which is the number vendors selling generators do not put on the slide. Anyone answering the question of how to build an LLM friendly website by pointing at file adoption charts is answering a different question than the one you asked, which was whether any of this changes what an agent does.

The honest position, and the studio's, is this: publish llms.txt because it costs an afternoon, it is what Chrome checks, and it puts your own description of your business on your own domain. Do not budget a quarter for it. Details on who actually fetches these files are in the crawlers that fetch your file.

Which Businesses Gain First, Sorted by Site Shape

Site shape predicts the return better than industry does, which is why how to build an LLM friendly website has a different answer for a documentation portal than for a law firm. Four shapes cover most of the market.

Documentation-heavy SaaS. Highest return, and the only shape where llms-full.txt earns its bytes. Your users ask agents implementation questions constantly, the answers live in your docs, and an agent that reads a curated index instead of crawling a navigation tree burns fewer tokens per answer. This is exactly the shape Cloudflare tested on itself.

Catalogue commerce. Medium return, high effort. Thousands of product pages make a curated index genuinely hard: an llms.txt that lists 40,000 URLs is a sitemap with worse ergonomics. The win here is structured data and content negotiation on product pages, plus an llms.txt that indexes categories, policies, and guides rather than items.

Service firms with a small page count. Highest return per hour spent, and the shape most of this studio's clients take. Thirty to eighty pages fit in a single well-annotated index. The whole file is an afternoon's work and it replaces a description an aggregator wrote about you with one you wrote yourself.

Multi-brand groups. The interesting hard case. Three domains, one company, and no mechanism in the specification to say so. An agent has no reliable way to learn that your three sites belong to the same group unless each file says it in prose. That specific problem is covered in three platforms, three files that disagree.

The ownership argument cuts across all four shapes and outlives every technical detail of how to build an LLM friendly website and is the reason a service firm with modest traffic should still publish. When an answer engine describes your company, it assembles that description from whatever it can find: a directory listing, a five-year-old press release, a competitor's comparison page. A file on your own domain, under your own control, edited by you, is the only input to that description you own outright. That is a sovereignty argument before it is a marketing one, and it is developed in who gets to describe your company to an AI.

How to Build an LLM Friendly Website: The Publishing Order That Does Not Waste a Quarter

This is the operational core of how to build an LLM friendly website: nine steps, in order, with the ones you should refuse to do at the end. Each step is finishable; none of them requires a platform migration.

1. Fix the accessibility tree first. It is the highest-value item and it is not new work. Semantic headings, labelled form controls, real button elements. LOCOMOTIVE's June 2026 sampling found 412 of 636 major sites failing this check, so the bar is low and the fix is well documented.
2. Confirm your pages render without JavaScript. AI crawlers do not execute JavaScript. A client-rendered single page application is invisible to them regardless of what you publish at the root. Server-side rendering or static generation is the prerequisite for everything below.
3. Delete the pages that exist only for crawlers. Localised directory listings, tag archives, paginated index pages with no content. Cloudflare removed roughly 450 of them from its own documentation. This step usually improves human experience too.
4. Write llms.txt by hand, or generate it and then edit it. An H1 with your company name, a blockquote summarising what you do in two sentences, then H2 sections grouping your genuinely useful pages with one line of context each. Twenty good links beat two hundred mediocre ones.
5. Check how the file is served. A 200 status, text/plain or text/markdown content type, UTF-8 encoding, no redirect chain, no login wall. A file that returns 403 to a crawler is worse than no file, because Chrome flags a server error and marks a plain 404 as not applicable. Walked through in checking how the file is served.
6. Add agents.md only if your product is a codebase. SDK, API client library, open-source tool, developer platform. Otherwise skip it entirely; it instructs coding agents and you have none to instruct.
7. Add content negotiation if your stack makes it cheap. Serving markdown on Accept: text/markdown puts you in the 3.9% Cloudflare measured. On a static site generator this is a build step. On a legacy CMS it is a project, so defer it.
8. Run the Lighthouse agentic browsing report and record the fraction. Not as a score to optimise, as a dated baseline you can compare against after each change. It needs Chrome 150 or later.
9. Instrument the server logs before you change anything else. Covered in the next section, and it is the only step that tells you whether any of the previous eight did anything.

Two things to refuse for now, because how to build an LLM friendly website includes deciding what not to build. Do not build an MCP server because an agent-readiness vendor scored you down for not having one: fewer than 15 sites in Cloudflare's 200,000 domain scan have shipped the related discovery surfaces, and there is no measured demand. Do not publish llms-full.txt unless you are documentation-heavy and can regenerate it automatically; a stale full-text dump is worse than none.

How to Build an LLM Friendly Website Without a Developer?

Steps 4, 5, 6, and 8 are all achievable by a non-developer on a modern platform. Writing the file is editorial work, not engineering: you are choosing which thirty pages matter and describing each in a sentence. Uploading it is a file drop on Netlify, Vercel, or a static folder in most CMS platforms. Steps 1, 2, 3, and 7 need engineering. Platform-by-platform specifics, including which hosts generate the file for you and which silently rewrite it, are in where your llms.txt comes from, platform by platform, and French-language readers can follow créer et déployer votre fichier llms.txt.

Three Site Shapes, Three Worked Examples

Abstract advice on how to build an LLM friendly website is cheap. Here is what how to build an LLM friendly website costs and returns on three concrete shapes, with the arithmetic traced.

A 62-page consultancy site, no engineering team, one afternoon. The index lists 24 URLs across five H2 sections: services, methodology, case studies, insights, contact. Effort is roughly four hours of editorial work and one file upload. Lighthouse result: the llms.txt check moves from not applicable to pass. The accessibility check is unchanged, because it depends on the theme, not the file. Expected retrieval effect, stated honestly: an agent that already fetches the domain saves several page loads. There is no evidence it produces citations that would not otherwise occur.

A 12,000-page catalogue, in-house developers, one sprint. Listing products is the wrong move. The index carries 38 links: category landing pages, sizing and returns policies, buying guides. The larger win is step 7, serving markdown on Accept: text/markdown for product pages, which moves the site into the 3.9% of the 200,000 domains Cloudflare found doing it. Effort is one sprint for the negotiation layer and half a day for the file.

A documentation-heavy developer product, continuous delivery, ongoing. The only shape where all nine steps pay. llms.txt indexes the docs tree, llms-full.txt is regenerated on every docs build so it is never stale, agents.md ships in the repository, and the directory-listing deletion of step 3 is the change with the largest measured effect, following Cloudflare's own reported result of 31% fewer tokens consumed and answers 66% faster than the average unrefined site.

For a first-party reference point rather than a hypothetical, the studio's own file at laboetie.io/llms.txt is 54 lines: an H1, a one-sentence blockquote, and nine H2 sections covering the company, four products, the tools surface, the lab, the insights archive, and a data and plans section. Fifty-four lines is not an accident of laziness. It is the size at which a file stays maintainable by hand, which is the only way it stays accurate.

Proving It Works: Reading Your Own Server Logs

The measurement problem in this field is that everyone reports adoption and nobody reports effect, which leaves how to build an LLM friendly website an unfalsifiable question for most buyers. You can fix that on your own domain, and it costs nothing but log retention.

The method that turns how to build an LLM friendly website into a measurable question has four parts. First, log the fetches. Record every request to /llms.txt, /llms-full.txt, and /agents.md with timestamp, user agent, and status code. Named AI crawlers identify themselves: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, plus a long tail of unnamed clients. Ahrefs found that 77% of bots reading these files were not AI tools, so segmenting by user agent is not optional.

Second, log the follow-through. The fetch alone means nothing. What matters is whether the agent then requested pages listed in your file. A request for /llms.txt followed within the same minute by three requests for URLs it references is an agent using the index as intended. A single fetch with no follow-through is a scraper filling a dataset.

Third, keep a dated question log. Ask the same ten questions about your business to the major answer engines on a fixed schedule, record the answers and the citations verbatim, and keep the dates. This is the only way to correlate a change on your side with a change in what engines say about you.

Fourth, correlate. A dated request for your file, followed later by a citation on the same question that the file's description covers, is the closest thing to causal evidence available in public today. It is weak evidence. It is also more than the vendors in this market typically publish, and running it for a quarter will tell you more about your own domain than any agent-readiness score will. The scoring and validation side is developed further in validating and watching the file after launch.

FAQ: How to Build an LLM Friendly Website

Does a missing llms.txt fail a Google or Chrome audit?

No. Chrome's Lighthouse documentation states that when the file returns a 404 the audit is marked Not Applicable rather than failed, "as providing the file is optional at the moment." A server error while retrieving the file is flagged, which means a broken or permission-blocked file is worse than no file at all. Google Search Central separately maintains that no special AI-facing files are required for Search.

Do AI systems actually read llms.txt?

Partially, and less than vendors claim. Ahrefs analysed 137,210 domains in June 2026 and found 97% of published llms.txt files received zero traffic in May 2026, with 77% of the bots that did read one not being AI tools. Cloudflare found that of seven agents tested in February 2026, only Claude Code, OpenCode, and Cursor requested markdown content by default. Treat publishing as one cheap step in how to build an LLM friendly website, justified by cost and ownership rather than by traffic.

What is the difference between llms.txt and llms-full.txt?

llms.txt is an index: an H1 with your name, a blockquote summary, then H2 sections of annotated markdown links pointing at your pages. llms-full.txt inlines the actual text of those pages into one document, so an agent needs no second fetch. The index suits any site. The full-text version suits documentation-heavy products that can regenerate it on every build, because a stale full-text dump misinforms agents rather than helping them.

How to build an LLM friendly website if my site is a JavaScript single page application?

Fix rendering before publishing any file. AI crawlers do not execute JavaScript, so a client-rendered application returns an empty shell regardless of what sits at your domain root. Move to server-side rendering or static generation for the pages you want read, then publish llms.txt pointing at those pages. Publishing an index that references unrenderable URLs wastes the agent's fetches and yours.

Should I build an MCP server for my website?

Almost certainly not yet. Cloudflare's April 2026 scan of the 200,000 most visited domains found MCP Server Cards and API Catalogs together on fewer than 15 sites, and WebMCP remains in Chrome origin trial. Build one when you have an identified agent workflow that needs to perform an action, such as booking, ordering, or querying an account, not because a readiness scorer deducted points for its absence.

How long should an llms.txt file be?

Short enough to maintain by hand. Twenty to forty well-annotated links beat several hundred unannotated ones, because the annotation is the part an agent uses to decide what to fetch. The studio's own file is 54 lines covering nine sections. If your index has grown past what one person can review in ten minutes, it has stopped describing your site and started duplicating your sitemap.

How La Boétie Ships the Agent-Facing Layer

The studio answers how to build an LLM friendly website by treating this layer as infrastructure the client owns, not as a marketing add-on. Three ways in, depending on how much you want to do yourself.

The llms.txt Generator, self-serve at $25 or 20 € a run, no account and no call. It produces a spec-compliant file from your domain, which you then edit. Editing is the point: a generated index is a first draft, and the sentence describing each page is the part that determines whether an agent fetches it.

Full agent-facing engagements, delivered by a flexible team of five to six engineers working across time zones and languages. This is the shape for the steps a generator cannot do: rendering fixes, content negotiation, deleting the 450-page equivalent of directory listings from your own site, and log instrumentation.

Cortex, the studio's SEO and GEO growth engine, available by application only and capped at three brands per industry. It runs the insights archive you are reading and produced the research behind this page.

Whichever route, the throughline holds: clients keep ownership of what gets built, including the description of their own company. Start with the llms.txt Generator if you want the file this week.

Conclusion

The agent-facing layer is real, small, and badly explained by the people selling it. Chrome has published which properties it thinks matter and has pointedly refused to turn them into a score. Cloudflare has published a large-scale measurement, showing both a genuine efficiency gain on its own restructured documentation and an internet where 3.9% of major sites support markdown content negotiation. Ahrefs has published the uncomfortable counterpart: most files nobody reads.

Taken together, those three datasets support a modest, defensible position. Fix your accessibility tree and your rendering, because those pay off for human readers, screen readers, and agents alike. Delete the pages that exist only to be crawled. Publish an llms.txt you wrote yourself, keep it short enough to maintain, and instrument your logs so that next quarter you are arguing from your own data rather than someone's adoption chart. That is what how to build an LLM friendly website actually means in 2026: a small amount of durable engineering, one honest file you own, and a measurement habit nobody else in this market has bothered to establish.

Sources

Further reading on this site:

• generating an llms.txt that earns its place
• where your llms.txt comes from, platform by platform
• the Chrome agentic browsing audit
• créer et déployer votre fichier llms.txt
• agents.md, WebMCP and letting agents act
• the file your platform generated for you
• validating and watching the file after launch
• three platforms, three files that disagree
• checking how the file is served
• four machine-readable files and the job each one does
• who gets to describe your company to an AI
• the crawlers that fetch your file

External references:

• Lighthouse agentic browsing scoring : Chrome for Developers, 2026
• llms.txt audit reference : Chrome for Developers, 2026
• Introducing the Agent Readiness score : Cloudflare, 2026
• The llms.txt specification : Answer.AI, Jeremy Howard, 2024
• AGENTS.md : agents.md, 2026
• Model Context Protocol introduction : Model Context Protocol, 2026
• La Boétie llms.txt : La Boétie, 2026
• Lighthouse Agentic Browsing Audit: What It Checks and Why It Matters : LOCOMOTIVE Agency, Vinicius Stanula, 2026
• Google Lighthouse Has A New Agentic Browsing Category : DebugBear, Matt Zeunert, 2026
• Google adds llms.txt check to Chrome Lighthouse : Search Engine Land, 2026