La BoétieInsights
llms.txt generation and platform setup

llms.txt Verify Your File: Status, Content Type and Encoding

By La BoétieUpdated August 6, 20269 min read
Terminal window filled with HTTP response header lines on a developer workstation at night

Your llms.txt is live. The URL loads in a browser, the Markdown looks right, and a public validator returns a green tick. None of that proves an agent can read it. The transport layer sits between your file and the crawler that requested it, and it fails quietly: a 301 redirect to a trailing slash, a text/html body from a platform fallback route, a byte order mark welded to the first character. The llms.txt verify your file routine below is five commands against the response headers, and it runs in under 60 seconds.

Key checks:

  • /llms.txt returns HTTP 200 directly. Chrome for Developers documentation states that Lighthouse flags the page when a server error occurs on retrieval, while a 404 marks the audit Not Applicable because the file is optional today.
  • The header reads Content-Type: text/plain; charset=utf-8. Dom Sipowicz reported on DEV Community on 28 August 2025 that deployments serving text/markdown; charset=utf-8 failed in ChatGPT and Gemini.
  • The first 3 bytes are not EF BB BF. The Unicode Consortium states that a byte order mark is neither required nor recommended for UTF-8.
  • Every link uses Markdown syntax. Slobodan Manic measured the Agentic Browsing score moving from 0.67 to 1.0 on No Hacks on 1 June 2026, a 49 % gain from that single change.

A File That Exists and a File That Is Served Are Different Things

The llms.txt specification, published by Jeremy Howard of Answer.AI on 3 September 2024, states that the spec is for files located in the root path /llms.txt of a website. It says nothing about the HTTP status code, nothing about the Content-Type header, and nothing about encoding beyond permitting an optional byte order mark ahead of the H1.

That silence is where implementations break. Structure checkers such as the MRS Digital llms.txt Validator read the bytes they are handed, so a file delivered over a redirect chain as HTML can still clear a structure pass. Scoring the contents is a separate exercise, covered in testing your llms.txt implementation.

The llms.txt Verify Your File Sequence, in Five Commands

Run the llms.txt verify your file sequence against the production domain, not a staging build. Each command returns a value you read straight off the response.

  1. Status. curl -sSI https://example.com/llms.txt returns HTTP/2 200 on the first line, with no location header beside it.
  2. Redirects. curl -sSIL https://example.com/llms.txt and count the hops. A 301 or 308 pointing at /llms.txt/ means a canonical rewrite intercepted the request.
  3. Content type. The content-type header reads text/plain; charset=utf-8.
  4. Encoding. curl -s https://example.com/llms.txt | head -c 3 | xxd shows the first character of your H1, never efbbbf.
  5. Link syntax. Every file list entry is a Markdown hyperlink in [name](url) form, which the specification names as required.

Status Codes and the Redirect Trap

The canonical path answers with 200 itself. A redirect at the root survives longest because it is invisible in a browser, which follows the hop and renders the destination. A crawler following the same hop into an HTML page receives navigation chrome and script tags instead of a Markdown index.

Contributors on the WordPress.org support forums documented WordPress adding a trailing slash to /llms.txt through its canonical redirect, issuing a 301 before the serving plugin ever ran. The same pattern appears wherever a catch-all route sits above static files.

On Nginx, an exact-match block above try_files removes the ambiguity: location = /llms.txt { default_type text/plain; charset utf-8; }. On Apache, exclude the path from the rewrite engine with RewriteCond %{REQUEST_URI} !^/llms\.txt$ ahead of the front-controller rule. Both keep the llms.txt verify your file status check honest by stopping the framework from claiming the request.

Two fibre cables on a patch panel, one routed straight and one looped back, illustrating a direct response against a redirect hop

Content Type, and What text/html Costs You

An agent decides how to parse your file from the header, not from the extension. Serve text/plain; charset=utf-8. Dom Sipowicz, writing on DEV Community on 28 August 2025, found that deployments serving text/markdown; charset=utf-8 failed in ChatGPT and Gemini.

text/markdown is a registered MIME type: the IETF published RFC 7763 in March 2016, authored by S. Leonard of Penango, as an Informational document. Registration is not adoption, and the crawlers that matter today were built against text/plain.

The worse outcome is text/html. A single-page application fallback or a 404 handler returning the site shell answers /llms.txt with a 200 and an HTML body. Slobodan Manic noted on No Hacks that the Lighthouse parser enforces Markdown formatting regardless of MIME type or extension, so an HTML shell yields zero recognised links. That is the failure the llms.txt verify your file content type check exists to catch.

Encoding: UTF-8, Byte Order Marks and Accented Characters

The byte order mark is 3 bytes, EF BB BF, prepended by Windows editors and several CMS export routines. The llms.txt specification lists it as an optional first element, so a BOM conforms. The Unicode Consortium is blunter, stating that use of a BOM is neither required nor recommended for UTF-8 and that it interferes with files required to begin with an ASCII sequence. A parser expecting # at offset zero misses your H1.

The accented-character check takes one second. Open the served file and look at a word carrying a diacritic. When a single accented letter arrives rendered as two Latin-1 characters, the export passed through windows-1252 and the file needs regenerating at source. Run the llms.txt verify your file encoding step against the served URL rather than the copy on disk, which is what catches a CDN transcoding step.

When the llms.txt Verify Your File Checks Disagree With Chrome

Google shipped the check into Lighthouse in May 2026. Slobodan Manic recorded that Lighthouse 13.3.0 moved the Agentic Browsing category into the default configuration. Danny Goodwin reported in Search Engine Land on 20 May 2026 that the category returns a fractional pass ratio rather than a score out of 100.

The audit result tells you which failure you have. Chrome for Developers documentation states that Lighthouse flags the page when a server error occurs while retrieving the file, and that a 404 marks the audit Not Applicable because providing the file is optional at the moment. A flagged page means your server answered and answered badly. Not Applicable means nothing was served.

Confirm the browser build before chasing a phantom. Chrome documents Chrome 150 or later as the requirement for the category, and issue 17082 on the GoogleChrome/lighthouse repository records PageSpeed Insights running HeadlessChromium 146.0.7680.177 and reporting a misleading error instead of Not Applicable. The Chrome Lighthouse agentic browsing audit covers the full audit set.

FAQ: Serving llms.txt Correctly

Does a 404 on /llms.txt fail the Lighthouse audit?

No. Chrome for Developers documentation states that when the file is not provided by the server, resulting in a 404, the audit is marked Not Applicable because providing the file is optional at the moment. A server error during retrieval flags the page instead. A broken deployment therefore scores worse than no deployment at all.

Should llms.txt be served as text/plain or text/markdown?

Use text/plain; charset=utf-8. Dom Sipowicz reported on DEV Community on 28 August 2025 that deployments serving text/markdown; charset=utf-8 failed in ChatGPT and Gemini. RFC 7763 registered text/markdown in March 2016, so the type is legitimate, and current crawler behaviour still favours plain text. The file content stays Markdown either way.

How often should you run the llms.txt verify your file checks?

Run them on every deployment touching routing, static file handling, CDN rules or the CMS canonical layer. Those four systems reclaim /llms.txt without anyone noticing. A scheduled curl every 7 days against the production URL, asserting a 200 status and a text/plain content type, covers the drift between deployments.

Does a redirect from /llms.txt break AI crawler access?

A redirect at the canonical path is a failure condition, because the crawler receives a 3xx where it expected the document. Where the redirect lands on an HTML page, the crawler parses site chrome rather than your index. Serve the file at /llms.txt with a 200 response and scope redirect rules to paths other than the root file.

How La Boétie Ships the Agent-Facing Layer

The llms.txt Generator. A self-serve run costs $25 or 20 € on the La Boétie llms.txt Generator, with no account and no call. It emits Markdown link syntax throughout, which is the shape the Lighthouse parser requires.

Studio builds. A flexible team of 5 to 6 engineers, multilingual and across timezones, ships the agent-facing layer as part of the site: the route, the header, the encoding and the deployment check that runs the llms.txt verify your file sequence on every release.

Ownership. Clients keep ownership of everything built. The routing rules, the generator output and the serving configuration stay in your repository, on your infrastructure.

Conclusion

Four of the five failures described here still return a 200 status to a browser, which is why visual inspection closes tickets that should stay open. Read the headers instead of the page. The llms.txt verify your file sequence costs 60 seconds per deployment and turns a file that exists into a file an agent can retrieve, parse and cite.

Sources

Also read:

External sources:

Questions

Does a 404 on /llms.txt fail the Lighthouse audit?

No. Chrome for Developers documentation states that when the file is not provided by the server, resulting in a 404, the audit is marked Not Applicable because providing the file is optional at the moment. A server error during retrieval is treated differently and flags the page. The practical reading is that a broken deployment scores worse than no deployment.

Should llms.txt be served as text/plain or text/markdown?

Use text/plain; charset=utf-8. Dom Sipowicz reported on DEV Community on 28 August 2025 that deployments serving text/markdown; charset=utf-8 failed in ChatGPT and Gemini. RFC 7763 registered text/markdown in March 2016, so the type is legitimate, and current crawler behaviour still favours plain text. The file content stays Markdown either way.

How often should you run the llms.txt verify your file checks?

Run them on every deployment that touches routing, static file handling, CDN rules or the CMS canonical layer. Those four systems are the ones that reclaim /llms.txt without anyone noticing. A weekly scheduled curl against the production URL, asserting a 200 status and a text/plain content type, covers the drift between deployments.

Does a redirect from /llms.txt break AI crawler access?

A redirect at the canonical path is a failure condition, because the crawler receives a 3xx where it expected the document. Where the redirect lands on an HTML page, the crawler parses site chrome rather than your index. Serve the file at /llms.txt with a 200 response and keep any redirect rules scoped to paths that are not the root file.

Work with the studio.