When a website is readable by an agent, too

A simple question can expose a website’s weak spots: “Can I return a discounted product bought from Germany?” The answer may be split across an FAQ, a product page, terms and conditions, and an expired promotion.

An agent can open all of those pages. The hard part is knowing which one states the rule, which one describes an exception, and which one is current.

That is why an agent-readable website does not begin with one isolated technical fix. It begins with content that has already been put in order for its readers.

Reachable, readable, trustworthy

Those are three different things.

A crawler needs to reach a page. A browser needs to load it. A reader, human or automated, needs to understand what the page says and how much weight to give it. robots.txt only addresses the first point: the RFC defining it is clear that it is not an access-control system.1

Page structure helps with the next step. A title, headings, main content and navigation are not merely visual details. W3C guidance explains that headings and landmarks make a page’s organisation available to assistive technologies.23 It is reasonable to draw a limited conclusion from that: explicit structure gives clearer clues to other software trying to orient itself on a page. It does not mean every agent will use those clues in the same way.

Trust calls for something else. The website must be able to point to its source.

Not a URL for every sentence, but a home for every rule

Not every piece of information needs its own URL. A durable rule does need a recognisable place to return to: a returns policy, documentation for a specific version, terms for a service, a product record.

That is an editorial principle, not a technical requirement. The source does not have to hold every detail on a single screen. It should make it possible to separate the rule from an example, marketing copy from contractual terms, and a current version from an older one.

Good content architecture also makes relationships explicit. This guide applies to this version. This exception changes that rule. This page explains the source of record. Internal links cannot solve everything, but they can make those relationships visible instead of leaving them in the heads of the people who maintain the website.

The site needs to decide which page contains the rule and which pages explain it. Easy to say; often hard to do. When the same sentence lives in four places, someone has to decide where it gets updated.

Structured data cannot fix confused content

This is where the technical vocabulary comes in. JSON-LD is a format for embedding structured data in a page: it can state, for example, that a text is an article or that a page describes a product. It is useful when it repeats, in a software-readable form, what the website already says well.

It should not become a second version of the website. Google requires structured data to represent the page content and notes that correct markup does not guarantee a particular result appearance.4 That is a Google Search rule, but the underlying lesson travels: if the visible text says one thing and the metadata says another, the problem has not been solved. It has been duplicated.

Clarify the page first. Add structured data where it helps.

What about llms.txt?

llms.txt is a recent proposal for a Markdown file at a website’s root. Its purpose is to give a language model brief context and links to useful resources.5 It can help on large technical sites or documentation full of interfaces, where an editorial map cuts through the noise.

It is still an index. It cannot decide which policy is valid, add a date to an offer, or reconcile two conflicting descriptions. The proposal itself does not specify how software should process the file.5

So the useful question is not, “Do we have an llms.txt file?” It is this: if an agent opened the pages we point it to, would it find an attributable, up-to-date answer? If not, the missing piece is not a file. It is the order that file would have been expected to summarise.

An agent-readable website does something very human: it does not make readers guess where its answers have gone.

Footnotes

  1. M. Koster et al., RFC 9309: Robots Exclusion Protocol, September 2022. ↩

  2. W3C Web Accessibility Initiative, Headings, accessed 1 August 2026. The guidance concerns accessibility; its application to agents is an inference. ↩

  3. W3C Web Accessibility Initiative, Landmarks Pattern, accessed 1 August 2026. It describes assistive navigation, not a requirement for language agents. ↩

  4. Google Search Central, General Structured Data Guidelines, updated 10 July 2026. The rules apply to Google Search. ↩

  5. Jeremy Howard, The /llms.txt file, 3 September 2024. It is a community proposal, not a rule every agent follows. ↩ ↩2