Serve Markdown to AI agents
Last updated Edit this page View as Markdown
Serve AI agents a rendered Markdown copy of your pages at the same URL, and synthesize an /llms.txt — off by default, nothing leaves your server.
On this page
AI agents and LLM crawlers fetch your pages to read, summarize, and act on them — but a JavaScript-rendered site often looks empty to a client that does not execute scripts. The mod_pagespeed optimizer worker can serve those agents a rendered, readable copy of each page, at the same URL, without changing anything a browser or search engine sees.
How it works
When an agent requests a page with Accept: text/markdown and you have enabled
agent optimization, the optimizer worker serves a Markdown
representation produced from the fully-rendered DOM — JavaScript executed,
layout settled. JS-heavy pages that are otherwise invisible to agents become
clean, structured text. Everyone else gets your normal optimized HTML at that
same URL.
The agent response is content-negotiated, with these guarantees:
Vary: Accept— shared caches never serve the Markdown copy to a browser, or the HTML to an agent.X-Robots-Tag: noindexandCache-Control: private— the agent copy stays out of search indexes and shared caches.- The page body and Content-Type for browsers are unchanged — standard
content negotiation on
Accept, not cloaking.
In the nginx module, while agent optimization is on,
the HTML responses of your pages name Accept in their Vary too (for example
Vary: Accept-Encoding, User-Agent, Accept), whether or not a Markdown copy
exists yet, so a CDN in front of nginx keys the two apart from the first
request. With agent optimization off, Vary is as before. Stylesheets,
scripts and images are not affected.
x-markdown-tokens(nginx module) — an estimate of the number of tokens in the Markdown body, so an agent can budget its context before reading it. The estimate is the body size in bytes divided by four, rounded down; no tokenizer is run, so a model’s own count differs. Treat it as an order of magnitude.content-signal(nginx module) — only when you configure one (see below).
The rendering runs in the optimizer worker process behind nginx, reusing the same headless-Chrome subsystem as browser analysis.
Off by default
The capability is present in every build, but it does nothing until you enable
agent optimization on the worker (--agent-optimize, or
PAGESPEED_AGENT_OPTIMIZE=true in the Docker images). It also requires browser
analysis (--enable-browser-analysis), since the Markdown copy is produced from
the rendered DOM.
While the flag is off, an agent request transparently receives the normal HTML of the same URL — never an error, never a half-rendered page. Turning the feature off is always safe.
State a content-use policy (content-signal)
A Markdown response can tell agents how you allow your content to be used,
with a content-signal header in the Content Signals
format. The nginx module sends it only when you set one; there is no default
policy, and the module never states one for you:
pagespeed_agent_content_signal "search=yes, ai-input=yes, ai-train=no";
The directive works in http, server and location blocks and is inherited
like other directives; pagespeed_agent_content_signal off; stops an inherited
value. The header goes out with the Markdown copy and with /llms.txt, not with
HTML. The value must be printable ASCII on one line, up to 512 bytes; nginx
refuses to start with any other value. The module does not read a
content-signal header from your origin’s response: the directive is the only
source. To state the same policy to crawlers that read robots.txt, add a
Content-Signal: line there as well.
Serve an /llms.txt site index
The optimizer worker can also publish a synthesized /llms.txt for your
site — a compact Markdown index that points AI agents at your important pages,
in the llmstxt.org format. The worker builds it from your own
sitemap.xml (intersected with the paths you allow), with a one-line title and
summary per page, and nginx serves it at /llms.txt with X-Robots-Tag: noindex,
Cache-Control: private, and Content-Type: text/markdown.
/llms.txt carries the same x-markdown-tokens estimate and content-signal
as the Markdown copy.
It is off by default and enabled with its own flag (--agent-optimize-llms-txt,
which requires --agent-optimize). Per-page summaries come for free: from a page’s
already-rendered agent copy when one exists, or from a cheap title/description
read otherwise. The file is never built by rendering pages on demand. The
index is generated and cached off the serving path and refreshed lazily when your
sitemap changes (or once a day at most).
Content is left out if your robots.txt tells AI crawlers to stay away, or if a
page opts out with noai / X-Robots-Tag: noai. A site that blocks AI crawlers
entirely gets no /llms.txt at all.
Like the rendered-Markdown copy, /llms.txt needs the optimizer worker and its
headless-Chrome substrate; the module alone does not produce it.
Nothing leaves your server
Rendering, cleaning, caching, and serving all happen on your own server. Serving a cached response — a page, the agent Markdown copy, or
/llms.txt — makes no third-party network call: no DNS, no telemetry, no
analytics, no beacon.
The only network activity happens off the serving path:
- Building the
/llms.txtindex fetches your ownsitemap.xmland pages. - Rendering may fetch a page’s own resources.
- Both go through the same SSRF-guarded fetcher, IP-pinned to your origin.
This makes agent optimization suitable for sovereign and regulated environments where content must not leave the premises.
What the Markdown copy contains
The Markdown copy reproduces the page’s own content. It is built from the page as the headless browser renders it, after the page’s scripts have run. Those scripts run only inside that browser, as they would for any visitor; the conversion step never executes them. The converter treats every piece of text as data: it does not run it, follow instructions written in it, or fetch anything it links to. Text and link targets are escaped, so the copy’s structure follows the page’s markup. The copy has a fixed size limit.
Content a reader is not meant to see is left out:
- elements hidden in their own markup (
style="display:none",style="visibility:hidden", thehiddenattribute, oraria-hidden="true"), with everything inside them; <noscript>and<template>content;- scripts, stylesheets, the document head, SVG graphics, and form drop-downs and labels;
- page-level navigation, header, and footer regions, unless the page’s main content sits inside them.
Text hidden by CSS classes or an external stylesheet stays in the copy, because the conversion works from the page’s markup, not from computed styles. Treat the copy as untrusted page content, not as instructions.
How the page cache treats signed-in visitors
The page cache is shared between visitors, so it follows rules about requests
that carry credentials or cookies. The cookie rules also apply to the Markdown
copy and /llms.txt, which follow the page they were made from.
Never stored. A response is not stored when it sets a cookie
(Set-Cookie), or when it carries Cache-Control: private or
Cache-Control: no-store. A response is stored only when its Vary names
nothing beyond Accept-Encoding, User-Agent, Accept and Save-Data. Any
other field, including Vary: Cookie, and Vary: *, means it is not stored. For
a response that varies on Accept, User-Agent or Save-Data, those request
headers take part in choosing which stored copy answers.
A response is stored only when its Vary names
nothing beyond Accept-Encoding. Any other field, including Vary: Accept,
Vary: User-Agent, Vary: Save-Data, Vary: Cookie and Vary: *, means it is
not stored, and each request is answered by your origin. This is the Vary
your origin sends; the Vary mod_pagespeed adds to its own responses (such as
Accept with agent optimization on) does not count.
Requests with a cookie. A request that carries a Cookie header is treated
by the kind of response:
- A page (every response except css, js and images) fetched for such a request
is stored and shared only when the origin marks it
Cache-Control: publicor gives a singles-maxagegreater than 0. A stored page is served to a request with a cookie only under the same mark. Otherwise the request goes to your origin. - A static asset (css, js, images) fetched for such a request is stored only when
the origin states a lifetime, with
max-ageors-maxageabove 0 (orExpires, when noCache-Controlis sent). Without one, nothing is stored.
Requests with Authorization. A request that carries an Authorization
header is served from a shared copy, and its response is stored, only under the
permit in RFC 9111 section 3.5: the response says public, must-revalidate
or s-maxage. Without one of those, the request goes to your origin and the
response is not stored. Once a copy is stale, it is served to such a request only
when the response said public. The Markdown copy is served to a request that
carries Authorization at most when the page itself was stored under that
permit. /llms.txt is never served to a request that carries Authorization.
When license enforcement (rsl_cap_enforcement) is on, a request whose
Authorization header carries valid credentials under that scheme is treated as an
agent, not as a signed-in visitor, and may be served the Markdown copy and
/llms.txt. See RSL-CAP for details.
What to do on your origin
- Mark every personalised page
Cache-Control: private, or sendVary: Cookie, which is never stored.publicors-maxageon a page that differs per visitor makes it shared. - A response of any type that differs per visitor must carry
Cache-Control: privateorno-store; a response with an explicit lifetime and noprivatemay be stored and shared. - If your site sets a cookie on every request (analytics, consent), pages that
are the same for every visitor are shared only when you mark them, for example
Cache-Control: public, max-age=600. Give css, js and images an explicitmax-age.
Enabling it
Start the worker with --agent-optimize (and --enable-browser-analysis), then
add --agent-optimize-llms-txt if you also want the synthesized /llms.txt.
Evaluate it in staging first: the capability is experimental and its flags may
change.
See also: Analyze pages with headless Chrome. For the design and rationale, read Serve Markdown to AI agents and the deep-dive on serving Markdown and a synthesized /llms.txt to AI crawlers.
Frequently asked questions
Does this change what browsers and search engines see?
No. Browsers and crawlers receive your normal HTML at the same URL. Only a request that explicitly negotiates the agent representation (Accept: text/markdown), on a deployment where you have enabled agent optimization, receives the rendered-Markdown copy. The response carries Vary: Accept so shared caches never mix the two.
Is it on by default?
No. The capability ships in every build but stays off until you enable it with the --agent-optimize worker flag. There is no separate build.
Does my content leave my server?
No. Rendering, cleaning, caching, and serving all happen on your own server. Serving a cached response performs zero outbound network I/O.
Does this work with the module alone, or does it need the optimizer worker?
It needs the optimizer worker. The rendered-Markdown variant runs in the optimizer worker behind nginx, using headless Chrome. The module recognizes and safely declines the agent request (it always serves normal HTML); it does not produce the rendered variant.
What is the synthesized /llms.txt, and is it your /llms.txt?
The optimizer worker can publish a synthesized /llms.txt for YOUR site — a Markdown index built from your own sitemap, off by default and enabled by its own worker flag. It is unrelated to the /llms.txt on modpagespeed.com, which describes our product. The index is built off the serving path and serving it performs zero outbound network I/O. It needs the worker: the module alone does not produce it.