Doing AEO at the edge, without rewriting your application

Every AEO recommendation ends the same way: change the application. Add JSON-LD to the template. Serve a new file at the apex. Vary the response by user agent. Log requests you currently discard.

Each of those is small. Together they are a cross-team project against a Next.js marketing site, a Rails app, a WordPress instance, and a docs platform nobody has touched in a year — four codebases, four deploy pipelines, four teams with their own roadmaps. That is why AEO work sits in the backlog: not because anyone disputes it, but because the unit of work is “coordinate four teams” rather than “write some markup”.

The request path is a way out. Everything on that list is a property of the HTTP exchange, not of your application logic.

What a proxy hop can do

Put a proxy between visitors and your origin and four capabilities become configuration instead of code.

Classify and record. Every request carries a user agent, a path, and a status. Classifying it as human, training crawler, or live AI fetcher, and recording that, is a read-only operation that touches nothing. It also solves the measurement problem, since crawlers never execute the analytics tag your application ships.

Serve new paths. /llms.txt is a static response at a fixed path. A proxy can answer it directly without the origin ever seeing the request — meaning no route to add, no build, no deploy, and no framework fighting you about a non-HTML extension.

Transform responses. HTML streaming rewriters can insert a <script type="application/ld+json"> block before </head> as bytes pass through. No template edit, no rebuild, applied uniformly across every origin behind the proxy.

Enforce rules. robots.txt is a request; an edge rule is a decision. Blocking, allowing, or redirecting a verified agent happens before your origin is touched, which also means the traffic you decline costs you nothing.

Where it should run

Not in a datacentre in one region. The proxy is in front of every request, so the hop needs to be short for everyone.

Shiftrank runs on Cloudflare Workers. The properties that matter for this workload:

• Proximity. Code executes at the edge location nearest the visitor, so the added hop is small relative to the origin round trip it fronts.
• Streaming rewrites. HTMLRewriter mutates HTML as it streams through, so time-to-first-byte is unaffected and memory does not scale with document size. Buffering an entire response to parse it is the failure mode this avoids.
• No cold-start tax. Isolate-based execution means the proxy does not add a container spin-up to a random slice of requests.
• Analytics Engine. High-cardinality event writes are non-blocking and queryable with SQL, which is what makes per-request classification affordable at all.

The three things that must be true

A proxy in front of everything is a serious position in your architecture. Three properties are non-negotiable, and it is worth checking any vendor against them.

Fail open. If the proxy errors, times out, or hits a limit, the request must still reach the origin and the response must still reach the visitor. A measurement layer that can take the site down is not worth having, and observe mode should be the default for exactly this reason.

Stream, never buffer. Transformation must operate on the stream. Buffering whole responses to parse them turns your fast site into a slow one and makes large documents a memory risk.

Roll out in observe mode. Record what a rule would have done before it does it. A crawler-blocking rule that is one path pattern too broad removes you from AI answers silently — nothing errors, nothing pages, and the traffic you lost was never in your dashboard. Seeing the counterfactual first is the difference between a config change and an incident.

What it does not solve

A proxy changes delivery. It does not change what your content says.

If your pages hedge, bury the claim in the fourth sentence, and avoid specifics, injecting structured data will not make them quotable — it will describe vague content precisely. The proxy handles access, machine-readable surfaces, measurement, and control. Whether a model finds your page worth quoting is still a writing problem. The AEO checklist separates the two explicitly.

It is also not a reason to leave robots.txt wrong. Edge rules enforce; robots.txt communicates intent to operators who honour it. You want both saying the same thing.

What adoption looks like

1. Verify the domain with a TXT record.
2. Point the hostname at the proxy with a CNAME. Traffic is forwarded unchanged.
3. Sit in observe mode for a week or two. You now have the baseline you never had: which agents fetch you, which paths, what they receive.
4. Turn on surfaces — llms.txt first, since it is additive and cannot break an existing page.
5. Add rules last, after the observe data tells you what they will affect.

The order matters more than the tooling. Most AEO mistakes are irreversible-looking because they are invisible; doing measurement before enforcement makes them neither.