How to Make Your Site Agent-Operable: The Complete Technical Guide
Ryan KingsFounder & CTOPublished June 2026 · 12 min read
Corrected 2026-11-04: earlier versions quoted Cloudflare, Lighthouse and FORKOFF scores that came from our own approximation of those tools, not from the tools themselves.
How to Make Your Site Agent-Operable: The Complete Technical Guide
Making a website agent-operable means deploying files and middleware that AI agents use to discover, understand, and interact with your site. Cloudflare's isitagentready.com groups its checks into five categories: Discoverability, Content Accessibility, Bot Access Control, Protocol Discovery, and Commerce. Google's Lighthouse Agentic Browsing category is experimental and does not produce a 0–100 score. It reports a pass ratio plus audits for the agent accessibility tree, WebMCP tool registration and schema validity, CLS, llms.txt, and an ARD manifest checked against ARD v0.91. This guide walks through the files to deploy, where each one goes, and how to verify it worked.
On 2026-11-04 the repaired check of aeoforged.com reported: Protocols found: 19 of 27 that apply to this site. Read score 100. 5/5 agent probes reached the homepage; discovery 4/4 (robots access, sitemap, machine-readable content, structured data). The June 2026 write-up stated a before figure from an earlier version of these checks. That version is what this correction replaced. This page does not repeat it, and it does not quote a Cloudflare, Lighthouse, or FORKOFF score.
What should you do before writing any agent operability code?
The first step is a baseline audit that tells you which files your site already has and which are missing. AEOForge's aeo_agent_ready tool (free, no account required) probes the live site and returns what it found.
The report produces a prioritised to-do list with specific file paths and fix guidance. After deploying each step below, re-run the scan to confirm the signal flipped green. Alternatively, aeo_make_agent_ready (8 credits) generates every artifact in this guide as platform-specific deployable code you can commit directly.
How do you configure robots.txt for AI agents in 2026?
robots.txt applies to every crawler through User-agent: * unless a bot has its own group. An explicit group is only needed when you want to treat that bot differently. Content-Signal is a Cloudflare convention that states search, AI input, and AI training preferences. It is separate from the X-Robots-Tag header.
User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
Sitemap: https://yourdomain.com/sitemap.xml
The three Content-Signal values above are search indexing, real-time retrieval for AI answers, and training dataset inclusion. Add a named User-agent group only when that bot should be treated differently from the wildcard.
How do you create a valid llms.txt file?
The llms.txt file is a plain markdown index at /llms.txt that gives agents and coding tools a curated map of important pages. The llms.txt proposal requires an H1. A summary blockquote and link sections are recommended. Google Search does not use it.
A usable llms.txt file:
- First line: an H1,
# Your Site Name(required by the proposal) - Next, a blockquote with a one-sentence description (recommended)
- Link sections with absolute URLs (recommended)
- Plain text, not HTML
# YourSite
> One-sentence description of what your site does.
## Core pages
- [Homepage](https://yourdomain.com/): Platform overview
- [Pricing](https://yourdomain.com/pricing): Plans and pricing
## Documentation
- [Getting Started](https://yourdomain.com/docs/start): Setup guide
/llms-full.txt is a longer companion used by documentation sites. Lighthouse's experimental agentic audits include llms.txt. Google Search does not use it.
How do you add auto-discovery link tags for agent protocols?
Auto-discovery link tags are <link> elements in your document <head> that point agents at protocol files. Point llms.txt, /.well-known/agents.json, and sitemap.xml. Do not add agent-instructions.md: it has no published spec.
<link rel="describedby" href="/llms.txt" type="text/plain" />
<link rel="describedby" href="/.well-known/agents.json" type="application/json" />
<link rel="sitemap" href="/sitemap.xml" type="application/xml" />
In Next.js App Router, add these to your root layout.tsx metadata or serve them as <link> elements in your head component. Lighthouse's experimental Agentic Browsing audits do not check these tags. They look at the agent accessibility tree, WebMCP tool registration and schema validity, CLS, llms.txt, and an ARD manifest.
How does markdown content negotiation work for AI agents?
When an agent sends Accept: text/markdown, your server returns clean markdown instead of HTML. This is the single highest-leverage implementation for agent content accessibility because it eliminates the parsing step entirely.
The approach is ordinary HTTP content negotiation. Your middleware checks the Accept header, rewrites the request to an internal markdown endpoint, and returns the response with appropriate headers:
import { NextRequest, NextResponse } from "next/server";
export function middleware(request: NextRequest) {
const accept = request.headers.get("accept") ?? "";
if (!accept.toLowerCase().includes("text/markdown")) {
const response = NextResponse.next();
response.headers.set("Vary", "Accept");
return response;
}
const url = request.nextUrl.clone();
url.pathname = "/api/markdown" + url.pathname;
const response = NextResponse.rewrite(url);
response.headers.set("Content-Type", "text/markdown; charset=utf-8");
response.headers.set("Vary", "Accept");
return response;
}
The /api/markdown/[...path] route fetches your own page as HTML, strips markup, and returns clean text. Set Content-Location to signal the alternate representation exists.
How do you deploy agents.json and MCP Server Card?
The agents.json file and MCP Server Card tell agents what is available on your site. agents.json belongs at /.well-known/agents.json. MCP Server Cards are listed in /.well-known/ai-catalog.json. The card itself lives at the server's URL plus /server-card and does not list tools (SEP-2127). Copy field names from the published agents.json repository. Do not reuse an older plugin-manifest shape, and do not put tools on the MCP card.
The MCP Server Card does not list tools. Cloudflare's checker and Lighthouse do not score that card as a shared percentage.
How do you enable agent-to-agent discovery with the A2A Agent Card?
The A2A Agent Card is published by the A2A project at /.well-known/agent-card.json. A 1.0 card declares supportedInterfaces. Do not also publish /.well-known/agent.json; that path is not the 1.0 card.
A2A 1.0 is a published specification from the A2A project. It is not a Google draft, and Lighthouse's experimental agentic audits do not score the card.
How do you verify your agent operability deployment?
Verification is a re-run of the same baseline audit. Each step maps to a named check in our probe:
- robots.txt (Step 1):
robots_txt_quality,content_signal - llms.txt (Step 2):
llms_txt,llms_txt_valid - Discovery links (Step 3):
auto_discovery_links - Content negotiation (Step 4):
content_negotiation - agents.json (Step 5):
agents_json,agent_permissions - MCP Server Card (Step 6):
mcp_server_card - A2A Agent Card (Step 7):
a2a_agent_card
Deploying these files does not by itself pass Cloudflare's checker or Lighthouse's experimental audits. Re-run the probe to see what it finds on the live site.
Key takeaways
- Cloudflare's checker and Lighthouse's experimental agentic audits look at parts of agent access. They are separate tools from this probe.
- The files in this guide are: robots.txt, llms.txt, auto-discovery link tags, markdown content negotiation, agents.json at
/.well-known/agents.json, an MCP Server Card listed in/.well-known/ai-catalog.json(SEP-2127), and an A2A Agent Card that declaressupportedInterfaces. - Content negotiation via
Accept: text/markdownis the single highest-leverage implementation for agent accessibility. - On 2026-11-04 the repaired check of aeoforged.com found 19 of 27 applicable protocols, with a read score of 100.
- Run a baseline audit before starting and re-audit after each deployment step to confirm signal detection.
- Most sites can deploy all seven components in 4-8 hours; automated generation tools reduce this to minutes.
- These are open standards, not proprietary requirements. Every file in this guide is a plain text file served from your existing infrastructure.