By Ryan Kings, Founder & CTO at AEOForged · Published June 2026 · 12 min read
How to Make Your Site Agent-Operable: The Complete Technical Guide
How to Make Your Site Agent-Operable: The Complete Technical Guide
Making a website agent-operable means deploying standards-compliant files and middleware that AI agents use to discover, understand, and interact with your site. As of June 2026, four independent scoring frameworks measure agent operability: Cloudflare's isitagentready.com, Google Lighthouse Agentic Browsing, FORKOFF, and AEOForge's 33-signal audit. This guide walks through the seven components you need to deploy, where each file goes, and how to verify it worked.
When we applied these steps to AEOForge's own site, we went from 11 detected signals to 28/28 applicable signals. Our Cloudflare score rose from 25% to 75%, Google Lighthouse Agentic Browsing from 50% to 100%, and FORKOFF from 46% (Tier D) to 100% (Tier A).
What should you do before writing any agent operability code?
The first step is a baseline audit that tells you exactly which of the seven components your site already has and which are missing. AEOForge's aeo_agent_ready tool (free, no account required) returns a 33-signal report covering all four 2026 scoring frameworks: Cloudflare isitagentready.com, Google Lighthouse Agentic Browsing, FORKOFF, and AEOForge's own checker.
The report produces a prioritised to-do list with specific file paths and fix guidance. After deploying each step below, re-run the scan to confirm the signal flipped green. Alternatively, aeo_make_agent_ready (8 credits) generates every artifact in this guide as platform-specific deployable code you can commit directly.
How do you configure robots.txt for AI agents in 2026?
The robots.txt file is the first thing every AI agent reads when it encounters your site. A properly configured robots.txt requires explicit rules for the six major AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, CCBot) and a Content-Signal directive. The Content-Signal directive, backed by Cloudflare and checked by their scoring framework as of 2026, declares what AI systems may do with your content: search indexing, real-time retrieval for AI answers, and training dataset inclusion.
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
User-agent: GPTBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
User-agent: ClaudeBot
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=no
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
The three Content-Signal values control search indexing (yes), real-time retrieval for AI answers (yes), and training dataset inclusion (no). This directive is distinct from the X-Robots-Tag header and applies specifically to AI systems.
How do you create a valid llms.txt file?
The llms.txt file is a plain markdown index at /llms.txt that gives agents a curated map of your site's most important pages. The llms.txt specification defines the format: it is the content equivalent of robots.txt, telling agents what to read rather than what to avoid.
A valid llms.txt file follows these rules (per the June 2026 spec):
- First line:
# Your Site Name - Second line: a blockquote with a one-sentence description
- All links must be absolute URLs with descriptions
- File size under 20KB, no HTML tags
# YourSite
> One-sentence description of what your site does.
## Core pages
- [Homepage](https://yourdomain.com/): Platform overview
- [Pricing](https://yourdomain.com/pricing): Plans and pricing
## Documentation
- [Getting Started](https://yourdomain.com/docs/start): Setup guide
You may also publish /llms-full.txt with expanded content for agents that want deeper context. Both files are checked by Cloudflare and Lighthouse.
How do you add auto-discovery link tags for agent protocols?
Auto-discovery link tags are <link> elements in your document <head> that point agents directly to your protocol files without requiring them to crawl or guess file locations. Four tags cover the core discovery surface: llms.txt, agents.json, agent-instructions.md, and sitemap.xml.
<link rel="describedby" href="/llms.txt" type="text/plain" />
<link rel="describedby" href="/agents.json" type="application/json" />
<link rel="describedby" href="/agent-instructions.md" type="text/markdown" />
<link rel="sitemap" href="/sitemap.xml" type="application/xml" />
In Next.js App Router, add these to your root layout.tsx metadata or serve them as <link> elements in your head component. Google Lighthouse Agentic Browsing checks for these tags as of 2026 and flags their absence.
How does markdown content negotiation work for AI agents?
When an agent sends Accept: text/markdown, your server returns clean markdown instead of HTML. This is the single highest-leverage implementation for agent content accessibility because it eliminates the parsing step entirely.
The approach follows standard HTTP content negotiation per RFC 9727. Your middleware checks the Accept header, rewrites the request to an internal markdown endpoint, and returns the response with appropriate headers:
import { NextRequest, NextResponse } from "next/server";
export function middleware(request: NextRequest) {
const accept = request.headers.get("accept") ?? "";
if (!accept.toLowerCase().includes("text/markdown")) {
const response = NextResponse.next();
response.headers.set("Vary", "Accept");
return response;
}
const url = request.nextUrl.clone();
url.pathname = "/api/markdown" + url.pathname;
const response = NextResponse.rewrite(url);
response.headers.set("Content-Type", "text/markdown; charset=utf-8");
response.headers.set("Vary", "Accept");
return response;
}
The /api/markdown/[...path] route fetches your own page as HTML, strips markup, and returns clean text. Set Content-Location to signal the alternate representation exists.
How do you deploy agents.json and MCP Server Card?
The agents.json file and MCP Server Card are two action manifests that tell agents what operations are available on your site. agents.json at your site root describes available actions with typed parameters and endpoints, while the MCP Server Card at /.well-known/mcp/server-card.json (defined by SEP-2127) enables MCP-compatible agents to discover your tools and their input schemas programmatically.
{
"schema_version": "v1",
"name_for_human": "Your Site Name",
"name_for_model": "your_site",
"description_for_model": "Machine-readable description for AI agents.",
"actions": [
{
"name": "search_content",
"description": "Search site content",
"parameters": {
"type": "object",
"properties": {
"query": { "type": "string" }
},
"required": ["query"]
},
"endpoint": "https://yourdomain.com/api/search",
"method": "GET"
}
]
}
The MCP Server Card should declare your tools, their input schemas, and authentication requirements. Both files are independently scored by Cloudflare and FORKOFF.
How do you enable agent-to-agent discovery with the A2A Agent Card?
The A2A Agent Card is Google's protocol for agent-to-agent discovery and communication, published at /.well-known/agent-card.json. It declares your agent's capabilities, supported protocols, and contact endpoints. A v0.3 compatibility version at /.well-known/agent.json ensures backward compatibility with older clients.
The Agent Card declares your agent's capabilities, supported protocols, and contact endpoints. As of June 2026, A2A is still in draft but already scored by FORKOFF and Google Lighthouse.
How do you verify your agent operability deployment?
Verification is a re-run of the same baseline audit you ran before starting. Each of the seven steps maps to specific named signals in the scoring frameworks, so you can confirm exactly which checks now pass:
- robots.txt (Step 1):
robots_txt_quality,content_signal - llms.txt (Step 2):
llms_txt,llms_txt_valid - Discovery links (Step 3):
auto_discovery_links - Content negotiation (Step 4):
content_negotiation - agents.json (Step 5):
agents_json,agent_permissions - MCP Server Card (Step 6):
mcp_server_card - A2A Agent Card (Step 7):
a2a_agent_card
A site that completes all seven steps passes all Cloudflare isitagentready.com checks, scores 100% on Google Lighthouse Agentic Browsing, and reaches Tier A on FORKOFF.
Key takeaways
- Agent operability in 2026 is measured by four independent frameworks: Cloudflare, Google Lighthouse, FORKOFF, and AEOForge's 33-signal audit.
- The seven required components are: robots.txt with Content-Signal, llms.txt, auto-discovery link tags, markdown content negotiation, agents.json, MCP Server Card (SEP-2127), and A2A Agent Card.
- Content negotiation via
Accept: text/markdownis the single highest-leverage implementation for agent accessibility. - AEOForge's own site went from 11/28 signals to 28/28 after deploying these seven steps, with Cloudflare score rising from 25% to 75% and Lighthouse from 50% to 100%.
- Run a baseline audit before starting and re-audit after each deployment step to confirm signal detection.
- Most sites can deploy all seven components in 4-8 hours; automated generation tools reduce this to minutes.
- These are open standards, not proprietary requirements. Every file in this guide is a plain text file served from your existing infrastructure.