- Published
- Updated
Agent & AI readiness for websites
Agent and AI readiness is the work of making a website understandable to search engines, AI-assisted search features, crawlers, and automated agents while keeping control of access, load, privacy, and business value.
It is not a separate trick layered on top of SEO. It sits on the same foundation: crawlable pages, clear text, useful internal links, structured content, accurate metadata, safe forms, and deliberate rules for bots.
For business websites, this has direct consequences. A property site needs agents to understand listings, location, availability signals, and enquiry routes. A booking business needs clear service pages, pricing context, and transaction boundaries. A local service company needs its area, services, contact route, and credibility signals to be visible without hiding useful details inside a script or image.
I have a weakness for scorecards: if a tool offers a 100% score, I usually aim for it, even when 100% is hardly ever necessary for a normal website. :-) The green result is not the real objective. Working through every check forces me to understand the protocol, decide whether it belongs on the site, and make sure anything I publish is connected to a working feature.
I used Webstudiet as the implementation project. The first complete scan classified the live site as Level 5, “Agent-Native,” but the score was 93%. The remaining scored gap was ARD: the site already had real MCP, OpenAPI, and Agent Skills resources, but no common manifest connecting them. That was a useful gap to close rather than a protocol added only for appearance.
Start With What Google Actually Requires
Google’s current guidance for AI features such as AI Overviews and AI Mode is conservative: the same SEO fundamentals still apply, and there are no extra technical requirements or special schema types needed just to appear in those features. Google also says a page needs to be indexed and eligible to show a snippet to be eligible as a supporting link in those AI features.
That keeps the priority order clear. Before adding new agent-specific files, make sure the page can be crawled, indexed, linked, rendered, understood, and shown with an appropriate snippet. That connects directly to the earlier guide on crawlability and indexation.
New machine-readable files can still be useful for other AI systems and agents. They are not a replacement for basic technical SEO.
Make Important Content Textual and Extractable
AI systems and agents are better at using content when the important information is present as ordinary text in the rendered page. Do not rely on images, carousels, hidden tabs, or JavaScript-only widgets to carry the core message.
A service page should clearly state what is offered, who it is for, where it applies, what problem it solves, what the next step is, and what constraints matter. Relevant details can include service areas, availability, pricing context, booking requirements, multilingual audiences, or support boundaries.
This also helps human visitors. If an agent can extract the service, area, price context, and enquiry route without guessing, a busy business owner can usually do the same.
Use Structure and Metadata to Reduce Ambiguity
Agent readiness depends on clean structure. Headings should describe the content below them. Internal links should point to useful next steps. Titles and descriptions should match the page. Canonicals, redirects, and language alternates should not contradict each other.
Structured data is useful when it reflects the visible page. Use Article, BreadcrumbList, Organization, LocalBusiness, Product, Offer, or Service only where the page genuinely supports that meaning. Google’s structured data guidelines are clear that structured data should represent the visible page, not hidden or misleading claims.
For the implementation details behind those signals, see the guide on structure, semantics, and metadata.
Handle Robots and AI Crawler Controls Deliberately
Robots rules should reflect a business decision, not an accident. Some crawlers help discovery. Some train models. Some create load without sending useful traffic. Some may try to reach URLs that should never be automated.
For Google Search AI features, Googlebot access and snippet controls are the relevant search controls. If you want to limit what appears from a page in Search, review nosnippet, data-nosnippet, max-snippet, and noindex. If you want to limit certain non-Search AI training or grounding uses, review the crawler-specific controls that apply to that system, such as Google-Extended for some Google AI uses outside Search.
For the broader web, each crawler may publish its own user agent and rules. Review server logs, rate limits, firewall rules, and CDN bot controls before assuming that a single directive covers every automated system.
Machine-Readable Files Are Optional Tools
Files such as /llms.txt, /llms-full.txt, markdown endpoints, feeds, API catalogs, and well-known discovery files can help some automated systems understand a site. They are most useful when they are accurate, maintained, and connected to real public content.
For a small service website, a concise llms.txt or markdown version of key pages may be enough. For a booking platform, SaaS product, marketplace, or API-driven project, deeper discovery may make sense: API documentation, OpenAPI files, MCP server cards, WebMCP, A2A agent cards, or skills-style descriptions.
The rule is simple: add machine-readable formats when they reduce ambiguity or support a real workflow. Do not add them as decorative SEO files that no one maintains.
Protect Actions, Forms, and Commercial Flows
Reading content is different from performing an action. An agent that summarizes a service page is low risk. An agent that submits a contact form, books a room, requests a quote, adds a product to a cart, or triggers a payment needs stronger boundaries.
Important flows should have clear form labels, validation, CSRF protection, abuse prevention, rate limits, confirmation steps, useful error messages, and logging. Private areas, admin endpoints, dynamic search URLs, expensive generated pages, and booking actions should not be open to unlimited automated access.
This overlaps with security, privacy, and protocol. Agent readiness should never mean opening everything.
What isitagentready.com Checks — and What Each Check Requires
Cloudflare’s isitagentready.com scanner currently offers separate presets for content sites and API or application sites, plus an all-checks scan. That distinction matters: passing a check means the scanner found the protocol signal it expects. It does not mean every website should implement every protocol.
The requirements below summarize the scanner’s own published implementation guides as of September 2026. Many of the agent-specific protocols are drafts or early standards, so confirm the current specification and scanner guidance before implementing them. The scanner’s checks are useful detection tests, not complete security or protocol-conformance audits.
Discoverability
| Check | What the scanner expects | When it is relevant |
|---|---|---|
robots.txt | /robots.txt returns HTTP 200 as text/plain, contains User-agent with Allow or Disallow, and references the sitemap when one exists. | Almost every public website. |
| Sitemap | /sitemap.xml returns valid XML with canonical <loc> URLs, stays current, and is referenced from robots.txt. | Any site with public pages that should be discovered. |
| Link response headers | The homepage response has an RFC 8288 Link header pointing to a real machine-readable resource with a relation such as api-catalog, service-desc, service-doc, or describedby. | Sites that publish an API catalog, specification, documentation, or another discovery document. |
| DNS for AI Discovery (DNS-AID) | Publish ServiceMode SVCB records, or HTTPS records for HTTPS endpoints, below the _agents namespace — for example _index._agents.example.com or _a2a._agents.example.com. Include alpn and connection parameters, use numeric keyNNNNN names for experimental custom parameters, and return DNSSEC-authenticated data. | Organizations that operate real agent, MCP, or A2A endpoints. It is unnecessary for an ordinary content-only site. |
On Webstudiet, I serve a plain-text robots.txt with the sitemap declaration, generate the sitemap as /sitemap-index.xml, and add response Link headers for the Markdown index, sitemap, API catalog, and Agent Skills index. For DNS-AID, I published an HTTPS record at _index._agents.webstudiet.com with priority 1 and alpn=“mcp,h2”. I also enabled DNSSEC so validating resolvers return an authenticated answer. Those four discoverability checks now pass on the live site.
DNSSEC is the easy-to-miss part of the DNS-AID check. The scanner’s DNS-AID guide requires public discovery zones to be signed so its DNS-over-HTTPS lookup can validate the answer. In practice, that normally means enabling DNSSEC for the domain’s DNS zone and completing the DS record at the registrar, not merely adding an HTTPS or SVCB record.
That scanner requirement is stricter than an unconditional requirement in the current DNS-AID Internet-Draft. The draft says DNS-AID records should be DNSSEC-signed; if TLSA records are used, they must be signed. For this scanner, however, unsigned DNS-AID records do not pass.
Content accessibility
| Check | What the scanner expects | When it is relevant |
|---|---|---|
| Markdown content negotiation | A request for the normal page URL with Accept: text/markdown returns a useful Markdown representation and Content-Type: text/markdown. Requests without that header still return HTML. An x-markdown-tokens response header is useful when available, but not the core requirement. | Content, documentation, and product sites where agents benefit from a lower-noise version of the same page. |
I implemented this in the shared Cloudflare Worker used by both Webstudiet sites. A request to an ordinary page with Accept: text/markdown is converted to Markdown on the fly, while the same URL still returns HTML to a browser. The response includes the Markdown content type and token-count headers. This is content negotiation on the same URL, not just a separate .md file.
The conversion preserves useful headings, text, lists, tables, links, image descriptions, and code while omitting navigation and interface noise. That makes the result smaller and easier for an agent to consume without creating a second content source that can drift out of date.
Bot access control
| Check | What the scanner expects | When it is relevant |
|---|---|---|
| AI bot rules | Explicit User-agent blocks in robots.txt for AI crawlers, with an Allow or Disallow policy. The scanner recognizes names including GPTBot, OAI-SearchBot, Claude-Web, Google-Extended, Amazonbot, anthropic-ai, Bytespider, CCBot, and Applebot-Extended. A wildcard block alone does not pass this check. | Any site that wants an explicit policy for AI crawlers. |
| Content Signals | A Content-Signal directive under the relevant User-agent block declaring preferences for ai-train, search, and ai-input, for example Content-Signal: ai-train=no, search=yes, ai-input=yes. | Publishers that want to state allowed uses more precisely than allow or disallow. It is an emerging convention, not an access-control mechanism by itself. |
| Web Bot Auth | A JWKS with at least one public verification key at /.well-known/http-message-signatures-directory. A bot operated by the site must also sign its outbound requests with Signature-Agent and Signature-Input headers for receiving sites to verify. | Sites that operate a bot or agent which makes requests to other sites. A content-only site does not need to invent an outbound bot identity. |
My robots.txt explicitly allows the main search and assistant crawlers rather than relying only on User-agent: *. I also publish Content-Signal: search=yes, ai-input=yes. I deliberately do not declare an ai-train preference until I have chosen that policy separately.
I have not published a Web Bot Auth key directory because Webstudiet does not operate an outbound bot that visits other sites under its own identity. The scanner therefore reports this as informational rather than failed. If I add such an agent later, it will need real signed requests as well as the public keys; an unused JWKS document would not provide meaningful authentication.
Protocol discovery
| Check | What the scanner expects | When it is relevant |
|---|---|---|
| MCP Server Card | JSON at /.well-known/mcp/server-card.json with HTTP 200, serverInfo.name, serverInfo.version, a transport endpoint, and the server’s real capabilities. | A site that operates a discoverable MCP server. |
| Agent Skills | JSON at /.well-known/agent-skills/index.json with $schema set to the v0.2.0 discovery schema. Every skill entry needs a lowercase hyphenated name, a type of skill-md or archive, a description, artifact url, and SHA-256 digest. | A site that publishes reusable instructions or skill packages for agents. |
| WebMCP | Page JavaScript calls navigator.modelContext.registerTool() on load. Each tool has a name, description, JSON Schema input, and an execution callback, and should be unregistered with an AbortController when no longer needed. The scanner detects the API in a real browser. | Interactive websites that want to expose existing browser actions as agent-callable tools. |
| API Catalog | /.well-known/api-catalog returns HTTP 200 with Content-Type: application/linkset+json. Its linkset entries identify each API with an anchor and relations such as service-desc and service-doc. | Services with one or more public APIs. |
| OAuth discovery | OIDC metadata at /.well-known/openid-configuration or OAuth authorization-server metadata at /.well-known/oauth-authorization-server, including issuer, authorization endpoint, token endpoint, JWKS URI, and supported grant and response types. | APIs or applications that use OAuth or OpenID Connect. |
| OAuth Protected Resource | JSON at /.well-known/oauth-protected-resource with the protected resource identifier and its authorization-server URLs. Supported scopes may also be listed; a 401 response can point to the document with WWW-Authenticate. | An OAuth-protected API or MCP resource. |
Auth.md | /auth.md returns Markdown whose H1 contains auth.md. It documents the agent audience, registration or provisioning endpoints, supported methods, and credential use; where OAuth metadata exists, the document should connect to it and describe the agent_auth registration flow. | Services that allow agents to register or obtain credentials. |
| ARD | /.well-known/ai-catalog.json returns JSON with Access-Control-Allow-Origin: *, a specVersion, and a host display name and stable identifier. It has at least one entry whose identifier follows urn:air:<fqdn>:<namespace>:<name>, with a display name, media type, exactly one of url or data, and two to five representative queries. | Services that need one catalog covering MCP servers, A2A agents, skills, or API tools. |
| A2A Agent Card | The current custom-scan interface also tests /.well-known/agent-card.json. The card needs a name, version, description, supported service interfaces, capabilities, and described skills. | A service that implements agent-to-agent communication. |
These documents must describe working services. An empty MCP card, fake API catalog, or OAuth document without a functioning authorization server may raise a score but makes the site less reliable for agents and can create security risk.
For Webstudiet, I publish an API catalog connected to the real URL Analyzer and its OpenAPI document, an Agent Skills index with a generated SHA-256 digest, an MCP Server Card, and discovery documents explaining that the current public access model is anonymous and requires no credentials. The Auth.md file says that directly instead of pretending the site issues OAuth tokens.
I Made the URL Analyzer Available Through MCP
The most useful protocol work was not the discovery files. I built an actual MCP endpoint for the Webstudiet URL Analyzer at /api/mcp. It exposes an analyze_url tool. An MCP-compatible agent can supply a public HTTP or HTTPS URL, run it through the same analyzer as a visitor, and receive the score together with the passing checks, warnings, and failures.
The MCP Server Card at /.well-known/mcp/server-card.json tells agents where that endpoint lives and which capability it exposes. I also register three browser-side WebMCP tools: one for site context, one for URL analysis, and one for contact guidance. That is why MCP and WebMCP are useful on this site: they connect agents to a working tool instead of existing only to satisfy a scanner.
I have not added an A2A Agent Card because Webstudiet does not currently operate an autonomous A2A service. ARD was different: it could describe capabilities that already existed. I therefore added /.well-known/ai-catalog.json with entries for the URL Analyzer MCP Server Card, the public OpenAPI description, and the Webstudiet Site Context skill. Each entry has a stable urn:air identifier, its native media type, exactly one URL, and three representative queries. That manifest closes the scored ARD gap without inventing a service.
Commerce
| Check | What the scanner expects | When it is relevant |
|---|---|---|
| x402 | A protected API route returns HTTP 402 with machine-readable x402 payment requirements, backed by a configured payment facilitator and receiving wallet. | APIs selling metered or pay-per-request access. |
| MPP | /openapi.json describes payable operations with x-payment-info, including an intent of charge or session, a supported payment method such as tempo, stripe, lightning, or card, and an amount. | APIs implementing the Machine Payment Protocol. |
| UCP | JSON at /.well-known/ucp with protocol_version, services, capabilities, and endpoints, with reachable referenced schemas and specifications. | Merchants implementing Universal Commerce Protocol. |
| ACP | JSON at /.well-known/acp.json with protocol name acp and version, an absolute API base URL, at least one transport, and at least one service capability. | Merchants implementing Agentic Commerce Protocol. |
I did not implement any of these commerce protocols on Webstudiet because it is not an automated storefront and the URL Analyzer is not sold per request. The scanner correctly marks the checks as neutral for a non-commerce site. A normal company website should not add payment endpoints merely to satisfy a score; they belong on systems that can quote, sell, authorize, and fulfil the transaction safely.
How to Interpret an Agent-Readiness Score
I still aim for 100% because the exercise exposes weak assumptions and half-finished integrations. But I read 100% as “everything relevant is implemented properly,” not “every possible protocol exists.” For a public content or service website, the practical baseline is a valid robots.txt, a current sitemap, useful HTML, an explicit crawler policy, and — where it can be maintained correctly — Markdown content negotiation. Link headers only help when there is a real discovery resource to link to.
MCP, A2A, Agent Skills, WebMCP, API catalogs, OAuth metadata, Auth.md, ARD, DNS-AID, and agentic commerce are capability signals. Implement them when the capability exists or is deliberately being built. A missing optional protocol is not automatically a defect, and publishing a non-functional document to collect a green check is worse than leaving it absent.
The Webstudiet URL Analyzer has its own broader agent-readiness checks for signals such as llms.txt, Markdown endpoints, crawler rules, machine-readable formats, discovery headers, MCP, A2A, skills, DNS-AID, NLWeb, WebMCP, and schema maps. It is not a replica of isitagentready.com, so its results and pass conditions should not be compared one for one.
What Not to Do
Avoid turning AI readiness into another thin-content pattern. Do not create doorway pages for every AI prompt variation. Do not hide content for machines that users cannot see. Do not mark up fake services, fake reviews, unsupported locations, or unavailable products. Do not mass-generate pages just because AI tools make it cheap. Those patterns move toward the kind of behavior covered in Google’s spam policies.
Good AI readiness makes the real website clearer. Bad AI readiness creates more ambiguity, more low-value URLs, and more risk.
A Readiness Checklist
- Make sure important pages are crawlable, indexable, internally linked, and snippet-eligible where visibility is wanted.
- Keep service, product, location, price, contact, and policy information visible as normal text.
- Use headings, links, metadata, canonicals, and language alternates consistently.
- Add structured data only where it matches the visible page.
- Review
robots.txt, meta robots, snippet controls, CDN bot controls, and server logs. - Add
llms.txt, markdown endpoints, feeds, API catalogs, or well-known discovery only when they will be maintained. - Protect forms, booking flows, checkout flows, admin areas, and expensive dynamic URLs from abuse.
- Monitor bot traffic, failed requests, crawl anomalies, and lead quality after changes.
- Recheck the site after content, routing, platform, or policy changes.
Frequently Asked Questions
Does a website need special schema or extra files to appear in AI Overviews or AI Mode?
No. Google’s current guidance says the same SEO fundamentals apply, with no extra technical requirements or special schema types needed. A page still has to be indexed and eligible to show a snippet.
Is llms.txt required for a website to be agent-ready?
No. It is an optional file that helps when it is accurate and maintained. A small service website may only need a concise version, while a booking platform or API-driven project may benefit from deeper discovery files.
Does DNS-AID require DNSSEC?
It does for the isitagentready.com check: the scanner expects the DNS-AID answer to be authenticated with DNSSEC. The current DNS-AID Internet-Draft uses “should” rather than an unconditional “must” for signing ordinary discovery records, while TLSA records must be signed. Treat scanner compliance and protocol conformance as related but distinct questions.
Does every website need MCP, OAuth, ARD, and commerce discovery?
No. Those checks describe capabilities, not a universal baseline. Publish them only when the website actually provides the corresponding server, API, authentication flow, agent catalog, or commercial transaction. A content-only website can be useful to agents without pretending to operate those services.
Can an agent use the Webstudiet URL Analyzer directly?
Yes. I expose the analyzer through an MCP endpoint at /api/mcp. Its analyze_url tool accepts a public URL and returns a concise account of the score, passing checks, warnings, and failures. The browser version also registers URL analysis as a WebMCP tool where that experimental API is available.
Should content be hidden from users but shown to crawlers or agents?
No. That approach creates ambiguity and risk instead of clarity, and it moves toward the patterns covered by Google’s spam policies. Content should be visible as ordinary text to people and machines alike.
What is the difference between robots.txt rules and AI crawler controls?
Robots.txt and meta robots directives mainly control search indexing and snippets. Separate crawler-specific controls, such as Google-Extended, govern non-Search AI training or grounding uses, and other crawlers may publish their own user agents and rules.
Why do forms and booking flows need extra protection when a site becomes agent-ready?
Reading a page is low risk, but submitting a form, booking a room, or triggering a payment is not. Those flows still need validation, CSRF protection, rate limits, and abuse prevention so readiness never turns into open automated access.
Readiness Means Clarity With Boundaries
For most websites, agent and AI readiness is worth doing, but only with boundaries. You want useful systems to understand your services, articles, products, location, contact options, and credibility signals. You do not want every automated client to scrape everything, submit forms freely, overload dynamic URLs, or blur the line between public content and private business process.
The answer is the same as good technical SEO: make the site clear, crawlable, structured, fast, secure, and explicit about what is public.
This guide is step 6 in the Technical SEO guide.
Use the Technical SEO URL Analyzer to check a live page against the same guide areas. If the website needs review and implementation help, see technical SEO implementation services or send the URL and the issue you want fixed.