Answer Engine Optimization

Ten operating principles

1. Optimize evidence rather than generic “AI-friendly” prose.
2. Model availability, discovery, retrieval, citation, representation, and action as separate stages.
3. Maintain one canonical source of truth for consequential entities and claims.
4. Publish unique, primary, verifiable information with explicit scope and limitations.
5. Use feeds and APIs for volatile facts such as price, inventory, events, and availability.
6. Separate search indexing, model training, user-directed retrieval, advertising, licensing, and transactions.
7. Treat crawler directives as preferences, not authentication or authorization.
8. Require runtime validation and conformance—not just schemas and TypeScript types.
9. Measure fidelity and business outcomes rather than citation count alone.
10. Place draft protocols behind adapters and maintain explicit version and migration policy.

Recommended scope

Use AEO as the enterprise umbrella. Treat SEO as its public-search foundation, GEO as its generative-answer measurement and experimentation subset, and agent experience as its execution layer.

Evidence hierarchy

1. Official platform documentation and first-party telemetry.
2. Standards documents, conformance suites, and production-derived code.
3. Controlled, reproducible experiments with released code and datasets.
4. Repeated multi-engine observational studies with transparent methods.
5. Single-engine snapshots and limited audits.
6. Vendor correlations, proprietary scoring models, and anecdotal checklists.

Practices supported by the strongest evidence

What the research does—and does not—show

Purpose-based crawler registry

The missing enterprise component: a policy compiler

Do not manually maintain robots.txt, HTTP headers, RSL documents, WAF rules, crawler allowlists, feeds, and agent permissions as disconnected files. Generate them from one governed policy model and reject contradictions during CI.

content_class: public-editorial

uses:
  conventional_search: allow
  answer_grounding: allow_with_attribution
  model_training: deny
  user_directed_fetch: allow
  commercial_republication: license_required

access:
  anonymous: true
  verified_bot_preferred: true
  rate_limit: 120/minute

licensing:
  discovery: /.well-known/rsl.xml
  training:
    mode: paid
    currency: USD
    unit: crawl

owners:
  policy: legal-content-rights
  technical: edge-platform

Illustrative crawler policy

# Public conventional search
User-agent: Googlebot
Allow: /

# ChatGPT search discovery
User-agent: OAI-SearchBot
Allow: /

# OpenAI foundation-model training
User-agent: GPTBot
Disallow: /

# User-directed retrieval still requires application authorization
User-agent: ChatGPT-User
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Commerce is a multi-plane system

Minimum agent-action controls

• Authenticated user or workload identity
• Least-privilege authorization and rights policy
• Typed inputs plus server-side validation
• Read/write and consequential-action classification
• Idempotency, replay protection, and transaction limits
• Explicit confirmation before consequential operations
• Structured success, failure, and recovery responses
• Auditable decisions, state transitions, and external evidence

Implementation lessons

Canonical claim object

Consequential facts should have identity, evidence, temporal validity, rights, and ownership independent of the page or protocol through which they are published.

interface CanonicalClaim {
  id: string;
  subjectId: string;
  predicate: string;
  statement: string;

  status: "draft" | "approved" | "deprecated";
  confidence: "verified" | "supported" | "provisional";

  validFrom?: string;
  validThrough?: string;
  lastReviewedAt: string;

  jurisdictions?: string[];
  audiences?: string[];
  exceptions?: string[];
  rightsClass?: string;

  evidence: Array<{
    sourceUrl: string;
    sourceType: "primary" | "secondary" | "internal";
    retrievedAt: string;
    excerptHash?: string;
  }>;

  owner: {
    team: string;
    approver?: string;
  };

  supersedes?: string;
}

Publication and release gates

1. Resolve the authoritative entity and claim identifiers.
2. Validate evidence, scope, dates, ownership, and rights.
3. Compile usage policy into every downstream carrier.
4. Generate HTML, JSON-LD, feeds, APIs, and agent representations.
5. Run type, runtime, semantic, parity, and platform validation.
6. Calculate affected URLs and derived representations.
7. Publish, invalidate caches, update sitemaps, and notify indexes.
8. Run protocol conformance and security tests.
9. Measure discovery, retrieval, fidelity, value, and negative outcomes.
10. Feed results into claims, policy, and adapter revisions.

Minimum evaluation protocol

1. Define representative user intents and query families.
2. Create controlled paraphrases rather than one exact prompt.
3. Test multiple engines, locales, sessions, and dates.
4. Record whether external search or retrieval activated.
5. Capture cited URLs, ordering, and visible presentation.
6. Compare answer claims with canonical claims and evidence.
7. Measure evidence absorption separately from citation.
8. Detect omissions, unsupported claims, and stale facts.
9. Track referrals, conversions, and zero-click value proxies.
10. Test agent actions independently with authorization and recovery.
11. Retain model, engine, date, locale, context, and experimental conditions.
12. Use holdouts, controlled changes, and rollback when testing rewrites.

Measurement cautions

• Run-to-run variability can exceed the effect being measured.
• Source overlap across engines may be low.
• Search activation may change by prompt wording and freshness.
• Paid placement, merchant feeds, and organic retrieval must be separated.
• A citation can be visible without materially supporting the answer.
• Increased visibility can amplify an incorrect or outdated claim.
• Competitive adoption can erase an isolated optimization advantage.

Required controls

Near-term platform backlog

Procurement questions for AEO vendors

1. Which pipeline stage does the product actually observe or control?
2. Which claims are supported by official telemetry versus inferred experiments?
3. Can results be reproduced across engines, sessions, locales, and dates?
4. How are citation, absorption, fidelity, referral, and conversion separated?
5. Does the system modify factual content, and how is equivalence validated?
6. Can the organization export prompts, evidence, raw observations, and history?
7. How are paid, organic, feed-driven, and sponsored results distinguished?
8. What rights, crawler, and data-retention policies apply to submitted content?
9. Does it provide an API, versioning, audit trail, and rollback?
10. What evidence would falsify the vendor’s recommendation?

Update cadence

Change-control rules

1. Record source publication date, retrieval date, and source type.
2. Prefer normative specifications and first-party documentation.
3. Separate platform claims from cross-platform conclusions.
4. Label drafts, community reports, industry specifications, and research distinctly.
5. Do not promote a tactic based on one engine, one run, or one vendor correlation.
6. Update maturity only when implementation, validation, conformance, or adoption evidence changes.
7. Retain superseded findings and the reason for revision in a changelog.
8. Re-run threat and rights analysis when adding any agent action or transaction.

Open questions to monitor

• Will AIPREF produce stable semantics for training, search, grounding, and inference use?
• Will verified bot identity converge on broadly interoperable HTTP signature profiles?
• Can citation provenance travel from generated sentence to canonical claim and licensed source?
• Will platforms expose standardized retrieval, citation, absorption, and fidelity events?
• How will paid placement, licensing, organic answers, and transactions be independently audited?
• Which agent and commerce protocols will achieve durable multi-platform conformance?
• How should multimodal evidence retain rights, source, transformation, and temporal validity?

Standards and foundational web controls

1. S01Google: Optimizing for generative AI featuresOfficial guidance on SEO foundations, query fan-out, technical eligibility, non-commodity content, generative reporting, and myths such as special markup, artificial chunking, and llms.txt.Primary
2. S02Google: How Search worksOfficial description of crawling, indexing, and serving.Primary
3. S03RFC 9309: Robots Exclusion ProtocolIETF Standards Track RFC defining REP parsing, matching, error handling, and caching expectations.Normative
4. S04RFC 9969: IAB Workshop on AI-CONTROLDocuments limitations and design concerns around content-control and AI-use signaling.Primary
5. S05Schema.org release historyCurrent published release listing; version 30.0 published March 19, 2026.Primary
6. S07IndexNow documentationOpen URL-change notification protocol, request formats, validation, and response semantics.Primary
7. S21IETF AIPREF vocabulary draftActive Internet-Draft defining vocabulary for automated processing preferences; work in progress.Draft
8. S22IETF AIPREF HTTP attachment draftEmerging mechanisms for associating AI-use preferences with content through HTTP and REP.Draft
9. S23The /llms.txt proposalCommunity proposal and implementation links for packaging site documentation into LLM-oriented context.Proposal

Platform controls and telemetry

1. S10Bing AI Performance in Webmaster ToolsOfficial citation telemetry and explicit limitations of citation counts.Primary
2. S11Bing expanded AI visibility insightsIntents, topics, citation share, and comparison capabilities in preview.Primary
3. S12OpenAI crawler overviewOfficial purpose separation for OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot, with published identity data.Primary
4. S13OpenAI Agentic Commerce ProtocolOfficial ACP overview for catalog ingestion, product discovery, and commerce integration.Primary
5. S14OpenAI product feed referenceStructured product ingestion, discovery, purchase, and ads eligibility fields.Primary
6. S24Anthropic crawler guidanceOfficial distinction among ClaudeBot, Claude-SearchBot, and Claude-User.Primary
7. S25Perplexity crawler guidanceOfficial PerplexityBot and Perplexity-User descriptions, published IP addresses, and WAF guidance.Primary

Agent, commerce, licensing, and public implementation

1. S06Google schema-dtsGenerated TypeScript definitions for Schema.org JSON-LD; not an officially supported Google product.Code
2. S08Microsoft IndexNow WordPress pluginAutomated CMS event integration for URL submission; public issues expose operational edge cases.Code
3. S09Google robots.txt parser and matcherPublic C++ parser and matcher library derived from Google’s crawler implementation context.Code
4. S15Universal Commerce Protocol specificationVersioned commerce capability discovery, authentication options, negotiation, identity, checkout, and lifecycle contracts.Specification
5. S16WebMCP Draft Community Group ReportJuly 21, 2026 draft enabling web applications to provide JavaScript-based tools to AI agents; not a W3C Standard.Draft
6. S18NLWeb reference implementationOpen protocols and proof-of-concept tools for natural-language web interfaces and MCP exposure.Code
7. S19RSL 1.0 specificationMachine-readable usage, licensing, legal, compensation, and reporting terms for digital assets.Industry spec
8. S20Cloudflare Web Bot AuthTypeScript and Rust examples for signed automated HTTP requests; repository notes that the software has not been audited.Code / draft
9. S31MCP conformance test frameworkExecutable client and server conformance framework demonstrating a stronger protocol-maturity model.Code

Research and benchmarks

1. S26GEO: Generative Engine OptimizationFoundational GEO paper and GEO-Bench. Reported gains are conditional on the experimental setting and candidate context.Research
2. S27Critical survey of GEO, 2023–2026Pipeline model, evidence hierarchy, portability limits, and the absence of stable longitudinal cross-platform causal proof in the reviewed corpus.Research
3. S28Measurement framework for citation selection and absorptionSeparates visible citation from material contribution to a generated answer.Research
4. S29C-SEO BenchMulti-domain, multi-task, multi-actor benchmark reporting limited effectiveness for many conversational SEO tactics.Research code
5. S30AutoGEOEngine- and domain-specific preference learning and content rewriting framework; illustrates conditional portability.Research code