Agent Docs
Write documentation that AI agents can efficiently consume. Based on Vercel benchmarks and industry standards (AGENTS.md, llms.txt, CLAUDE.md).
The Hybrid Context Hierarchy
Three-layer architecture for optimal agent performance:
Layer 1: Constitution (Inline)
Always in context. 2,000–4,000 tokens max.
# AGENTS.md
> Context: Next.js 16 | Tailwind | Supabase
## 🚨 CRITICAL
- NO SECRETS in output
- Use `app/` directory ONLY
## 📚 DOCS INDEX (use read_file)
- Auth: `docs/auth/llms.txt`
- DB: `docs/db/schema.md`
Include:
- Security rules, architecture constraints
- Build/test/lint commands (top for primacy bias)
- Documentation map (where to find more)
Layer 2: Reference Library (Local Retrieval)
Fetched on demand. 1K–5K token chunks.
- Framework-specific guides
- Detailed style guides
- API schemas
Layer 3: Research Assistant (External)
Gated by allow-lists. Edge cases only.
- Latest library updates
- Stack Overflow for obscure errors
- Third-party llms.txt
Why This Works
Vercel Benchmark (2026):
| Approach | Pass Rate |
|---|---|
| Tool-based retrieval | 53% |
| Retrieval + prompting | 79% |
| Inline AGENTS.md | 100% |
Root cause: Meta-cognitive failure. Agents don't know what they don't know—they assume training data is sufficient. Inline docs bypass this entirely.
Core Principles
1. Compressed Index > Full Docs
An 8KB compressed index outperforms a 40KB full dump.
Compress to:
- File paths (where code lives)
- Function signatures (names + types only)
- Negative constraints ("Do NOT use X")
2. Structure for Chunking
RAG systems split at headers. Each section must be self-contained:
## Database Setup ← Chunk boundary
Prerequisites: PostgreSQL 14+
1. Create database...
Rules:
- Front-load key info (chunkers truncate)
- Descriptive headers (agents search by header text)
3. Inline Over Links
Agents can't autonomously browse. Each link = tool call + latency + potential failure.
| Approach | Token Load | Agent Success |
|---|---|---|
| Full inline | ~12K | ✅ High |
| Links only | ~2K | ❌ Requires fetching |
| Hybrid | ~4K base | ✅ Best of both |
4. The "Lost in the Middle" Problem
LLMs have U-shaped attention:
- Strong: Start of context (primacy)
- Strong: End of context (recency)
- Weak: Middle of context
Solution: Put critical rules at TOP of AGENTS.md. Governance first, details later.
5. Signal-to-Noise Ratio
Strip everything that isn't essential:
- No "Welcome to..." preambles
- No marketing text
- No changelogs in core docs
Formats like llms.txt and AGENTS.md mechanically increase SNR.
llms.txt Standard
Machine-readable doc index for agents:
# Project Name
> One-line project description.
## Authentication
- [Setup](docs/auth/setup.md): Environment vars and init
- [Server](docs/auth/server.md): Cookie handling
## Database
- [Schema](docs/db/schema.md): Full Prisma schema
Location: /llms.txt at domain root
Companion: /llms-full.txt — full concatenated docs, HTML stripped
Security Considerations
Inline = Trusted
AGENTS.md is part of your codebase. Controlled, version-pinned.
External = Attack Surface
- Indirect prompt injection via hidden text
- SSRF risks if agents can browse freely
- Dependency on external uptime
Mitigation: Domain allow-lists, human-in-the-loop for external retrieval.
Anti-Patterns
- Pasting 50 pages — triggers "Lost in the Middle"
- "See external docs" — agents can't browse autonomously
- Generic advice — "Write clean code" (use specific constraints)
- TOC-only docs — indexes without content
- Trusting retrieval alone — 53% vs 100% pass rate
Advanced Patterns
For detailed guidance on RAG optimization, multi-framework docs, and API templates, see references/advanced-patterns.md.
Validation Checklist
- Critical governance at TOP of doc
- Total inline context under 4K tokens
- Each H2 section self-contained
- No external links without inline summary
- Negative constraints explicit ("Do NOT...")
- File paths and signatures, not full code