AI-Native Product Development

"AI products aren't deterministic. They require continuous calibration, not just A/B tests."

This skill covers AI-Native Product Development — the overlay that modifies discovery, architecture, and delivery when AI is at the core. It addresses the unique challenges of building products where AI agents perform tasks autonomously.

Part of: Modern Product Operating Model — a collection of composable product skills.

Related skills: product-strategy, product-discovery, product-architecture, product-delivery, product-leadership

When to Use This Skill

Use this skill when:

• Building AI agents that act on behalf of users
• Adding LLM-powered features to existing products
• Designing human-AI interaction patterns
• Deciding how much autonomy to give AI
• Setting up eval strategies and calibration loops
• Managing the "agency-control tradeoff"

Not needed for: Traditional software products, ML models used only for backend optimization (no user-facing autonomy)

What Makes AI Products Different

Traditional Software vs. AI Products

The Core Challenge

AI products must navigate a fundamental tension:

More autonomy = More value (fewer steps, faster outcomes)
More autonomy = More risk (errors affect real work)

This is the Agency-Control Tradeoff.

Framework: The CCCD Loop

Credit: Aishwarya Goel & Kiriti Gavini

AI products require a Continuous Calibration and Confidence Development (CCCD) loop:

┌─────────────────────────────────────────────────────────────────┐
│                        CCCD LOOP                                │
│                                                                 │
│    CALIBRATE → CONFIDENCE → CONTINUOUS DISCOVERY → CALIBRATE   │
│         ↓           ↓              ↓                 ↓         │
│     Eval and    Build user    Observe AI       Update evals    │
│     adjust AI    trust over   interactions     and models      │
│     behavior     time         at scale                         │
└─────────────────────────────────────────────────────────────────┘

CCCD Components:

The Agency-Control Progression

Five Levels of AI Agency

Progression Strategy

Start lower, earn higher:

Level 1 → Build trust → Level 2 → Demonstrate reliability → Level 3 → ...

Graduation Criteria:

Never fully autonomous for:

• Irreversible actions (delete, send, purchase)
• High-stakes decisions (financial, legal, health)
• Novel situations outside training distribution
• Actions affecting third parties

AI-Native Discovery

Standard discovery practices need adaptation for AI products.

Modified Discovery Focus

AI-Specific Discovery Questions

Delegation appetite:

• "Which parts of this task feel tedious vs. require your judgment?"
• "If AI made an error here, what would the consequences be?"
• "How would you want to verify AI's work?"

Trust calibration:

• "What would AI need to demonstrate before you'd trust it to [action]?"
• "Have you used AI tools before? What built or broke your trust?"
• "Would you prefer AI to do more but occasionally err, or do less perfectly?"

Failure tolerance:

• "What kinds of errors are annoying vs. damaging?"
• "How quickly do you need to catch and fix AI mistakes?"
• "What's your 'undo' option if AI gets it wrong?"

Observing AI Interactions

In addition to interviews, AI discovery includes:

AI-Native Architecture

Solution Brief Additions

For AI features, add to standard solution brief:

AI-SPECIFIC SECTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

AGENCY LEVEL
Target: [Level 1-5]
Graduation path: [How might this evolve?]

FAILURE MODES
• [Failure mode 1]: [Consequence] → [Mitigation]
• [Failure mode 2]: [Consequence] → [Mitigation]

EVAL STRATEGY
• [Eval type 1]: [What we measure, how often]
• [Eval type 2]: [What we measure, how often]

CALIBRATION PLAN
• Initial calibration: [Approach]
• Ongoing calibration: [Cadence, triggers]

CONFIDENCE BUILDING
• How AI explains itself: [Approach]
• How users verify: [Mechanisms]
• Trust-building milestones: [Progression]

AI Bet Categories

In addition to standard bet categories:

AI-Native Delivery

Eval Strategy (Replaces Traditional Testing)

Eval Types:

Eval Metrics:

Eval Cadence:

Code change → Unit evals (automated)
Daily → Behavioral evals (automated)
Weekly → Human evals (sample)
Release → Adversarial evals (red team)
Continuous → Production evals (monitoring)

Staged Rollout for AI Features

AI features require more cautious rollout:

AI-Specific Rollout Gates:

Calibration Loop

Continuous calibration process:

OBSERVE → IDENTIFY → CALIBRATE → VALIDATE → DEPLOY
   ↑                                           │
   └───────────────────────────────────────────┘

Calibration Triggers:

• Eval metrics below threshold
• New failure pattern identified
• User feedback trend (negative)
• Model update available
• New use case discovered

AI Metrics Hierarchy

LAGGING
├── User retention (AI users vs. non-AI users)
├── Task completion rate (with AI assist)
└── Revenue from AI features

CORE
├── User acceptance rate
├── Override rate
├── Time-to-completion (with AI)
└── User-reported satisfaction

LEADING
├── Eval metrics (accuracy, hallucination, etc.)
├── Interaction volume
├── Feature discovery rate
└── Feedback sentiment

GUARDRAILS
├── Harmful output rate
├── Latency P95
├── Error rate
└── Cost per interaction

AI-Specific Anti-Patterns

Templates

This skill includes templates in the templates/ directory:

• agency-assessment.md — Determine appropriate agency level
• eval-strategy.md — Design eval suite for AI feature
• calibration-plan.md — Set up continuous calibration

Using This Skill with Claude

Ask Claude to:

1. Assess agency level: "What agency level should [AI feature] have?"
2. Design agency progression: "Create a graduation path from assist to autonomous for [feature]"
3. Identify failure modes: "What could go wrong with [AI feature]? How do we mitigate?"
4. Design eval strategy: "Design an eval suite for [AI feature]"
5. Plan calibration: "Create a calibration plan for [AI feature]"
6. Adapt discovery: "What AI-specific questions should I ask in discovery for [use case]?"
7. Design confidence building: "How should [AI feature] show its reasoning?"
8. Plan AI rollout: "Create a staged rollout plan for [AI feature]"
9. Set AI metrics: "What metrics should we track for [AI feature]?"
10. Review AI brief: "Critique this solution brief for AI considerations"

Connection to Other Skills

Quick Reference: AI Product Checklist

Before shipping AI features:

• Agency level defined — Clear level for this feature
• Graduation criteria set — How we'll earn higher autonomy
• Failure modes mapped — Know what can go wrong
• Evals in place — Automated quality checks
• Human evals scheduled — Subjective quality review
• Calibration loop running — Continuous improvement process
• Confidence mechanisms built — Users can verify AI work
• Guardrails active — Prevent harmful outputs
• Rollout staged — More cautious than traditional features
• Override tracking — Learning from user corrections

Sources & Influences

• Aishwarya Goel & Kiriti Gavini — CCCD Loop, Agency-Control Trade-off
• Anthropic — Constitutional AI, RLHF approaches
• OpenAI — Eval best practices
• Google DeepMind — AI safety frameworks

Part of the Modern Product Operating Model by Yannick Maurice