To build great AI products, you must transition from subjective "vibe checks" to systematic measurement. This process identifies exactly where an LLM is failing and creates a feedback loop for continuous improvement.

Phase 1: Open Coding (The "Benevolent Dictator" Phase)

Before automating, you must manually ground yourself in the data. Appoint one "Benevolent Dictator"—typically the Product Manager or domain expert—to define "good" taste.

• Sample the Data: Extract 50–100 "traces" (logs of full LLM interactions) from your observability tool (e.g., Braintrust, LangSmith, Phoenix).

• Note the Upstream Error: Read each trace. If something is wrong, write a brief, informal note (an "Open Code") describing the first thing that went wrong.

• Rule: Don't overthink it. Use specific language (e.g., "hallucinated virtual tour," "didn't confirm call transfer") rather than just "bad."

• Stop at Saturation: Continue until you stop learning new ways the system fails (Theoretical Saturation).

Phase 2: Axial Coding (Categorization)

Synthesize your mess of notes into actionable categories using an LLM.

• Export Notes: Put your open codes into a CSV or spreadsheet.

• Synthesize Failure Modes: Use an LLM (Claude or ChatGPT) to group your notes into 5–7 "Axial Codes" (failure categories).

• Prompt Pattern: "Analyze these manual notes from AI traces and group them into actionable failure categories (Axial Codes). Each category should represent a specific product problem."

• Map Back: Use a spreadsheet formula or LLM to categorize every trace into one of these buckets.

• Prioritize: Create a pivot table to count the frequency of each category. Focus your engineering efforts on the highest-frequency or highest-risk buckets.

Phase 3: Build the "LLM as Judge"

For complex, subjective failures (like "human handoff quality"), create an automated evaluator.

• Write the Judge Prompt: Create a separate prompt for an LLM whose only job is to evaluate one specific failure mode.

• Enforce Binary Scoring: Require the judge to output only True or False.

• Note: Avoid 1–5 or 1–10 scales. They result in "weasel" metrics (e.g., a score of 3.7) that provide no clear direction for improvement.

• Define Rules: Include specific criteria from your "Benevolent Dictator" notes.

• Example: "Output True if the user explicitly asked for a human and the assistant responded with a tool call without acknowledging the request."

Phase 4: Alignment & Validation

Never ship an eval until you know the judge matches human judgment.

• Create an Agreement Matrix: Compare the Judge's True/False labels against your manual labels from Phase 1.

• Review Mismatches: Specifically look at:

• False Positives: Judge said error, Human said no error.

• False Negatives: Human said error, Judge said no error.

• Iterate: Refine the Judge's prompt until it aligns with the "Benevolent Dictator" at least 80–90% of the time.

Examples

Example 1: Real Estate AI Assistant

• Context: AI is supposed to book apartment tours.

• Open Code: "AI told the user a virtual tour was available when the property only offers in-person tours."

• Axial Code: "Capability Misrepresentation."

• Judge Logic: "Check the 'Property Context' tool output. If 'virtual_tour' is False, but the LLM response contains 'virtual tour,' output True (Error)."

Example 2: Customer Support Handoff

• Context: AI should hand off to a human for sensitive issues.

• Open Code: "User said they were frustrated with a leak, AI just gave a generic maintenance link."

• Axial Code: "Handoff Protocol Violation."

• Judge Logic: "Search for sentiment indicating frustration or emergency. If found, did the AI offer a human transfer? If no, output True (Error)."

Common Pitfalls

• Likert Scales: Using 1–5 scales makes it impossible to know if a change in score is meaningful. Use binary True/False.

• Automating Too Early: Do not let an LLM do the initial "Open Coding." It lacks the product context to know what "janky" looks like for your specific business.

• Committee Judging: Don't use a committee to define "good." Appoint one person with the best domain taste to be the final arbiter (The Benevolent Dictator).

• Chasing Generic Metrics: Don't rely on generic evals like "hallucination score" or "cosine similarity." They rarely correlate with product-specific success.