LLM Evaluation

Evaluate and validate LLM outputs for quality assurance using RAGAS and LLM-as-judge patterns.

Quick Reference

LLM-as-Judge Pattern

async def evaluate_quality(input_text: str, output_text: str, dimension: str) -> float: response = await llm.chat([{ "role": "user", "content": f"""Evaluate for {dimension}. Score 1-10. Input: {input_text[:500]} Output: {output_text[:1000]} Respond with just the number.""" }]) return int(response.content.strip()) / 10

Quality Gate

QUALITY_THRESHOLD = 0.7

async def quality_gate(state: dict) -> dict: scores = await full_quality_assessment(state["input"], state["output"]) passed = scores["average"] >= QUALITY_THRESHOLD return {**state, "quality_passed": passed}

Hallucination Detection

async def detect_hallucination(context: str, output: str) -> dict: # Check if output contains claims not in context return {"has_hallucinations": bool, "unsupported_claims": []}

RAGAS Metrics ()

Metric Use Case Threshold

Faithfulness RAG grounding ≥ 0.8

Answer Relevancy Q&A systems ≥ 0.7

Context Precision Retrieval quality ≥ 0.7

Context Recall Retrieval completeness ≥ 0.7

Anti-Patterns (FORBIDDEN)

❌ NEVER use same model as judge and evaluated

output = await gpt4.complete(prompt) score = await gpt4.evaluate(output) # Same model!

❌ NEVER use single dimension

if relevance_score > 0.7: # Only checking one thing return "pass"

❌ NEVER set threshold too high

THRESHOLD = 0.95 # Blocks most content

✅ ALWAYS use different judge model

score = await gpt4_mini.evaluate(claude_output)

✅ ALWAYS use multiple dimensions

scores = await evaluate_all_dimensions(output) if scores["average"] > 0.7: return "pass"

Key Decisions

Decision Recommendation

Judge model GPT-5.2-mini or Claude Haiku 4.5

Threshold 0.7 for production, 0.6 for drafts

Dimensions 3-5 most relevant to use case

Sample size 50+ for reliable metrics

Detailed Documentation

Resource Description

references/evaluation-metrics.md RAGAS & LLM-as-judge metrics

examples/evaluation-patterns.md Complete evaluation examples

checklists/evaluation-checklist.md Setup and review checklists

scripts/evaluator-template.py Starter evaluation template

Related Skills

• quality-gates

• Workflow quality control

• langfuse-observability

• Tracking evaluation scores

• agent-loops

• Self-correcting with evaluation

Capability Details

llm-as-judge

Keywords: LLM judge, judge model, evaluation model, grader LLM Solves:

• Use LLM to evaluate other LLM outputs

• Implement judge prompts for quality

• Configure evaluation criteria

ragas-metrics

Keywords: RAGAS, faithfulness, answer relevancy, context precision Solves:

• Evaluate RAG with RAGAS metrics

• Measure faithfulness and relevancy

• Assess context precision and recall

hallucination-detection

Keywords: hallucination, factuality, grounded, verify facts Solves:

• Detect hallucinations in LLM output

• Verify factual accuracy

• Implement grounding checks

quality-gates

Keywords: quality gate, threshold, pass/fail, evaluation gate Solves:

• Implement quality thresholds

• Block low-quality outputs

• Configure multi-metric gates

batch-evaluation

Keywords: batch eval, dataset evaluation, bulk scoring, eval suite Solves:

• Evaluate over golden datasets

• Run batch evaluation pipelines

• Generate evaluation reports

pairwise-comparison

Keywords: pairwise, A/B comparison, side-by-side, preference Solves:

• Compare two model outputs

• Implement preference ranking

• Run A/B evaluations