Evaluation & Monitoring

Evaluation determines how well an agent performs (correctness, helpfulness, safety), usually on a test dataset. Monitoring determines how the system is running (latency, errors, cost) in a live environment. Both are essential for the lifecycle management of AI systems.

When to Use

• CI/CD: Rejecting code changes if they drop accuracy below a threshold.

• A/B Testing: Comparing Prompt A vs. Prompt B to see which users prefer.

• Cost Auditing: Understanding which agents or tools are driving up the bill.

• Drift Detection: Noticing if the model starts hallucinating more often on new data.

Use Cases

• LLM-as-a-Judge: Using GPT-4 to grade the answers of a smaller model.

• Latency Tracking: Measuring the time-to-first-token (TTFT) and total generation time.

• Topic Clustering: Analyzing user queries to see what topics are trending or failing.

Implementation Pattern

def evaluate_agent(agent, test_set): score = 0 total = len(test_set)

for case in test_set:
    # Run agent
    prediction = agent.run(case.input)
    
    # Evaluate vs Golden Answer
    # Simple exact match or fuzzy match
    if is_correct(prediction, case.expected):
        score += 1
    else:
        # Semantic Evaluation using an LLM Judge
        judge_score = llm_judge.evaluate(
            prediction, 
            case.expected
        )
        score += judge_score
        
return score / total