Eval Harness Updater

Refresh eval harnesses to keep live + fallback modes actionable under unstable environments.

Focus Areas

• Prompt and parser drift

• Timeout/partial-stream handling

• SLO and regression gates

• Dual-run fallback consistency

Workflow

• Resolve harness path.

• Research test/eval best practices (Exa + arXiv — see Research Gate below).

• Add RED regressions for parsing and timeout edge cases.

• Patch minimal harness logic.

• Validate eval outputs and CI gates.

• Resolve companion artifact gaps (see Cross-Reference table below).

Research Gate (Exa + arXiv — BOTH MANDATORY)

Before proposing harness changes, gather current best practices:

• Use Exa for implementation and ecosystem patterns:

• mcp__Exa__web_search_exa({ query: 'LLM eval harness 2025 best practices' })

• mcp__Exa__get_code_context_exa({ query: 'eval harness parser reliability timeout handling' })

• Search arXiv for academic research on evaluation methodology (mandatory):

• Via Exa: mcp__Exa__web_search_exa({ query: 'site:arxiv.org LLM evaluation harness 2024 2025' })

• Direct API: WebFetch({ url: 'https://arxiv.org/search/?query=LLM+evaluation+harness&searchtype=all&start=0' })

• Record decisions, constraints, and non-goals in memory learnings.

arXiv is mandatory (not fallback) when topic involves: LLM evaluation, agent evaluation, SLO gates, regression testing methodology, or parser reliability.

Cross-Reference: Creator Ecosystem

This skill is part of the Creator Ecosystem. When research uncovers gaps, trigger the appropriate companion creator:

Gap Discovered Required Artifact Creator to Invoke When

Domain knowledge needs a reusable skill skill Skill({ skill: 'skill-creator' })

Gap is a full skill domain

Existing skill has incomplete coverage skill update Skill({ skill: 'skill-updater' })

Close skill exists but incomplete

Capability needs a dedicated agent agent Skill({ skill: 'agent-creator' })

Agent to own the capability

Existing agent needs capability update agent update Skill({ skill: 'agent-updater' })

Close agent exists but incomplete

Domain needs code/project scaffolding template Skill({ skill: 'template-creator' })

Reusable code patterns needed

Behavior needs pre/post execution guards hook Skill({ skill: 'hook-creator' })

Enforcement behavior required

Process needs multi-phase orchestration workflow Skill({ skill: 'workflow-creator' })

Multi-step coordination needed

Artifact needs structured I/O validation schema Skill({ skill: 'schema-creator' })

JSON schema for artifact I/O

User interaction needs a slash command command Skill({ skill: 'command-creator' })

User-facing shortcut needed

Repeated logic needs a reusable CLI tool tool Skill({ skill: 'tool-creator' })

CLI utility needed

Narrow/single-artifact capability only inline Document within this artifact only Too specific to generalize

Iron Laws

• ALWAYS run the Exa + arXiv research gate before updating any eval harness — updating without current external knowledge produces stale evaluation criteria.

• NEVER remove existing evaluation criteria without replacing them with equivalent or better ones — reducing test coverage in an eval harness is a regression.

• ALWAYS cross-reference the creator ecosystem for gaps before declaring the harness complete — missing companion artifacts (skills, agents, schemas) leave the harness unable to test new capabilities.

• NEVER update eval harness in isolation from the skill/agent it evaluates — harness and artifact must stay synchronized or the harness tests the wrong behavior.

• ALWAYS preserve backward compatibility in eval scoring — changing scoring semantics without migrating historical baselines makes trend analysis impossible.

Anti-Patterns

Anti-Pattern Why It Fails Correct Approach

Updating eval harness without research gate Criteria based on outdated knowledge; misses recent evaluation methodology advances Always run Exa + arXiv research before updating any eval criteria

Removing test cases to simplify the harness Silently reduces coverage; regressions pass undetected Only remove test cases when the behavior they tested has been deliberately removed

Harness and artifact in separate PRs Harness tests wrong behavior the moment artifact changes; immediate test drift Always update harness and artifact in the same commit

Changing scoring scale mid-project Historical baselines become incomparable; trend analysis breaks Define scoring scale once; create a migration if it must change

Declaring harness complete without companion check Missing skills or schemas leave evaluation gaps Always run companion artifact check before marking harness update complete

Memory Protocol (MANDATORY)

Before starting: Read .claude/context/memory/learnings.md

After completing:

• New evaluation pattern → .claude/context/memory/learnings.md

• Evaluation gap found → .claude/context/memory/issues.md

• Scoring decision made → .claude/context/memory/decisions.md

ASSUME INTERRUPTION: If it's not in memory, it didn't happen.