Skill Evolver

Analyze skill execution traces to discover issues, identify improvement opportunities, and apply fixes to skill files.

Trace Format

Traces are JSON with this structure:

{ "id": "uuid", "request": "user's original request", "skills_used": ["skill-name"], "success": true/false, "total_turns": 2, "total_input_tokens": 5000, "total_output_tokens": 200, "duration_ms": 7000, "steps": [ {"role": "assistant", "content": "...", "tool_name": null}, {"role": "tool", "tool_name": "...", "tool_input": {}, "tool_result": "..."} ], "llm_calls": [ {"turn": 1, "stop_reason": "tool_use", "input_tokens": 2500, "output_tokens": 50} ] }

Workflow

This skill can receive two types of input (at least one required):

• Traces: Execution trace data from real skill runs — provides data-driven problem discovery

• Feedback: User-written improvement suggestions — provides directed guidance for changes

When both are provided, combine insights: use traces to validate/discover issues and feedback to prioritize and guide fixes.

Step 1: Analyze Inputs

If traces are provided, run the analysis script:

scripts/analyze_traces.py <traces.json> [--skill <name>] [--format json|text]

Output includes:

• Success rate

• Average turns, duration, tokens

• Common issues and warnings

• Recommendations

If feedback is provided, identify the user's improvement goals and map them to actionable changes.

If both are provided, cross-reference: does the feedback align with trace-discovered issues? Use feedback to prioritize which trace-identified problems to fix first.

Step 2: Extract Issue Details

For failed or problematic traces, extract full context:

scripts/extract_issue_context.py <traces.json> --failed scripts/extract_issue_context.py <traces.json> --trace-id <id> --show-llm scripts/extract_issue_context.py <traces.json> --high-turns

Skip this step if only feedback was provided (no traces).

Step 3: Identify Root Causes

Map issues to skill components using references/issue-patterns.md:

Issue Type Likely Fix Location

execution_failure scripts/, error handling

high_turn_count SKILL.md clarity, add examples

tool_errors scripts/, input validation

high_token_usage SKILL.md verbosity, progressive disclosure

repeated_tool_calls SKILL.md decision trees

For feedback-only input, map the user's suggestions directly to the appropriate skill components.

Step 4: Apply Fixes

Read the target skill and apply changes based on analysis:

• For script errors: Fix scripts, add validation, improve error messages

• For efficiency issues: Add examples, decision trees, clearer instructions

• For token issues: Reduce SKILL.md, move content to references/

• For trigger issues: Update frontmatter description

• For feedback-guided changes: Apply the user's specific suggestions

Scope constraints — strictly follow:

• Only modify the target skill's existing files (SKILL.md, scripts/, references/)

• Do NOT create new reference files, templates, or guides

• Do NOT search the web for domain-specific content

• Do NOT generate CHANGELOG, improvement reports, or other extra deliverables

• The evolved skill files themselves are the sole deliverable

Quick Reference

Issue Severity Levels

• high: Failures, max_tokens, tool errors → Fix immediately

• medium: High turns, high tokens, retries → Optimize

• low: Long duration → Consider optimization

Key Metrics Thresholds

Metric Warning Action

success_rate <90% Review failures

avg_turns

4 Simplify workflow

avg_tokens

30000 Reduce context

duration_ms

60000 Optimize scripts

Common Fixes

Low success rate:

• Add error handling in scripts

• Add input validation

• Clarify ambiguous instructions

High turn count:

• Add decision tree

• Provide more examples

• Use scripts for multi-step operations

High token usage:

• Reduce SKILL.md lines (<500)

• Move details to references/

• Remove redundant examples