Skill Development

Guide for creating and refining skills in the skills/ directory of this repository. Skills are packaged instructions that teach AI agents how to work with Phoenix features.

Directory Structure

skills/ {skill-name}/ # kebab-case, prefixed with "phoenix-" for Phoenix features SKILL.md # Required: skill definition and index README.md # Optional: human-oriented overview rules/ # Optional: detailed rule files {prefix}-{topic}.md {prefix}-{topic}-{lang}.md

Existing Skills

Skill Purpose Has Rules

phoenix-tracing

OpenInference instrumentation and span types Yes (31 files)

phoenix-evals

Evaluator development and validation Yes (33 files)

phoenix-cli

CLI debugging and analysis No (single SKILL.md)

Creating a New Skill

1. Plan the Scope

Determine whether the skill needs rule files:

• Single SKILL.md (like phoenix-cli ): Self-contained topics, command references, single-workflow tools

• SKILL.md + rules/ (like phoenix-tracing , phoenix-evals ): Multi-faceted topics with language-specific guides, multiple workflows, or extensive reference material

2. Create SKILL.md

Every skill requires a SKILL.md with YAML frontmatter:

name: {skill-name} description: {What it does. When to use it. Include trigger phrases.} license: Apache-2.0 metadata: author: oss@arize.com version: "1.0.0" languages: Python, TypeScript # omit if language-agnostic

Frontmatter rules:

• name : kebab-case, match directory name

• description : Third person, specific, includes trigger terms

• license : Always Apache-2.0

• metadata.author : Use oss@arize.com for Phoenix skills

• metadata.version : Semver string (e.g., "1.0.0" )

• metadata.languages : Only include if skill has language-specific content

• metadata.internal : Set to true for skills not intended for public listing (e.g., skills in .agents/skills/ ). Omit for public skills in skills/ .

3. Structure the SKILL.md Body

SKILL.md serves as the index and entry point. The agent reads this first and navigates to rule files as needed.

Required sections:

Section Purpose

Title (# Name ) Skill name with brief description

Quick Reference Table mapping tasks to rule files

Rule Categories Table of prefix patterns

Optional sections (include as relevant):

Section Purpose

When to Apply Trigger scenarios

Workflows Step-by-step paths through rules

Key Principles Core decision-making guidance

Common Attributes Reference tables

How to Use Navigation guidance

References External docs and API links

4. Organize Rule Files

Naming Conventions

Pattern: {prefix}-{topic}[-{language}].md

Prefix categories (reuse these across skills):

Prefix Purpose Example

setup-*

Installation, configuration setup-python.md

fundamentals-*

Core concepts, reference fundamentals-overview.md

instrumentation-*

Auto/manual setup instrumentation-auto-python.md

span-*

Span type specifications span-llm.md

evaluators-*

Evaluator types, patterns evaluators-code-python.md

experiments-*

Datasets, running experiments experiments-running-typescript.md

production-*

Deployment, monitoring production-python.md

annotations-*

Feedback, scoring annotations-overview.md

validation-*

Calibration, testing validation-calibration-python.md

Language suffixes:

• -python.md — Python-specific content

• -typescript.md — TypeScript-specific content

• No suffix — Language-agnostic or overview (e.g., span-llm.md , evaluators-overview.md )

Overview files: Use -overview.md suffix for conceptual introductions (e.g., fundamentals-overview.md , production-overview.md ).

Flat Structure

All rule files go directly in rules/ — no subdirectories. Use prefixes for organization, not folders.

rules/ setup-python.md # Good: flat with prefix setup-typescript.md span-llm.md # Good: no language suffix (language-agnostic) evaluators-code-python.md # Good: prefix-topic-language

5. Write Rule Files

Standard Structure

Title

Brief description of what this rule covers.

Metadata (optional)

Quick Start / Basic Pattern

Minimal working example.

Detailed Sections

Expanded content with examples.

See Also

• related-rule.md — Brief description
• External docs

Code Examples

• Always use fenced blocks with language tags: python , typescript , bash , json

• Show working, copy-pasteable examples

• Include both minimal and production-ready patterns when relevant

Cross-References

• Reference other rule files by filename: setup-python.md , span-llm.md

• Reference external docs with full URLs

• Keep references one level deep (SKILL.md → rule file, not rule → rule → rule)

Refining Existing Skills

Adding a Rule File

• Choose the correct prefix from the table above (or establish a new one)

• Follow the naming convention: {prefix}-{topic}[-{language}].md

• Add an entry in SKILL.md under Quick Reference and Rule Categories

• Cross-reference related rules in the new file's "See Also" section

Updating SKILL.md

When adding content to SKILL.md, keep it as an index — move detailed content to rule files. SKILL.md should stay under 500 lines.

Improving Consistency

Common issues to fix when refining:

Issue Fix

Missing metadata table Add Priority, Setup Time if applicable

Inconsistent headings Standardize to ## Quick Start , ## See Also

Inline code dumps Extract to rule files, link from SKILL.md

Missing cross-references Add ## See Also with related rules

Vague descriptions Make frontmatter description specific with trigger terms

Quality Checklist

SKILL.md

• Frontmatter has all required fields (name , description , license , metadata )

• metadata.internal: true is set for skills in .agents/skills/ ; omitted for skills in skills/

• Description is third person, specific, includes trigger terms

• Quick Reference table maps tasks to rule files

• Rule Categories table lists all prefixes used

• Under 500 lines

• No detailed content that belongs in rule files

Rule Files

• Follow {prefix}-{topic}[-{language}].md naming

• Have a clear # Title and brief description

• Include working code examples with language tags

• Cross-reference related rules in ## See Also

• Use consistent heading structure

Overall Skill

• Flat rules/ directory (no subdirectories)

• Consistent terminology throughout all files

• Language-specific content split into -python.md / -typescript.md files

• Language-agnostic content has no language suffix

• No time-sensitive information (no dates, version caveats)

Anti-Patterns

Putting too much in SKILL.md. SKILL.md is an index. If a section exceeds ~30 lines of content, extract it to a rule file.

Deep reference chains. Rule files should not require reading other rule files to be useful. Each rule should be self-contained enough to act on independently.

Generic rule names. Use evaluators-code-python.md not code-eval.md . Prefixes enable discovery via ls rules/{prefix}-* .

Mixing languages in one file. Split into -python.md and -typescript.md unless the content is truly language-agnostic.

Forgetting SKILL.md updates. Every new rule file must be reflected in the SKILL.md Quick Reference and Rule Categories tables.

Workflow Summary

1. Plan scope → single SKILL.md or SKILL.md + rules/?
2. Create directory: skills/{skill-name}/
3. Write SKILL.md with frontmatter and index
4. Create rules/ directory (if needed)
5. Write rule files following naming conventions
6. Update SKILL.md index to reference all rules
7. Run quality checklist