Headless Adapters

Purpose

Schema-driven adapter for headless CLI agents. No code required - just define a JSON schema describing how to interact with the CLI.

Use Case Tool

Wrap headless CLI agent headless command

Create new schemas Schema Creation Guide

Quick Start

• Create a schema for your CLI agent using the Schema Creation Guide

• Run the adapter: bunx @plaited/agent-eval-harness headless --schema ./my-agent-headless.json

Any CLI agent that outputs JSON can be wrapped — no agent-specific code required.

CLI Commands

headless

Schema-driven adapter for ANY headless CLI agent.

bunx @plaited/agent-eval-harness headless --schema <path>

Options:

Flag Description Required

-s, --schema

Path to adapter schema (JSON) Yes

Schema Format:

{ "version": 1, "name": "my-agent", "command": ["my-agent-cli"], "sessionMode": "stream", "prompt": { "flag": "-p" }, "output": { "flag": "--output-format", "value": "stream-json" }, "autoApprove": ["--allow-all"], "outputEvents": [ { "match": { "path": "$.type", "value": "message" }, "emitAs": "message", "extract": { "content": "$.text" } }, { "match": { "path": "$.type", "value": "tool_use" }, "emitAs": "tool_call", "extract": { "title": "$.name", "status": "'pending'", "input": "$.input" } }, { "match": { "path": "$.type", "value": "tool_result" }, "emitAs": "tool_call", "extract": { "title": "$.name", "status": "'completed'", "output": "$.content" } } ], "result": { "matchPath": "$.type", "matchValue": "result", "contentPath": "$.content" } }

Session Modes:

Mode Description Use When

stream

Keep process alive, multi-turn via stdin CLI supports session resume

iterative

New process per turn, accumulate history CLI is stateless

Creating a Schema

• Run the CLI's --help to identify prompt, output format, and auto-approve flags

• Capture sample JSON output from the CLI

• Map JSONPath patterns to output events (including input /output for tool calls)

• Create the schema file

• Test with headless command

See Schema Creation Guide for the complete workflow.

Security Considerations

Trust Boundary: CLI Output is Untrusted

The headless adapter parses JSON output from CLI agents. This output may contain content from external sources (web searches, file reads, API responses) that flows into trajectory data:

CLI Agent → JSON stdout → JSONPath extraction → ParsedUpdate → TrajectoryStep

Trajectory fields — especially tool_call.input and tool_call.output — should be treated as untrusted content by downstream consumers (graders, LLM-as-judge, analysis scripts). Do not:

• Execute trajectory content as code

• Use trajectory content in unsanitized shell commands

• Pass trajectory content to LLMs without injection-aware prompting

autoApprove Flags

The autoApprove field bypasses the CLI agent's safety confirmation prompts. Use the least permissive flags your evaluation requires:

Risk Level Example When to Use

High ["--dangerously-skip-permissions"]

Only in isolated containers (Docker, CI)

Medium ["--allowedTools", "Read,Write,Glob"]

Scoped to specific tools

Low ["--auto-approve", "read-only"]

Read-only evaluations

Never run high-risk autoApprove flags outside isolated environments. Use --workspace-dir or Docker for evaluations that modify the filesystem.

Troubleshooting

Common Issues

Issue Likely Cause Solution

Tool calls not captured JSONPath not iterating arrays Use [*] wildcard syntax - see guide

Tool input/output missing Extract config missing input /output fields Add input /output paths - see guide

"unexpected argument" error Stdin mode misconfigured Use stdin: true

• see guide

401 Authentication errors API key not properly configured Set the correct API key environment variable for your agent

Timeout on prompt JSONPath not matching Capture raw CLI output, verify paths - see guide

Empty responses Content extraction failing Check extract paths - see guide

Complete troubleshooting documentation: Troubleshooting Guide

External Resources

• AgentSkills Spec: agentskills.io

Related

• agent-eval-harness skill - Running evaluations against adapters