Analyzing SPAA Files

SPAA (Stack Profile for Agentic Analysis) is an NDJSON file format for representing sampled performance stack traces. Each line is a self-contained JSON object with a type field.

Before analyzing an SPAA file, read the full specification at references/SPEC.md.

Quick Start

Examine the header

The first line is always the header. It tells you what profiler generated the data and how to interpret metrics:

head -1 profile.spaa | jq .

Key header fields:

source_tool : The profiler that generated this data (e.g., "perf", "dtrace")
frame_order : Either "leaf_to_root" or "root_to_leaf"
determines how to read stack frames
events[].sampling.primary_metric : The authoritative metric for weighting (e.g., "period" for perf, "samples" for DTrace)

Understand the record types

Count records by type

grep -o '"type":"[^"]*"' profile.spaa | sort | uniq -c | sort -rn

Common types:

header
File metadata (exactly one, always first)
dso
Shared libraries and binaries
frame
Individual stack frame definitions
thread
Thread/process info
stack
Aggregated call stacks with weights (the main data)
sample
Individual sample events (optional, for temporal analysis)

Find performance hotspots

Extract the heaviest stacks by the primary metric:

For perf data (uses "period" metric)

grep '"type":"stack"' profile.spaa |
jq -s 'sort_by(-.weights[] | select(.metric=="period") | .value) | .[0:10]'

For DTrace data (uses "samples" metric)

grep '"type":"stack"' profile.spaa |
jq -s 'sort_by(-.weights[] | select(.metric=="samples") | .value) | .[0:10]'

Find hot functions (exclusive time)

Exclusive time shows where the CPU actually spent time, not just functions on the call path:

grep '"type":"stack"' profile.spaa |
jq -s '[.[] | select(.exclusive) | {frame: .exclusive.frame, value: (.exclusive.weights[] | select(.metric=="period") | .value)}] | group_by(.frame) | map({frame: .[0].frame, total: (map(.value) | add)}) | sort_by(-.total) | .[0:20]'

Then look up the frame IDs to get function names:

Get frame details for a specific ID

grep '"type":"frame"' profile.spaa | jq 'select(.id == 101)'

Analyzing Memory Profiles

SPAA also supports heap/allocation profilers. Memory events use different metrics:

Find top allocation sites by bytes allocated

Find potential memory leaks (high live_bytes)

Key memory metrics:

alloc_bytes / alloc_count
Total allocations
live_bytes / live_count
Currently unreleased memory (potential leaks)
peak_bytes
High-water mark

Reconstructing Call Stacks

Stack records contain frame IDs. To see the actual function names:

Extract a stack and resolve its frames

STACK_FRAMES=$(grep '"type":"stack"' profile.spaa | head -1 | jq -r '.frames | @csv')

Build a frame lookup table, then query it

grep '"type":"frame"' profile.spaa | jq -s 'INDEX(.id)' > /tmp/frames.json echo $STACK_FRAMES | tr ',' '\n' | while read fid; do jq --arg id "$fid" '.[$id] | "(.func) ((.srcline // "unknown"))"' /tmp/frames.json done

Common Analysis Patterns

Filter by thread/process

grep '"type":"stack"' profile.spaa | jq 'select(.context.tid == 4511)'

Filter by event type

grep '"type":"stack"' profile.spaa | jq 'select(.context.event == "cycles")'

Find kernel vs userspace time

Kernel stacks

grep '"type":"stack"' profile.spaa | jq 'select(.stack_type == "kernel")'

Or check frame kinds

grep '"type":"frame"' profile.spaa | jq 'select(.kind == "kernel")' | head -20

Temporal analysis (if sample records exist)

Check if raw samples are included

grep -c '"type":"sample"' profile.spaa

Plot sample distribution over time

grep '"type":"sample"' profile.spaa | jq -s 'group_by(.timestamp | floor) | map({time: .[0].timestamp | floor, count: length})'

Tips for Performance Analysis

Start with the header - Understand the profiler, sampling mode, and time range
Check the primary metric - Use period for perf, samples for DTrace
Look at exclusive time first - This shows actual hotspots, not just callers
Cross-reference frame IDs - Build a lookup table for readable output
Filter by context - Narrow down by thread, CPU, or event type
For memory issues - Focus on live_bytes to find leaks, alloc_bytes for churn

spaa

Safety Notice

Copy this and send it to your AI assistant to learn