Deep Research

Iterative multi-round research across Web, Reddit, X/Twitter, GitHub, Hacker News, Substack, Financial Media, LinkedIn, and more with structured output frameworks (comparison, landscape, deep-dive, decision).

Architecture

Source Tool Cost

Web Claude Code WebSearch

• xAI web_search

Free + $0.005/call

Reddit Claude Code WebSearch (site:reddit.com) + xAI web_search

Free + $0.005/call

X/Twitter xAI x_search only $0.005/call

GitHub Claude Code WebSearch (site:github.com) + xAI web_search

Free + $0.005/call

Hacker News Claude Code WebSearch (site:news.ycombinator.com) + xAI web_search

Free + $0.005/call

Substack Claude Code WebSearch (site:substack.com) + xAI web_search

Free + $0.005/call

Financial Media Claude Code WebSearch (site-specific) + xAI web_search

Free + $0.005/call

Wall Street Oasis Claude Code WebSearch (site:wallstreetoasis.com) Free

LinkedIn Claude Code WebSearch (site:linkedin.com) + xAI web_search

Free + $0.005/call

Crunchbase Claude Code WebSearch (site:crunchbase.com) Free

YouTube Claude Code WebSearch (site:youtube.com) Free

Tech Blogs Claude Code WebSearch (site-specific) Free

Results from multiple sources are merged and deduplicated for comprehensive coverage.

Prerequisites

Required Tools

Tool Purpose Install

uv Python package manager (handles dependencies) curl -LsSf https://astral.sh/uv/install.sh | sh

Optional Tools

Tool Purpose Install

scrapling Headless browser fallback for sites that block WebFetch (403, captcha, empty responses) uv tool install 'scrapling[all]'

API Keys

Service Purpose Required Get Key

xAI X/Twitter search + supplemental web/GitHub/HN search Recommended https://console.x.ai

Note: The skill works without xAI key (web-only mode via Claude Code), but X/Twitter data and broader coverage require xAI.

Keychain Setup (One-Time, for xAI)

1. Create a dedicated keychain

security create-keychain -p 'YourPassword' ~/Library/Keychains/claude-keys.keychain-db

2. Add keychain to search list

security list-keychains -s ~/Library/Keychains/claude-keys.keychain-db ~/Library/Keychains/login.keychain-db /Library/Keychains/System.keychain

3. Store your xAI API key

echo -n "Enter xAI API key: " && read -s key && security add-generic-password -s "xai-api" -a "$USER" -w "$key" ~/Library/Keychains/claude-keys.keychain-db && unset key && echo

Before using: security unlock-keychain ~/Library/Keychains/claude-keys.keychain-db

Workflow Overview

Step Action Purpose

0 Detect xAI key Determine Full vs Web-Only mode

1 Parse query Extract TOPIC, FRAMEWORK, DEPTH

2 Round 1: Broad search Discover entities, themes, initial findings

3 Gap analysis Identify missing perspectives, unverified claims

4 Round 2: Targeted follow-up Fill gaps, verify claims, deepen coverage

5 Round 3: Verification (deep only) Primary source verification

6 Synthesis Structure findings into framework template

7 Expert mode Answer follow-ups from cached results

Step 0: Detect xAI Key

MANDATORY — run before every research session.

security find-generic-password -s "xai-api" -w ~/Library/Keychains/claude-keys.keychain-db 2>/dev/null && echo "XAI_AVAILABLE=true" || echo "XAI_AVAILABLE=false"

• XAI_AVAILABLE=true: Use Full mode — Claude WebSearch AND xAI scripts in parallel.

• XAI_AVAILABLE=false: Use Web-Only mode — Claude WebSearch only. Append note suggesting xAI setup.

This step is NOT optional. Always check before starting research.

Step 1: Parse Query

Extract from user input:

1a. TOPIC

The subject being researched. Strip framework indicators and depth modifiers.

1b. FRAMEWORK

Detect output framework from query patterns:

Framework Detection Patterns Example

COMPARISON "X vs Y", "compare X and Y", "X or Y", "which is better" "React vs Vue for enterprise apps"

LANDSCAPE "landscape", "ecosystem", "market", "what's out there", "overview of" "AI agent frameworks landscape"

DEEP_DIVE "deep dive", "how does X work", "explain", "tell me about", "what is" "Deep dive into WebAssembly"

DECISION "should I/we", "evaluate", "which should we use", "recommend" "Should we use Kafka or RabbitMQ?"

Explicit override: User can force a framework with [comparison] , [landscape] , [deep-dive] , or [decision] anywhere in query.

Default: If no framework detected, use DEEP_DIVE.

1c. DEPTH

Depth Trigger Rounds Target Sources

quick "quick", "brief", "overview" 1 10-15

default (none) 2 25-40

deep "deep", "comprehensive", "thorough" 3 60-90

Step 2: Round 1 — Broad Search

Query Generation

Generate 6-9 queries covering different angles of the TOPIC:

• Direct query: "{TOPIC}" — the topic as stated

• Temporal query: "{TOPIC} 2026" or "{TOPIC} latest"

• Reddit query: site:reddit.com "{TOPIC}"

• GitHub query: site:github.com "{TOPIC}"

• HN query: site:news.ycombinator.com "{TOPIC}"

• Framework-specific query:

• COMPARISON: "{Alt A} vs {Alt B}"

• LANDSCAPE: "{TOPIC} ecosystem" OR "{TOPIC} landscape"

• DEEP_DIVE: "how {TOPIC} works" OR "{TOPIC} explained"

• DECISION: "{TOPIC}" experience OR recommendation

• Substack query: site:substack.com "{TOPIC}"

• Financial media query: site:tradingview.com OR site:benzinga.com OR site:seekingalpha.com "{TOPIC}" (for finance/economics topics)

• LinkedIn query: site:linkedin.com "{TOPIC}" (when topic involves people or companies)

Parallel Execution

Run searches simultaneously:

Claude Code (free):

• WebSearch : direct query

• WebSearch : temporal query

• WebSearch : Reddit-targeted query

• WebSearch : GitHub-targeted query

• WebSearch : HN-targeted query

• WebSearch : Substack-targeted query

• WebSearch : Financial media-targeted query (for finance/economics topics)

• WebSearch : LinkedIn-targeted query (when topic involves people/companies)

• WebSearch : YouTube-targeted query

• WebSearch : WSO-targeted query (for finance topics)

• WebSearch : Crunchbase-targeted query (for company/startup topics)

xAI scripts (if available, run as background Bash tasks):

uv run scripts/xai_search.py web "{TOPIC}" --json & uv run scripts/xai_search.py reddit "{TOPIC}" --json & uv run scripts/xai_search.py x "{TOPIC}" --json & uv run scripts/xai_search.py github "{TOPIC}" --json & uv run scripts/xai_search.py hn "{TOPIC}" --json & uv run scripts/xai_search.py substack "{TOPIC}" --json & uv run scripts/xai_search.py finance "{TOPIC}" --json & uv run scripts/xai_search.py linkedin "{TOPIC}" --json &

Merge and Deduplicate

MERGED_WEB = dedupe(claude_web + xai_web) MERGED_REDDIT = dedupe(claude_reddit + xai_reddit) MERGED_GITHUB = dedupe(claude_github + xai_github) MERGED_HN = dedupe(claude_hn + xai_hn) MERGED_SUBSTACK = dedupe(claude_substack + xai_substack) MERGED_FINANCE = dedupe(claude_finance + xai_finance) MERGED_LINKEDIN = dedupe(claude_linkedin + xai_linkedin) X_RESULTS = xai_x_results (no Claude equivalent) WSO_RESULTS = claude_wso (WebSearch only) CRUNCHBASE_RESULTS = claude_crunchbase (WebSearch only) YOUTUBE_RESULTS = claude_youtube (WebSearch only)

Deduplication by URL, keeping the entry with more metadata.

Round 1 Internal Notes

Record (internally, NOT in output):

KEY_ENTITIES: [specific tools, companies, people discovered] THEMES: [recurring themes across sources] GAPS: [what's missing — feed into Step 3] CONTRADICTIONS: [conflicting claims] LEADS: [URLs worth deep-reading via WebFetch (or scrapling fallback) in Round 2]

Step 3: Gap Analysis

After Round 1, run gap analysis. See references/iterative-research.md for full checklist.

Gap Categories

Gap Check Action

Missing perspective Have developer, operator, and business views? Target missing perspective

Unverified claims Any claims from only 1 source? Seek corroboration

Shallow coverage Any entity mentioned but unexplained? Deep-search that entity

Stale data Key facts > 12 months old? Search for recent updates

Missing source type Missing Reddit / GitHub / HN / X / blogs? Target that platform

Missing financial/business context Missing Substack / financial media / LinkedIn / Crunchbase? Target that platform

Plan Round 2

Select top 4-6 gaps. Generate targeted queries for each. See references/search-patterns.md for multi-round refinement patterns.

Skip to Step 6 if depth is quick (single round only).

Step 4: Round 2 — Targeted Follow-Up

Query Rules

• Never repeat Round 1 queries

• Entity-specific queries — target names/tools discovered in Round 1

• Source-type specific — target platforms underrepresented in Round 1

• Framework-adapted — see targeting table in references/iterative-research.md

Execution

Same parallel pattern as Round 1, but with targeted queries.

Additionally, use WebFetch for high-value URLs discovered in Round 1:

• Official documentation pages

• Benchmark result pages

• Engineering blog posts

• Comparison articles with methodology

Maximum 4-6 WebFetch calls in Round 2.

Scrapling fallback: If WebFetch returns 403, empty content, a captcha page, or a blocked response, retry using the auto-escalation protocol from cli-web-scrape:

• scrapling extract get "URL" /tmp/scrapling-fallback.md → Read → validate content

• If content is thin (JS-only shell, no data, mostly nav links) → scrapling extract fetch "URL" /tmp/scrapling-fallback.md --network-idle --disable-resources → Read → validate

• If still blocked → scrapling extract stealthy-fetch "URL" /tmp/scrapling-fallback.md --solve-cloudflare

• All tiers fail → skip URL and note "scrapling blocked"

Confidence Update

After Round 2, re-assess all claims:

Before New Evidence After

[LOW] Second source found [MEDIUM]

[MEDIUM] Third source found [HIGH]

Any Contradicted Note conflict, present both sides

Skip to Step 6 if depth is default .

Step 5: Round 3 — Verification (Deep Only)

Round 3 is for verification only. No new discovery.

Budget

Maximum 6-10 WebFetch lookups targeting:

Target Purpose Max Calls

Primary sources for key claims Verify accuracy 3-4

Independent benchmark sites Validate performance claims 1-2

Both sides of contradictions Resolve conflicts 1-2

Official sites for versions/dates Confirm recency 1-2

Scrapling fallback: Same auto-escalation protocol as Round 2 — try HTTP tier, validate content, escalate to Dynamic/Stealthy if thin or blocked.

Rules

• Verify, don't discover — no new topic exploration

• Target highest-impact claims — those that would change the recommendation

• Check primary sources — go to the original, not summaries

• Update confidence — upgrade or downgrade based on findings

• Trust primary over secondary — if primary contradicts secondary, note it

Step 6: Synthesis

Framework Selection

Load references/output-frameworks.md and select the template matching the detected FRAMEWORK.

Filling the Template

• Header block — Framework type, topic, depth, source count, date

• Core content — Fill framework sections with research findings

• Confidence indicators — Mark each claim: [HIGH] , [MEDIUM] , or [LOW]

• Community perspective — Synthesize Reddit/X/HN/GitHub sentiment

• Statistics footer — Source counts and engagement metrics

• Sources section — Organized by platform with URLs and metrics

Engagement-Weighted Synthesis

Weight sources by signal strength. See references/iterative-research.md for full weighting table.

Signal Threshold Weight

Reddit upvotes 100+ High

X engagement 50+ likes High

GitHub stars 1000+ High

HN points 100+ High

Substack likes 50+ High

Multi-platform agreement 3+ sources High

Dual-engine match Claude + xAI High

Seeking Alpha comments 20+ Medium

WSO upvotes 10+ Medium

YouTube views 10K+ Medium

Recent (< 7 days) Any Medium

Single source only Any Low

Statistics Footer Format

Research Statistics ├─ Reddit: {n} threads │ {upvotes} upvotes ├─ X: {n} posts │ {likes} likes │ {reposts} reposts ├─ GitHub: {n} repos │ {stars} total stars ├─ HN: {n} threads │ {points} total points ├─ Substack: {n} articles │ {likes} likes ├─ Financial: {n} articles │ {sources} ├─ LinkedIn: {n} profiles/articles ├─ YouTube: {n} videos ├─ WSO: {n} threads ├─ Crunchbase: {n} profiles ├─ Web: {n} pages │ {domains} ├─ Merged: {n} from Claude + {n} from xAI └─ Top voices: r/{sub1} │ @{handle1} │ {blog1}

Web-Only Mode footer:

Research Statistics ├─ Reddit: {n} threads (via Claude WebSearch) ├─ GitHub: {n} repos (via Claude WebSearch) ├─ HN: {n} threads (via Claude WebSearch) ├─ Substack: {n} articles (via Claude WebSearch) ├─ Financial: {n} articles (via Claude WebSearch) ├─ LinkedIn: {n} profiles/articles (via Claude WebSearch) ├─ YouTube: {n} videos (via Claude WebSearch) ├─ Web: {n} pages └─ Top sources: {site1}, {site2}

Add X/Twitter + broader coverage: Set up xAI API key (see Prerequisites)

Step 7: Expert Mode

After delivering research, enter Expert Mode:

• Answer follow-ups from cached results

• No new searches unless explicitly requested

• Cross-reference between sources

New search triggers (exit Expert Mode):

• "Search again for..."

• "Find more about..."

• "Update the research..."

• "Look deeper into..."

Operational Modes

Mode Sources When

Full Claude WebSearch + xAI (web + X + Reddit + GitHub + HN + Substack + Finance + LinkedIn) Step 0 returns XAI_AVAILABLE=true

Web-Only Claude WebSearch only Step 0 returns XAI_AVAILABLE=false

Mode is determined by Step 0 — never skip it or assume Web-Only without checking.

Depth Control

Depth Rounds Sources xAI Calls (Full) Use Case

quick 1 10-15 8 Fast overview, time-sensitive

default 2 25-40 13 Balanced research

deep 3 60-90 18 + 6-10 WebFetch Comprehensive analysis, important decisions

Cost Awareness

Action Cost

Claude Code WebSearch Free (subscription)

xAI search call (any type) $0.005/call

WebFetch (built-in) Free

scrapling fallback (optional) Free

Estimated cost per research session:

Depth Full Mode Web-Only

quick ~$0.04 (8 xAI calls) Free

default ~$0.065 (13 xAI calls) Free

deep ~$0.09 (18 xAI calls) Free

Cost-Saving Strategy:

• Claude WebSearch handles most needs (free)

• xAI adds X/Twitter (unique value) + broader coverage per platform

• WebFetch for deep-reading specific URLs (free), with scrapling fallback for blocked sites

Critical Constraints

DO:

• Run Step 0 (xAI key detection) before every research session

• If xAI key exists: run Claude WebSearch AND xAI scripts in parallel (Full mode)

• If xAI key missing: use Claude WebSearch only (Web-Only mode)

• Run gap analysis between rounds — never skip it

• Merge and deduplicate results by URL

• Exclude archived GitHub repositories from results — they are unmaintained and read-only

• Mark every claim with confidence: [HIGH] , [MEDIUM] , or [LOW]

• Ground synthesis in actual research, not pre-existing knowledge

• Cite specific sources with URLs

• Use --json flag when calling xAI scripts for programmatic parsing

• Load framework template from references/output-frameworks.md

DON'T:

• Skip Claude Code WebSearch (it's free)

• Run sequential searches when parallel is possible

• Display duplicate results from different search engines

• Quote more than 125 characters from any single source

• Repeat queries across rounds — each round targets gaps from previous

• Add Round 3 for quick or default depth — it's deep-only

• Discover new topics in Round 3 — verification only

Troubleshooting

xAI key not found:

security find-generic-password -s "xai-api" ~/Library/Keychains/claude-keys.keychain-db

If not found, run keychain setup above.

Keychain locked:

security unlock-keychain ~/Library/Keychains/claude-keys.keychain-db

No X/Twitter results: Requires valid xAI API key. Check at https://console.x.ai

WebFetch blocked (403/captcha/empty): Install scrapling and follow the auto-escalation protocol from cli-web-scrape (HTTP → validate → Dynamic → Stealthy):

uv tool install 'scrapling[all]' scrapling install # one-time: install browser engines for Dynamic/Stealthy tiers

Script errors: Ensure uv is installed: which uv . If missing: curl -LsSf https://astral.sh/uv/install.sh | sh

References

• references/output-frameworks.md — Framework templates (comparison, landscape, deep-dive, decision)

• references/search-patterns.md — Search operators and multi-round query patterns

• references/iterative-research.md — Gap analysis, round procedures, cross-referencing methodology