XCrawl Crawl

Overview

This skill orchestrates full-site or scoped crawling with XCrawl Crawl APIs. Default behavior is raw passthrough: return upstream API response bodies as-is.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: ~/.xcrawl/config.json

{
  "XCRAWL_API_KEY": "<your_api_key>"
}

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits. If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/. After registration, they can activate the free 1000 credits plan before running requests.

Tool Permission Policy

Request runtime permissions for curl and node only. Do not request Python, shell helper scripts, or other runtime permissions.

API Surface

• Start crawl: POST /v1/crawl
• Read result: GET /v1/crawl/{crawl_id}
• Base URL: https://run.xcrawl.com
• Required header: Authorization: Bearer <XCRAWL_API_KEY>

Usage Examples

cURL (create + result)

API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"

CREATE_RESP="$(curl -sS -X POST "https://run.xcrawl.com/v1/crawl" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${API_KEY}" \
  -d '{"url":"https://example.com","crawler":{"limit":100,"max_depth":2},"output":{"formats":["markdown","links"]}}')"

echo "$CREATE_RESP"

CRAWL_ID="$(node -e 'const s=process.argv[1];const j=JSON.parse(s);process.stdout.write(j.crawl_id||"")' "$CREATE_RESP")"

curl -sS -X GET "https://run.xcrawl.com/v1/crawl/${CRAWL_ID}" \
  -H "Authorization: Bearer ${API_KEY}"

Node

node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",crawler:{limit:300,max_depth:3,include:["/docs/.*"],exclude:["/blog/.*"]},request:{locale:"ja-JP"},output:{formats:["markdown","links","json"]}};
fetch("https://run.xcrawl.com/v1/crawl",{
  method:"POST",
  headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
  body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'

Request Parameters

Request endpoint and headers

• Endpoint: POST https://run.xcrawl.com/v1/crawl
• Headers:
• Content-Type: application/json
• Authorization: Bearer <api_key>

Request body: top-level fields

crawler

proxy

request

js_render

output

output.formats enum:

• html
• raw_html
• markdown
• links
• summary
• screenshot
• json

webhook

Response Parameters

Create response (POST /v1/crawl)

Result response (GET /v1/crawl/{crawl_id})

data[] fields follow output.formats:

• html, raw_html, markdown, links, summary, screenshot, json
• metadata (page metadata)
• traffic_bytes
• credits_used
• credits_detail

Workflow

1. Confirm business objective and crawl boundary.

• What content is required, what content must be excluded, and what is the completion signal.

2. Draft a bounded crawl request.

• Prefer explicit limits and path constraints.

3. Start crawl and capture task metadata.

• Record crawl_id, initial status, and request payload.

4. Poll GET /v1/crawl/{crawl_id} until terminal state.

• Track pending, crawling, completed, or failed.

5. Return raw create/result responses.

• Do not synthesize derived summaries unless explicitly requested.

Output Contract

Return:

• Endpoint flow (POST /v1/crawl + GET /v1/crawl/{crawl_id})
• request_payload used for the create request
• Raw response body from create call
• Raw response body from result call
• Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

Guardrails

• Never run an unbounded crawl without explicit constraints.
• Do not present speculative page counts as final coverage.
• Do not hardcode provider-specific tool schemas in core logic.
• Highlight policy, legal, or website-usage risks when relevant.