Crawler Skill

Playwriter exploration -> API/Network analysis -> Documentation -> Code generation

Templates: document-templates.md · code-templates.md Checklists: pre-crawl-checklist.md · anti-bot-checklist.md References: playwriter-commands.md · crawling-patterns.md · selector-strategies.md · network-crawling.md

<trigger_conditions>

</trigger_conditions>

<mandatory_reasoning>

Mandatory Sequential Thinking

• Always use the sequential-thinking tool before starting crawl design, extraction strategy, or code generation decisions.
• Run sequential-thinking for each major phase: discovery, method selection, and implementation planning.
• If sequential-thinking is unavailable, stop and report the blocker instead of continuing without structured reasoning.

</mandatory_reasoning>

<quick_commands>

# Create session + open page
playwriter session new
playwriter -s 1 -e "state.page = await context.newPage(); await state.page.goto('https://target.com')"

# Understand structure
playwriter -s 1 -e "console.log(await accessibilitySnapshot({ page: state.page }))"

# Intercept API responses
playwriter -s 1 -e $'
state.responses = [];
state.page.on("response", async res => {
  if (res.url().includes("/api/")) {
    try { state.responses.push({ url: res.url(), body: await res.json() }); } catch {}
  }
});
'

# Extract auth material
playwriter -s 1 -e "console.log(JSON.stringify(await context.cookies(), null, 2))"
playwriter -s 1 -e "console.log(await state.page.evaluate(() => localStorage.getItem('token')))"

# Convert selector
playwriter -s 1 -e "console.log(await getLocatorStringForElement(state.page.locator('aria-ref=e14')))"

</quick_commands>

<method_selection>

</method_selection>

<output_structure>

.hypercore/crawler/[site-name]/
├── ANALYSIS.md      # Site structure
├── SELECTORS.md     # DOM selectors
├── API.md           # API endpoints
├── NETWORK.md       # Auth/network details
└── CRAWLER.ts       # Generated crawler code

Templates: document-templates.md

</output_structure>

✅ Playwriter session created
✅ Structure analyzed with accessibilitySnapshot
✅ API interception attempted
✅ Selector extraction validated
✅ Findings documented under .hypercore/crawler/
✅ Crawler code generated
✅ sequential-thinking trace recorded for major phases

# User: /crawler crawl products from https://shop.example.com

# 1. Session
playwriter session new  # => 1
playwriter -s 1 -e "state.page = await context.newPage(); await state.page.goto('https://shop.example.com/products')"

# 2. Structure analysis
playwriter -s 1 -e "console.log(await accessibilitySnapshot({ page: state.page }))"
# => list "Products" [ref=e5]: listitem [ref=e6]: link "Product A" [ref=e7]

# 3. API detection (scroll trigger)
playwriter -s 1 -e "await state.page.evaluate(() => window.scrollTo(0, 9999))"
playwriter -s 1 -e "console.log(state.responses.map(r => r.url))"
# => ["/api/products?page=2"]

# 4. Documentation -> .hypercore/crawler/shop-example-com/
# 5. Generate API-based crawler