Web crawler

All-in-one scraping skill. Routes requests to the best backend automatically — the user does not need to know which API is used.

Use this when:

• Normal web_fetch fails, returns boilerplate, or a site blocks basic fetching
• The user asks for YouTube video content/transcript
• The user wants to scrape, fetch, or extract data from any social media platform
• The user mentions social media profiles, posts, comments, transcripts, ads, trending content, or engagement metrics

Prefer native web_fetch first for simple pages; paid fallback calls should be deliberate and scoped.

What each service is for

ScrapeCreators — Social media data extraction (27+ platforms)

Use for any request involving social media profiles, posts, videos, comments, transcripts, search, ads, trending content, or engagement metrics. Covers TikTok, Instagram, YouTube, LinkedIn, Facebook, Twitter/X, Reddit, Threads, Bluesky, Pinterest, Snapchat, Twitch, Kick, Truth Social, TikTok Shop, Google search, and link-in-bio services (Linktree, Komi, Pillar, Linkbio, Linkme, Amazon Shop).

Base URL: https://api.scrapecreators.com Auth: x-api-key: $SCRAPECREATORS_API_KEY header Method: All endpoints use GET requests with query params. Responses are JSON.

SerpApi — YouTube-only retrieval (legacy)

• engine=youtube to search YouTube and discover candidate videos.
• engine=youtube_video to fetch video metadata, description, chapters, related videos, and the transcript discovery link.
• engine=youtube_video_transcript to fetch timestamped transcript segments for AI analysis.

Do not use SerpApi for Google/Bing/general SERP scraping, shopping, maps, news, or any non-YouTube engine. The transparent proxy blocks those engines.

Firecrawl — Fallback web page scraper

Only a fallback crawler for one web page when ordinary fetching fails. Use POST /v2/scrape with a single url and focused formats like markdown, html, rawHtml, links, summary, or constrained json/question/highlights extraction.

Do not use Firecrawl crawl/map/search/agent/browser endpoints. Do not request screenshots, audio, branding, images, or browser actions unless the proxy policy is expanded later.

ScrapeCreators — Intent routing

Map user intent to the right endpoint. Endpoint paths use the pattern /v1/platform/action.

Important: After selecting an endpoint from the tables below, fetch its OpenAPI spec at https://docs.scrapecreators.com/{path}/openapi.json for full parameter details, types, and example response before making the actual API call. For example: https://docs.scrapecreators.com/v1/tiktok/profile/openapi.json

Profiles / User Info

Posts / Content Feeds

Single Post / Video Details

Comments

Transcripts

Search

Ad Libraries

Trending / Popular

Followers / Following / Live (TikTok only)

TikTok Shop

Link-in-Bio / Other

ScrapeCreators pagination

Paginated endpoints return a cursor/token in the response. Pass it back as a query param to get the next page.

ScrapeCreators known limitations

• Handles: pass without the @ symbol. Use charlidamelio not @charlidamelio. Applies to TikTok, Instagram, Twitter, Threads, Bluesky, Snapchat, Twitch, Pinterest, Truth Social
• YouTube handles: pass without the @ symbol. Use ThePatMcAfeeShow not @ThePatMcAfeeShow. You can also pass a channelId or full URL instead
• Hashtags: pass without the # symbol. Use fyp not #fyp. Applies to TikTok and YouTube hashtag search endpoints
• Twitter: returns ~100 most popular tweets, not chronological/latest
• Threads: only last 20-30 posts visible publicly
• Facebook posts: only 3 posts per page (API limitation)
• Facebook group posts: only 3 posts per page (same limitation)
• LinkedIn company posts: max 7 pages
• Instagram play counts: IG-only views (excludes cross-posted FB views)
• Truth Social: only prominent users (Trump, Vance, etc.) work publicly
• Transcripts: all transcript endpoints require video under 2 minutes
• Reddit subreddit names: case-sensitive! Use "AskReddit" not "askreddit"

Access patterns

ScrapeCreators (social media)

Call the ScrapeCreators API directly using curl or WebFetch. Authenticate with x-api-key header.

curl -s "https://api.scrapecreators.com/v1/tiktok/profile?handle=charlidamelio" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"

Or with fetch:

const res = await fetch(
  "https://api.scrapecreators.com/v1/tiktok/profile?handle=charlidamelio",
  { headers: { "x-api-key": process.env.SCRAPECREATORS_API_KEY } }
);
const data = await res.json();

Each endpoint has its own OpenAPI spec at https://docs.scrapecreators.com/{path}/openapi.json. Always fetch the per-endpoint spec first to get full parameter details before making the actual API call. The full spec is at https://docs.scrapecreators.com/openapi.json (large file — prefer per-endpoint specs).

Common optional params:

• trim (boolean): reduces response payload size. Use when you only need key metrics.
• region (string): 2-letter country code for proxy location. Does NOT filter by region — just routes through that country's proxy.

SerpApi (YouTube via transparent proxy)

Use Python scripts with core.http_client.proxied_get; include a typed SC-CALLER-ID header for cost tracking.

from core.http_client import proxied_get

headers = {"SC-CALLER-ID": "chat:youtube-transcript"}

search = proxied_get(
    "https://serpapi.com/search.json",
    params={"engine": "youtube", "search_query": "topic keywords"},
    headers=headers,
).json()

video = proxied_get(
    "https://serpapi.com/search.json",
    params={"engine": "youtube_video", "v": "VIDEO_ID"},
    headers=headers,
).json()

transcript = proxied_get(
    "https://serpapi.com/search.json",
    params={"engine": "youtube_video_transcript", "v": "VIDEO_ID", "language_code": "en"},
    headers=headers,
).json()

Firecrawl (web page fallback via transparent proxy)

from core.http_client import proxied_post

headers = {"SC-CALLER-ID": "chat:web-crawl-fallback"}

page = proxied_post(
    "https://api.firecrawl.dev/v2/scrape",
    json={
        "url": "https://example.com/article",
        "formats": ["markdown", "links"],
        "onlyMainContent": True,
        "timeout": 60000,
    },
    headers=headers,
).json()

Decision rules

Route every request to the right backend. The user should never need to specify which API to use.

Social media request (profile, posts, comments, search, ads, trending, transcripts)

Use ScrapeCreators. Match the user's intent to an endpoint from the routing tables above. Strip @ from handles and # from hashtags before calling. Fetch the per-endpoint OpenAPI spec first for full param details.

YouTube URL or YouTube content request

Use SerpApi for YouTube search/discovery and transcript retrieval via the transparent proxy. Alternatively, ScrapeCreators also covers YouTube (channel info, videos, shorts, playlists, comments, transcripts, search, trending shorts) — use whichever is more convenient for the specific request.

For a YouTube URL, extract the 11-character video id and call youtube_video_transcript first if the user's goal is content analysis, summarization, quote extraction, or topic mining. This returns timestamped transcript segments directly.

If the transcript call returns empty, unavailable, or the wrong language, call youtube_video next to inspect title, description, channel, publication date, and any advertised transcript metadata. Try one obvious language_code change only when the desired language is clear. If no transcript exists, summarize from metadata only and say the transcript was not available.

For a YouTube topic query, call engine=youtube, choose relevant video_results, then call metadata/transcript only for the videos needed. Avoid fetching many transcripts by default.

Blocked or JS-heavy web page

Use Firecrawl once with formats:["markdown","links"] and onlyMainContent:true. Treat the returned Markdown as the extraction substrate, not as final truth: parse the title, price/value fields, specs, body description, image URLs, outbound links, and obvious contact/location hints from the page structure.

General web-page extraction lessons:

• Many listing/detail pages render important content with JavaScript, image galleries, hidden sections, or repeated UI labels. web_fetch may return boilerplate while Firecrawl can still recover the real main content.
• Do not hard-code site-specific labels. Convert page text into a generic structured summary: what it is, where it is, key numbers, evidence snippets, media/links, and caveats.
• Preserve source URLs for images and links when they help verify the page, but do not download or batch-process every media asset unless the user asks.
• If Markdown misses important layout or structured fields, retry once with rawHtml; use json, question, or highlights only when the user asked for narrow extraction and the schema/prompt is specific.

Cost discipline

ScrapeCreators — most endpoints cost 1 credit per request. Exceptions: /v1/tiktok/user/audience costs 26 credits; /v1/tiktok/video/transcript with use_ai_as_fallback=true costs +10 credits; /v1/google/company/ads with get_ad_details=true costs 25 credits. Warn users before calling expensive endpoints.

SerpApi — billed per successful search-like call, regardless of result count.

Firecrawl — billed per page plus expensive modifiers.

Keep calls tight: one page, one video, or a small shortlist. Never batch-crawl whole websites or bulk-scrape entire feeds with this skill.

If the proxy returns 403, the request is outside the allowed use case. Change the approach instead of retrying.

If the proxy returns 429, back off; do not parallelize around the limit.

If the upstream returns a failure, report the exact failure and avoid repeated paid retries unless one parameter change is clearly justified.