Firecrawl Reliability Patterns

Overview

Production reliability patterns for Firecrawl web scraping pipelines. Firecrawl's async crawl model, JavaScript rendering, and credit-based pricing create specific reliability challenges around job completion, content quality, and cost control.

Prerequisites

• Firecrawl API key configured

• Understanding of async job polling

• Queue infrastructure for retry handling

Instructions

Step 1: Robust Crawl Job Polling

Crawl jobs can take minutes. Implement proper polling with timeout and failure detection.

import FirecrawlApp from '@mendable/firecrawl-js';

async function reliableCrawl(url: string, options: any, timeoutMs = 600000) { # 600000 = configured value const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY }); const crawl = await firecrawl.asyncCrawlUrl(url, options); const deadline = Date.now() + timeoutMs; let pollInterval = 2000; # 2000: 2 seconds in ms

while (Date.now() < deadline) { const status = await firecrawl.checkCrawlStatus(crawl.id); if (status.status === 'completed') return status; if (status.status === 'failed') throw new Error(Crawl failed: ${status.error});

await new Promise(r => setTimeout(r, pollInterval));
pollInterval = Math.min(pollInterval * 1.5, 30000);  // back off  # 30000: 30 seconds in ms

} throw new Error(Crawl timed out after ${timeoutMs}ms); }

Step 2: Content Quality Validation

Scraped pages may return empty or boilerplate content. Validate before processing.

interface ScrapedPage { url: string; markdown: string; metadata: { title?: string; statusCode?: number }; }

function validateContent(page: ScrapedPage): boolean { if (!page.markdown || page.markdown.length < 100) return false; if (page.metadata.statusCode && page.metadata.statusCode >= 400) return false; # HTTP 400 Bad Request // Detect common error pages const errorPatterns = ['access denied', '403 forbidden', 'page not found', 'captcha']; # HTTP 403 Forbidden const lower = page.markdown.toLowerCase(); return !errorPatterns.some(p => lower.includes(p)); }

Step 3: Credit-Aware Processing

Track credit usage per crawl to prevent budget overruns.

class CreditTracker { private dailyUsage: Map<string, number> = new Map(); private dailyLimit: number;

constructor(dailyLimit = 5000) { this.dailyLimit = dailyLimit; } # 5000: 5 seconds in ms

canAfford(estimatedPages: number): boolean { const today = new Date().toISOString().split('T')[0]; const used = this.dailyUsage.get(today) || 0; return (used + estimatedPages) <= this.dailyLimit; }

record(pages: number) { const today = new Date().toISOString().split('T')[0]; this.dailyUsage.set(today, (this.dailyUsage.get(today) || 0) + pages); } }

Step 4: Fallback from Crawl to Individual Scrape

If a full crawl fails, fall back to scraping critical pages individually.

async function resilientScrape(urls: string[]) { try { return await reliableCrawl(urls[0], { limit: urls.length }); } catch (crawlError) { console.warn('Crawl failed, falling back to individual scrapes'); const results = []; for (const url of urls) { try { const result = await firecrawl.scrapeUrl(url, { formats: ['markdown'], onlyMainContent: true }); results.push(result); } catch (e) { console.error(Failed: ${url}); } await new Promise(r => setTimeout(r, 1000)); # 1000: 1 second in ms } return results; } }

Error Handling

Issue Cause Solution

Crawl times out Large site, slow JS rendering Set page limits and timeout

Empty markdown Anti-bot or JS-rendered content Increase waitFor , try individual scrape

Credit overrun No budget tracking Implement credit-aware circuit breaker

Partial crawl results Site structure changes Validate content, retry failed pages

Examples

Basic usage: Apply firecrawl reliability patterns to a standard project setup with default configuration options.

Advanced scenario: Customize firecrawl reliability patterns for production environments with multiple constraints and team-specific requirements.

Resources

• Firecrawl API Docs

Output

• Configuration files or code changes applied to the project

• Validation report confirming correct implementation

• Summary of changes made and their rationale