firecrawl-scraper

Firecrawl Web Scraper Skill

Status: Production Ready ✅ Last Updated: 2025-10-24 Official Docs: https://docs.firecrawl.dev API Version: v2

---

What is Firecrawl?

Firecrawl is a Web Data API for AI that turns entire websites into LLM-ready markdown or structured data. It handles:

• JavaScript rendering - Executes client-side JavaScript to capture dynamic content
• Anti-bot bypass - Gets past CAPTCHA and bot detection systems
• Format conversion - Outputs as markdown, JSON, or structured data
• Screenshot capture - Saves visual representations of pages
• Browser automation - Full headless browser capabilities

---

API Endpoints

1. /v2/scrape - Single Page Scraping

Scrapes a single webpage and returns clean, structured content.

Use Cases:

• Extract article content
• Get product details
• Scrape specific pages
• Convert HTML to markdown

Key Options:

• formats: ["markdown", "html", "screenshot"]
• onlyMainContent: true/false (removes nav, footer, ads)
• waitFor: milliseconds to wait before scraping
• actions: browser automation actions (click, scroll, etc.)

2. /v2/crawl - Full Site Crawling

Crawls all accessible pages from a starting URL.

Use Cases:

• Index entire documentation sites
• Archive website content
• Build knowledge bases
• Scrape multi-page content

Key Options:

• limit: max pages to crawl
• maxDepth: how many links deep to follow
• allowedDomains: restrict to specific domains
• excludePaths: skip certain URL patterns

3. /v2/map - URL Discovery

Maps all URLs on a website without scraping content.

Use Cases:

• Find sitemap
• Discover all pages
• Plan crawling strategy
• Audit website structure

4. /v2/extract - Structured Data Extraction

Uses AI to extract specific data fields from pages.

Use Cases:

• Extract product prices and names
• Parse contact information
• Build structured datasets
• Custom data schemas

Key Options:

• schema: Zod or JSON schema defining desired structure
• systemPrompt: guide AI extraction behavior

---

Authentication

Firecrawl requires an API key for all requests.

Get API Key

1. Sign up at https://www.firecrawl.dev
2. Go to dashboard → API Keys
3. Copy your API key (starts with fc-)

Store Securely

NEVER hardcode API keys in code!

# .env file
FIRECRAWL_API_KEY=fc-your-api-key-here

# .env.local (for local development)
FIRECRAWL_API_KEY=fc-your-api-key-here

---

Python SDK Usage

Installation

pip install firecrawl-py

Latest Version: firecrawl-py v4.5.0+

Basic Scrape

import os
from firecrawl import FirecrawlApp

# Initialize client
app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Scrape a single page
result = app.scrape_url(
    url="https://example.com/article",
    params={
        "formats": ["markdown", "html"],
        "onlyMainContent": True
    }
)

# Access markdown content
markdown = result.get("markdown")
print(markdown)

Crawl Multiple Pages

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Start crawl
crawl_result = app.crawl_url(
    url="https://docs.example.com",
    params={
        "limit": 100,
        "scrapeOptions": {
            "formats": ["markdown"]
        }
    },
    poll_interval=5  # Check status every 5 seconds
)

# Process results
for page in crawl_result.get("data", []):
    url = page.get("url")
    markdown = page.get("markdown")
    print(f"Scraped: {url}")

Extract Structured Data

import os
from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

# Define schema
schema = {
    "type": "object",
    "properties": {
        "company_name": {"type": "string"},
        "product_price": {"type": "number"},
        "availability": {"type": "string"}
    },
    "required": ["company_name", "product_price"]
}

# Extract data
result = app.extract(
    urls=["https://example.com/product"],
    params={
        "schema": schema,
        "systemPrompt": "Extract product information from the page"
    }
)

print(result)

---

TypeScript/Node.js SDK Usage

Installation

npm install @mendable/firecrawl-js
# or
pnpm add @mendable/firecrawl-js
# or use the unscoped package:
npm install firecrawl

Latest Version: @mendable/firecrawl-js v4.4.1+ (or firecrawl v4.4.1+)

Basic Scrape

import FirecrawlApp from '@mendable/firecrawl-js';

// Initialize client
const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Scrape a single page
const result = await app.scrapeUrl('https://example.com/article', {
  formats: ['markdown', 'html'],
  onlyMainContent: true
});

// Access markdown content
const markdown = result.markdown;
console.log(markdown);

Crawl Multiple Pages

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Start crawl
const crawlResult = await app.crawlUrl('https://docs.example.com', {
  limit: 100,
  scrapeOptions: {
    formats: ['markdown']
  }
});

// Process results
for (const page of crawlResult.data) {
  console.log(`Scraped: ${page.url}`);
  console.log(page.markdown);
}

Extract Structured Data with Zod

import FirecrawlApp from '@mendable/firecrawl-js';
import { z } from 'zod';

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

// Define schema with Zod
const schema = z.object({
  company_name: z.string(),
  product_price: z.number(),
  availability: z.string()
});

// Extract data
const result = await app.extract({
  urls: ['https://example.com/product'],
  schema: schema,
  systemPrompt: 'Extract product information from the page'
});

console.log(result);

---

Common Use Cases

1. Documentation Scraping

Scenario: Convert entire documentation site to markdown for RAG/chatbot

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

docs = app.crawl_url(
    url="https://docs.myapi.com",
    params={
        "limit": 500,
        "scrapeOptions": {
            "formats": ["markdown"],
            "onlyMainContent": True
        },
        "allowedDomains": ["docs.myapi.com"]
    }
)

# Save to files
for page in docs.get("data", []):
    filename = page["url"].replace("https://", "").replace("/", "_") + ".md"
    with open(f"docs/{filename}", "w") as f:
        f.write(page["markdown"])

2. Product Data Extraction

Scenario: Extract structured product data for e-commerce

const schema = z.object({
  title: z.string(),
  price: z.number(),
  description: z.string(),
  images: z.array(z.string()),
  in_stock: z.boolean()
});

const products = await app.extract({
  urls: productUrls,
  schema: schema,
  systemPrompt: 'Extract all product details including price and availability'
});

3. News Article Scraping

Scenario: Extract clean article content without ads/navigation

article = app.scrape_url(
    url="https://news.com/article",
    params={
        "formats": ["markdown"],
        "onlyMainContent": True,
        "removeBase64Images": True
    }
)

# Get clean markdown
content = article.get("markdown")

---

Error Handling

Python

from firecrawl import FirecrawlApp
from firecrawl.exceptions import FirecrawlException

app = FirecrawlApp(api_key=os.environ.get("FIRECRAWL_API_KEY"))

try:
    result = app.scrape_url("https://example.com")
except FirecrawlException as e:
    print(f"Firecrawl error: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

TypeScript

import FirecrawlApp from '@mendable/firecrawl-js';

const app = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY
});

try {
  const result = await app.scrapeUrl('https://example.com');
} catch (error) {
  if (error.response) {
    // API error
    console.error('API Error:', error.response.data);
  } else {
    // Network or other error
    console.error('Error:', error.message);
  }
}

---

Rate Limits & Best Practices

Rate Limits

• Free tier: 500 credits/month
• Paid tiers: Higher limits based on plan
• Credits consumed vary by endpoint and options

Best Practices

1. Use onlyMainContent: true to reduce credits and get cleaner data
2. Set reasonable limits on crawls to avoid excessive costs
3. Handle retries with exponential backoff for transient errors
4. Cache results locally to avoid re-scraping same content
5. Use map endpoint first to plan crawling strategy
6. Batch extract calls when processing multiple URLs
7. Monitor credit usage in dashboard

---

Cloudflare Workers Integration

⚠️ Important: SDK Compatibility

The Firecrawl SDK cannot run in Cloudflare Workers due to Node.js dependencies (specifically axios which uses Node.js http module). Workers require Web Standard APIs.

✅ Use the direct REST API with fetch instead (see example below).

Alternative: Self-host with workers-firecrawl - a Workers-native implementation (requires Workers Paid Plan, only implements /search endpoint).

---

Workers Example: Direct REST API

This example uses the fetch API to call Firecrawl directly - works perfectly in Cloudflare Workers:

interface Env {
  FIRECRAWL_API_KEY: string;
  SCRAPED_CACHE?: KVNamespace; // Optional: for caching results
}

interface FirecrawlScrapeResponse {
  success: boolean;
  data: {
    markdown?: string;
    html?: string;
    metadata: {
      title?: string;
      description?: string;
      language?: string;
      sourceURL: string;
    };
  };
}

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    if (request.method !== 'POST') {
      retu

---

*Content truncated.*