firecrawl-prod-checklist

Firecrawl Production Checklist

Overview

Pre-deployment validation checklist for applications using Firecrawl's scrape, crawl, map, and extract APIs. Covers credential management, crawl safety limits, error handling, monitoring, and rollback.

Prerequisites

• Staging environment tested and passing
• Production API key from firecrawl.dev/app
• Monitoring infrastructure ready

Pre-Deployment Checklist

Credentials & Security

• Production FIRECRAWL_API_KEY in secure vault (not in code or .env)
• Key starts with fc- and is scoped to production
• Different API keys for dev/staging/production
• .env files in .gitignore
• Webhook secrets stored securely
• Git history scanned for leaked keys

Crawl Safety

• All crawlUrl calls have limit parameter set
• maxDepth configured to prevent unbounded crawling
• includePaths / excludePaths filters applied where appropriate
• Credit budget tracking implemented (daily limit alerts)
• No hardcoded URLs in production code

Error Handling

• 429 rate limit handling with exponential backoff
• 402 credit exhaustion handled gracefully (no crash)
• 401 auth failure logged and alerted
• Async crawl jobs have timeout with deadline
• Fallback from crawl to individual scrape on failure
• Empty markdown detection (JS rendering issues)

Monitoring & Alerting

• Scrape success/failure rate tracked
• Credit consumption monitored
• Crawl job completion rate tracked
• Alert on credit balance below threshold
• Alert on error rate > 5%
• Webhook delivery failures logged

Instructions

Step 1: Verify API Connectivity

set -euo pipefail
# Test production key
curl -s https://api.firecrawl.dev/v1/scrape \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}' | jq '.success'

# Check credit balance
curl -s https://api.firecrawl.dev/v1/team/credits \
  -H "Authorization: Bearer $FIRECRAWL_API_KEY_PROD" | jq .

Step 2: Health Check Endpoint

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

export async function healthCheck() {
  const start = Date.now();
  try {
    const result = await firecrawl.scrapeUrl("https://example.com", {
      formats: ["markdown"],
    });
    return {
      status: result.success ? "healthy" : "degraded",
      latencyMs: Date.now() - start,
      hasContent: (result.markdown?.length || 0) > 0,
    };
  } catch (error: any) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      error: error.statusCode || error.message,
    };
  }
}

Step 3: Production-Safe Crawl Wrapper

export async function productionCrawl(url: string, opts: {
  maxPages: number;
  paths?: string[];
  timeout?: number;
}) {
  // Hard credit safety — never exceed configured limit
  const limit = Math.min(opts.maxPages, 500);

  const job = await firecrawl.asyncCrawlUrl(url, {
    limit,
    maxDepth: 3,
    includePaths: opts.paths,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });

  // Poll with timeout
  const deadline = Date.now() + (opts.timeout || 600000);
  let pollInterval = 2000;
  let status = await firecrawl.checkCrawlStatus(job.id);

  while (status.status === "scraping" && Date.now() < deadline) {
    await new Promise(r => setTimeout(r, pollInterval));
    pollInterval = Math.min(pollInterval * 1.5, 30000);
    status = await firecrawl.checkCrawlStatus(job.id);
  }

  if (status.status !== "completed") {
    throw new Error(`Crawl ${job.id} did not complete: ${status.status}`);
  }
  return status;
}

Step 4: Rollback Procedure

set -euo pipefail
# Immediate rollback — disable Firecrawl integration
kubectl set env deployment/app FIRECRAWL_ENABLED=false
kubectl rollout restart deployment/app

# Verify rollback
curl -s https://app.example.com/health | jq '.services.firecrawl'

Alerting Rules

Resources

• Firecrawl Dashboard
• Firecrawl Rate Limits
• Firecrawl API Reference

Next Steps

For version upgrades, see firecrawl-upgrade-migration.