fireflies-incident-runbook

Fireflies.ai Incident Runbook

Overview

Rapid incident response procedures for Fireflies.ai integration failures. Covers API outages, authentication problems, webhook issues, and rate limiting.

Severity Levels

Quick Triage (Run First)

set -euo pipefail
echo "=== Fireflies.ai Incident Triage ==="
echo ""

# 1. Can we reach the API?
echo "--- API Connectivity ---"
curl -s -o /dev/null -w "HTTP %{http_code} (%{time_total}s)\n" \
  -X POST https://api.fireflies.ai/graphql \
  -H "Authorization: Bearer $FIREFLIES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ user { email } }"}'

# 2. What error are we getting?
echo ""
echo "--- API Response ---"
curl -s -X POST https://api.fireflies.ai/graphql \
  -H "Authorization: Bearer $FIREFLIES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ user { email is_admin } }"}' | jq .

# 3. Can we list transcripts?
echo ""
echo "--- Transcript Access ---"
curl -s -X POST https://api.fireflies.ai/graphql \
  -H "Authorization: Bearer $FIREFLIES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ transcripts(limit: 1) { id title date } }"}' | jq '.data.transcripts[0] // .errors[0]'

Decision Tree

API returning errors?
├─ YES: What error code?
│   ├─ auth_failed (401) → API key revoked or invalid
│   │   └─ Fix: Regenerate key at app.fireflies.ai > Integrations
│   ├─ too_many_requests (429) → Rate limited
│   │   └─ Fix: Enable backoff, check for runaway loops
│   ├─ account_cancelled (403) → Subscription expired
│   │   └─ Fix: Renew subscription, contact billing
│   └─ 5xx errors → Fireflies platform issue
│       └─ Fix: Enable fallback mode, wait for resolution
└─ NO: Webhook issues?
    ├─ Not receiving webhooks → Check dashboard registration
    ├─ Invalid signature → Webhook secret mismatch
    └─ Processing failures → Check your webhook handler logs

Remediation by Error Type

auth_failed (401) -- P1

set -euo pipefail
# Verify API key format (should be non-empty)
echo "Key length: ${#FIREFLIES_API_KEY}"

# Test with explicit key
curl -s -X POST https://api.fireflies.ai/graphql \
  -H "Authorization: Bearer $FIREFLIES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "{ user { email } }"}' | jq '.errors[0].code // "OK"'

# Fix: Regenerate at app.fireflies.ai > Integrations > Fireflies API
# Then update your secret store:
# - GitHub: gh secret set FIREFLIES_API_KEY --body "new-key"
# - Vercel: vercel env rm FIREFLIES_API_KEY && vercel env add FIREFLIES_API_KEY
# - GCP: echo -n "new-key" | gcloud secrets versions add fireflies-api-key --data-file=-

too_many_requests (429) -- P2

set -euo pipefail
# Check if we have a request loop
# Free/Pro: 50/day limit. Business: 60/min limit.
echo "Check application logs for request volume"

# Immediate mitigation: reduce request rate
# Long-term: implement exponential backoff (see fireflies-rate-limits skill)

Webhook Not Firing -- P2

set -euo pipefail
# Verify webhook is registered
echo "Check: app.fireflies.ai > Settings > Developer settings"
echo "Webhook URL should be your HTTPS endpoint"

# Test by uploading audio (triggers webhook when done)
curl -s -X POST https://api.fireflies.ai/graphql \
  -H "Authorization: Bearer $FIREFLIES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "mutation($input: AudioUploadInput) { uploadAudio(input: $input) { success message } }",
    "variables": { "input": { "url": "https://example.com/test.mp3", "title": "Webhook Test" } }
  }' | jq .

# Remember: webhooks only fire for meetings YOU own (organizer_email)

Invalid Webhook Signature -- P3

// Debug signature verification
import crypto from "crypto";

function debugWebhookSignature(payload: string, receivedSig: string, secret: string) {
  const computed = crypto.createHmac("sha256", secret).update(payload).digest("hex");

  console.log("Received signature:", receivedSig);
  console.log("Computed signature:", computed);
  console.log("Match:", receivedSig === computed);
  console.log("Secret length:", secret.length, "(must be 16-32)");

  // Common issues:
  // 1. Secret doesn't match what's in Fireflies dashboard
  // 2. Payload was parsed/modified before verification (use raw body)
  // 3. Secret has trailing whitespace
}

Communication Templates

Internal (Slack)

P[1-4] INCIDENT: Fireflies.ai Integration
Status: INVESTIGATING / MITIGATED / RESOLVED
Error: [error code and message]
Impact: [what users are affected]
Action: [what we're doing]
ETA: [next update time]

Postmortem Template

## Incident: Fireflies.ai [Error Type]
**Date:** YYYY-MM-DD | **Duration:** Xh Ym | **Severity:** P[1-4]

### Summary
[1-2 sentences]

### Timeline
- HH:MM - Issue detected via [alert/user report]
- HH:MM - Triage started
- HH:MM - Root cause identified: [cause]
- HH:MM - Fix applied
- HH:MM - Verified resolved

### Root Cause
[Technical explanation]

### Action Items
- [ ] [Prevention measure] - Owner - Due date

Error Handling

Output

• Issue identified and categorized by severity
• Remediation applied based on error code
• Stakeholders notified via templates
• Evidence collected for postmortem

Resources

• Fireflies API Docs
• Fireflies Webhooks

Next Steps

For data handling and compliance, see fireflies-data-handling.