superpowers-rest-automation

REST Automation Skill

This skill enforces reliability and safety when building automations that call REST APIs.

When to use this skill

Use whenever the task involves:

• calling external REST APIs (CRUD, search, sync)
• integrating 2+ systems (ETL, iPaaS-like flows)
• webhooks, polling, or scheduled jobs
• data ingestion, normalization, enrichment, deduplication

Default design principles

• Idempotent by design: repeats should not create duplicates or corrupt data.
• Observable: logs/metrics correlate each run and each API call.
• Fail safe: handle partial failures; avoid silent data loss.
• Rate-limit aware: backoff and respect vendor limits.
• Least privilege: handle secrets safely, avoid overbroad scopes.

Checklist (apply unless irrelevant)

1) Define the contract

• Inputs (format, required fields, validation)
• Outputs (where data goes, expected shape)
• Success criteria (what “done” means)
• Non-goals (what the automation will not do)

2) Authentication & secrets

• Identify auth type: API key, OAuth2, JWT, mTLS
• Never hardcode secrets in code or logs
• Support secret injection via env vars / secret manager
• Plan token refresh if applicable (OAuth2)

3) Idempotency & deduplication

Pick at least one:

• Use provider idempotency keys (if supported)
• Use stable external IDs (e.g., external_id field) for upserts
• Keep a local/state store mapping source IDs -> target IDs
• Use deterministic hashes for dedupe when no stable ID exists Document the idempotency strategy explicitly.

4) Pagination & incremental sync

• Detect pagination style: next link, cursor, page+limit, offset+limit
• Ensure loops terminate safely (max pages / max time)
• Prefer incremental sync using updated_since/ETag/If-Modified-Since when possible
• Handle out-of-order updates and late-arriving events

5) Retries, backoff, and timeouts

• Set timeouts for connect/read
• Retry on transient errors: network failures, 429, 5xx (with limits)
• Use exponential backoff with jitter if possible
• Do not retry on most 4xx (except 408/409/429 depending on semantics)
• Cap retries and surface failures clearly

6) Rate limits & quotas

• Respect Retry-After and rate-limit headers
• Implement adaptive backoff on 429
• Consider batch endpoints to reduce call volume
• Avoid bursty concurrency unless explicitly safe

7) Data mapping & validation

• Explicit mapping layer (source -> normalized -> target)
• Validate required fields and types
• Normalize common formats (dates, enums, currency, locales)
• Handle nullability and partial payloads
• Record rejected records with reasons (don’t silently drop)

8) Error handling strategy

Choose and document per error class:

• Skip with log (non-critical record)
• Retry (transient)
• Quarantine (store failing payload for later)
• Fail the run (systemic issue) Ensure the workflow reports a clear summary at the end.

9) Observability & audit trail

Minimum:

• Run ID / correlation ID
• Per-request logs: method, path (not full secrets), status, latency, attempt count
• Counters: processed, created, updated, skipped, failed Prefer structured logs (JSON) if possible.

10) Webhooks (if involved)

• Verify signature (if provided)
• Handle replay (idempotency for event IDs)
• Respond quickly; process async if needed
• Store raw event payloads (optional but recommended)

11) Safety controls

• Dry-run mode (no writes)
• Limit scope (max records per run)
• “Kill switch” config flag
• Backups/rollback plan for destructive operations

Output requirements (when producing a solution)

Include:

• Idempotency strategy (1–3 bullets)
• Retry/backoff policy
• Pagination/incremental sync approach (if relevant)
• Error handling strategy + what gets logged/quarantined
• Verification plan (tests or a safe sandbox run plan)