api-ops

api-ops

Patcher's API service (api/patcher_api/) is the only piece intended to run on a Linux host. The documented deploy is uvicorn under systemd, fronted by a cloudflared tunnel. No inbound ports beyond 22/80/443 are opened. SQLite + SQLAlchemy 2.0 async is the backing store. /apps* and /stats are public; /admin/* is gated by a deploy token.

This skill has two modes: a proactive status sweep and reactive troubleshooting. A sweep that finds a problem flows straight into the matching troubleshooting gotcha below.

Everything here is read-only. Report findings and recommend fixes; let the user run anything that changes state (including Healthchecks.io dashboard edits).

Host layout (the map)

The patcher-api.service unit lives only on the box. Discover its port, user, and ExecStart at runtime (systemctl cat patcher-api.service) rather than assuming, the app binds a localhost port that cloudflared fronts.

Mode A: status sweep

Run these read-only checks in order and report a per-component verdict. Don't stop at the first yellow, complete the sweep so the summary is whole.

Run the checks sequentially, and append || true to any check whose non-zero exit is benign, notably systemctl is-failed, which exits non-zero precisely when a unit is healthy. One un-guarded non-zero exit can cancel a parallel command batch and abort the sweep mid-run.

Privileges (no passwordless sudo assumed)

This sweep runs unprivileged and must never invoke sudo or hang on a password prompt. systemctl reads, curl to localhost, and df need no privilege. Two capabilities unlock the rest:

• Journal reads (journalctl -u ...) require the invoking user to be in the systemd-journal group. Grant once: sudo usermod -aG systemd-journal <user>, then re-login. Without it, journalctl for a unit returns nothing or permission errors.
• DB-file inspection (step 7) needs read/traverse on /var/lib/patcher-api (owned by patcher). Add the user to the patcher group, or skip the raw file check and lean on /stats for freshness.

If a check needs elevation the sweep user doesn't have, do not run sudo. Print the exact sudo ... command for the human, mark that line "needs elevation" in the summary, and continue the rest of the sweep.

1. Discover the running config

systemctl cat patcher-api.service | grep -iE 'ExecStart|User|Environment'

Note the bind port (feeds the endpoint checks) and the service user.

2. Core service health

systemctl is-active patcher-api.service cloudflared.service
systemctl show patcher-api.service -p NRestarts -p ActiveEnterTimestamp
journalctl -u patcher-api.service -p warning --since "6 hours ago" --no-pager

• active for both → green.
• A climbing NRestarts or a recent restart cluster → possible restart loop (see G6).
• Warnings/errors in the journal → pattern-match against the gotchas below.

3. Timers (scheduled jobs)

systemctl list-timers 'patcher-*' --no-pager
# `is-failed` has INVERTED exit codes: non-zero when HEALTHY (a oneshot at rest
# reads "inactive"). Guard with `|| true` or it cancels a parallel batch.
systemctl is-failed patcher-catalog-refresh.service patcher-schema-audit.service || true

Confirm patcher-catalog-refresh.timer and patcher-schema-audit.timer are active with a sane NEXT. A oneshot reading inactive/dead at rest is normal; only a literal failed is a problem. Then check each last run:

journalctl -u patcher-catalog-refresh.service -n 15 --no-pager
journalctl -u patcher-schema-audit.service -n 15 --no-pager

• Catalog refresh should end with a Stitch summary: ... failed=0.
• Schema audit should log Schema audit clean: live database matches the models. A non-zero exit means schema drift, investigate against the ORM models before the next deploy.

4. Deploy path unit

systemctl is-active patcher-deploy.path
journalctl -u patcher-deploy.service -n 20 --no-pager

The .path unit should be active (watching the sentinel). A stale "last deploy" here is NORMAL — deploys are event-driven, see the monitoring notes below. Only a /fail (non-zero deploy) is a real problem.

5. cloudflared tunnel

systemctl status cloudflared.service --no-pager | head -20

Tunnel should be connected. If the public URL 502s but the app answers on localhost, that's G2, not a real outage.

6. App endpoints (use the discovered port; default 8000)

PORT=8000   # replace with the discovered bind port
curl -fsS "http://127.0.0.1:${PORT}/health"
curl -fsS "http://127.0.0.1:${PORT}/stats"
curl -fsS "http://127.0.0.1:${PORT}/apps" | head -c 200

• /health should return OK.
• /stats carries last_refresh (ingest freshness). If last_refresh is older than ~26h, the daily catalog refresh has been missing — the same threshold the external Lambda monitor alerts on. Cross-check step 3.
• /apps returning [] with no errors → G7 (empty catalog / ingest never populated).

7. Database + disk

df -h /var/lib/patcher-api /opt/patcher                # no privilege needed
stat -c '%U:%G' /var/lib/patcher-api/patcher_api.db    # needs read on the dir

• df always works. The stat needs read/traverse on /var/lib/patcher-api (owned by patcher); if the sweep user isn't in the patcher group, skip it and note "DB file check needs elevation" rather than sudo-prompting. Freshness is already covered by /stats in step 6.
• DB owner mismatch → G4 (write failures incoming).
• Low disk on the DB or code volume → flag; SQLite + backups + logs grow.

8. Healthchecks.io monitors (reference — read the dashboard)

These are external and their config lives only in the dashboard (not in the repo). Report whether each matches the intended config in the monitoring table below; a mismatch is itself a finding (it's how the schema-audit flap crept in).

Status sweep output

Patcher API status — <date/time>

  patcher-api.service    ✅ active, 0 restarts in 6h
  cloudflared            ✅ connected
  catalog-refresh timer  ✅ last run clean, next 04:00 UTC
  schema-audit timer     ✅ audit clean, next 06:00 UTC
  deploy path            ✅ watching (last deploy 3d ago — normal, event-driven)
  /health                ✅ ok
  /stats last_refresh    ✅ 8h ago (< 26h)
  /apps                  ✅ non-empty
  database               ✅ patcher:patcher, not locked
  disk                   ✅ 62% used on /var/lib
  Healthchecks config    ⚠️  schema-audit schedule drift — see note

Overall: 🟡 one config drift, no service impact.
<then: the specifics + recommended fix for any yellow/red>

Green when everything's nominal. Yellow for config drift or soft signals with no user impact. Red for an actual outage or failed job. Always end with the concrete fix for anything not green.

Monitoring topology & "don't panic" notes

The three Healthchecks.io dead-man's switches plus one external Lambda. The intended config below is the source of truth — Healthchecks stores it only in its dashboard, so drift between this table and the dashboard is a real finding.

External: an AWS Lambda probes /stats and alerts if last_refresh > 26h — independent liveness/freshness watch, separate from the box-side pings.

Don't-panic note 1 — patcher-deploy is outcome-only, not cadence. Deploys fire only when CI (push to main under api/**) POSTs /admin/deploy, which touches the sentinel and runs patcher-deploy.service. Deploys are therefore event-driven; there is no cadence. The check's long period exists so that a quiet stretch with no merges does not alarm. "Deploy DOWN: success signal did not arrive on time" with a week+ since the last ping is a false positive from a mis-modeled period, not a broken deploy. A real deploy problem shows up as a /fail ping (immediate) or a /start with no finish inside grace (a hang). Only those warrant action.

Don't-panic note 2 — schema-audit schedule must track the 06:00 UTC timer. If the audit flaps DOWN then UP within seconds each day, the Healthchecks schedule has drifted from the timer's actual run time. The timer runs at 06:00 UTC, so the schedule must be 0 6 * * *. A grace shorter than (schedule-to-actual-ping gap) causes the momentary flap. This is a dashboard fix, not a code fix.

Why config drift is invisible: systemd unit timings live in version control (api/deploy/*.timer); Healthchecks schedules/grace live only in the dashboard. When a timer moves in git and the dashboard doesn't follow, nothing catches it. Treat the table above as the reconciliation source.

Mode B: troubleshooting

Input shapes

Accept either:

1. Symptom description - "API returns 502 through cloudflared", "service won't start", "keyring import fails on boot", "admin endpoints return 401 with the right token".
2. Pasted logs - `journalctl -u

Content truncated.