skill-observability

skill-observability - Measurement event contract and emit substrate

A referenced companion that defines the contract and tools underpinning the measurement layer of the skill improvement loop (install → use → reflect). Provides the SSOT for the shape of events (schema) and the convention for writing them (emit), along with the trace-derivation hook (obs-derive.mjs) that derives invocations after the fact from the transcript. Aggregation (/skill-metrics) and reflection are built separately on top of this contract.

Principles

• Trace derivation as the main axis: Measurement is derived after the fact, as much as possible, from gh/git ground truth and transcript traces (avoiding self-report bias). Inline emit from within a skill prompt is secondary, limited to semantic signals that don't show up in traces (fallback, HITL rejection, loop cap reached, etc.).
• fail-open: A failed emit must not stop the skill being measured. obs-emit.mjs never throws — whether absent, failing, or hitting a validation error — it prints a warning to stderr and exits 0.
• privacy (strictest mode): Events are metadata only. event.schema.json's additionalProperties: false mechanically rejects unknown fields such as payload / diff / token / path. Repo identifiers are never stored raw — only a hash (repo_hash). Sending (reflection) is always explicit opt-in / HITL (this skill has no send path of its own).
• Count-based reasoning: Because a single author's low-frequency data volume doesn't reach statistical significance, this contract does not force rates or confidence intervals. The aggregation side is built primarily around counts + notable events.

Event log

~/.agents/observability/events.jsonl   # append-only, 1 line per event, self-contained without OTel dependency

HOME-anchored (outside the skills dir). Survives a dogfood mirror rebuild. Readable by any consumer / hook.

Event contract (event.schema.json is the SSOT)

event.schema.json (a sibling of this skill) is the sole SSOT. Both obs-emit.mjs and the tests validate by reading this file, so doc and code never drift apart. Field names lean toward the shape of the OpenTelemetry GenAI semantic conventions (skill≈gen_ai.agent.name / operation≈gen_ai.operation.name). Since the conventions are experimental, this skill does not couple tightly to them.

Required fields: schema_version(=1) / ts(ISO 8601) / adapter / session_id / skill / event.

Types of event:

event	Purpose	Additional required field
start	Skill invocation	—
phase	Phase transition (implement / ship / review, etc.)	—
outcome	End state	status ∈ {completed, aborted, fallback}
signal	Semantic counter (a transition that doesn't show up in traces)	name (fixed vocabulary)
heartbeat	Records that "observation ran" (prevents misreading absent data as success)	—

Fixed vocabulary (initial) for signal.name: review.loop_iter / review.deep_to_quick_fallback / usage_guard.fail_open / hitl.rejected / loop.hit_cap.

privacy: repo_hash is 12-digit hex (a sha256 prefix) only. The contract makes it impossible to write raw repo names, cwd, or raw PR numbers (additionalProperties: false).

Emit substrate (obs-emit.mjs)

obs-emit.mjs (sibling, a CLI that runs on all adapters) is the build→validate→append write primitive. Both the trace-derivation hook and inline emit ultimately go through this to append a single event.

# Examples
node obs-emit.mjs --skill=drive  --event=outcome --status=completed
node obs-emit.mjs --skill=review --event=signal  --name=review.loop_iter --value=2
node obs-emit.mjs --skill=drive  --event=heartbeat
node obs-emit.mjs --skill=drive  --event=outcome --status=merged --repo="$(git rev-parse --show-toplevel)"

Arguments: --skill / --event (effectively required), --status / --name / --value / --phase / --operation / --reason / --run (optional), --repo (hashed and stored in repo_hash), --adapter / --session (resolved from env by default).

Any event that fails validation, and any other failure, is not appended — it warns and exits 0 (fail-open). Does not break callers chained with &&.

Trace-derivation hook (obs-derive.mjs)

obs-derive.mjs (sibling, a SessionEnd hook) is the primary capture path. After a session ends, it reads the transcript, derives after the fact which skill fired, and records it via the obs-emit substrate. Because it reconstructs from traces rather than asking the model for a mid-run self-report, it avoids self-report bias (the worst-case runs that abort being exactly the ones that drop the emit).

The reliable core it derives:

• One heartbeat per session (a record that "observation ran." Makes an empty window distinguishable between "zero invocations" and "the hook didn't fire").
• One start per skill invocation found in the transcript. 2 channels:
  • A model-invoked Skill tool_use → operation: invoke_agent
  • A user-entered /slash-command → operation: slash_command (only for skills that actually exist. Built-in commands with no sibling skill dir, such as /clear /compact, are excluded to prevent data contamination)

The skill's arguments (which may be sensitive) are not recorded (only the skill name and channel).

Deferred (not derived by this hook): the merge/abort outcome. The merge state at session end is not yet settled — it requires session→PR linkage plus deferred re-evaluation, and abort inference ("ended without a PR") is hard to distinguish from a human interruption or an idempotent resume, and is noisy. To keep things reliable and low-noise, this is split off into a separate increment.

Enabling the SessionEnd hook (manual opt-in)

This repo does not distribute settings/hooks (same policy as the usage-guard hook). Add a SessionEnd entry to ~/.claude/settings.json (or settings.local.json). command must be the absolute path to obs-derive.mjs (differs between user-scope ~/.claude/skills/skill-observability/... and dogfood <repo>/.claude/skills/skill-observability/... — fill in your own path):

{
  "hooks": {
    "SessionEnd": [
      {
        "hooks": [
          { "type": "command", "command": "node /home/<you>/.claude/skills/skill-observability/obs-derive.mjs" }
        ]
      }
    ]
  }
}

Receives the SessionEnd JSON on stdin (session_id / transcript_path / cwd / reason), always exits 0, and stays lightweight (lines are pre-filtered by substring before JSON.parse).

Scope of application

This skill provides the contract + emit substrate + trace-derivation hook (invocation capture). Layers built on top of this contract (capture → aggregate → reflect → consume):

• /skill-metrics (aggregation, shipped): read-only aggregation of events.jsonl into counts + notable events.
• Reflection channel (reflection, shipped): the metrics-primed version of lessons-triage. Privacy-cleansed rollups are reflected into a backlog-pointer issue via HITL (sending is opt-in).
• backlog (consumption, shipped): passes that issue to drive as a priority index.
• outcome derivation (separate PR, pending): folds merge/abort state into the rollup using gh/git merge ground truth + session→PR linkage (the next increment of the trace-derivation hook).

Notes

• Does not read .env files.
• Events must not include verbatim logs, secrets, or raw private repo names/paths/PR values (the schema mechanically rejects these).