debug

Debug

Goals

• Find why a run is stuck, retrying, or failing.
• Correlate Linear issue identity to a Codex session quickly.
• Read the right logs in the right order to isolate root cause.

Log Sources

• Primary runtime log: log/symphony.log
  • Default comes from SymphonyElixir.Observability.LogFile (log/symphony.log).
  • Includes orchestrator, agent runner, and Codex app-server lifecycle logs.
• Rotated runtime logs: log/symphony.log*
  • Check these when the relevant run is older.

Correlation Keys

• issue_identifier: human ticket key (example: MT-625)
• issue_id: Linear UUID (stable internal ID)
• session_id: Codex thread-turn pair (<thread_id>-<turn_id>)

elixir/docs/logging.md requires these fields for issue/session lifecycle logs. Use them as your join keys during debugging.

Quick Triage (Stuck Run)

1. Confirm scheduler/worker symptoms for the ticket.
2. Find recent lines for the ticket (issue_identifier first).
3. Extract session_id from matching lines.
4. Trace that session_id across start, stream, completion/failure, and stall handling logs.
5. Decide class of failure: timeout/stall, app-server startup failure, turn failure, or orchestrator retry loop.

Long-Running Service Checks

When verifying that the main Symphony service stays resident, do not treat a background process launched from a transient tool shell as authoritative. For example, nohup ./bin/symphony ... & inside exec_command may disappear when that tool shell returns because the execution environment cleans up child processes. That is not evidence that Symphony exited because the queue was empty.

Use one of these verification modes instead:

• Run ./bin/symphony ... in a foreground long-running tool session and observe at least two or three poll cycles before stopping it explicitly.
• For real daemon behavior, use a real process supervisor such as launchd, systemd, Docker/release runner, tmux, or screen.

Acceptable evidence for a healthy empty-queue resident service:

• the dashboard/API port remains listening when configured
• log/symphony.log* continues to record poll_cycle_completed status=ok
• candidate_count=0 still leads to later poll_cycle_started events

If a background process vanishes after the parent tool shell exits, first classify it as a launch-method artifact. Only investigate supervisor or child lifecycle failure after finding service-level evidence such as service_stopped, child crash logs, a nonzero foreground exit, or missing follow-up poll cycles while the parent shell is still alive.

Commands

# 1) Narrow by ticket key (fastest entry point)
rg -n "issue_identifier=MT-625" log/symphony.log*

# 2) If needed, narrow by Linear UUID
rg -n "issue_id=<linear-uuid>" log/symphony.log*

# 3) Pull session IDs seen for that ticket
rg -o "session_id=[^ ;]+" log/symphony.log* | sort -u

# 4) Trace one session end-to-end
rg -n "session_id=<thread>-<turn>" log/symphony.log*

# 5) Focus on stuck/retry signals
rg -n "Issue stalled|scheduling retry|turn_timeout|turn_failed|Codex session failed|Codex session ended with error" log/symphony.log*

Investigation Flow

1. Locate the ticket slice:
   • Search by issue_identifier=<KEY>.
   • If noise is high, add issue_id=<UUID>.
2. Establish timeline:
   • Identify first Codex session started ... session_id=....
   • Follow with Codex session completed, ended with error, or worker exit lines.
3. Classify the problem:
   • Stall loop: Issue stalled ... restarting with backoff.
   • App-server startup: Codex session failed ....
   • Turn execution failure: turn_failed, turn_cancelled, turn_timeout, or ended with error.
   • Worker crash: Agent task exited ... reason=....
4. Validate scope:
   • Check whether failures are isolated to one issue/session or repeating across multiple tickets.
5. Capture evidence:
   • Save key log lines with timestamps, issue_identifier, issue_id, and session_id.
   • Record probable root cause and the exact failing stage.

Reading Codex Session Logs

In Symphony, Codex session diagnostics are emitted into log/symphony.log and keyed by session_id. Read them as a lifecycle:

1. Codex session started ... session_id=...
2. Session stream/lifecycle events for the same session_id
3. Terminal event:
   • Codex session completed ..., or
   • Codex session ended with error ..., or
   • Issue stalled ... restarting with backoff

For one specific session investigation, keep the trace narrow:

1. Capture one session_id for the ticket.
2. Build a timestamped slice for only that session:
   • rg -n "session_id=<thread>-<turn>" log/symphony.log*
3. Mark the exact failing stage:
   • Startup failure before stream events (Codex session failed ...).
   • Turn/runtime failure after stream events (turn_* / ended with error).
   • Stall recovery (Issue stalled ... restarting with backoff).
4. Pair findings with issue_identifier and issue_id from nearby lines to confirm you are not mixing concurrent retries.

Always pair session findings with issue_identifier/issue_id to avoid mixing concurrent runs.

Notes

• Prefer rg over grep for speed on large logs.
• Check rotated logs (log/symphony.log*) before concluding data is missing.
• If required context fields are missing in new log statements, align with elixir/docs/logging.md conventions.