triage-logs

Telemetry 2.0 Issue Triage Skill

Purpose

Systematically correlate device log bundles with Telemetry 2.0 source code to identify root causes, characterize impact, and propose unit-test and functional-test reproduction scenarios — for any behavioral anomaly reported by the user.

---

Usage

Invoke this skill when:

• A device log bundle is available under logs/ (or attached separately)
• The user describes a behavioral anomaly (examples: daemon stuck, reports missing, too many reports sent, reports arriving late, high CPU, high memory, unexpected profile activation, marker counts wrong)
• You need to write a reproduction scenario for an existing or proposed fix

The user's stated issue drives the investigation. Do not assume a specific failure mode — read the issue description first, then follow the steps below.

---

Step 1: Orient to the Log Bundle

Log bundle layout (typical RDK device):

logs/<MAC>/<SESSION_TIMESTAMP>/logs/
    telemetry2_0.txt.0        ← Primary T2 daemon log (start here)
    GatewayManagerLog.txt.0   ← WAN/gateway state machine
    WanManager*.txt.0         ← WAN interface transitions
    PAMlog.txt.0              ← Platform/parameter management
    SelfHeal*.txt.0           ← Watchdog and recovery events
    top_log.txt.0             ← CPU/memory snapshots (useful for perf issues)
    messages.txt.0            ← Kernel and system messages

Include any log files surfaced by the user's issue description (e.g., cellular*.txt.0 for connectivity issues, syslog for OOM events).

Log timestamp prefix format: YYMMDD-HH:MM:SS.uuuuuu

• Session folder names are local-time snapshots (format: MM-DD-YY-HH:MMxM)
• Log lines inside use device local time — always confirm via [Time] field in telemetry reports ("Time":"2026-03-06 07:24:23")
• Report JSON "timestamp" fields are Unix epoch UTC

Session ordering: Sort session folders chronologically. Multiple sessions may represent reboots. Alphabetical sort does NOT equal chronological order.

---

Step 2: Map Profiles and Threads

Read the startup section of telemetry2_0.txt.0 (first ~50 lines) to identify:

What to find	Log pattern
Profile name	Profile Name : <name>
Reporting interval	Waiting for <N> sec for next TIMEOUT
Timeout thread TID	TIMEOUT for profile - <name> (first occurrence)
CollectAndReport TID	CollectAndReport ++in profileName : <name> (first occurrence)
Send mechanism	methodName = Device.X_RDK_Xmidt.SendData (rbus) or HTTP_CODE (curl)

Thread role map (look for TID in TimeoutThread context):

• TimeoutThread per profile — fires TIMEOUT for profile log lines
• CollectAndReport / CollectAndReportXconf — one per profile, generates/sends reports
• asyncMethodHandler — short-lived rbus handler thread, called when SendData is dispatched

---

Step 3: Identify the Anomaly Window

Based on the user's stated issue, search for the relevant evidence pattern:

Hang / Stuck Daemon

A reporting hang manifests as a timestamp gap between CollectAndReport ++in and the next report-related log line from the same TID.

grep -n "CollectAndReport" telemetry2_0.txt.0 | head -40

Gap > 1 reporting interval = anomaly. During the gap, check:

• Is asyncMethodHandler ever logged? (no → rbus provider unresponsive)
• Does TIMEOUT for profile still fire? (yes → TimeoutThread alive but CollectAndReport stuck)

Under-Reporting / Missing Reports

Look for expected TIMEOUT for profile events that never trigger a CollectAndReport:

grep -n "TIMEOUT for profile\|CollectAndReport ++in\|Return status" telemetry2_0.txt.0

• Count TIMEOUT events vs. CollectAndReport entries over a time window
• Check for SendInterruptToTimeoutThread logged as failed (EBUSY path) — signals silently dropped
• Check for profile deactivation or reload during expected report window

Over-Reporting / Duplicate Reports

Look for multiple CollectAndReport ++in within a single interval:

grep -n "CollectAndReport ++in\|TIMEOUT for profile" telemetry2_0.txt.0

• Multiple TIMEOUT signals in one interval → concurrent interrupt and scheduler fire
• Report-on-condition (T2ERROR_SUCCESS after a marker event) firing alongside periodic report
• Check signalrecived_and_executing global flag race (concurrent profile callbacks)

CPU / Memory Spikes

Correlate top_log.txt.0 timestamps with T2 activity:

grep -n "telemetry2" top_log.txt.0

• Identify what T2 was doing (profile scan, DCA grep, report generation, rbus call) at spike time
• Check DCA log grep operations (dca.c, dcautil.c) for large log files causing high CPU
• Check marker accumulation in t2markers.c for memory growth
• Check if multiple profiles overlap their CollectAndReport window

Profile / Configuration Anomalies

• Unexpected profile changes: grep -n "profile\|xconf" telemetry2_0.txt.0 | grep -i "receiv\|updat\|activ"
• Marker count mismatches: compare report JSON marker values against grep patterns in dca.c
• Wrong reporting interval: confirm Waiting for <N> sec matches profile definition

---

Step 4: Correlate with Other Component Logs

Based on the anomaly window identified in Step 3, cross-reference with other logs:

Issue Type	Companion Log	What to Look For
Hang / rbus block	GatewayManagerLog.txt.0	WAN/interface state changes within hang window
Hang / rbus block	WanManager*.txt.0	Interface up/down transitions
Under-reporting	SelfHeal*.txt.0	Watchdog restarts of telemetry2_0 process
Over-reporting	PAMlog.txt.0	Parameter changes triggering report-on-condition
CPU spike	top_log.txt.0	CPU% at anomaly timestamps
Memory growth	messages.txt.0	OOM killer events, slab usage
Profile changes	Any xconf response log	Profile push or xconf poll activity

A tight coupling between an external event (state change, parameter update, restart) and the T2 anomaly window is the primary indicator of cause vs. coincidence.

---

Step 5: Locate the Code Path

Navigate to the relevant source based on the anomaly type. Key modules:

Scheduler (source/scheduler/scheduler.c)

Controls when profiles fire. Key paths:

• TimeoutThread — per-profile thread; calls timeoutNotificationCb while holding tMutex
• SendInterruptToTimeoutThread — uses pthread_mutex_trylock; if tMutex is held (callback in progress), the interrupt is silently dropped (EBUSY returns T2ERROR_FAILURE)
• signalrecived_and_executing — global flag with no atomic protection; susceptible to concurrent-write races under multi-profile load

Profile / Report Generation (source/bulkdata/profile.c, profilexconf.c, reportprofiles.c)

• CollectAndReport / CollectAndReportXconf hold plMutex or reuseThreadMutex for the entire report lifecycle (collection + send)
• rbus send (rbusMethod_Invoke / rbusMethod_InvokeAsync) has no timeout — a blocked rbus provider blocks the entire thread indefinitely
• Report-on-condition logic in reportprofiles.c can fire concurrently with a periodic send if synchronization is missing

Data Collection / CPU (source/dcautil/dca.c, dcautil.c, dcacpu.c, dcamem.c)

• DCA log-grep is I/O and CPU intensive; large log files can cause CPU spikes
• dcacpu.c and dcamem.c sample system resources; misreads can cause false markers
• Marker accumulation without cleanup (t2markers.c) can grow heap over time

Profile Configuration (source/t2parser/, source/bulkdata/profilexconf.c)

• Profile reception, parsing, and activation path for xconf-sourced profiles
• Incorrect interval parsing or duplicate profile names can cause over-scheduling or silent deactivation

Transport Layer (source/protocol/http/, source/protocol/rbusMethod/)

• HTTP send failures, retry logic, and cached-report replay
• rbus method provider registration and response handling

---

Step 6: Characterize Root Cause

Use this matrix to classify the issue based on observed evidence:

Observed Pattern	Issue Class	Primary Code Location
rbus call blocks > 10s, no asyncMethodHandler logged	Rbus provider unresponsive	profile.c / rbus transport
Signal Thread To restart logged but no report follows	Interrupt signal dropped (EBUSY on tMutex)	scheduler.c:SendInterruptToTimeoutThread
TIMEOUT fires but CollectAndReport never starts	Thread pool exhausted or profile in error state	scheduler.c, reportprofiles.c
TIMEOUT entries missing for > 2 intervals	TimeoutThread stuck, exited, or profile deregistered	scheduler.c:TimeoutThread
Multiple CollectAndReport ++in within one interval	Over-scheduling: concurrent interrupt + periodic fire	scheduler.c, reportprofiles.c
Long gap between ++in and --out with HTTP errors	Network failure; cached report retry loop	profilexconf.c, HTTP transport
Report JSON marker counts lower than expected	DCA grep miss, log rotation during scan, or marker not registered	dca.c, t2markers.c
Report JSON marker counts higher than expected	Duplicate marker registration or over-counting in DCA	dca.c, t2markers.c
signalrecived_and_executing logic inconsistency	Unsynchronized global flag race	scheduler.c (global variable)
CPU spike during report window	Large log file DCA grep or concurrent profile collection	dcautil.c, dca.c
Memory growth over sessions	Marker list not freed, profile not cleaned up on deregister	t2markers.c, profile.c
Profile activated/deactivated unexpectedly	xconf push race or profile name collision	profilexconf.c, t2parser/

---

Step 7: Assess L1 (Unit) Test Coverage

Location: source/test/

Existing coverage (representative):

• schedulerTest.cpp: basic SendInterruptToTimeoutThread, `TimeoutT

---

Content truncated.