vscuse-case-diagnosis

vscuse Case Diagnosis

Goal

Run an existing local vscuse test case against an explicitly selected product image, show the real execution through the integrated browser, and classify any failure as one of:

• product bug: the extension or generated app behavior is wrong.
• test plan drift: the product behavior is correct but the vscuse plan/group/assertion/precondition is stale.
• setup failure: credentials, Docker, GHCR, vscuse runner, env vars, or local image setup blocked execution.
• flake: timing, tooltip, transient service state, or visual instability made an otherwise-correct flow unreliable.

Do not make a case green by hiding the classification. The final answer must say what failed, why it failed, what was changed, and what passed afterward.

Required Inputs

• Plan name, case title, failing step id, report path, or feature workflow.
• Product image mode: either current local repository bits or a pinned published image from a specific successful Actions run. Local mode uses TEMPLATE_VERSION=local and a branch-specific image tag such as VSCUSE_VSCODE_IMAGE=vscuse-atk-dev:local; published mode uses an exact provenance-checked ghcr.io/officedev/vscuse-atk-vscode:<tag> and the originating run's template routing.
• Any case-specific feature flags declared in plan_metadata.tags as feature_flag:<NAME>=<VALUE>, or in the docs scenario that owns the case.

If local setup, image build, runner install, or credentials are missing, use the shared setup guidance in local-vscuse-validation first.

Core Rules

• Before launching a local run, compare the plan with its owning docs/scenario and the current implementation. Repair deterministic drift first: changed question order, removed flows, required feature flags, generated-file expectations, and over-broad shared groups. Use live execution to validate that static repair and to investigate visual or timing ambiguity, not to rediscover differences already explicit in source.
• Use the narrowest login group required by the scenario. A Microsoft 365-only scaffold/provision flow must use an M365-only login group; include Azure login only when the case creates, reads, or deploys Azure resources.
• Use a real vscuse execute run. Do not mock the UI flow.
• After the first failure is classified, start vscuse-ui before any second full CLI run. It is the required workbench for live inspection, recording changed steps, refreshing visual checks, and demonstrating the repaired flow.
• Open the integrated browser to the Web UI at http://127.0.0.1:6082 and keep its embedded noVNC view visible while repairing. Also provide http://127.0.0.1:6080/vnc.html when a separate full-size live view is useful. Tell the user both URLs as soon as the services are ready.
• Do not repeatedly run the complete create/login/provision/deploy workflow through vscuse execute while diagnosing plan drift. Use Run, Continue, and Next in vscuse-ui to iterate near the failing area, then use one clean CLI run for final validation.
• Still finish with vscuse execute. A plan is not validated until it passes a clean CLI run or the remaining failure is explicitly classified.
• For every local vscuse execute, use the repository-root artifact command from local-vscuse-validation: create a unique .local/test-reports/<timestamp>-<plan-name>/ directory, pass it with --report-dir, and stream unbuffered output to run.log. Never overwrite the first failure with a retry.
• Show the live execution through the integrated browser at http://localhost:6080/vnc.html when the user asks to demonstrate the process.
• noVNC is a live view of the running container, not a replay. Open it while execution is still running.
• Do not screenshot or print secrets, passwords, tokens, tenant secrets, or generated API keys.
• --start and --end only work when the required UI state already exists. They do not recreate prior setup.
• Plan/group JSON is read from disk at execution time; edits apply on the next run without rebuilding the image.
• The local plan source and product image source are independent. It is valid to diagnose a dev plan against a pinned published image when that version is the requested product baseline.
• Never replace a requested Actions-run image with latest, a branch-local image, or a guessed neighboring tag. Verify the pulled image's org.opencontainers.image.revision against the Actions run head_sha and record the digest.
• A feature-flagged plan is not self-contained if the flags only live in a previous shell command. The plan should declare them with feature_flag:* tags, and the run must apply them before vscuse starts the container and VS Code extension host.
• When the user identifies a docs scenario as the source of truth, compare the case to that doc before preserving old steps. If the doc removes a shipped sub-flow, classify the old steps as test plan drift and do not keep executing them merely because they appear in the existing plan.
• Prefer key_press and type_text over click when a command palette, quick pick, focused button, or default selection can be driven by keyboard. Coordinate clicks with screenshot preconditions are more brittle under layout, zoom, focus, and tooltip changes.
• Treat keyboard-driven quick-pick steps as stateful interactions, not unconditional commands. If a step presses enter to select a row, keep a precondition/wait that proves the intended row is visible and highlighted; otherwise the key can fire before the quick pick loads or select the wrong existing item.
• For semantically equivalent UI states, such as a Teams app button showing either Add or Open, do not preserve a single-label visual precondition. Accept both states in the assertion, use a semantic/OCR-backed action when keyboard is not available, and validate the shared outcome.
• For optional UI prompts, do not use force_run:true. Use prompt-specific preconditions, key_press where possible, a short precondition_wait_timeout:X tag, and continue_on_error only after confirming the runner treats it as a non-failing skip.
• After plan or runner edits, do not trust stale failure rows in an already-open vscuse-ui tab. Reload the tab or click Restart/Clear, then confirm the UI has reloaded the edited plan before classifying a new failure.

Procedure

1. Locate the Owning Case

Read the owning docs/scenario and current implementation, then find the plan and any shared groups it expands. Before running, statically compare the documented question order, feature flags, generated artifacts, removed legacy flows, login/resource prerequisites, and the plan's execution order. Fix clear mismatches immediately, then use the first local run as validation rather than discovery.

Prefer the smallest owning surface:

• Edit the plan when only that case is affected.
• Edit a shared group when the drift belongs to a reused flow and should affect all consumers.
• Edit product code when noVNC/report evidence shows the product behavior is wrong.

2. Prepare the Runtime and Select the Product Image

Choose exactly one mode before starting the container.

Local product mode validates the current checkout's VSIX:

From the repository root:

.\set-azure-env.ps1 -Env atk06
$env:TEMPLATE_VERSION = "local"
$branchName = (git branch --show-current).Trim()
$imageBranch = $branchName -replace '^release/', '' -replace '[^A-Za-z0-9_.-]', '-'
$env:VSCUSE_VSCODE_IMAGE = "vscuse-atk-${imageBranch}:local"

$dockerCliDir = "C:\Program Files\Docker\Docker\resources\bin"
if (Test-Path (Join-Path $dockerCliDir "docker.exe")) {
   $env:PATH = "$dockerCliDir;$env:PATH"
}

Published product mode runs the local or dev plan files against a requested CI-built image. Resolve the exact tag and verify provenance using local-vscuse-validation, then set:

$env:VSCUSE_VSCODE_IMAGE = "ghcr.io/officedev/vscuse-atk-vscode:<verified-tag>"
Remove-Item Env:TEMPLATE_VERSION -ErrorAction SilentlyContinue

Only set TEMPLATE_VERSION in published mode when the originating CI run or scenario explicitly used it. A local plan path does not imply local template routing.

If the plan declares feature flags, apply them before running vscuse execute:

$planPath = "packages/tests/vscuse/vscode-test-cases/plans/<plan-name>.json"
$plan = Get-Content $planPath -Raw | ConvertFrom-Json
@($plan.plan_metadata.tags) | Where-Object { $_ -like "feature_flag:*" } | ForEach-Object {
   $pair = $_ -replace "^feature_flag:", ""
   $name, $value = $pair.Split("=", 2)
   Set-Item -Path "Env:$name" -Value $value
}

If those flags are not propagated into the VS Code container by the config used for the run, classify the result as a setup failure or precondition miss, not a product bug. Use a temporary run config that injects only the plan-declared flags under docker.environment, or update the shared config deliberately when the pass-through is meant to be stable for this repository.

Check only whether required secrets are set, never their values:

"M365_ACCOUNT_NAME","M365_ACCOUNT_PASSWORD" | ForEach-Object {
   [PSCustomObject]@{ Name = $_; IsSet = -not [string]::IsNullOrWhiteSpace([Environment]::GetEnvironmentVariable($_)) }
}

3. Execute the Focused Plan

Use the complete repository-root artifact command under Execute a Local Test Plan in local-vscuse-validation, substituting the focused plan path. Confirm the resulting run directory contains both run.log and test_report.html before interpreting or editing the case.

When the user needs to see the process, open the integrated browser to http://localhost:6080/vnc.html, click Connect, and keep it visible near the failing area.

4. Gather Failure Evidence

Use the report and terminal logs to capture:

• Plan id and execution id.
• Failing expanded step id and nearby previous step.
• Repository-relative paths to the run-specific `.local/test-reports

Content truncated.