vscuse-scenario-authoring

vscuse Scenario Authoring

Goal

Use docs, PRDs, scenarios, or mockups as the source of truth for vscuse coverage. Use vscuse-ui and noVNC as the primary authoring surface: execute the scenario in a real vscuse desktop, decide whether product behavior matches the docs, and then update existing test coverage or generate a new plan. Use vscuse execute afterward as the final CI-parity validation.

This workflow is for authoring or refreshing vscuse coverage. For diagnosing an already-failing case, use vscuse-case-diagnosis.

Required Inputs

• Docs/mockup/scenario path or quoted scenario text.
• The feature area or expected user workflow.
• Whether the expected UI already exists in product code. vscuse-ui cannot execute future UI that has not been implemented.
• Feature flags from the docs scenario, especially any ## Feature flags section or acceptance criteria that depends on featureFlag(...).
• The intended validation case boundary: start state, one primary behavior, and success state.

Core Rules

• Docs/mockups define expected behavior, but vscuse-ui does not automatically understand or execute docs. The agent or human must perform the scenario in the live noVNC desktop.
• Prefer vscuse-ui for creating or repairing case steps. It is the fastest way to inspect live UI state, record changed interactions, refresh screenshots, and edit steps. Treat the CLI as the final verifier, not the primary authoring surface.
• Use the integrated browser at http://localhost:6080/vnc.html as the live evidence surface.
• Treat vscuse-ui generated plans as drafts until cleaned, parsed, and rerun with vscuse execute.
• Archive every local vscuse execute using the repository-root artifact command in local-vscuse-validation: a unique gitignored .local/test-reports/<timestamp>-<plan-name>/ directory containing run.log and test_report.html.
• If product behavior does not match the docs, classify it as a product gap or bug before changing coverage.
• When a draft scenario explicitly replaces shipped behavior, treat the draft doc as the source of truth for the covered case. Existing vscuse steps that encode the old behavior are test plan drift, even if they used to pass.
• A product journey, product scenario, and validation case are different units. Follow the definitions in docs/01-product/scenarios/README.md; do not turn a complete journey into one long plan by default.
• Apply the independent-execution rules below to newly authored or materially refreshed cases. Existing plans remain valid until they are migrated; do not perform metadata-only rewrites across the current plan library.
• A new or materially refreshed validation case must be independently executable and retryable: it creates or loads its own declared start state, owns setup/assertions/cleanup, and does not consume another case's workspace or ephemeral cloud resources.
• Split cases at deterministic, reconstructable state boundaries, not mechanically at every command. A remote-preview case may still provision and deploy when those steps share inseparable ephemeral resources.
• Do not generate the Cartesian product of lifecycle × language × authentication × feature flags. Add a case only when a dimension changes user-visible UI, generated output, runtime behavior, or a meaningful failure domain.
• Do not commit empty r_*.json recordings or exploratory generated plans.
• Do not hardcode app names, paths, tenant data, API keys, or secrets captured during recording.
• Scenario feature flags must enter the target plan as plan_metadata.tags entries such as feature_flag:TEAMSFX_MCP_FOR_DA_DT=true. Do not leave them only in a local shell command or narrative notes.
• Feature-flag tags document the requirement but do not inject the container environment. Confirm the runner/config passes each flag before classifying behavior, and count a flag as covered only when the case reaches and asserts its gated branch.
• Prefer keyboard-driven steps over pointer-driven steps when authoring or cleaning recordings. Quick picks, command palette commands, focused default buttons, and simple selections should usually become key_press/type_text steps instead of coordinate click steps.
• When converting quick-pick clicks to key_press enter, preserve or create the wait condition that proves the intended row is loaded and highlighted. A keyboard action without a prompt-specific precondition can run before the list appears and type subsequent input into the wrong quick pick.
• When recording captures equivalent labels for the same action, such as Teams Add versus Open, author the case around the shared user intent: assertion accepts either label, the action is semantic/OCR-backed when keyboard is unavailable, and the next assertion verifies the converged outcome.
• When a prompt is conditional, record the observed UI first, then convert it into an optional guarded step: prompt-specific preconditions, short precondition_wait_timeout:X, no force_run:true, and continue_on_error only if the runner is known to honor it.

Procedure

1. Extract the Scenario From Docs

Read the docs/mockup and write down the testable contract:

• Source path and scenario title.
• Start state and required prerequisites.
• User actions in order.
• Expected visible UI at each important state.
• Expected generated files, env files, debug target, command labels, or Teams/browser behavior.
• Required feature flags and expected values.
• Negative/error states if they are part of the scenario.

If the docs are ambiguous, ask for the intended scenario rather than inventing hidden product behavior.

2. Map Existing Coverage

Search existing vscuse assets under packages/tests/vscuse/vscode-test-cases for:

• Plan names and descriptions.
• Shared group names.
• Command labels and visible UI strings.
• Template names, generated file names, debug configuration names, and provider names.

Decision rules:

• Same workflow with changed UI/assertions/waits: update the existing plan or group.
• Shared flow changed: update the shared group, then rerun one focused consumer plan.
• New workflow, provider, debug target, or major branch: create a new plan.
• Existing workflow but different feature-flag preconditions: update the plan metadata with feature_flag:* tags, and split into a separate plan when flag-on and flag-off behavior both need coverage.
• Future UI not implemented: report product gap; do not create a passing case that hides the gap.

2a. Define Independently Executable Cases

Map one approved product scenario to one or more validation cases. For a newly authored or materially refreshed case, record this authoring contract in existing plan_metadata.tags where a tag convention already exists and in the final authoring report for the remaining dimensions. Do not invent unsupported plan JSON fields. The contract includes:

• stable scenario ID and a unique validation ID;
• lifecycle phase and execution target;
• start state, one primary behavior, and success state;
• authentication or other behavior variant;
• exact feature-flag values;
• account/resource profile and whether Azure or M365 login is required;
• automation level: contract, managed-integration, or full-e2e;
• prerequisite owner: Toolkit, test fixture, tenant administrator, or external service;
• setup, assertions, cleanup, and execution gate.

Use these automation levels consistently:

• contract validates prompts and generated outputs without requiring the external system to run end-to-end.
• managed-integration starts from infrastructure or identity state owned and health-checked by the test environment.
• full-e2e crosses the real supported external boundaries and is usually scheduled when it is expensive or fragile.

Account requirements belong to the execution target. For example, local debug of a DA in Copilot requires an M365 account; do not infer account_profile:none from the word "local." An Entra SSO create case may stop at the generated admin handoff, while a separate scheduled integration uses a managed tenant fixture.

Source-of-Truth Override: DA Create With MCP Server

For SCN-DA-CREATE-WITH-MCP-SERVER, resolve the requested contract from Scenario ID, Status, and Proposal key; do not infer lifecycle from a directory name. The current dynamic-tool-discovery contract is docs/01-product/scenarios/da/create-da-with-mcp-server.md. With TEAMSFX_MCP_FOR_DA_DT=true, the create-flow case must not continue the legacy fetch-tools/static operation selection path:

• Do not click Fetch action from MCP, Fetch from MCP, or Select Operation(s) Copilot can interact with as part of the create scenario.
• Validate the generated dynamic project shape instead: no appPackage/mcp-tools-1.json; appPackage/ai-plugin.json has an MCP server runtime for the entered URL, empty functions, no mcp_tool_description, and run_for_functions: ["*"]; appPackage/declarativeAgent.json references action_1; .vscode/mcp.json points at the MCP server URL.
• For None auth, m365agents.yml must not inject oauth/register or dcr/register.
• Setting the DCR flag is not DCR coverage. Only a case that selects oauth-dynamic and asserts dcr/register covers the DCR-on branch; use a separate DT-on/DCR-off case only when validating that the dynamic-registration option is absent.
• Treat Entra SSO create as contract coverage through the generated client-ID/admin handoff. Run the real SSO path only from a managed tenant fixture on an appropriate scheduled gate.
• Keep create, local debug, and remote debug independently executable. Do not split provision/deploy/launch further when they must share resources created in the same remote-debug case.
• Run the edited plan end-to-end in vscuse-ui/noVNC first. Use CLI only after the UI flow has demonstrated that the saved case reaches the expected generated-projec

Content truncated.