cli-e2e-testing

Aspire CLI End-to-End Testing with Hex1b

This skill provides patterns and practices for writing end-to-end tests for the Aspire CLI using the Hex1b terminal automation library.

Overview

CLI E2E tests use the Hex1b library to automate terminal sessions, simulating real user interactions with the Aspire CLI. Tests run in CI with asciinema recordings for debugging.

Location: tests/Aspire.Cli.EndToEnd.Tests/

Supported Platforms: Linux only. Hex1b requires a Linux terminal environment. Tests are configured to skip on Windows and macOS in CI.

Key Components

Core Classes

• Hex1bTerminal: The main terminal class from the Hex1b library for terminal automation
• Hex1bTerminalAutomator: Async/await API for driving a Hex1bTerminal — the preferred approach for new tests
• Hex1bAutomatorTestHelpers (shared helpers): Async extension methods on Hex1bTerminalAutomator (WaitForSuccessPromptAsync, AspireNewAsync, etc.)
• CliE2EAutomatorHelpers (Helpers/CliE2EAutomatorHelpers.cs): CLI-specific async extension methods on Hex1bTerminalAutomator (PrepareDockerEnvironmentAsync, InstallAspireCliAsync, etc.)
• CellPatternSearcher: Pattern matching for terminal cell content
• SequenceCounter (Helpers/SequenceCounter.cs): Tracks command execution count for deterministic prompt detection
• CliE2ETestHelpers (Helpers/CliE2ETestHelpers.cs): Environment variable helpers and terminal factory methods
• TemporaryWorkspace: Creates isolated temporary directories for test execution
• Hex1bTerminalInputSequenceBuilder (legacy): Fluent builder API for building sequences of terminal input/output operations. Prefer Hex1bTerminalAutomator for new tests.

Test Architecture

Each test:

1. Creates a TemporaryWorkspace for isolation
2. Builds a Hex1bTerminal with headless mode and asciinema recording
3. Creates a Hex1bTerminalAutomator wrapping the terminal
4. Drives the terminal with async/await calls and awaits completion

Test Structure

public sealed class SmokeTests(ITestOutputHelper output)
{
    [Fact]
    public async Task MyCliTest()
    {
        var repoRoot = CliE2ETestHelpers.GetRepoRoot();
        var strategy = CliInstallStrategy.Detect(output.WriteLine);
        var workspace = TemporaryWorkspace.Create(output);

        using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace);

        var counter = new SequenceCounter();
        var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500));
        await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, output, TestContext.Current.CancellationToken);

        await auto.PrepareDockerEnvironmentAsync(counter, workspace);
        await auto.InstallAspireCliAsync(strategy, counter);

        await auto.TypeAsync("aspire --version");
        await auto.EnterAsync();
        await auto.WaitForSuccessPromptAsync(counter);
    }
}

TerminalRun Pattern

Always use CliE2ETestHelpers.StartRun to wrap the terminal run. This returns a TerminalRun (implements IAsyncDisposable) that automatically:

1. Captures Aspire diagnostics via CaptureAspireDiagnosticsAsync (best effort)
2. Types exit and presses Enter to close the terminal
3. Awaits the pending run task

This eliminates the need for manual exit/await pendingRun at the end of every test and ensures diagnostics are always captured, even when tests fail.

// DO: Use StartRun for consistent diagnostics capture and cleanup
using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace);

var counter = new SequenceCounter();
var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500));
await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, output, TestContext.Current.CancellationToken);

// ... test body — no exit/pendingRun needed at the end

// DON'T: Manually handle exit and pendingRun
var pendingRun = terminal.RunAsync(TestContext.Current.CancellationToken);
// ... test body ...
await auto.TypeAsync("exit");
await auto.EnterAsync();
await pendingRun;

Running Tests Locally

CLI E2E tests run inside Docker containers on Linux. The workflow is: build a portable archive with localhive, then point the tests at it. This is the primary way to iterate on E2E tests during development.

Prerequisites

• Docker Desktop running (with Linux containers)
• .NET 10 SDK (installed via ./restore.sh or .\restore.cmd)

Quick Start (macOS / Linux)

# 1. Build a portable archive with CLI + packages + bundle
#    Use linux-arm64 on Apple Silicon, linux-x64 on Intel/Linux
./localhive.sh -o /tmp/aspire-e2e -r linux-arm64 --archive

# 2. Run a specific test
ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \
  dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \
  -- --filter-method "*.CreateAndRunAspireStarterProject"

# 3. Run all tests in a class
ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \
  dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \
  -- --filter-class "*.SmokeTests"

Quick Start (Windows / PowerShell)

# 1. Build a portable archive (Docker Desktop uses linux-x64 via WSL2)
.\localhive.ps1 -o C:\tmp\aspire-e2e -r linux-x64 -Archive

# 2. Run a specific test
$env:ASPIRE_E2E_ARCHIVE = "C:\tmp\aspire-e2e.tar.gz"
dotnet test tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj `
  -- --filter-method "*.CreateAndRunAspireStarterProject"

# 3. Run all tests in a class
dotnet test tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj `
  -- --filter-class "*.SmokeTests"

Choosing the Right RID

The archive must match the Docker container's architecture:

Development Workflow

The typical loop when writing or debugging E2E tests:

# 1. Make your code changes (CLI, hosting, templates, etc.)

# 2. Rebuild the archive (picks up all changes — ~3 min)
./localhive.sh -o /tmp/aspire-e2e -r linux-arm64 --archive

# 3. Run the specific test you're working on
ASPIRE_E2E_ARCHIVE=/tmp/aspire-e2e.tar.gz \
  dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \
  -- --filter-method "*.YourTestName"

# 4. If it fails, check the asciinema recording
#    Recordings are saved under the test output TestResults/recordings/ directory
#    Play with: asciinema play /path/to/YourTestName.cast

# 5. Fix and repeat from step 1 or 2

Install Modes

The CliInstallStrategy class auto-detects how to install the CLI in the test container. You can override via environment variables:

LocalHive (via ASPIRE_E2E_ARCHIVE) is the recommended mode for local development — it uses your locally-built CLI, packages, and bundle so you test exactly what you've changed.

Testing Against Released Versions

Useful for verifying tests pass against shipped versions or catching regressions:

# Test against latest GA release
ASPIRE_E2E_QUALITY=release dotnet test tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \
  -- --filter-method "*.CreateAndRunAspireStarterProject"

# Test against daily builds
ASPIRE_E2E_QUALITY=dev dotnet test ...

# Test against a specific version
ASPIRE_E2E_VERSION=13.2.1 dotnet test ...

Emulated channel matrix tests (identity sidecar)

A set of tests validates the CLI identity sidecar — the ability to make a locally built CLI emulate a different channel/version via ASPIRE_CLI_* env vars. They form an AppHost-language × channel-emulation matrix (one test per language because C# and TypeScript scaffold through different code paths and have diverged before):

EmulatedLocalReleaseBuildTests is the all-local "future release" row: it emulates a version (e.g. 13.5.0) that exists only in a locally built hive, so a successful resolve proves the CLI consulted ASPIRE_CLI_PACKAGES rather than nuget.org. Run it by building a stable-shaped archive with localhive --version:

# 1. Build a stable-shaped archive (note: --version X.Y.Z, NOT a prerelease suffix)
./localhive.sh --version 13.5.0 -o /tmp/aspire-localrelease -r linux-arm64 --archive

# 2. Run the all-local class
ASPIRE_E2E_ARCHIVE=/tmp/aspire-localrelease.tar.gz \
  dotnet test --project tests/Aspire.Cli.EndToEnd.Tests/Aspire.Cli.EndToEnd.Tests.csproj \
  -- --filter-class "*.EmulatedLocalReleaseBuildTests"

These tests skip unless the CLI was installed from a LocalHive archive and that archive is stable-shaped (no prerelease suffix). In default CI the archive is a prerelease `LocalArchive

Content truncated.