windsurf-observability

Windsurf Observability

Overview

Monitor Windsurf AI IDE adoption, feature usage, and productivity impact across your team. Covers Admin Dashboard analytics, custom tracking via extensions, and ROI measurement.

Prerequisites

• Windsurf Teams or Enterprise plan
• Admin dashboard access at windsurf.com/dashboard
• Team members actively using Windsurf

Authentication

Use an authenticated organization-admin session for dashboards. If analytics API access is enabled, use the organization-issued credential from an approved secret store, request read-only scope where available, and never include the credential or raw member data in reports.

Tool Use

• Use Read to inspect only the repository files and configuration needed for the request.
• Use Write only for a new artifact the user requested; never write credentials or unreviewed production configuration.
• Use Edit for bounded, reviewable changes and preserve unrelated user work.

Instructions

Step 1: Access Admin Dashboard Analytics

Navigate to Admin Dashboard > Analytics for team-wide metrics:

# Key metrics available in Windsurf Admin Dashboard
core_metrics:
  adoption:
    active_users_daily: "Unique developers using Windsurf per day"
    seat_utilization: "Active users / total seats (target set by the organization)"
    feature_adoption: "Which AI features each user uses"

  quality:
    completion_acceptance_rate: "Supercomplete suggestions accepted vs shown"
    cascade_flow_success_rate: "Cascade tasks completed vs failed"

  consumption:
    quota_utilization: "Daily and weekly included usage, when exposed"
    on_demand_usage: "Approved variable usage beyond included quota"

  efficiency:
    tasks_per_session: "Average Cascade interactions per session"
    time_saved_estimate: "Based on task complexity and completion speed"

Step 2: Set Up Usage Alerts

Monitor for underutilization and overuse:

# Alert thresholds for team management
alerts:
  low_adoption:
    condition: "seat_utilization < 50% for 7 days"
    action: "Schedule team training session"

  low_acceptance_rate:
    condition: "completion_acceptance_rate < 20% for 7 days"
    action: "Review .devin/rules/project.md — AI suggestions not matching project patterns"

  high_cascade_failures:
    condition: "cascade_success_rate < 50% for 3 days"
    action: "Check workspace config — .codeiumignore may be too aggressive"

  variable_usage_risk:
    condition: "on-demand usage approaches the approved spending limit"
    action: "Notify the billing owner and review model/context choices"

  inactive_seats:
    condition: "user has <10 interactions in 30 days"
    action: "Offer training or downgrade to Free tier"

Step 3: Build Custom Extension for Detailed Tracking

// windsurf-analytics-extension/src/extension.ts
import * as vscode from "vscode";

interface UsageEvent {
  event: string;
  timestamp: string;
  userId: string;
  file?: string;
  metadata?: Record<string, unknown>;
}

const events: UsageEvent[] = [];

export function activate(context: vscode.ExtensionContext) {
  // Track Cascade usage patterns
  const cascadeListener = vscode.workspace.onDidSaveTextDocument((doc) => {
    events.push({
      event: "file_save_after_cascade",
      timestamp: new Date().toISOString(),
      userId: vscode.env.machineId,
      file: doc.fileName,
      metadata: { languageId: doc.languageId, lineCount: doc.lineCount },
    });
  });

  // Flush events periodically
  setInterval(() => {
    if (events.length > 0) {
      const batch = events.splice(0);
      sendToAnalytics(batch);
    }
  }, 60000); // Flush every minute

  context.subscriptions.push(cascadeListener);
}

async function sendToAnalytics(batch: UsageEvent[]) {
  const endpoint = vscode.workspace
    .getConfiguration("windsurf-analytics")
    .get<string>("endpoint");
  if (!endpoint) return;

  await fetch(endpoint, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ events: batch }),
  }).catch(() => {}); // Silently fail — analytics should never block work
}

Step 4: Measure Productivity Impact

# Weekly team productivity report template
productivity_report:
  period: "Week of YYYY-MM-DD"
  team_size: 10
  active_windsurf_users: 8

  ai_metrics:
    total_cascade_tasks: 150
    cascade_success_rate: "78%"
    completion_acceptance_rate: "32%"
    quota_utilization: "source-defined value"

  productivity_proxies:
    commits_per_developer: 12       # vs baseline 8 pre-Windsurf
    pr_turnaround_hours: 6          # vs baseline 12 pre-Windsurf
    code_review_comments: 45        # quality indicator

  estimated_time_saved:
    per_developer_per_week: "3 hours"
    total_team_per_week: "24 hours"
    monthly_value: "$7,200"         # 24hrs * 4wks * $75/hr

  roi_calculation:
    monthly_subscription_cost: "from invoice"
    estimated_value_generated: "calculation with stated assumptions"
    roi: "derived only after finance approves the model"

Step 5: Dashboard Visualization

Track these metrics over time in your preferred dashboard tool:

## Recommended Dashboard Panels

1. Daily Active Users vs Total Seats (line chart)
   - Shows adoption trend
   - Alert when utilization drops below 70%

2. Completion Acceptance Rate (line chart, 7-day rolling avg)
   - Higher = better .devin/rules/project.md quality
   - Drop = rules need updating or team needs training

3. Cascade Success Rate (bar chart, weekly)
   - Tracks agentic task effectiveness
   - Low rate = prompts too vague or workspace too large

4. Included vs On-Demand Usage (bar chart, billing period)
   - Shows where variable spend occurs
   - Guides seat and spending-limit decisions

5. Top Workflows Used (table)
   - Shows which automated workflows team uses most
   - Identifies candidates for new workflows

Output

Produce a dated adoption and reliability report with source definitions, cohort size, active usage, quota or on-demand exposure, acceptance and rework signals, incidents, limitations, and recommended experiments. Avoid claiming causation from editor telemetry alone.

Error Handling

Issue	Cause	Solution
Low acceptance rate	AI suggestions don't match project style	Update .devin/rules/project.md with project conventions
Cascade flow failures	Insufficient tool permissions or context	Check workspace config, .codeiumignore
Seat utilization low	Team not adopted	Training session, share productivity data
Analytics data missing	Not on Teams/Enterprise plan	Upgrade for admin analytics
Custom extension conflicts	Extension interferes with Cascade	Ensure extension doesn't register completions

Examples

Quick Adoption Check

Admin Dashboard > Analytics > Overview
Look for: active users, acceptance rate, quota utilization, and on-demand usage

Monthly Seat Optimization

steps:
  1. Export member usage from Admin Dashboard
  2. Sort by authorized usage for the review period (ascending)
  3. Bottom 20%: offer training or downgrade to Free
  4. Top 10%: interview for best practices to share
  5. Reallocate freed seats to new team members

Resources

• Focused first-party references
• Windsurf Admin Guide
• Windsurf Enterprise

Related Skill

Continue with windsurf-incident-runbook to turn monitoring thresholds into owned response actions, rollback criteria, and durable incident evidence.