feat: stream Agent Manager browser previews with public HTTPS - #13804

What Problem This Solves

Agent Manager's browser is limited to HTTP loopback applications. Its iframe preview is separate from the Chromium page used for automation, so framing restrictions can prevent a page from appearing and the visible page is not necessarily the page being inspected. Local applications also need public CDN modules and other secure resources.

This implements the public HTTPS use case tracked in #13618. It deliberately changes the preview architecture instead of removing framing headers. It also expands resource access beyond that issue's original same-origin-only proposal: public HTTPS/WSS resources may load across origins, while new document origins require approval.

Why This Change Was Made

• Render the same controlled Chromium page in Agent Manager through a high-resolution CDP stream. Keep frame delivery bounded, coalesce wheel input, and reject stale navigation, viewport, and session identities.
• Preserve native CORS preflight decisions with browser-target Fetch interception. Unowned pages are rejected before their first HTTP request, including warm popups and directly created targets.
• Keep the broker authenticated and loopback-only. Per-context proxies validate DNS answers, dial validated addresses, and prevent public pages from reaching private or loopback destinations. Certificate validation stays enabled.
• Retain the existing Playwright integration and browser-installation model. No new browser installer or separate automation framework is added.

Architecture

The preview is not an iframe, and it does not use VS Code's integrated browser API. A headless Chrome process runs outside the webview; only encoded frames and input events cross the boundary. This is the same family of design as the older VS Code browser-preview extensions: the extension host owns the browser and enforces policy where connections are actually made, while the webview only paints pixels and forwards input.

flowchart LR
  subgraph HOST["VS Code extension host (Node.js)"]
    BRK["BrowserBroker<br/>authenticated loopback API"]
    STR["BrowserStream<br/>CDP screencast + input"]
    NET["BrowserNetwork + BrowserProxy<br/>request policy, DNS pinning"]
  end
  subgraph WEB["Agent Manager webview (SolidJS)"]
    VP["StreamViewport<br/>canvas + input capture"]
  end
  subgraph CHROME["Headless Chrome (separate OS process)"]
    CTX["BrowserContext per session<br/>one page + per-context proxy"]
  end
  CLI["CLI agent<br/>browser_open tool"]

  VP -- "postMessage: viewport / interact" --> BRK
  BRK -- "postMessage: frame (JPEG + identity)" --> VP
  STR -- "newCDPSession, Input.*" --> CTX
  STR -- "Page.startScreencast / ack" --> CTX
  NET -- "proxy + Fetch gate" --> CTX
  CLI -- "Bearer token over 127.0.0.1" --> BRK

One Chrome process is shared by the extension host, and each session gets its own isolated BrowserContext. Chrome sends JPEG frames over a CDP screencast; the webview decodes them onto a canvas and drops stale frames by identity. Pointer, keyboard, wheel, IME, and clipboard events travel back and are replayed with CDP input calls.

Policy is enforced at the connection boundary, not by inspecting page content:

flowchart TB
  subgraph CONTENT["Page content (untrusted)"]
    PUB["Public HTTPS document"]
    LOCAL["Approved localhost app"]
  end
  FG["Browser-target Fetch gate<br/>rejects unowned frames before first request"]
  PX["Per-context authenticated proxy<br/>validates DNS, dials pinned IP"]
  PUB --> FG
  LOCAL --> FG
  FG --> PX
  PX -->|public origins| EXT["Public HTTPS / WSS"]
  PX -->|approved origin only| LOOP["localhost:port"]
  PX -.->|denied| PRIV["Private / link-local / LAN"]

Local HTTP applications remain limited to their approved origin, public pages cannot reach private or loopback destinations, navigation to a new origin requires approval, and certificate validation stays with Chromium. While the agent sandbox is enabled, the agent cannot drive this browser at all, so the stream cannot become a network bypass; the manual Browser panel still works.

Key files: BrowserBroker, BrowserStream, BrowserNetwork, BrowserProxy, StreamViewport, and the browser_open tool.

User Impact

• Open public HTTPS pages from the Browser panel and browser_open, alongside HTTP localhost applications. Bare public hostnames in the address bar use HTTPS.
• Local HTTP/WS traffic and document navigation stay on the approved local origin. Local pages can load public HTTPS/WSS resources, including CDN-hosted Three.js modules. Public pages cannot access private or loopback destinations.
• Use pointer, keyboard, IME, and plain-text clipboard input in the streamed page. Clipboard operations remain ordered and respect page handlers. Closing a stalled renderer no longer waits behind its pending input evaluation.
• Go back and forward with toolbar buttons. The buttons follow the real page history and ignore the initial blank page.
• Recover from a browser crash with Refresh. If Chromium exits, every open panel shows "The browser stopped unexpectedly. Refresh to start it again." and the next Refresh or open starts a new browser. A crashed page is replaced the same way.
• See "Cannot connect to . Make sure the local server is running." when a local server is not running, instead of a misleading HTTP 403.
• Open http://0.0.0.0:<port> URLs printed by dev servers. They open as http://localhost:<port>, because 0.0.0.0 is a listen address, not a destination, and localhost is the origin dev tools usually configure.
• See first-use browser requirements and localized missing-browser recovery actions in 21 languages. Chrome is the default; a compatible installed Playwright browser is the alternative. The experimental flag remains off by default and workspace trust is still required.

Known limits:

• Native Windows/Linux operation has not been exercised in this environment.
• Headless Chromium does not throttle hidden pages. A hidden worktree's page keeps running (about 7-8% of a core for a small animated page) until its browser closes. Page freeze, window minimize, background tabs, and CPU throttling have no useful effect in headless mode.
• Each local origin gets its own browser context, because the per-context proxy is locked to one local origin. Opening a different local port, or switching between local HTTP and public HTTPS, starts a fresh context without cookies. Public HTTPS pages share one context.
• Native macOS select popups are not captured. Clipboard events are synthetic and support plain text, not full native rich-clipboard behavior.
• The pinned Playwright 1.57/Chromium 143 combination has an upstream nested-frame locator/evaluation limitation after process swaps. The compatibility regression uses native DOM and HTTP assertions for that case; it does not claim the upstream locator problem is fixed.
• This is not an OS-enforced network sandbox around Chromium. Public TLS preconnects can occur without an HTTP request. Corporate proxy/PAC-only networks are not supported.

Evidence

Latest changes (4ef8aeb55b), verified in an isolated VS Code instance with two Agent Manager worktrees on macOS:

• Two worktrees keep separate cookies, storage, and input. Only the visible worktree streams; with the panel closed, zero frames are sent. Worktree switch to first frame: 11 ms. One visible stream runs at about 60 fps.
• After Chromium was killed, both worktrees showed the new error and recovered with Refresh in one new Chrome process.
• An agent-style browser_open with the panel closed streamed in 3 of 3 runs (0 of 3 before the fix). The cause was a Playwright screenshot restoring the viewport during a concurrent stream resize; screenshots now run in the stream queue.
• Back and forward buttons enabled and disabled from the real navigation history and navigated correctly.
• Browser test files: 316 passed, 0 failed. Extension lint, typecheck, and Knip passed. CLI sandbox/browser tests: 14 passed.

Fresh checks after updating the branch to 1536aef0fb:

• Extension unit suite: 5,070 passed, 4 skipped, 0 failed, 26,667 assertions across 389 files. Skips cover a Windows-only case and opt-in external probes.
• CLI sandbox/browser tests: 13 passed, 80 assertions. Destination classifier tests: 3 passed, 47 assertions.
• Extension and CLI typechecks, extension/CLI lint, production build, Knip, formatting, Kilo marker guard, source-link extraction, and upstream annotation checks passed.
• The local CLI build used the active Bun runtime after the pinned bunx bootstrap failed; the resulting binary passed its version, models, and sandbox-worker smoke tests.

During implementation, controlled real-Chromium regressions verified rejected preflights send no mutation, allowed preflights complete, private/loopback redirects and workers make no forbidden contact, and unowned popup/target requests are blocked. A focused run on both installed Chrome and pinned Chromium 143 passed 139 tests, with 3 opt-in external probes skipped.

Isolated VS Code checks verified the missing-browser settings/retry recovery flow, local CDN/WebGL loading, copy/cut followed by custom paste handling, and stream visibility behavior. A real Chromium page with an eight-second blocking copy handler closed in 14 ms and reopened in a fresh context. These are scoped macOS results, not cross-platform performance guarantees.

Streamed local fixture after copy/cut and a page-handled paste:

Missing-browser recovery guidance:

Manual verification: enable Browser Automation in a trusted workspace; open localhost and an approved HTTPS page; test clipboard cancellation/custom handling; close and reopen a busy page; and check the recovery actions when the selected browser is unavailable.