#16627

Instrument third-party harness preparation and quiet the Claude CLI

Draftkevinyang372wants to merge intofrom

Description

Part of the Oz cloud-agent startup latency work. The A1 baseline shows setup_third_party_harness_preparation for Claude at 15.2s p50 on a firewalled self-hosted Kubernetes worker, of which only ~2.1s is the two plugin installs that are instrumented today; the same code path is ~0.7s on Warp-hosted workers and Codex on the same cluster shows no excess. The leading hypothesis is that the Claude CLI cold start in claude auth status --json (and possibly in the harness launch) stalls on blocked egress — auto-updater, Statsig telemetry, error reporting — until internal timeouts fire. This PR makes the remainder attributable, removes the suspected traffic, and bounds the preflight so a slow check cannot hold the launch.

1. Sub-step events inside setup_third_party_harness_preparation

Four new run-scoped setup events, recorded with the existing SetupClientEventReporter and no renames of existing events:

  • setup_third_party_harness_preparation_prompt_resolution — the resolve-prompt call (0ms for local prompts)
  • setup_third_party_harness_preparation_mcp_resolution — resolve_mcp_specs_to_json, which reaches the managed-MCP endpoint when the run has managed servers
  • setup_third_party_harness_preparation_config_write — build_runner: .claude.json / settings.json / config.toml writes, skill publication, git exclude updates
  • setup_third_party_harness_preparation_auth_preflight — the auth check command on the PTY

Together with the existing plugin install/update events these partition the preparation step, so the ~13s can be read off directly. prepare_harness now takes the reporter; both driver tests that call it pass a noop reporter.

2. CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1 and DISABLE_AUTOUPDATER=1 for every claude the agent runs

Single source: CLAUDE_UNATTENDED_ENV in plugin_manager/claude.rs. Placement and what each covers:

  • harness_env_vars(Harness::Claude, _) in harness/mod.rs adds both variables to the run's resolved environment, which is the PTY launch environment. That covers (a) the auth preflight, which runs as a PTY command; (b) the PTY harness launch (claude --session-id ...); (c) claude --version for transcript metadata, which runs in the PTY session; and (d) the ACP adapter, because the bridge command runs in the PTY and exec_agent_command / relay_agent spawn claude-code-acp with the inherited environment, and the adapter spawns the claude CLI from there. Both transports get it; Codex, Gemini, Pi are untouched.
  • ClaudeCodePluginManager::run_logged passes both variables to every claude plugin ... invocation through LocalCommandExecutor (notification plugin install/update and platform plugin install/update). This is the only Claude invocation that does not inherit the PTY environment.

Side effects to be aware of: harness_env_vars is also used by local_harness_launch.rs, so local child Claude panes launched from the desktop app get the same two variables (their hidden pane cannot use /bug and does not auto-update; the user's own sessions are unaffected). The local dormant-child wake command (wake_driver.rs) is not changed; it is a desktop-side path, not an agent invocation.

What the variables disable, per Claude Code's documentation: DISABLE_AUTOUPDATER stops the update check and download at startup; CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC is the umbrella for DISABLE_AUTOUPDATER, DISABLE_BUG_COMMAND, DISABLE_ERROR_REPORTING (Sentry), and DISABLE_TELEMETRY (Statsig). Both are set explicitly so the intent survives a future change to the umbrella's definition. Neither affects API or OAuth traffic, plugin marketplace fetches, or MCP. The latency effect on the firewalled cluster is a hypothesis; the sub-step events above are how it gets confirmed.

3. Auth preflight timeout

The preflight was already bounded (PREFLIGHT_CHECK_TIMEOUT, 30s) and is fatal today for every harness that defines an auth check (Claude, Codex, and ACP manifests): a non-zero exit or a timeout surfaced as HarnessAuthCheckFailed. It is not advisory. This PR keeps the non-zero-exit semantics exactly and changes only the timeout:

  • PREFLIGHT_CHECK_TIMEOUT is now 10s.
  • On timeout the command is interrupted (Ctrl-C to the PTY) and the run continues with a warning: a slow check is not an auth verdict, and a real credential problem still surfaces through the harness's runtime error patterns (Not logged in, Invalid API key, ...) exactly as it does for anything the preflight does not catch. The interrupt is required because TerminalDriver admits one command at a time; without it the harness launch would fail with InvalidRuntimeState.
  • If the interrupted command does not release the terminal within PREFLIGHT_INTERRUPT_GRACE (5s), the run fails with the same HarnessAuthCheckFailed { detail: "command timed out" } it would have produced before, since the terminal is wedged either way.

A timed-out-and-continued preflight shows up in the new ..._auth_preflight event as a ~10s success; the warning log and a tracing::warn! with harness mark it explicitly.

4. ACP: starting the adapter during workspace prep (not done; design note)

Not cheap, so not attempted. The ACP adapter is deliberately launched as a command in the driver's terminal session (see the acp module doc) so it inherits the shell state setup commands established (nvm use, exports, PATH changes) and appears as a block in the shared session. The PTY runs one command at a time and is busy with clone and setup commands during workspace prep, so the adapter cannot be started there earlier. The alternatives are:

  • Spawn the adapter from the agent process (as the plugin manager does with LocalCommandExecutor) during prep, do initialize there, and defer session/new until harness_working_dir exists. This loses setup-command environment inheritance, which is load-bearing for customers whose setup commands install or select the toolchain the CLI needs, and loses the visible block. It would need an opt-in or a re-exec-into-PTY handoff, neither of which is a small change.
  • Keep the PTY launch but reduce harness_start itself: on staging ACP harness_start is ~5.1s p50 for Claude versus 0.17s PTY, which is the adapter's Node cold start plus the Claude SDK spawning the CLI plus initialize + session/new. The traffic variables in this PR apply to that CLI spawn too, so re-measure after this lands before designing further; a sidecar-side warm cache for the adapter's module graph is the next likely lever and is outside this repo.

Linked Issue

  • The linked issue is labeled ready-to-spec or ready-to-implement.
  • Where appropriate, screenshots or a short video of the implementation are included below (especially for user-visible or UI changes).

Testing

  • cargo check -p warp --tests — clean, no warnings
  • cargo nextest run -p warp -E 'test(/agent_sdk::(driver|setup_observability)/) | test(/plugin_manager/) | test(/local_harness_launch/)' — 856 passed, 0 failed
  • cargo clippy -p warp --all-targets --tests -- -D warnings — clean
  • cargo fmt -p warp with the repo's import config

New tests: claude_env_vars_disable_nonessential_traffic_on_both_transports and codex_env_vars_are_untouched_by_the_claude_traffic_switches in harness/mod_tests.rs; claude_model_env_vars_depend_on_transport no longer asserts an empty map for an unconfigured Claude run and asserts the absence of the model variables instead. setup_observability_tests::EXPECTED lists the four new steps. The preflight timeout path is not unit-tested: it needs a live PTY command that outlives the timeout, which is integration-level in this repo.

Not done: a run on the affected cluster. The point of the sub-step events is to make that measurement possible; recommend comparing ..._auth_preflight and ..._config_write on the self-hosted worker before/after the environment variables take effect.

This branch is independent of #16625 and #16626 and will conflict with #16625 in driver.rs (prepare_harness signature and the third-party arm of run_internal); whichever lands second needs a small manual merge.

  • I have manually tested my changes locally with ./script/run

Agent Mode

  • Warp Agent Mode - This PR was created via Warp's AI Agent Mode

CHANGELOG-OZ: Claude cloud agent runs no longer wait on the Claude CLI's auto-updater or telemetry at startup, and a slow auth check no longer blocks the launch.

Activity

1

This pull request is a draft

Merging is unavailable until it is marked ready for review.