#33

fix: keep synthetic pointer coordinates in CSS pixels while recording

Openrileyjshawwants to merge intofrom

Summary

drag steps silently do nothing whenever zoom is set to anything other than 1, and every recorded frame pairs the UI with a cursor position from slightly earlier. Both come from the recording pipeline rather than the drag code itself.

Pointer coordinates are device-scaled during recording

runVideo emulates deviceScaleFactor: zoom so Page.captureScreenshot returns frames at the output size. While the capture loop is running, that also makes Chrome interpret Input.dispatchMouseEvent coordinates as device pixels. Measured with a page that logs what it receives, at zoom: 0.638:

statesentpage received
before the capture loop starts1000,5001000,500
capture loop running1000,5001566,783
capture loop running638,319 (= x zoom)1000,500
after the loop stops1000,5001000,500

Clicks and typing are unaffected, because clickAt blocks the CDP-generated events and re-dispatches the chain from page JS (the comment there already calls CDP input unreliable). dragFromTo also fires dragstart/dragover/drop from JS at correct CSS coordinates, which is why examples/drag-and-drop still works despite running at zoom: 2: HTML5 drag and drop never needs the real pointer. Anything driven by actual pointer events, such as a Radix slider that takes pointer capture on the thumb, gets the raw CDP coordinates, so a press aimed at a slider thumb at (2936, 395) arrived at (4600, 620): outside the popover being dragged in, which dismissed it, left the drag doing nothing, and made the following step fail to resolve its target.

The fix applies the zoom with clip.scale at capture time and leaves the emulated scale factor at 1. Frames come out at the same size, capture is marginally faster in a 10-frame benchmark (422ms vs 461ms), and coordinates stay 1:1 for the whole recording. screenshot steps pass the same scale so their output size does not change.

The cursor overlay trails the frame it is drawn on

The capture loop advanced the cursor before capturing, but captureScreenshot resolves with pixels sampled at the moment it returns, not when it was called. Timing it against a marker moving at a known speed: pixels are dated 45ms after the call starts and within 2ms of when it returns (mean round trip 47ms).

So each frame paired UI from time T with a cursor position from T minus one round trip. Measured on a page whose dot follows the real pointer exactly, at 1920x1024:

mean gap between pointer and drawn cursor
before50.7px at 56px/frame, about 30ms
after0.9px, about 1ms

The overlay now advances after the capture. Duplicate frames keep the previous cursor state, since they stand in for time that has already passed.

Two smaller drag fixes

  • Drag ghosts are deep clones and kept the id, name and data-* attributes of their source, so for the 200ms before the ghost is removed a selector could match the ghost instead of the real element. This is how the failure above surfaced: the next step resolved [data-testid="slider-weight"] against the leftover ghost and then failed on the other endpoint, which pointed away from the actual cause.
  • A drag that cannot resolve an endpoint now says which end it was.

Test plan

  • pnpm test (105 core, 109 webreel), pnpm type-check, pnpm lint, pnpm format:check
  • New unit tests for the capture parameters and the ghost attribute stripping
  • Recorded a 25-step config at zoom: 0.638 that previously failed at the first drag: all steps complete and the five slider drags land on the values they should
  • Re-recorded examples/drag-and-drop (zoom: 2) with both builds and compared the final frames: identical

No changeset here, per the release process in AGENTS.md; happy to send one in a follow-up PR.

Found while recording a variable-font explorer with sliders. Generated with Claude Code

Activity

2
vercel[bot]2mo ago

@rileyjshaw is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.