#22

Docs: tell agents how to keep full-resolution screenshots from phone_observe

Openferrraridave-coderopened 6h ago

Why

Agents that need App Store–style captures often ask for phone_observe { screenshot: true }, then save whatever a vision model shows them (commonly a downscaled preview such as ~473×1024) — or they expect Pinky to write into Downloads/…. Pinky already returns a full-resolution JPEG in the tool result (same size as the screen WxH line); the miss is on the agent side.

Verified on Pinky 2026.1109.5, iPhone USB, portrait: HTTP MCP phone_observe { screenshot: true } returned a JPEG 1125×2436 (~299 KB) while a client vision preview of the same frame was 473×1024.

Ask

Add one short tip so agents (including remote HTTP ones that never load SKILL.md) keep the original.

1. docs/SKILL.md Tips — replace the existing screenshot tip with:

- `phone_observe { screenshot: true }` only when the element list is not enough; screenshots are large. The JPEG in the tool result is the phone's full resolution (same size as the `screen WxH` line). If you need that original on disk, save the image block's bytes from the tool result yourself (decode its base64 `data`): Pinky does not write screenshots into your folders, and a smaller preview a model shows you is not the file — do not re-encode or resize before saving.

2. phone_observe screenshot jsonschema in internal/mcp/tools.go (this is what every MCP client sees, including ones that skip skills), e.g.:

Also return a JPEG screenshot at the phone's full resolution (same size as the screen WxH line). Use it only when the element list is not enough. To keep the original file, save the image block's bytes from the tool result; Pinky does not write into the agent's folders, and a smaller preview a model shows is not that file.

3. Optional: one matching sentence under phone_observe / screenshot in docs/mcp.md.

Happy to send a PR if useful.

Activity

0

No activity yet.