#119

iPadOS qualification: AX-backed MCP tasks blocked on iPad Air 5 over USB

OpenScottH1122opened 12d ago

What you want to do

Qualify the same safe Mac Poki MCP tasks on an iPad Air 5 as on an iPhone: read the foreground test app, identify its controls through AX, and retain native field/focus confirmation for text input. This is a request to align the supported iPadOS path before changing native control semantics.

What happens today

On one owner's single-display iPad Air 5 over USB, screenshot capture, OCR, and portrait coordinate input work, but native AX-backed tasks are blocked in the tested configuration.

Tested with main 1911231 and 834a433 (the relevant native AX reader is unchanged), macOS 27.3, Xcode 27.2, Node 24.16.0 and iPadOS 27.3. The owner confirmed Trust, Developer Mode, UI Automation and an unlocked device. Only synthetic test apps were used.

  • list_elements with source: "accessibility" and get_foreground_app repeatedly reach Native accessibility tree did not arrive within 20 seconds.
  • The same screen is readable through explicit OCR; screenshot capture works. An OCR-targeted Increment changes the fixture's independent state file from count 0 to 1. OCR-targeted double tap and long press show the expected fixture states without changing its text/count file.
  • The failure persists after portrait/landscape read-only comparison, a fresh signed fixture install and launch, an exact per-device Mac helper restart, and an owner-reported device restart/UI Automation check. No landscape input was attempted.
  • An independent one-shot native AX request reproduces the failure. Metadata-only diagnostics receive AX updates with one root and one node; the root role is Other, sized to the current 820×1180-point portrait viewport. The reader's readiness predicate never becomes true. This is not evidence that no AX response arrived, nor proof of the underlying device-side cause.
  • Increasing only the per-subscription wait from 2 to 5 seconds, with the total budget unchanged, does not resolve it. That experimental change was not deployed as a fix.
  • The same AX failure occurs in both the universal UIKit fixture and a freshly signed Poki Gym running in iPhone compatibility mode. Gym OCR/capture work. Compatibility mode does not establish native iPad support.

Minimal reproduction after ordinary authorized USB preparation:

  1. Open the disposable local.pokimcp.fixture on the unlocked iPad in portrait.
  2. Call list_elements with { "device": "<IPAD_UDID>", "source": "accessibility" }.
  3. Compare with explicit source: "ocr" and take_screenshot on the same synthetic screen.
  4. For native isolation, run the built poki-native --udid <IPAD_UDID> with { "op": "ax_tree", "include_offscreen": false } on stdin.

An iPhone 16 Pro on the same host/source/USB setup returns usable AX and passes the synthetic field/gesture/alert/navigation tests. Real-model attempts were not accepted as end-to-end successes; their failures do not establish an additional iPad product defect.

Area

Poki MCP

Proposal

Please align whether to investigate the existing Mac native AX service on iPad first, or prioritize the separate standalone App/PokiCore path. The standalone Poki App was not installed or tested here, so these results do not establish a failure in that path.

Keep incomplete AX data and unanswered subscriptions distinguishable. Do not mark a one-node tree complete merely to remove the timeout, bypass physical field verification, or declare iPad supported from OCR success alone. Follow-up acceptance should cover both portrait orientations, both landscapes, viewport/reference invalidation and window/focus changes, using an iPad-capable version of the existing Gym rather than a separate benchmark framework.

Related #20 / #76 cover two-panel iPhone routing. This observation is from a single-display iPad with working capture and portrait HID; the diagnostic root is not identified as SleepLockScreen. This is maintainer alignment with real qualification evidence, not a claim that the root cause or a safe native fix is already established. No device identifiers, pairing material, screenshots or private runtime artifacts are attached.

Activity

5
ScottH112212d ago

Additional bounded qualification on the same 834a433 runtime: the computer-use look → do protocol does work on this iPad through OCR fallback. In one persistent MCP session, I explicitly selected the iPad, read the foreground synthetic fixture, used the latest unique numbered Increment reference, and sent exactly one tap. The independent fixture state file changed count 1 → 2 with text unchanged. look reported ocr and accessibility did not answer within 2500 ms (2.728 s); do returned without an error in 21.750 s. The iPhone control passed using accessibility (look 1.361 s, do 3.486 s).

This narrows the limitation: legacy AX/foreground reads and AX-dependent field verification remain blocked; basic OCR-backed CUA control is usable but slow in this sample. It is not a complete iPad qualification, a real-model Agent success, or standalone App evidence. I did not test text entry through CUA or weaken native field guards. This is one bounded action per device, not a performance benchmark.

ScottH112212d ago

R4 follow-up on unchanged main 834a433: metadata-only diagnostics show that the incomplete AX root PID belongs to the system ScreenshotServicesService, while the foreground synthetic fixture has a different PID. The owner confirmed there was no visible screenshot preview. With explicit owner approval, ending only that transient system process through devicectl (no elevation, no App reinstall or data reset) restored a complete native fixture AX tree in 1.75 s. Subsequent ordinary captures did not immediately reproduce the failure; standard MCP AX reads, verified Unicode field input, invalid-field batch stop, drag, pinch, swipe and scroll now pass on the same physical iPad in portrait.

This identifies the immediate stale AX target/service state in this setup, not the mechanism that originally caused it or a general automatic-recovery fix. The parser, completeness predicate and 20-second budget were unchanged; no singleton tree was accepted. I am not proposing automatic termination of system services. Raw protocol/device identifiers and screenshots remain private. iPad keyboard-toggle and orientation qualification are being kept separate from this AX recovery evidence.

JosssphZhou11d ago

We read both follow-ups, including the ScreenshotServicesService finding and the successful AX/field/gesture checks after that stale service was ended. This usefully separates a stale system AX target from an unanswered subscription; OCR success alone was not proof of full iPad support.

The current native reader retries subscriptions and requires tree.ready() before accepting a tree. It does not identify or recover this stale system-service state. Better diagnostics and a bounded, non-destructive recovery path belong in our code, but the initiating cause and a safe automatic recovery are not established by this one recovery. We should keep this issue open for that investigation, preserve the completeness and field-verification guards, and keep orientation/keyboard qualification separate. We will not turn system-service termination into automatic recovery on this evidence. If it recurs, sanitized service-role and foreground-role observations plus the triggering steps would help; no raw device identifiers or private screen contents are needed.

ScottH112211d ago

Thanks for the thoughtful review and guidance!

Agreed that the stale system-service state needs further investigation before considering any automatic recovery mechanism. I'll keep #119 open and continue investigating as part of the broader physical-device and iPadOS qualification work.

I'll focus on capturing the conditions that lead to the stale AX target, along with sanitized service-role and foreground-state evidence, while keeping the existing completeness and input-safety guarantees intact.

This will be an ongoing effort alongside other device and functionality testing, so progress may be incremental. I'll update this issue when I have new reproducible findings or a well-supported fix.

JosssphZhou8d ago

Thanks for the follow-up about ScreenshotServicesService on the iPad. Recovering after manually terminating the service is useful diagnostic evidence, but it is not a product fix. This issue should remain open while the trigger and a safe detection and recovery path are investigated. The display-routing PRs should not be treated as proof that this stale-service case is resolved.