#356
iPhone history: let Poki observe the screen without the automation watermark
What you want to do
Let Poki watch what happens on my iPhone while it is not operating it, so that it can later:
- learn from how I do things myself (watch me do a task once, then offer it as a routine),
- keep an iPhone history I can ask about ("what was that address I looked at this morning?"), in the spirit of ChatGPT's computer history and the old Rewind app.
This must not show iOS's "Automation Running" watermark, since nothing is being automated, and it must be fully under the owner's control.
What happens today
Poki sees the screen only while an agent is working, and on the in-process path every read goes through XCTest's Automation Mode:
- From the iOS 27 developer disk image (build 27B52),
testmanagerdchecks Automation Mode before every snapshot, attribute read and element lookup, and the call that loads accessibility turns Automation Mode on itself. So the element tree cannot be read in-process with the watermark off. - The work on
dev/automation-session-rebindgoes the other way on purpose: the session, and the watermark, exist only while a task runs.
There is a second way to the screen that does not use XCTest at all, Apple's developer services reached over the loopback and the pairing. Poki's bridge code (native/bridge) already talks to three of them:
| Service | What it gives | Measured (iPhone 17 Pro, Nov 6 2026) | Cost |
|---|---|---|---|
| Screenshot service | One picture on request | about 0.8 s each once warm | Low |
| Screen video (display service media session) | Continuous HEVC, about 59 units a second even on a still screen | picture age median 36 ms | Holds a screen sharing session: the camera is unavailable to every app and apps that restrict screen sharing stop working. Decoding takes over one CPU core |
Accessibility audit service (axAuditDaemon.remoteAXService) | The element tree: text and structure | not measured for this use | Low |
Not known yet:
- Whether iOS 27 shows any indicator while the screenshot or accessibility services are used. Only the video session's side effects are recorded.
- Whether these can run on the phone itself, in the background, for hours. The bridge was written for the Mac side, and iOS suspends Poki in the background.
- Battery and storage cost of sampling all day.
Area
Poki iPhone app or Poki Setup
Proposal
A dream for later, not a plan for the next release. No pull request is planned yet.
- Loopback as the off switch. These services need the loopback and the pairing. That is a feature here: with the loopback off, Poki cannot observe anything, whatever its own settings say. The owner can block the whole feature from outside Poki, in their proxy app or by turning Poki's tunnel off, and can see that it is blocked.
- Sample, do not stream. Take a screenshot and an element tree when the app in front changes and every few seconds while the screen changes. No video session: it blocks the camera and breaks apps that restrict screen sharing.
- Text first. Keep the element tree's text and structure as the searchable record, and pictures only where the tree is empty (games, video, canvases) or for a short time.
- Private by design. Off by default. Everything stays on the iPhone. A clear pause control and a visible sign in Poki while observing. Never record secure text fields.
- Owner controls. Poki knows which app is in front before it samples, so each of these is a check made before anything is captured, not a filter applied afterwards:
- Excluded apps: a list the owner picks from their installed apps, with banking, passwords and messages suggested. Nothing is captured while one is in front. An allow list (only these apps) as the stricter choice.
- Access times: a schedule (for example weekdays 9 to 6) and a pause for an hour, until tomorrow, or until turned on again.
- Retention: keep history for a day, a week, a month, with deletion of any span or any one app's history.
- A plain record of when Poki was observing, so the owner can check it kept to the rules.
- Two uses on the same record.
- History: ask Poki about something seen earlier, answered from the stored text.
- Learning: Poki notices a sequence the owner repeats and offers to save it as a routine, with the owner confirming.
- First step. A Testing Options button that takes one screenshot and one element tree through these services with Automation Mode off, and records what iOS shows (any indicator, any prompt) and how long each takes. That answers whether the idea is possible on iOS 27 before anything else is built.
Thanks for the thorough proposal, and for measuring the services on a real iPhone.
This stays open as an idea for later. It is not planned for a particular release. The first experiments are in draft #360: it reads the element tree through the accessibility audit service with no automation session and nothing drawn on screen, and adds an "iPhone History" section to Testing Options to time it. The first results from an iPhone 17 Pro on the iOS 27.3 beta are in that pull request.
The open questions from your issue still decide whether the rest is worth building: whether iOS 27 shows any indicator for these services, whether sampling can keep running in the background for hours, and the battery and storage cost. If you can add measurements for any of them (iPhone model, iOS version, Poki build), please post them here or in #360.