#484

Work in progress and plan, Nov 18

OpenJosssphZhouopened 3d ago

This is a snapshot of what we are working on and what comes next, written so you can pick up the parts that suit you. Status was checked against GitHub on Nov 18, evening.

1. Shipping tonight

Release approach: cut an Alpha first so the maintainers can try it on a real iPhone, then promote it to Beta.

Going in:

  • Screen History feeds Poki's memory, so a vague request such as "play a song" picks the app the user usually uses. This is #435 (the Screen History query tool and top-level settings) followed by #465 (learning the usual music app). Both are open, #435 must merge first, then #465.
  • Result cards, first card: the 12306 train list, plus the card behavior shared by all apps. Branch dev/poki-result-card-12306, pushed, no PR yet. It is a work-in-progress commit that adds the shared card, the card view and RailwayTrains.swift.
  • Daily tasks, #480. It only goes in if the second review passes.

Already merged into main today:

  • #450: sign-in asks for at most one verification code and stops on rate limits
  • #477: talk to Poki from the Action Button without leaving the current app
  • #448: keep WeChat mini program names in task text
  • #453: refresh home shortcuts when their source content changes
  • #476: glossary of Poki product terms
  • #469: contributor levels and when to open an issue first
  • #470: regrouped labels, reworked issue forms and PR template
  • #454 and #455: test fixture updates (iOS startup locale, macOS notification and signing)

2. In progress

  • Gemini Live realtime voice, so people without an OpenAI account can use voice. Branch dev/poki-gemini-live-voice (local only so far, nothing pushed yet).
  • USB setup wizard in the graphical Poki Setup, including guided steps for Developer Mode and the UI automation switch, so a user without an agent can install Poki over a cable. Branch dev/poki-usb-setup-wizard, one WIP scaffold commit (87dfa68d), not mergeable yet.
  • Loopback on demand: after install, the loopback connection is switched on and off automatically only on restart and on signing renewal. Branch dev/poki-loopback-on-demand, not started.
  • Memory page: reorganize it by app ("About you", an app list, "Recently updated"), each entry showing its basis and being editable and deletable. Design is in review, build is in progress. No PR yet.
  • README rewrite across all six languages: #473, open, review requested.
  • External PRs, all still waiting for review:
    • #446 (draft): revoke in-flight Screen History queries on privacy changes. It does not block tonight.
    • #449: recognize NetEase Cloud Music short names.
    • #474: use the active iPhone Duo panel for screenshots and taps after folding or rotation.
    • #460: defer phone automation until a chat uses phone tools.

3. Planned next, and where help is welcome

Result cards for the other 14 apps (best place to jump in)

After the 12306 card, 14 apps are left: WeChat, X, Taobao, JD, Pinduoduo, Xianyu, Meituan Waimai, Dianping, Xiaohongshu, Amap, Baidu Maps, Didi, Ctrip and iOS Calendar.

Once the shared part of the 12306 card branch is merged into main, these can be built in parallel: one card per app, one file per app, next to RailwayTrains.swift. The look of each card is rebuilt from real screenshots of that app on a test phone, so each one feels like it ships with the app.

Please reply here with the apps you would like to take, so we do not collide.

Design rules, in short:

  • The card has exactly one piece of state: the selected item, defaulting to Poki's recommendation. Button labels and actions are derived from it.
  • One primary button per card. At most one next-step action, as a native blue text button, and none if there is no clear next step.
  • Tapping a button is the same as the user sending one sentence in the chat (for example "send it to the work travel group"). After that the normal task flow handles progress, approvals and the Live Activity. The card never shows "sending" or "sent".
  • "View" opens the app page directly when a deep link exists. Otherwise it sends a sentence and Poki opens it.
  • Buttons are disabled while a task is running, derived from the existing task state.
  • Cards only appear when there are two or more items to choose between. A single result is one sentence of text plus one native button such as "View in Taobao", with no card.
  • No promo badges, no deep-link capsule in the corner, no payment actions, and no features we cannot do yet (such as price alerts).
  • If the card data is malformed or missing, fall back to a plain text reply with the full result.
  • The selected item is saved with the conversation and survives reopening it. Old cards stay tappable.
  • Anything the other app already shows live, such as a driver's position, gets no card.
  • Native SwiftUI controls, verified in the simulator.

Known Screen History issues shipping with tonight's build

These do not block tonight and are logged for a later fix:

  1. Collection only starts from a manual one-time read or a time-limited read in the test options. Turning on the two switches alone does not start collection, so everyday use does not teach Poki anything.
  2. The word "listen" inside a dictation request is treated as a music request, so a learned music hint can be attached to it.
  3. "Play some music" can carry both "the user named Apple Music" and "the inferred usual app", so the model may open Apple Music.
  4. A saved note that mentions a music app name, or contains substrings such as "app" or "use", permanently blocks the inference (substring matching).
  5. When the usage summary becomes empty, an old conclusion never expires, for example after 30 days without a music app or after excluding it.
  6. Screen History query answers are not asked to summarize first and use local time, and the first song request of each day waits for one extra model round trip.

If you want to take any of these, say so in a comment before starting.

@ZhehanZhang

Activity

8
JosssphZhou3d ago

@ZhehanZhang here is where we are after tonight, and a few places where your help would make the biggest difference.

Tonight's progress

  • Alpha 0.4.5 (1735) is out.
  • Result cards: the shared card behavior and the 12306 train list are merged (#486).
  • Poki Setup no longer spins after the verification code is accepted. The fix is in review (#494). Once it passes review, Poki Setup 0.2.6 for Mac goes out tonight.
  • Open PRs: we are only reviewing them and listing what each one needs. Nothing from external contributors has been touched.
  • Realtime voice fixes in progress: the Dynamic Island no longer says the completion twice, the action button never jumps back into Poki, voice can see the screen, the opening line follows the interface language, replies are shorter, apps open directly by name, and more voice models.
  • The chat input box closes when you tap an empty area.
  • The shortcut is renamed to Poki Anytime.

Ask: result card designs for more apps

The idea: when a task ends with two or more options from one app, Poki shows them as a card that looks like that app's own screen. The card keeps one state (the selected item), has one primary button and at most one next step, and a button tap simply sends one sentence as the user's message. The full rules and the card interface are in the design README.

Reference files, on the branch design/result-cards-reference (not a PR, nothing to merge):

Could you take this as the reference and design the matching cards for other everyday apps, for example food delivery, ride hailing, maps, calendar and music? Rough sketches pasted in this issue or a draft PR are both fine, and we will pick them up and build them. Please look at each app on a real phone first, so the card feels native to that app.

The card interface that exists today (merged in #486):

  • finish_task takes an optional card with app, kind, items, recommended, primary and an optional next. Each item needs id and title; the other fields belong to that app's look.
  • A button's label and say use {field} placeholders that are filled from the selected item, for example "say": "Open {title} in the app".
  • A new look is one SwiftUI view plus one entry in ResultCardLooks.all. Files under native/ios-app/Poki/Chat/: ResultCard/ResultCard.swift (model and looks table), ResultCard/ResultCardView.swift (shared shell and buttons), ResultCard/RailwayTrains.swift (the 12306 look, the template to copy), and Harness/Harness.swift (the finish_task schema and the model's instructions).

Ask: more help

  1. Fresh-account sign-in test. Once Poki Setup 0.2.6 is out (the fix is #494), please run it from the start on your own Mac with an Apple account that has never signed in to Poki Setup, covering sign-in and install. If you have a Windows computer, please do the same there. A fresh account is hard for us to get, so this is where we need you most. If anything goes wrong, paste the exact error text from Poki Setup into this issue.
  2. External PRs. About half of the open PRs come from outside contributors, and tonight we only listed them. Please pick a few you know well, reply to the author and do a first review. If you think one can merge, say so on the PR and we will review it again and merge.
  3. Your own PRs. Our review comments are already on #331 and #464. Please update them following the comments and push, and we will re-review the same day. #464 is currently closed, so tell us whether you want to reopen it or carry it on in another PR.

What we will do next

We will finish the voice fixes and release 0.2.6, then build the first cards for the next apps from whatever designs you post here.

JosssphZhou3d ago

@ZhehanZhang one more ask, to add to the list above and to put first.

Ask: recordings of Poki in English apps

We need footage for the website, the promo video on X and the animated images in the repo README. Could you record as many real scenes as you can of Poki operating English-language apps, on an iPhone whose system language is English?

  • One task per recording. Start when the request is spoken or typed, and end when Poki has finished and the result is on screen. 15 to 45 seconds each.
  • Scene ideas, pick the ones you can do: ordering food (Uber Eats, DoorDash), ride hailing (Uber, Lyft), map navigation (Google Maps, Apple Maps), adding a calendar event, playing a song in Spotify or Apple Music, sending a message (iMessage, WhatsApp), shopping (Amazon), turning a setting on or off.
  • Use the built-in iOS screen recording, portrait, original quality, with sound on. For realtime voice, the recording must pick up the spoken request.
  • Before recording, clear notifications and hide private information: no real names, phone numbers, home addresses, payment details or private chats. Use a test account where you can.
  • With each recording, add one line: which app, what was asked, how long Poki took, and whether anything went wrong.
  • Please do not upload large files to GitHub. Paste a cloud drive or Slack link in this issue.
JosssphZhou3d ago

@ZhehanZhang one more note for tonight: Joseph has gone to sleep. It has been a very long couple of days for us, so thank you for picking these up, we really appreciate it.

While he sleeps, the main agent keeps going through the night: reviewing and merging the sign-in fix (#494) so Poki Setup 0.2.6 can ship, and working through the open PRs. Anything you push tonight will get a review in the morning at the latest.

If you want a faster answer, ask the Poki agent in our Slack workspace once it is online there. We will post here when it is ready.

ZhehanZhang3d ago

Plan for today: JosssphZhou/poki#331 (open apps by name) and JosssphZhou/poki#350 (links ask first) passed on my phone, and recordings are coming. Fast Decisions is now chosen per account and in Local Models across JosssphZhou/poki#490, JosssphZhou/poki#463 and JosssphZhou/poki#487, with one combined test build next. I’m also fixing review findings in JosssphZhou/poki#436, JosssphZhou/poki#475 and JosssphZhou/poki#486.

On this issue: loopback on demand would leave Screen History nothing to read, since it reads over the loopback. I’d like to claim known issue 1 (collection only starts by hand) and design it together with loopback on demand. JosssphZhou/poki#464 is folded into JosssphZhou/poki#331, so no reopen.

JosssphZhou3d ago

@ZhehanZhang two bigger pieces we would love you to own, since they need deep iOS digging and we are running low on agent quota:

1. Screen History that keeps recording. Today Screen History only reads for a short run (manual start, one read every 5 seconds for a few minutes) and stops when the phone locks, when an agent task opens a session, or when iOS ends Poki's background time. Joseph turned it on, used other apps for a few minutes, and Poki said it saw nothing. The goal: once it is on, it keeps recording while the phone is unlocked, without a manual start, and pauses cleanly during a task instead of stopping for good. The code is in native/ios-app/Poki/History/HistoryCapture.swift. The hard part is iOS background time; please start with a short note in this issue on which approach you think works (and what it costs in battery and permissions) before building.

2. More voice models for realtime voice. We started Gemini Live support plus a shared audio layer that other voice models (including Chinese ones) can plug into. The work in progress is on branch dev/poki-gemini-live-voice; a draft PR with what is done and what is left will be up shortly. Please take it from there.

Smaller voice fixes (Dynamic Island showing "Done" twice, stop button, approving without leaving the current app, voice handing the actual phone work to the main agent) are already in progress on our side, so no need to touch those.

JosssphZhou3d ago

@ZhehanZhang these are our notes on making install easier, for reference since you are working on it; use whatever helps, ignore the rest. Checked against main and open PRs on Nov 19. "Verified" means we read the code on main; "unverified" means we did not run it or could not confirm it.

Where a new user gets stuck today

  • The README and install guide lead with "send this link to Claude Code or Codex". Poki Setup is the no-agent path but is not first.
  • Windows without Apple Devices or iTunes: the driver message no longer contains "usbmuxd", but phones() in commands.rs only reports the driver note when it does, so the device list likely stays blank (verified by reading, not run on Windows).
  • Poki Setup copies the disk image into Poki (seed_ddi) but never mounts it on the phone. device::ddi::mount is only called from the "is Poki running" check (verified).
  • First open on the iPhone: startSession, prepareForTask and canStartFromHome all require setup.loopback != .none (verified). This is where #35, #69, #191, #377, #382 and #397 land: a proxy that answers on 62078 while the remote pairing port refuses, or a 10.7.0.1 placeholder that was never saved.
  • After a restart the image must be mounted again, and every 7 days the certificate renews. SelfSigner.canPrepare needs phone() != nil, so even preparing the next build waits for the loopback (verified; whether the Apple requests work without it is unverified).

Options we looked at

  • USB install, mount the image from the computer. Cheapest first step. Daily automation in process does not use 10.7.0.1 (no loopback address in InProcess.m or InProcessWDA.swift; verified by reading). Cost: the image is gone after a phone restart. Gap on main: ddi::mount and mounted only handle the personalized image, there is no Cryptex1 branch for iOS 26.4 and later (verified). Whether idevice can do the cryptexd install from the computer is unverified; the launcher does it on the phone with idevice::cryptexd::install_ddi.
  • Same-Wi-Fi computer remounts and renews. Poki Setup already reaches phones over Wi-Fi (Phone::over_wifi, renew_due). A new Bonjour service may not be needed. Risks: AP isolation, Windows firewall, changing DHCP addresses (unverified on real networks).
  • LocalDevVPN on for a few seconds only (App Intents or a Shortcut), instead of asking people to edit proxy rules: https://github.com/jkcoxson/LocalDevVPN. Unverified, needs a real phone with a proxy app.
  • Keep PokiTunnel for paid accounts. sign/mod.rs drops the packet tunnel extension for everyone. For free accounts this matches Apple's capability list (https://developer.apple.com/support/app-capabilities/, we could not read the table cells, so unverified). Paid accounts: untested.
  • Not workable: TestFlight, App Store, EU marketplaces, enterprise certificates, because of guidelines 2.5.1 and 2.5.2 (https://developer.apple.com/app-store/review/guidelines/#software-requirements); the debug entitlement and disk image needs are unverified on our side.
  • iloader writes the pairing file to /Documents/ and has no Poki entry in its app list (#35); its Windows driver error cards are a good model: https://github.com/nab138/iloader (MIT).

Relation to your PRs and issues

  • Already in main from you: #441, #227, #357, #379, #403, #440. #350 (links ask before changing relay or pairing file) touches the same hand-off.
  • Possible overlap: your note on this issue about loopback on demand together with Screen History collection. History reads through the loopback today (HistoryCapture requires loopback.isWorking), so a "loopback only on restart and renewal" design must decide what History does. We have not started it, so it is yours if you want it.
  • Not install related: #490, #489, #488, #487, #468.
  • Close but not yours: #478 (loopback prompt while the mounted image takes tasks), #291 (sessions on cellular), #456 (Chinese in the Mac Poki Setup; an outside contributor was told to go ahead).

Our two branches

  • dev/poki-usb-setup-wizard: pushed, no PR, one commit (87dfa68d, Nov 18, marked do not merge). Scaffold with failing tests and stubs: wizard state machine, Windows and Mac step screens, virtual adapter filtering in pairshare.rs, default image mount in setup.rs, iloader style error cards (MIT notice added), a large Mac Chinese strings file. About 7,200 lines over 34 files.
  • dev/poki-loopback-on-demand: not pushed yet, no commits, not started.

Not solved yet

  • Restart on pure cellular with no Wi-Fi: iOS refuses the loopback there (#35), so one Wi-Fi moment is needed to mount the image.
  • Apple sign-in loops (#135, #445) and the lost passcode notice (#320, #377).
  • Whether a stranger can finish the whole install alone. Nobody has measured it.
  • Proxy apps that make the loopback look healthy while the remote pairing port refuses. A real handshake check would catch it; how many apps do this is unverified.
ZhehanZhang2d ago

@JosssphZhou an update on the install notes above. The Poki Setup part is tested and works.

Poki Setup, tested

#509, ready for review. I tested a notarized Mac build (0.2.7, build 1752) on my iPhone, and install works. All CI checks pass.

  • Windows without Apple Devices or iTunes. The device list no longer stays blank. You were right that phones() matched on "usbmuxd" and the driver message no longer contained it. A missing device service is now its own error, Error::DeviceService, with kind usbmuxd, so nothing depends on the wording. Windows shows a card that explains the driver, with Get Apple Devices (Microsoft Store) and Download iTunes. Any other device-list error now shows its text too. This is not run on Windows yet.
  • The disk image is mounted during install. After seeding the image into Poki, Setup now mounts it from the computer.
    • iOS 26.4 and later get the Cryptex1 cryptex through cryptexd over the CoreDevice tunnel. It is the same install_ddi call the launcher makes on the phone, so idevice can do it from the computer.
    • ddi::mounted reads the CoreDevice service list, so it also sees a cryptex.
    • If the mount fails, the install still finishes with a note, and Poki mounts the image itself.

Already in 0.2.6 and 0.2.7 through #379: the Developer section (GitHub account, test builds from Actions artifacts, auto sync), Devices instead of iPhones, and the install progress bar. It also has the check before an update that Poki is closed on the device. If Poki is running, Setup asks to close it and closes it through CoreDevice's app service. Test Builds has a switch to close it without asking.

Loopback and Screen History

All of this is merged into dev/screen-history-always-on. Nothing here is compiled yet. The first build of that branch will tell.

  • Remount after a restart. The accessibility audit service comes with the disk image, so after a restart Screen History read nothing until a task mounted the image again. Now, when a History tunnel finds the image missing, it mounts it from the copy Poki already has and opens a new tunnel. The supervisor and History never mount at the same time.
  • A prompt when Poki opens. If Screen History is on and the loopback is off, a small sheet says when Poki last read and offers one step: open the proxy app chosen in Setup, or open Setup. On cellular it offers the reconnect shortcut if Setup has not offered it yet. Not Now hides it for an hour.
  • Poki's own connection is no longer offered. Enable Local Connection is gone from the loopback step. An iPhone that still uses it only sees its status and a way to move to its proxy app.
  • The cellular shortcut is offered in the same Setup step as the loopback. This is my rule now: loopback on cellular is set up together with the loopback.

Your other notes, not started

These need a real phone first: remounting over the same Wi-Fi from the computer, LocalDevVPN on for a few seconds, keeping PokiTunnel for paid accounts, and a real handshake check against proxy apps that only look healthy.

The fresh-account sign-in test of 0.2.7 is still on me, because I need an Apple account that has never signed in to Poki Setup. I'll post the result here.

JosssphZhou2d ago

@ZhehanZhang thank you for the update, and for testing the notarized 0.2.7 build on a real iPhone.

#509 is queued for review now. #500 is queued too: you merged main into it overnight, and we had said we would review it as soon as it was current. We will post the findings on each pull request.

The loopback and Screen History work on dev/screen-history-always-on has no pull request yet, so we have not looked at it. Take your time with the first build and tell us what it shows.