Summary of changes
Live voice now only listens and speaks. Work on the phone goes to Poki's chat agent, the same model, harness, approvals and Live Activity as a typed message.
voice model ── run_task {task} ──▶ ChatModel.runSpoken ──▶ pi loop (look, do, report_plan, finish_task)
◀─ result when the task ends ─┘
── run_task while running ──▶ ChatModel.addSpoken ──▶ PiLoop.steer (read before the next turn)
── check_task ─────────────▶ plan, step under way, approval waiting
── stop_task ──────────────▶ ChatModel.stop (as Stop in the chat or on the Live Activity)- The Realtime session no longer offers
do. It keepslookandget_ios_version(read-only) and gainsrun_task,check_taskandstop_task(Voice/VoiceTasks.swift). The Mac voice bridge's default tool set changes the same way;--mac-toolsis unchanged. - The
run_taskcall stays open until the chat's task ends, so the talk shows Acting and does not end on its idle timer while the task runs. - The task stands in the chat as a grey step line, not as the owner's message, so the approval check does not read the voice model's wording as the owner's request. The owner's own transcribed words still reach it:
AppModelpasses the talk's lines toApprovalClassifier.decide, which adds them asOwner (live call). - Words added during a task are steered into the running pi loop; the steering queue is cleared when the task ends, so nothing leaks into the next task.
- Stop, approvals and the Live Activity follow the chat's task, because it is the chat's task.
Live voice in Chinese on a physical iPhone
A first talk on a physical iPhone in Chinese was close to unusable: a short English sound as the talk opened turned it to English, "open WeChat" took 27 s, a mis-heard "scroll down" was taken for "translate", and a backchannel cut Poki off mid-sentence. The talk now follows these rules:
- Language: it hears and speaks the iPhone's language, not the one chosen in Poki's settings.
gpt-transcribeis told that language and English (for app names and words like "hello"), and nothing the tools return changes it. To talk English, set the iPhone to English. - Opening an app:
open_appis offered again. It launches the app by name on the phone (VoiceApps) without Poki's agent. The 27 s were about 1 s of voice and 26 s of the agent (13 s of model turns and 13 s on the phone, 4.5 s of that a WeChat element-tree timeout). The launch is now logged, so the time from the end of speech to the app on screen can be read off the log. - Replies: a clear request gets one word ("OK") and the step. A message to someone is restated in one sentence and sent without waiting for a yes. Poki says nothing while a task runs and answers how it is going only when asked. "Stop" stops the running task.
wait_for_userlets the model end a turn on a backchannel or a cough without answering. - Backchannels: over WebRTC the server drops the rest of an answer at any speech it detects, with
interrupt_response: falsetoo (measured). SoVoiceModelturns speech detection off (session.update,turn_detection: null) fromresponse.createduntil the answer's audio has played, then restores the session's own setting. A backchannel no longer cuts an answer, and an answer that is only a tool call turns detection back on at once. The cost: speech over an answer is not heard. Answers are one short sentence, and "stop" during a task works. - Follow-up: after the answer the talk waits 15 s, not 8, for the person to go on.
- Mac bridge:
bin/voice-bridge.mjscarries the same instructions and tools word for word.
Motivation (no linked issue)
On a physical iPhone the realtime model drove look and do itself and was less precise than the chat agent. Since the voice model calls tools on the phone directly (since "Voice acts on the phone itself"), every task depended on a speech model operating the UI.
Test plan
Live voice in Chinese (this update):
-
native/ios-app/scripts/voice-eval.pyplays spoken Chinese (macOSsay, nothing played aloud) into realgpt-realtime-2.1sessions over WebRTC. The sessions are minted by the voice bridge from this branch. The client toggles speech detection asVoiceModeldoes, and the tools are stubbed with English results. Final run, 4 runs each:- "open WeChat":
open_appwith WeChat and "WeChat is open" in Chinese, 4/4. One word before the step, 3/4: one run skipped it. - Then "scroll down":
run_task, no translation, no English, 4/4. - "Send hello to File Transfer": restated in one Chinese sentence,
run_taskin the same turn without waiting for a yes, 4/4. - "Stop" while that task runs:
stop_task, 4/4. - A backchannel 0.3 s into an answer: the answer plays to its end, is not said again and stays in Chinese, 4/4.
- A short English sound as the talk opens, then "open my WeChat": every answer in Chinese, 4/4.
- "Scroll a little" (heard as "translate" before): scroll, 4/4.
- "open WeChat":
-
The same eval on the code before this update: the opening English sound gave English answers in 3 of 4 runs. "Scroll a little" was taken for translate in 4 of 4, and "scroll" in 3 of 4. A backchannel cut the answer in 4 of 4.
-
Unit tests that fail on the code before this update, in
VoiceModelTests.swift:VoiceLanguageTests.testTheLiveSessionHearsAndSpeaksTheIPhonesLanguageAndOpensAppsItself: the session's instructions, tools, local tools and transcription.testSpeechDetectionIsOffWhilePokiAnswers.testASoundWithNoRequestInItIsLeftUnanswered.- The follow-up tests, now at 15 s.
-
Two existing test files did not compile against this branch (a phone answer now carries a screenshot); both are fixed here.
-
Simulator (iPhone 17 Pro):
VoiceLanguageTests,VoiceModelTestsandChatHarnessTestspass, 61 tests. The new and changed tests fail on the code before this update (26 tests, 27 failures). -
Not yet: a run on the physical iPhone.
-
Simulator (iOS 27.0), PokiTests injected into Poki: all of
VoiceModelTests(21 passed), one existing chat harness case, and two newChatHarnessTestscases, which pass:testATalksTaskRunsInPokisChatTakesWhatIsAddedAndReportsItsEnd: a talk'srun_taskruns as one chat task on a local OpenAI-compatible fake; an addition during the task reaches the model's next request; the result reaches the voice only after the task ends; the task is not owner context for approvals.testStopInATalkStopsTheChatsTaskAndTheTalkHearsItStopped.
-
Deliberate regressions after the change: with
run_taskno longer handed to the chat, both new cases fail; with the handed-over task standing in the chat as the owner's message, the first fails. (Before the change the cases do not compile: there is norun_task.) -
The handoff has not yet run end to end in the app with a real model; the cases above use a local fake model and a fake Realtime connection.
-
Real Realtime API (
gpt-realtime-2.1) with this session's instructions and tools, typed user turns, three runs: the model calledrun_taskfor "open WeChat and send hello to File Transfer",run_taskagain for a change while the first was open,check_taskfor progress andstop_taskfor stop, 3 of 3. -
node --test test/voice-bridge.test.mjspasses. -
Not tested on a physical iPhone. On the test iPhone (iPhone 16 Pro, iOS 27.3) a typed task on the current build stopped at iOS's passcode prompt for UI Automation, which only the owner can answer, and the phone's microphone did not pick up speech played from the Mac. The before/after comparison of one spoken task is still to be recorded.
Screenshots or screen recording
N/A: the device comparison is still to be recorded (see Test plan).
AI assistance: Claude Code. I have reviewed and understand these changes.
- I removed passcodes, tokens, and private screen contents from this PR.
- I release this contribution under the AGPL-3.0-only license.
🤖 Generated with Claude Code
Activity
9654d4a0