#495
feat(ios): live voice on a Gemini key through a shared WebSocket voice layer
Summary of changes
Draft, handed over unfinished. The goal is live voice on a Gemini API key through Gemini Live, built on a WebSocket audio layer that other realtime voice models (Qwen Omni, GLM Realtime) can share later.
Done so far:
VoiceSocketLinkis a provider-neutral live voice connection. It handles the WebSocket, microphone capture at 16 kHz PCM with the hardware echo canceller on, and 24 kHz PCM playback that a barge-in clears at once. The microphone stays open while Poki speaks, so the owner can interrupt. A provider plugs in throughVoiceSocketAdapter.GeminiVoiceAdapteris the first adapter. It translates Gemini Live messages into the OpenAI Realtime eventsVoiceModelalready handles, so tool runs, approvals and transcripts go through the existing code.- Settings: Accounts › Gemini › Live Voice turns it on. Each live voice account keeps its own model list and choice (Gemini 3.8 Live by default).
- OpenAI voice keeps working as before. An OpenAI key still wins unless the owner turns Gemini on, and a saved Mac voice bridge keeps talking until then.
- The second commit fixes barge-ins (an
interruptedturn ends withturnCompletewhile the owner still talks) and reports a rejected key from the close frame. It also moves the key from the URL to thex-goog-api-keyheader and stops forcing non-blocking tool calls. Debug simulator builds take-pokiGeminiLiveURLto talk to a local stand-in.
Motivation
Live voice needed an OpenAI API key. Many owners only have a free Google AI Studio key, and owners in mainland China need models they can reach without a proxy.
Test plan
- Commit 1 was built for the simulator.
GeminiVoiceTestsandVoiceModelTestsran there: 25 tests, all passed. - Commit 2 is not built yet, because the build queue was held when it was committed. Build it first.
- The two regression tests:
testGeminiToolCallTranslatesToVoiceModelExecutionAndSendsToolResponse: a Gemini tool call runs once inVoiceModeland its result goes back as atoolResponse. The earlier run reported it red before the adapter existed (not rerun here). It passes on commit 1.testGeminiBargeInClearsPlaybackQueueAndMarksTalkInterrupted: a barge-in clears queued playback and the talk shows Interrupted. The earlier run reported it red before the adapter existed (not rerun here). It passes on commit 1. Commit 2 adds the trailingturnCompleteto it. The commit 2 version has not run yet, and it should fail on commit 1.
- Commit 2 adds
testARejectedGeminiKeySaysToCheckItAndTheKeyTravelsOnlyInAHeader. It has not run yet. - I probed the real endpoint with a fake key. With the key in the
x-goog-api-keyheader or in?key=, it closes with 1007, "API key not valid". With no key, it closes with 1008. - Not tested: a conversation with a real Gemini key, the simulator end-to-end run, and a physical iPhone.
Still to do
- Build commit 2 and run
GeminiVoiceTests,VoiceModelTests,VoiceLanguageTestsandConnectionCheckTests. Fix whatever does not compile. - With a real Gemini key on a simulator: say "open Settings", and Poki opens Settings and answers out loud. Then speak over Poki and check that it stops and listens.
- Settle the TODOs that need a real key:
- context notes sent as
clientContentwithturnComplete: false - speech start inferred from the first transcription chunk
- Extended Thinking's
interactionStatus
- context notes sent as
- Restart the audio engine after a route change or an audio session interruption (TODO in
VoiceSocketLink). - Check on a physical iPhone, then add the next adapter on
VoiceSocketAdapter.
Screenshots or screen recording
N/A for now. The live voice runs above are still to do.
AI assistance: Claude Code. I have reviewed and understand these changes.
- I removed passcodes, tokens, and private screen contents from this PR.
- I release this contribution under the AGPL-3.0-only license.
🤖 Generated with Claude Code
Activity
3This pull request is a draft
Merging is unavailable until it is marked ready for review.
@ZhehanZhang as discussed in #484, this draft is yours to take over (multiple live voice models over a shared WebSocket layer). Reuse whatever helps; we will not push further commits here. It still needs a Gemini test key to try end to end.