#495

feat(ios): live voice on a Gemini key through a shared WebSocket voice layer

DraftJosssphZhouwants to merge intofrom

Summary of changes

Draft, handed over unfinished. The goal is live voice on a Gemini API key through Gemini Live, built on a WebSocket audio layer that other realtime voice models (Qwen Omni, GLM Realtime) can share later.

Done so far:

  • VoiceSocketLink is a provider-neutral live voice connection. It handles the WebSocket, microphone capture at 16 kHz PCM with the hardware echo canceller on, and 24 kHz PCM playback that a barge-in clears at once. The microphone stays open while Poki speaks, so the owner can interrupt. A provider plugs in through VoiceSocketAdapter.
  • GeminiVoiceAdapter is the first adapter. It translates Gemini Live messages into the OpenAI Realtime events VoiceModel already handles, so tool runs, approvals and transcripts go through the existing code.
  • Settings: Accounts › Gemini › Live Voice turns it on. Each live voice account keeps its own model list and choice (Gemini 3.8 Live by default).
  • OpenAI voice keeps working as before. An OpenAI key still wins unless the owner turns Gemini on, and a saved Mac voice bridge keeps talking until then.
  • The second commit fixes barge-ins (an interrupted turn ends with turnComplete while the owner still talks) and reports a rejected key from the close frame. It also moves the key from the URL to the x-goog-api-key header and stops forcing non-blocking tool calls. Debug simulator builds take -pokiGeminiLiveURL to talk to a local stand-in.

Motivation

Live voice needed an OpenAI API key. Many owners only have a free Google AI Studio key, and owners in mainland China need models they can reach without a proxy.

Test plan

  • Commit 1 was built for the simulator. GeminiVoiceTests and VoiceModelTests ran there: 25 tests, all passed.
  • Commit 2 is not built yet, because the build queue was held when it was committed. Build it first.
  • The two regression tests:
    • testGeminiToolCallTranslatesToVoiceModelExecutionAndSendsToolResponse: a Gemini tool call runs once in VoiceModel and its result goes back as a toolResponse. The earlier run reported it red before the adapter existed (not rerun here). It passes on commit 1.
    • testGeminiBargeInClearsPlaybackQueueAndMarksTalkInterrupted: a barge-in clears queued playback and the talk shows Interrupted. The earlier run reported it red before the adapter existed (not rerun here). It passes on commit 1. Commit 2 adds the trailing turnComplete to it. The commit 2 version has not run yet, and it should fail on commit 1.
  • Commit 2 adds testARejectedGeminiKeySaysToCheckItAndTheKeyTravelsOnlyInAHeader. It has not run yet.
  • I probed the real endpoint with a fake key. With the key in the x-goog-api-key header or in ?key=, it closes with 1007, "API key not valid". With no key, it closes with 1008.
  • Not tested: a conversation with a real Gemini key, the simulator end-to-end run, and a physical iPhone.

Still to do

  1. Build commit 2 and run GeminiVoiceTests, VoiceModelTests, VoiceLanguageTests and ConnectionCheckTests. Fix whatever does not compile.
  2. With a real Gemini key on a simulator: say "open Settings", and Poki opens Settings and answers out loud. Then speak over Poki and check that it stops and listens.
  3. Settle the TODOs that need a real key:
    • context notes sent as clientContent with turnComplete: false
    • speech start inferred from the first transcription chunk
    • Extended Thinking's interactionStatus
  4. Restart the audio engine after a route change or an audio session interruption (TODO in VoiceSocketLink).
  5. Check on a physical iPhone, then add the next adapter on VoiceSocketAdapter.

Screenshots or screen recording

N/A for now. The live voice runs above are still to do.

AI assistance: Claude Code. I have reviewed and understand these changes.

  • I removed passcodes, tokens, and private screen contents from this PR.
  • I release this contribution under the AGPL-3.0-only license.

🤖 Generated with Claude Code

Activity

3
JosssphZhou3d ago

@ZhehanZhang as discussed in #484, this draft is yours to take over (multiple live voice models over a shared WebSocket layer). Reuse whatever helps; we will not push further commits here. It still needs a Gemini test key to try end to end.

This pull request is a draft

Merging is unavailable until it is marked ready for review.