#513
feat(ios): show connecting and failed voice states on the island and play a ready cue
Summary of changes
- Extend
PokiState.Voicewith.connectingand.failedso the Live Activity and Dynamic Island reflect the real lifecycle of a voice session instead of showingListening(聆听中) before the Realtime session is established. - While a voice session is exchanging its ephemeral token, negotiating WebRTC, and waiting for the Realtime API's
session.createdevent, displayConnecting(连接中/連線中) with the waiting bear pose (.thinking) and a quiet compact waveform on the Dynamic Island, Lock Screen, and in-appTalkBar(including background starts fromTalkHereIntent). - Transition to
Listening(聆听中) and triggerVoiceReadyCueat the exact momentVoiceModelreceivessession.createdfrom the Realtime API (wasConnecting && holdingMicrophone), never by a fixed delay:- Synthesize a short, soft two-note PCM chime (180 ms, 587 Hz to 880 Hz with gentle attack and exponential decay) and play it as a UI system sound (
kAudioServicesPropertyIsUISound = 1), which respects the iPhone hardware Silent Switch. - When the completion callback reports that iOS muted the sound in silent mode (
elapsed < 0.05 s), trigger a single light impact vibration (UIImpactFeedbackGenerator(style: .light)) instead.
- Synthesize a short, soft two-note PCM chime (180 ms, 587 Hz to 880 Hz with gentle attack and exponential decay) and play it as a UI system sound (
- When connecting fails or the connection drops, keep the Live Activity visible in the
.failedvoice state (Connection failed/连接失败/連線失敗, subtitleCheck the network, then try again./请检查网络后重试。, bear pose.tripped) with aTry Again(重试/重試) button (RetryTalkIntentover Darwin notificationTalkSignal.retry) that retriesvoice.start()directly from the Dynamic Island or Lock Screen. - Treat
state.voice != old.voiceas critical inActivityController.isCriticalso voice phase transitions update the Dynamic Island immediately without waiting for the 700 ms progress coalescing window. - Coordination with PR #508 (
fix/island-listening-clipped): PR #508 adjusts the expanded island's bottom clip andActivityMetrics.sizesforface.endsTalkvoice states, while this PR adds.connecting,.failed,RetryTalkButton, andVoiceReadyCuewithout duplicating layout sizing changes..connectingsetsface.endsTalk == true(so it automatically inherits PR #508's 142 pt expanded-island height),.faileduses.tallwith.retryTalk, and both branches merge cleanly in either order (git merge-treeverified with zero conflicts).
Motivation
Starting a live voice session requires an ephemeral token exchange, WebRTC ICE/DTLS negotiation, and waiting for OpenAI's session.created event. Previously, TalkActivity.state(of:) mapped .connecting to .listening (聆听中) and TalkActivity.startHere() displayed .listening before voice.start() began. As a result, the Dynamic Island immediately read Listening during cold start while early speech was still dropped, and a failed connection disappeared from the Dynamic Island instead of offering a retry action.
Test plan
- Built and ran
PokiTestson the iPhone 16 Pro simulator (0FFFE228-FF90-4E6E-B6A2-06D8C5B84A3A):VoiceModelTests: 26 tests executed, 0 failures (verifying thatTalkActivityshows.connectingbeforesession.created, switches to.listeningand firesreadyCuestrictly onsession.created, does not firereadyCueon failure or silent test runs, transitions to.failedon connection failure, and retries viaTalkSignal.retry).LiveActivityTests: 17 tests executed, 0 failures (verifyingActivityFaceand localized strings for.connectingand.failedacrossen,zh-Hans, andzh-Hant).LanguageTests: 12 tests executed, 0 failures.
- Verified on the iPhone 16 Pro simulator (
nz.zheha.poki -uiLanguage zh-Hans -pokiTalkRehearsal 1,offline) with screen recording andpoki-log.txtinspection:- Tapping the voice button shows
连接中with the waiting bear on bothTalkBarand the Dynamic Island, then switches to聆听中and logs[voice] session created, session 1,[voice] ready cue triggered, and[voice] ready cue playedat the same timestamp whensession.createdarrives. - Under offline failure (
URLError.notConnectedToInternet), the Dynamic Island transitions from连接中to连接失败with the重试button.
- Tapping the voice button shows
Screenshots or screen recording
Recorded on the iPhone 16 Pro simulator showing the transition from 连接中 (waiting bear) to 聆听中 upon session.created and from 连接中 to 连接失败 with 重试 when offline.
AI assistance: Other. I have reviewed and understand these changes.
- I removed passcodes, tokens, and private screen contents from this PR.
- I release this contribution under the AGPL-3.0-only license.