Problem / Motivation
Model failover already works, including across providers. But when it fires, users on chat channels (QQ, Telegram, Discord, Slack, …) get no signal at all — the reply simply comes back from a different model. The observer hook exists and is already invoked for every channel; it just throws non-WebUI turns away:
nanobot/cli/gateway_runtime.py:418builds one observer for the whole gateway and attaches it to theFallbackProviderat:423— so it is called for every channel, not only the WebUI.nanobot/session/webui_turns.py:520then discards everything else:context = current_request_context() if context is None or context.channel != "websocket": return- The event it publishes,
TurnModelUpdatedEvent(nanobot/bus/outbound_events.py:105), has exactly one consumer: the WebUI projection (nanobot/webui/outbound_projection.py:169).
Two consequences:
- Silent degradation. In my deployment a gateway provider disabled one model mid-flight. Every turn for hours was served by the fallback provider and both the user and the operator saw nothing — the only trace was a log line. Users assume the bot is answering with the model they configured; it wasn't.
- Silent provider switch, which is also a privacy question.
docs/guides/configure-model-fallback.mditself says "Different providers may have different data handling policies." Today a chat user cannot learn from the conversation that their prompt went to a different provider — the WebUI badge is the only surface that says so.
Proposed Solution
Make the failover signal channel-agnostic, opt-in, and edge-triggered:
- Publish the existing
TurnModelUpdatedEventfor whichever channel the turn belongs to, instead of onlywebsocket. (The builder namebuild_webui_fallback_model_observerwould stop matching its job.) - Gate channel delivery behind a config flag, default off, so nobody gets surprise
messages — e.g.
channels.notifyModelFallback: truewith a per-channel override, reusing the same inheritable-boolean shape as otherchannels.*flags. - Edge-triggered, not per-turn: one notice when a fallback first serves a turn, then silence until a turn completes without a fallback. Per-turn notices are noisy during an outage, and the notice's own turn runs through the same failing chain.
- Leave the WebUI behavior and the websocket wire format unchanged.
Alternatives Considered
- Do nothing; use the WebUI badge. Only helps if you are watching the WebUI, and the badge only labels named presets — inline fallback objects are never labeled (same doc). Chat-only users get nothing.
- User-side workaround (what I run today). Tail the journal for
Fallback '<fb>' succeeded after primary '<primary>' failedand re-inject a message viananobot trigger. It works, but it needs a bespoke daemon plus a hand-created trigger per deployment, and delivering a notification burns a real agent turn. Fine as a stopgap, not as the product answer. - Push model-switch notices to everything. Rejected — this is why I'd want it opt-in and edge-triggered rather than on by default.
Additional Context
Verified on nanobot d0d0a44e on Linux (systemd gateway, QQ channel), primary
opencode_go/deepseek-flash with a named-preset fallback on a second provider — failover
itself worked the whole time; only the channel-facing notification is missing:
INFO | turn=... | Primary model 'deepseek-v4-flash' failed: Error: {'type': 'server_error', 'message': 'Upstream request failed: Model access is disabled'}; trying fallback 'deepseek-flash'
INFO | turn=... | Fallback 'deepseek-flash' succeeded after primary 'deepseek-v4-flash' failed
INFO | qq | turn=... | Turn completed channel=qq outcome=completed ... response=<normal reply>Note the last line still reports provider=opencode_go model=deepseek-v4-flash even though
the fallback produced the answer, so even log-based detection has to match the middle line
specifically.
Things I am not sure about
- The right config surface: a global
channels.notifyModelFallbackvsagents.defaults.*vs per-channel only. I pickedchannels.*because the delivery target is the channel, but happy to follow whatever fits the existing hierarchy. - Whether maintainers consider a chat-channel notice desirable at all, or whether the intended answer is "watch the WebUI". If it's the latter, feel free to close this — the no-op is cheap.
I'm happy to implement it once the shape is agreed.
Verified the mechanism trace against current main (
6ff96f2) — all four refs hold:cli/gateway_runtime.py:418builds a singlefallback_model_observer, and_observe_providerattaches it to everyFallbackProvidersnapshot, so it already fires for all channels — the only filter is thecontext.channel != "websocket"early return inbuild_webui_fallback_model_observer(session/webui_turns.py).TurnModelUpdatedEvent(bus/outbound_events.py:105) has exactly one production consumer:webui/outbound_projection.py:171(send_turn_model_updated). Note there's a second producer of the same event atwebui_turns.py:684(_handle_turn_runtime_admitted, also websocket-gated) — worth keeping in mind if the event gains a chat-channel rendering.Three implementation notes on the proposed shape:
Config surface. The
channels.*inheritable-boolean pattern you sketched already exists:showCompactionNotices(from #5780) isshow_compaction_notices: bool = Falseinconfig/schema.py:36, registered inchannels/manager.py:72, resolved per channel via_resolve_bool_override(manager.py:242-245), with per-channelbool | Nonefields (e.g.channels/qq/runtime.py:199) andfield("bool", inheritable=True)in the manifests (channels/qq/manifest.py:14). Mirroring it gives you global-default-off + per-channel override, plus themsg.event.notify or self.<flag>delivery gate (AgentEvent.notify,events.py:42).Edge-triggering can't live in the current observer signature.
FallbackModelObserverisCallable[[FallbackModelSelection], Awaitable[None]](providers/fallback_provider.py:110): it fires per successful fallback LLM call (the "publish only after this response is known to be usable" comment atfallback_provider.py:625), not per turn, and it never observes non-fallback turns — so it cannot detect "a turn completed without a fallback" to reset the latch. It's also built once gateway-wide with no per-chat state. The reset signal exists one layer out:TurnCompletedis on the bus (bus/runtime_events.py:61) and responses already recordresponse_is_fallback(providers/base.py:276, set at:1635/:1900). So the latch wants a new builder holding per-(channel, chat_id)state, set on fallback selection and cleared on a non-fallbackTurnCompleted.Two delivery details. (a)
_event_content()(bus/outbound_events.py) returns""forTurnModelUpdatedEvent, so publishing the event as-is to a chat channel delivers an empty message — the notice needs explicitcontent, whichoutbound_message_for_eventaccepts. (b)FallbackModelSelectioncarries only the fallbackmodel(+reauth_provider); the primary model name isn't in it (the provider logs both at the call site,fallback_provider.py:629-633, but only passes the fallback name to the observer). A "primary X failed, served by Y" notice would need the selection type extended — it's currently documented as a display-safe fallback result.