#6031

Notify chat channels when a fallback model serves a turn (currently WebUI-only)

OpenGZY-SUPER-HACKERopened 26d ago

Problem / Motivation

Model failover already works, including across providers. But when it fires, users on chat channels (QQ, Telegram, Discord, Slack, …) get no signal at all — the reply simply comes back from a different model. The observer hook exists and is already invoked for every channel; it just throws non-WebUI turns away:

  • nanobot/cli/gateway_runtime.py:418 builds one observer for the whole gateway and attaches it to the FallbackProvider at :423 — so it is called for every channel, not only the WebUI.
  • nanobot/session/webui_turns.py:520 then discards everything else:
    context = current_request_context()
    if context is None or context.channel != "websocket":
        return
  • The event it publishes, TurnModelUpdatedEvent (nanobot/bus/outbound_events.py:105), has exactly one consumer: the WebUI projection (nanobot/webui/outbound_projection.py:169).

Two consequences:

  1. Silent degradation. In my deployment a gateway provider disabled one model mid-flight. Every turn for hours was served by the fallback provider and both the user and the operator saw nothing — the only trace was a log line. Users assume the bot is answering with the model they configured; it wasn't.
  2. Silent provider switch, which is also a privacy question. docs/guides/configure-model-fallback.md itself says "Different providers may have different data handling policies." Today a chat user cannot learn from the conversation that their prompt went to a different provider — the WebUI badge is the only surface that says so.

Proposed Solution

Make the failover signal channel-agnostic, opt-in, and edge-triggered:

  • Publish the existing TurnModelUpdatedEvent for whichever channel the turn belongs to, instead of only websocket. (The builder name build_webui_fallback_model_observer would stop matching its job.)
  • Gate channel delivery behind a config flag, default off, so nobody gets surprise messages — e.g. channels.notifyModelFallback: true with a per-channel override, reusing the same inheritable-boolean shape as other channels.* flags.
  • Edge-triggered, not per-turn: one notice when a fallback first serves a turn, then silence until a turn completes without a fallback. Per-turn notices are noisy during an outage, and the notice's own turn runs through the same failing chain.
  • Leave the WebUI behavior and the websocket wire format unchanged.

Alternatives Considered

  • Do nothing; use the WebUI badge. Only helps if you are watching the WebUI, and the badge only labels named presets — inline fallback objects are never labeled (same doc). Chat-only users get nothing.
  • User-side workaround (what I run today). Tail the journal for Fallback '<fb>' succeeded after primary '<primary>' failed and re-inject a message via nanobot trigger. It works, but it needs a bespoke daemon plus a hand-created trigger per deployment, and delivering a notification burns a real agent turn. Fine as a stopgap, not as the product answer.
  • Push model-switch notices to everything. Rejected — this is why I'd want it opt-in and edge-triggered rather than on by default.

Additional Context

Verified on nanobot d0d0a44e on Linux (systemd gateway, QQ channel), primary opencode_go/deepseek-flash with a named-preset fallback on a second provider — failover itself worked the whole time; only the channel-facing notification is missing:

INFO | turn=... | Primary model 'deepseek-v4-flash' failed: Error: {'type': 'server_error', 'message': 'Upstream request failed: Model access is disabled'}; trying fallback 'deepseek-flash'
INFO | turn=... | Fallback 'deepseek-flash' succeeded after primary 'deepseek-v4-flash' failed
INFO | qq   | turn=... | Turn completed channel=qq outcome=completed ... response=<normal reply>

Note the last line still reports provider=opencode_go model=deepseek-v4-flash even though the fallback produced the answer, so even log-based detection has to match the middle line specifically.

Things I am not sure about

  • The right config surface: a global channels.notifyModelFallback vs agents.defaults.* vs per-channel only. I picked channels.* because the delivery target is the channel, but happy to follow whatever fits the existing hierarchy.
  • Whether maintainers consider a chat-channel notice desirable at all, or whether the intended answer is "watch the WebUI". If it's the latter, feel free to close this — the no-op is cheap.

I'm happy to implement it once the shape is agreed.

Activity

1
ToniAdreal25d ago

Verified the mechanism trace against current main (6ff96f2) — all four refs hold:

  • cli/gateway_runtime.py:418 builds a single fallback_model_observer, and _observe_provider attaches it to every FallbackProvider snapshot, so it already fires for all channels — the only filter is the context.channel != "websocket" early return in build_webui_fallback_model_observer (session/webui_turns.py).
  • TurnModelUpdatedEvent (bus/outbound_events.py:105) has exactly one production consumer: webui/outbound_projection.py:171 (send_turn_model_updated). Note there's a second producer of the same event at webui_turns.py:684 (_handle_turn_runtime_admitted, also websocket-gated) — worth keeping in mind if the event gains a chat-channel rendering.

Three implementation notes on the proposed shape:

  1. Config surface. The channels.* inheritable-boolean pattern you sketched already exists: showCompactionNotices (from #5780) is show_compaction_notices: bool = False in config/schema.py:36, registered in channels/manager.py:72, resolved per channel via _resolve_bool_override (manager.py:242-245), with per-channel bool | None fields (e.g. channels/qq/runtime.py:199) and field("bool", inheritable=True) in the manifests (channels/qq/manifest.py:14). Mirroring it gives you global-default-off + per-channel override, plus the msg.event.notify or self.<flag> delivery gate (AgentEvent.notify, events.py:42).

  2. Edge-triggering can't live in the current observer signature. FallbackModelObserver is Callable[[FallbackModelSelection], Awaitable[None]] (providers/fallback_provider.py:110): it fires per successful fallback LLM call (the "publish only after this response is known to be usable" comment at fallback_provider.py:625), not per turn, and it never observes non-fallback turns — so it cannot detect "a turn completed without a fallback" to reset the latch. It's also built once gateway-wide with no per-chat state. The reset signal exists one layer out: TurnCompleted is on the bus (bus/runtime_events.py:61) and responses already record response_is_fallback (providers/base.py:276, set at :1635/:1900). So the latch wants a new builder holding per-(channel, chat_id) state, set on fallback selection and cleared on a non-fallback TurnCompleted.

  3. Two delivery details. (a) _event_content() (bus/outbound_events.py) returns "" for TurnModelUpdatedEvent, so publishing the event as-is to a chat channel delivers an empty message — the notice needs explicit content, which outbound_message_for_event accepts. (b) FallbackModelSelection carries only the fallback model (+ reauth_provider); the primary model name isn't in it (the provider logs both at the call site, fallback_provider.py:629-633, but only passes the fallback name to the observer). A "primary X failed, served by Y" notice would need the selection type extended — it's currently documented as a display-safe fallback result.