Lifecycle event processor overflows after long idle backoff

Open

Summary

A long-running CAR hub can start repeatedly logging OverflowError: (34, 'Numerical result out of range') from the lifecycle event processor after enough idle/backoff iterations.

Observed on

Package: codex-autorunner
Installed version when observed: 2.2.2
Runtime: Python 3.12, pipx install
Service mode: codex-autorunner hub serve --host 127.0.0.1 --port 4517 --base-path /car --path /Users/dazheng/car-workspace
Hub process had been running for several days.

Log excerpt

Error in lifecycle event processor
Traceback (most recent call last):
  File "/Users/dazheng/.local/share/pipx/venvs/codex-autorunner/lib/python3.12/site-packages/codex_autorunner/core/hub_lifecycle.py", line 301, in _process_loop
    current_interval = self._current_interval()
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/dazheng/.local/share/pipx/venvs/codex-autorunner/lib/python3.12/site-packages/codex_autorunner/core/hub_lifecycle.py", line 261, in _current_interval
    self._BACKOFF_GROW_FACTOR**self._idle_streak
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^~~~~~~~~~~~~~~~~~
OverflowError: (34, 'Numerical result out of range')

The same traceback repeated approximately every 30 seconds.

Expected behavior

The lifecycle processor's idle/backoff calculation should be bounded/clamped so long-running idle hubs do not eventually overflow. Once max backoff is reached, it should stay at max interval rather than continuing to exponentiate an unbounded idle streak.

Impact

The hub service remains active, but logs are noisy and lifecycle/event processing may be degraded or confusing while diagnosing unrelated agent failures.

Suggested fix

Clamp either:

_idle_streak before exponentiation, or
the computed interval with a safe exponent/math path that cannot overflow.

A regression test could set _idle_streak to a very large value and assert _current_interval() returns the configured maximum interval without raising.

Activity

Git-on-my-level1mo ago

Already fixed on the default branch: #2051 rewrote HubLifecycleWorker._current_interval() in src/codex_autorunner/core/hub_lifecycle.py to grow the interval iteratively and early-return at _max_poll_interval_seconds, removing the _BACKOFF_GROW_FACTOR ** _idle_streak exponentiation that raises OverflowError (float ** raises, while * saturates to inf). It also added test_hub_lifecycle_worker_backoff_saturates_without_overflow (tests/core/test_hub_lifecycle.py), which sets _idle_streak = 10**9 and asserts the interval equals the 3.0s max. The fix shipped in v2.2.3, so the report against 2.2.2 predates it — upgrading resolves this, and the issue can be closed.

by AI on behalf of David — mention @Git-on-my-level if you need a human.