#452

Gemini 429 rate limits are not retried

Openlalith0407opened 23d ago

Affected Component

External Integrations (LLM/Search APIs)

Describe the bug

The provider wrapper retries 429s up to 10 times (backend/pkg/providers/provider/wrapper.go:165-185), but only when isTooManyRequestsError (wrapper.go:381) recognizes the error. It checks AWS error types and text like "status code: 429" or "rate limit exceeded".

The Gemini provider gets its errors from google.golang.org/genai, which returns genai.APIError by value. Its text is Error 429, Message: You exceeded your current quota ..., Status: RESOURCE_EXHAUSTED, ..., which none of those checks match. So with Gemini, the first 429 fails the call right away, while OpenAI, Anthropic and Bedrock get the backoff.

This was behind the task title failure in #446: the 429 came from Gemini and was never retried.

Steps to Reproduce

  1. Use Gemini on the free tier (15 requests/min).
  2. Start a flow, or two back to back.
  3. The first 429 fails the step. The logs show no wrapper retries.

Logs

level=error msg="failed to prepare a flow" error="failed to get flow provider: failed to select primary docker image via llm call: gave up after 3 attempts: Error 429, Message: You exceeded your current quota ..., Status: RESOURCE_EXHAUSTED ..."

Expected Behavior

Gemini 429s get the same retry as other providers.

System Configuration

PentAGI 2.2.0-ce (843dfeb), Docker 29.9.1, Linux, Gemini (gemini-3.1-flash-lite / gemini-3.5-flash-lite), google.golang.org/genai v1.42.0

Activity

1
XinwangTao22d ago

This is a clean root-cause report — the mismatch is exactly between the text-matching heuristic and Gemini's error shape. A couple of suggestions for the fix: 1) prefer structural checks over text matching: genai.APIError has an HTTPStatus/code field (429) and the status string RESOURCE_EXHAUSTED, so inspect the error type and its fields first, then fall back to text matching for non-typed providers; 2) for the backoff, use the provider's Retry-After header or the documented quota reset time when available instead of a fixed delay — Gemini free-tier resets are often seconds-long and a uniform 10-retry window is wasteful; 3) consider covering "RESOURCE_EXHAUSTED" / "quota exceeded" substrings in the text matcher too, since other SDKs may wrap errors in different formats. This also closes the loop on #446 — it would be good to add a regression test that feeds a synthetic genai.APIError 429 through the wrapper and asserts retry happens.