#11527

Bedrock: a new BedrockRuntimeClient is created for every request

OpenDiegoGBrisaopened 11d ago

What happened?

bedrock-converse-stream.ts does new BedrockRuntimeClient(config) inside stream(), so every model call gets a fresh client: new credential provider and new HTTP/2 session.

With an SSO profile that means every call does a GetRoleCredentials round trip to portal.sso plus a new TLS handshake to bedrock-runtime. I measured 2 new TLS handshakes per call, around 0.35 s before the request even starts (0.24 s SSO + 0.11 s TLS from eu-west-1). In an agent loop with ~35 calls per run that adds up fast. The old SSO sockets also stay open, I had 29 idle connections to portal.sso in one process.

I patched it locally to keep one client per config (region, profile, endpoint, hashed credentials, proxy) in a small LRU with an idle timeout, and dropping the cached clients on credential/auth errors so a refreshed or rotated key is picked up. Time to first byte on EU Haiku went from ~1.25 s to ~0.69 s, and calls after the first one do 0 handshakes.

Steps to reproduce

  1. Use amazon-bedrock with an SSO profile (AWS_PROFILE).
  2. Run any session with a few tool calls.
  3. Watch lsof -i (or count TLS handshakes): every call opens a new connection to portal.sso.<region>.amazonaws.com and bedrock-runtime.<region>.amazonaws.com.

Expected behavior

The client, credential provider and HTTP/2 connection are reused across requests with the same config.

Version

1.4.0, still the same on main (cc9c52310)

Happy to open a PR for this if you want it.

Activity

0

No activity yet.