[BUG] Post-processing opens a new HTTPS connection for every dictation (about 1 s extra in my measurement)

Before You Submit

I searched existing issues, PRs and discussions. Related but different: discussions #1953 (hedged requests against tail latency) and #2217 (warming up a local model). This report asks for no new option: it is about the fixed cost of a new connection on every request.

Bug Description

Every post-processed dictation opens a new TCP + TLS connection to the LLM provider, because llm_client.rs builds a new reqwest::Client (and with it a new connection pool) on every call: create_client() is called from send_chat_completion_with_schema, send_anthropic_message and fetch_models. The Handy log shows starting new connection for every request (see Logs below).

https://github.com/cjpais/Handy/blob/73c095857bc4eadff097f64767c40ad129cc372d/src-tauri/src/llm_client.rs#L218-L226

In my measurement this costs about 1 s per dictation, roughly the same for short and long recordings, so short dictations suffer most: for the shortest ones (1-7 words) it almost doubles the post-processing time.

I measured it with a Python script sending the request body Handy sends (provider OpenAI, model chat-latest, my post-processing prompt as the system message, the transcription as the user message, response_format json_schema transcription_output, strict) on my own recordings. The two columns differ only in whether the client session, and so the HTTPS connection, is reused. "Reused" is the best case: a warm connection with requests back to back.

Median client-side time of the request, seconds (4-5 recordings per row, 2-3 runs each):

I have not broken the 1 s down into DNS, TCP and TLS. It will vary with network and location, but the extra round trips are there for everyone.

Steps to reproduce: enable post-processing with a cloud provider, dictate a few words twice in a row, and check the log: both requests show starting new connection.

Expected: the second request reuses the connection.

Actual: every request opens a new one, about 1 s each in my measurement.

Possible fix

I could not find whether a fresh client per request is intentional (for example, so that a changed API key or provider takes effect at once). If it is, that still works with a shared client, because the headers can be set per request. So the fix could be: keep one reqwest::Client for the app lifetime (for example in a OnceLock or in app state) and pass the provider headers per request (RequestBuilder::headers) instead of as client default_headers, so requests to the same host reuse pooled connections.

reqwest closes idle pooled connections after 90 s by default (or sooner if the server closes them), so this helps dictations that follow each other; after a longer pause the handshake still happens.

System Information

App Version: 0.9.8

Operating System: Windows 11 Pro (Insider build 26220)

CPU: AMD Ryzen 5 4500U

GPU: AMD Radeon integrated (not involved here)

Logs

One dictation with post-processing, abridged:

[16:57:35][handy_app_lib::actions][DEBUG] Starting LLM post-processing with provider 'openai' (model: chat-latest)
[16:57:35][handy_app_lib::actions][DEBUG] Using structured outputs for provider 'openai'
[16:57:35][handy_app_lib::llm_client][DEBUG] Sending chat completion request to: https://api.openai.com/v1/chat/completions
[16:57:35][reqwest::connect][DEBUG] starting new connection: https://api.openai.com/
[16:57:38][handy_app_lib::llm_client][DEBUG] Chat completion response received with status 200 OK over HTTP/1.1 from https://api.openai.com/v1/chat/completions
[16:57:38][handy_app_lib::actions][DEBUG] Structured output post-processing succeeded for provider 'openai'. Output length: 363 chars

The investigation and this text were done with Claude Code; the logs and measurements are from my machine.