#98

feat: local prompt caching (opt-in disk cache + ai cache commands)

Openmoah0911opened 1mo ago

Repeated prompts hit the Gateway every time. Running git diff | ai text "summarize" or ai image "a logo" twice pays twice, even though the result is the same. There is no local cache today.

Propose an opt-in disk cache.

  • Off by default. Enable per-run with `--cache` or globally with `AI_CLI_CACHE=1`. Disable with `--no-cache` when fresh output matters.
  • Cache key is model + prompt + system prompt + image references + relevant options (temperature, max tokens, etc.). Different models or options do not collide.
  • Stored under `~/.cache/ai-cli` (override with `AI_CLI_CACHE_DIR` / `XDG_CACHE_HOME`). Include `AI_CLI_CACHE_TTL` to control expiry, default 7 days.
  • Add `ai cache` helpers: `ai cache status`, `ai cache clear`, `ai cache prune --max-size 500M`.

Example:

```bash git diff | ai text "summarize these changes" --cache

first run: calls Gateway

git diff | ai text "summarize these changes" --cache

second run: Cache hit for openai/gpt-5.5 (0.02s)

```

Applies to `text`, `image`, `video`, `audio`, and `evaluate`. Failures are not cached. Works with `count` and multi-model runs.

I have a fork ready (`moah0911/ai-cli` branch `feat/prompt-cache`) and can open a PR if you approve the approach.

Activity

0

No activity yet.