Add file descriptor budgeting and backpressure for RocksDB, blob I/O, and replication

Problem

The cache service currently uses file descriptors from a shared process-wide pool without explicit budgeting between:

• RocksDB open files
• foreground blob reads
• upload/temp-file writes
• background replication
• inbound and outbound sockets

In a highly concurrent environment this can lead to EMFILE / file descriptor exhaustion, which would cause the service to fail in unpredictable ways.

Current state

A few places that consume file descriptors directly:

• src/store.rs: RocksDB is opened with default options and does not set max_open_files
• src/http.rs: blob serving opens files per request for streaming
• src/replication.rs: replication opens local blob files and outbound HTTP connections
• src/utils.rs: request bodies are persisted to temp files during uploads

We also do not currently expose the active FD count or enforce any workload-specific concurrency budgets.

What we should do

• set an explicit process/container nofile limit in deployment
• add a configurable RocksDB max_open_files
• introduce bounded concurrency / semaphores for major workload classes
  • foreground blob reads
  • uploads/temp-file work
  • replication
• reserve headroom so background work cannot starve foreground traffic
• add backpressure / load shedding when FD usage crosses configured watermarks
• export observability for open FD count and budget usage

Suggested approach

1. Add config for RocksDB file budget, per-workload concurrency budgets, and watermark thresholds.
2. Export a metric for current open FDs, e.g. by counting /proc/self/fd on Linux.
3. Give replication a lower priority and pause/throttle it first when usage is high.
4. Reject non-critical work with 503 rather than allowing the process to exhaust descriptors.
5. Log the effective RLIMIT_NOFILE at startup.

Acceptance criteria

• RocksDB open file usage is capped by config.
• Foreground and background file-consuming workloads have separate bounded concurrency controls.
• The service exposes an open-FD metric.
• Replication is throttled or paused before foreground traffic is impacted.
• The service sheds load before hitting EMFILE.
• Configuration and operational expectations are documented.

Notes

This does not need a perfect fairness model. The important part is explicit budgeting, reserved headroom, and predictable degradation under pressure.