Ship a usable memory → OpenDAL cache → caller source path.
Review and merge order
Already merged to main:
Land the remaining drafts in this order:
- Start here: #1360 — engine, tests, and needed CI wiring, based on
main. Review and merge this first. - Then: #1361 — README and Fs example, based on #1360. The #1361 diff is docs and the example only; it depends on the engine from #1360.
Current step: #1360.
#1360 now carries crate tests in foyer-opendal/src/{codec.rs,namespace.rs} (#[cfg(test)]) and foyer-opendal/src/engine_tests.rs (from lib.rs under #[cfg(all(test, not(madsim)))]), plus Redis CI (foyer-opendal redis). Reproduce with cargo test -p foyer-opendal and FOYER_OPENDAL_REDIS_URL=redis://127.0.0.1:6379 cargo test -p foyer-opendal --features test-redis -- --test-threads=1. #1357 is the engine delivery checklist for #1360. #1358 is integration and example evidence for #1361. #1351 and #1353–#1356 remain first-version checklists for that engine, not extra PRs and not a second merge sequence.
First-version contract
- Each key identifies immutable content, including source namespace, object/version and range where needed. The caller fetches that exact version. New content uses a new key; old work may still fill its old key.
- Support the documented key/value representation and write policies on Fs and real Redis. Reuse Foyer's source-fetch coalescing and
get_or_fetchfallback; plaingetretains its error behavior. - Use a fresh, exclusive namespace, a process-local index and
RecoverMode::None. Document who cleans up objects left after close, crashes or failed deletion. Caller/operator-owned cleanup is acceptable. - Cache admission may be rejected and failed writes may be discarded. Cache misses and refills are expected. Do not require durable publication, strict application deletion or cancellation of all older work.
- Capacity eviction and best-effort physical reclamation remain part of the engine. Resource and failure behavior must be bounded and explicit; logical capacity is not a guarantee of physical Fs/Redis usage.
First-version blockers
Only the concrete behaviors below block delivery. Implement and verify them alongside #1360; close each checklist when its first-version criteria are satisfied, without waiting for its possible follow-ups.
| Area | Required behavior | Tracking |
|---|---|---|
| Engine delivery | Public construction, optional dependencies, memory/secondary/source path and documented supported scope | #1357 |
| Duplicate submissions | Preserve pending data and version identity; retain the existing write coalescing and verify retry/refill eligibility | #1351 |
| Resource limits | Bound admitted payloads, object buffers/concurrency, metadata and queued work, including cleanup/error recording; reject cache admission without failing a successful source fetch | #1353 |
| Failure handling | Release registrations and budgets, never return another key/version's bytes, bound shutdown, and expose background/cleanup failures | #1354 |
| Namespace contract | Exclusive ownership, no recovery, safe cleanup scope and an explicit owner for abandoned objects | #1355 |
| Representation | Correct round trips, key verification and bounded handling of oversized, malformed or corrupt data | #1356 |
| Integration evidence | Verify those behaviors through the actual public API on Fs/Redis, preserve relevant BlockEngine behavior and provide a working example | #1358 |
Completed prerequisites
- #1359 — Keeper identity. This does not by itself complete engine-side coalescing or the engine delivery.
- #1352 — Device-independent engine boundary, merged in #1364.
Possible follow-ups — not first-version blockers
These are candidates, not commitments. Open a dedicated follow-up when a concrete caller or measurement justifies it.
- Exact process RSS or backend physical-usage accounting, adaptive admission and advanced metrics (#1353).
- Durable retry queues, guaranteed eventual publication/deletion, and reconciliation of uncertain remote outcomes (#1354).
- Restart recovery, shared namespaces, multiple writers and automatic orphan sweeping (#1355; related #390).
- Streaming/range reads, a broader codec abstraction and cross-version recovery compatibility (#1356; related #1026 and #1242).
- Further coalescing optimizations and a broad backend/workload performance matrix (#1351 and #1358). Performance claims still require relevant measurements; no speedup is assumed.
- Strict invalidation for mutable keys remains independent in #1085; it is neither fixed nor blocked by this work.
Completion
Close this tracker when the first-version engine, relevant regression coverage and usage documentation are merged and the blockers above are satisfied. A limited, explicit contract is sufficient; possible follow-ups do not keep this tracker open. Reuse valid existing evidence and add checks for actual gaps rather than creating a separate exhaustive acceptance project.
Background: #686; the Fs/Redis prototype established feasibility, not complete delivery. Broader Device/object-engine discussions (#1278, #1200, #1082) are related work, not additional prerequisites.
Activity
0No activity yet.