erofs-freebsd-out-tree/docs/pre9-lzma-cache-implementation.md

Pre9 LZMA decoded extent cache

Decision

Pre7, Pre8, and Pre9 all reproduced the same LZMA read-liveness symptom. The manual evidence shows that a single large read completes quickly while a full file read made up of successive small reads remains in the LZMA decode path. The source path creates and destroys a decoded extent for every mapped read; the existing FreeBSD buffer cache only retains compressed block buffers.

A cache local to z_erofs_read_uio() would not cross the VOP/read boundary between successive small user reads. Commit 61f4709 therefore kept one decoded extent on each compressed-file vnode. Review found that retained memory could then grow with the number of open vnodes. This follow-up moves the cache to struct erofs_mount: each mount retains at most one decoded extent until unmount, so ordinary open-file count cannot expand the cache. A single entry can thrash between files and concurrent readers; that is an accepted first-stage tradeoff.

The cache is deliberately limited to complete, non-partial MicroLZMA extents. Partial references keep the existing path because their decoded length and reference semantics are different. Other codecs are also unchanged in this phase: the measured failure is LZMA-specific, and broadening the change would make the validation and regression attribution less precise.

Data and lifecycle

struct erofs_zextent_cache stores the decoded allocation and the mapping identity needed to prove that it can be reused:

struct erofs_zextent_cache {
	void *data;
	erofs_nid_t m_nid;
	erofs_off_t m_pa;
	erofs_off_t m_la;
	uint64_t m_plen;
	uint64_t m_llen;
	unsigned int m_deviceid;
	unsigned int m_flags;
	unsigned char m_algorithmformat;
};

The cache mutex is initialized immediately after allocating struct erofs_mount. Every mount failure path reaches erofs_sb_free(), and normal unmount calls the same helper after vflush(). There is no cache field or cache lifecycle dependency in struct erofs_node. The cache key includes the inode NID because one mount entry is shared by all regular file vnodes. NID zero remains a valid key; data != NULL is the validity bit.

EROFS is read-only, so a decoded extent does not need write invalidation. vflush() completes before unmount frees the mount object, and z_erofs_extent_cache_fini() releases the single mount-owned allocation. EROFS_MAP_META is explicitly excluded, so metadata-backed tailpacking keeps the previous path. packed_inode and metabox_en are also excluded because they are mount-private backing objects, not user file vnodes.

Concurrency

Decompression and block reads occur without holding z_extent_cache_lock. A reader first takes the lock only to compare the complete key and copy a cache hit. On a miss, it builds a private decoded extent. It then takes the lock again:

1. If another reader published the same key, copy that published extent and discard the duplicate private allocation.
2. Otherwise replace the one cached extent, copy the requested range, unlock, and free the old allocation.

This permits duplicate construction under concurrent misses but keeps all shared pointer access under the mutex. The lock is never held across bread(), allocation, or decompression, all of which may sleep. The cache allocation is never freed while a reader is copying it because replacement, hit copying, and pointer clearing are serialized by the same mutex.

The helpers are reached only when want <= MAXPHYS and contain KASSERT checks for that contract. Thus the largest lock-held copy is the FreeBSD MAXPHYS request size, not the 12 MiB on-disk extent cap. Calls that can pass more than MAXPHYS use the existing uncached copy path. This bounded mutex copy is accepted for the first stage; a refcounted immutable entry is deferred until runtime evidence shows that this copy is materially contended.

Bounds and unchanged paths

The existing mapping sanity checks cap a mapped compressed extent at Z_EROFS_PCLUSTER_MAX_DSIZE before the read path. z_erofs_read_extent() also checks the compressed and decoded lengths before allocation. The read path now explicitly checks mapoff and the extent length before converting them to size_t; this protects the cache offset arithmetic on platforms where size_t is narrower than the on-disk fields.

Cache use requires all of the following:

compressed mapped extent
not EROFS_MAP_PARTIAL_REF
Z_EROFS_COMPRESSION_LZMA
initialized mount cache
not `EROFS_MAP_META`
not a mount-private backing inode
request length no greater than `MAXPHYS`

Fragments, holes, partial references, non-LZMA codecs, uncompressed files, metadata reads, and mount-private backing inodes retain their previous code paths and error handling.

Rejected alternatives for Pre9

• A function-local cache was rejected because it cannot span successive VOP reads that caused the observed amplification.
• A per-vnode cache was rejected after the 61f4709 review: open file count could retain one decoded extent per vnode without a system-wide bound.
• A larger cross-vnode cache was rejected because it would require an eviction policy and larger memory accounting. The one-entry per-mount cache is the controlled compromise: it has a fixed mount-scoped bound and simple teardown.
• A Linux page/folio/XArray/workqueue port was rejected because those are not FreeBSD vnode/buf primitives and would create an unnecessary compatibility layer.
• A decoder stream pool was deferred: it may reduce allocator overhead but does not remove repeated full extent decompression.
• Changing MAXPHYS, changing the disk format, bypassing the buffer cache, or adding decoder retries was rejected because none addresses the demonstrated decoded-result reuse and each changes unrelated behavior or resource bounds.

Static validation and required runtime matrix

This change is static-only in the source phase. The follow-up test agent must run the unchanged LZMA fixture against Pre9 and require:

1. bounded single reads at 4 KiB, 16 KiB, 64 KiB, and 1 MiB with source-range hash equality;
2. repeated small reads spanning the same extent, with completion and hash equality;
3. complete sequential SHA-256 with recorded exit status and elapsed time;
4. concurrent reads of the same file and close/reopen reads;
5. partial-reference, plain, LZ4, DEFLATE, and ZSTD regression coverage;
6. clean unmount, md detach, and module unload after every case.

The automation verdict and DUT verdict must remain separate. A timeout or missing final hash remains a failure or blocked result; it must not be promoted to PASS because bounded reads succeed.