erofs-freebsd-out-tree/tests/results/manual/2026-08-08T2306Z-compression-p0/manual-test-report.md

repo22 compression P0 WIP integration manual report

Date: 2026-08-08 23:06 UTC Baseline: 78182686e968c659932458cbe7a1e0889397f20b Guest: FreeBSD 15.0-RELEASE-p8 amd64, QEMU TCG Build reference: /work/dev-freebsd-releng (REVISION=15.0, source branch RELEASE-p9) Linux behavior reference: /work/dev-src-linux/fs/erofs Host production tool: erofs-utils 1.8.6

Result

PASS, with the extent-record guest case accurately recorded as MKFS-UNAVAILABLE rather than PASS.

The retained ten-file WIP was reviewed and integrated without reset, checkout, stash, or revert. Both module configurations build and load. DEFLATE, ZSTD, and MicroLZMA full and partial references read correctly; targeted corruption returns EIO. A fragment-backed compressed metabox now works after packed inode initialization, while loop, recursive-NID, and range mutations fail closed. HEAD2 and interlaced images pass real FreeBSD kernel reads. LZ4, xattr/metabox, chunk, external compressed multi-device, and NFS regressions also pass.

No CI or test runner was added. Only source, Markdown manual tests, and this report are intended for the commit.

Instruction and baseline audit

• find /work -name AGENTS.md -type f -print returned no paths. There were no applicable AGENTS.md files.
• Initial local HEAD and xdm/main both resolved to 78182686e968c659932458cbe7a1e0889397f20b.
• The exact ten tracked WIP files were present: build.sh, src/Makefile, src/decompressor.c, src/deflate.c, src/internal.h, src/lzma.c, src/super.c, src/zdata.c, src/zmap.c, and src/zstd.c.
• Untracked build, fixture, overlay, artifact, and other-repository paths were not staged.

Initial guest module cleanup

Before any build under test was loaded, the guest reported:

Id Refs Address                Size Name
 5    1 0xffffffff82822000     a690 erofs-nozstd.ko

kldstat -v -i 5 proved the path was ./erofs-nozstd.ko and the contained module name was erofs. The cleanup used the observed KLD ID, not a guessed filename:

kldunload -i 5
kldstat
kldstat | grep -i erofs

The second kldstat contained no EROFS entry and the final grep printed none.

Source review

Packed inode and metabox order

• packed_nid and metabox_nid are decoded and checked before carrier loads.
• A packed NID with the metabox selector bit is rejected before any metadata recursion can begin.
• The packed inode is loaded before the metabox inode, matching the Linux dependency order and allowing a compressed metabox to terminate in a packed fragment.
• The packed inode must be a regular, non-fragment inode. This rejects a packed carrier that would recurse back through itself.
• A fragment-backed metabox must have a distinct loaded packed inode, non-zero size, a real fragment tail mapping, and a range wholly inside the packed inode.
• Fragment recursion checks compare inode NIDs rather than object pointers, which also catches separately allocated erofs_node objects for the same on-disk inode.

DEFLATE, ZSTD, and MicroLZMA

• DEFLATE uses inflate(..., Z_SYNC_FLUSH) until the requested output is full, detects no-progress loops, accepts Z_OK for partial output, and requires Z_STREAM_END plus complete input consumption for full output.
• ZSTD uses FreeBSD's formal <contrib/zstd/lib/zstd.h> API with ZSTD_createDCtx_advanced, ZSTD_DCtx_setParameter, ZSTD_decompressStream, ZSTD_isError, and ZSTD_freeDCtx.
• ZSTD partial decoding stops after the requested output is produced. Full decoding requires frame completion and complete input consumption.
• MicroLZMA partial decoding accepts XZ_OK or XZ_STREAM_END after exact requested output. Full decoding requires XZ_STREAM_END and buffer.in_pos == srclen.
• All decoder failures are translated to the filesystem read error EIO and all allocated decoder/output buffers are released.

ZSTDIO build gate

• src/Makefile consumes opt_zstdio.h and adds FreeBSD's zstd compatibility include directory for zstd.c.
• build.sh accepts only EROFS_ZSTDIO=0 or 1 and creates the corresponding option header.
• The disabled translation unit contains only the availability result and a local stub; it references no ZSTD_* symbol.
• A filesystem advertising ZSTD is rejected during compression-config parsing when the module lacks ZSTDIO, before any file read can reach the stub.

HEAD2, interlaced, and extent records

• HEAD2 selects z_algorithmtype[1] and respects the HEAD2 big-pcluster bit.
• Plain records with interlaced advise use the interlaced byte rotation before returning data.
• Extent records retain Linux ordering and responsibilities for 4-, 8-, 16-, and 32-byte records, partial references, explicit algorithm format, shifted and interlaced plain data, and final fragments.
• Explicit extent tables with a non-empty file and zero extent count fail closed.
• Final fragment mappings are checked against the loaded packed inode before reads.

Build matrix

Host commands:

EROFS_ZSTDIO=0 ./build.sh
cp build/erofs.ko /tmp/repo22-erofs-nozstd-probe.ko
nm -u /tmp/repo22-erofs-nozstd-probe.ko

EROFS_ZSTDIO=1 ./build.sh
cp build/erofs.ko /tmp/repo22-erofs-zstd-probe.ko
nm -u /tmp/repo22-erofs-zstd-probe.ko

Build	Size	SHA256	ZSTD_* undefined	bcmp undefined	Guest KLD
no ZSTDIO	68744	c96f4e1620f005a19d7ed581f98f1894969b38a34d7021f54dbce99919888ef9	none	none	load/unload PASS
ZSTDIO	69984	7f8cb03afb9c76b535709af1a06bf993cbab74bb5899b2f4646a6fd20b00d15c	five formal API names	none	load/unload PASS

The enabled module's unresolved ZSTD set was exactly:

ZSTD_DCtx_setParameter
ZSTD_createDCtx_advanced
ZSTD_decompressStream
ZSTD_freeDCtx
ZSTD_isError

Both were loaded by path, identified with kldstat -v -i ID, and unloaded by that exact ID.

ZSTDIO behavior

With the disabled module, a normal LZ4 image still read successfully:

nozstd_lz4_sha256=370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52
nozstd_lz4_dmesg_delta=empty

The same module rejected the ZSTD image:

nozstd_mount_rc=1
mount: /dev/md0: erofs: ZSTD compression requires ZSTDIO support: Operation not supported

The ZSTDIO module mounted the image and produced the expected complete SHA256 shown below.

Fixture construction and proof

Fixtures and build outputs remained outside the repository commit. Authentication used the existing private askpass workflow; its contents and credentials were not printed or recorded.

Partial-reference images

dump.erofs -s and dump.erofs --path=PATH -e established compressor, layout, logical lengths, physical starts, and pcluster lengths. Byte-level patching asserted the original inode NID, map-header offset, HEAD record, and pblk before setting Z_EROFS_LI_PARTIAL_REF or redirecting a reused pblk. CRC32C was recomputed for every superblock mutation.

Image	SHA256
deflate-partial-ref.erofs	8bc720a1250ef28794d091b6264e76060fbf01d66ea706c59c24c1960fa347ac
deflate-partial-ref-corrupt.erofs	6ff9dff9b3b5aba7da9b4a93f5b068270b4916267d2fe0fd3702c02be4c19fae
lzma-partial-ref.erofs	2f4bd89d2340273dd4052ea73aa9ac43802db431258c118c5dfede7727be9408
lzma-partial-ref-corrupt.erofs	bfbbd304a401c091d63e8b8e42b2b1984d9750760e940aded89acc909551d838
zstd-partial-ref.erofs	7f5ee4f8a20f20d32eaf8780a7081229f7dbf6b896ad23986da36e1a1c580519
zstd-partial-ref-corrupt.erofs	0c6e6b312b879297e2f71f406dcd35236654800e7cd115df0cb9a748297c48c1

The LZMA and ZSTD full file was 1048576 bytes. Their partial file was 700000 bytes and reused the complete source pcluster. The DEFLATE fixture contained multiple real compressed extents, including reused physical extents in the partial file.

Fragment-backed compressed metabox

A METABOX-capable inspector proved:

packed_nid=40
metabox_nid=38
metabox inode: regular, compressed-full, size=20480
metabox fragment header=0x800000000004e800
fragmentoff=321536
packed inode: regular, plain, size=342016
321536 + 20480 = 342016

Positive image SHA256: 8a9a62bd203994711b8272192915d811e6c3de23e07ad9607dd63e66cc109bcd.

The negative images changed one proven field each:

Negative	Exact mutation	SHA256
self-loop	packed_nid: 40 -> 38	e9501ed9d149e2d735d95156669d144e733bf4be620bb69ea8a0c41f996b436c
range	fragmentoff: 321536 -> 342016	3fdcf2a41f50da93a5931edcef0d86ff2f576ecba781833036eaa930d99197d5
metabox recursion	set bit 63 in metabox_nid	b873e0cf4d2892d2d154a5769f494660f47f4429161e7be001b833da6dd0d705
packed recursion	set bit 63 in packed_nid	4fa368d1b1d47bb56b82132e6055d105ed2508b179a8df2a98ec5a728f91c9ac

HEAD2

The HEAD2 fixture patch asserted the original bytes and made only these semantic changes, plus the resulting CRC32C bytes:

feature_incompat: 0x00000003 -> 0x0000000b
map h_advise:     0x0002 -> 0x0006
first di_advise:  HEAD1 (1) -> HEAD2 (3)

Image SHA256: fc70cbef0442a86ac2f507aebd7ac7bcfbdfcc3d0b9b3cf2ff5231334583d816. The targeted corrupted copy SHA256 was f0d9ad645804565c5aca7a92128df3ae9c81717a08ad7e8692489cd2ae606700.

Interlaced

This image was generated specifically with installed erofs-utils 1.8.6:

mkfs.erofs -zlz4 -C4096 -Efragments -T0 \
  interlaced-1.8.6.erofs source

dump.erofs -e reported 105 real extents, with 4096-byte plain extents interspersed with compressed extents. Image SHA256: d77d86f874361bae86cae9e8f6d05d9b6c68ef04cd5c8779368aef033fa485c3.

Extent metadata

Installed mkfs.erofs -V reported 1.8.6. Exact source-tree search found no on-disk Z_EROFS_ADVISE_EXTENTS, z_erofs_extent_recsize, or struct z_erofs_extent { definition in the 1.8.6 include/lib tree. Its internal struct z_erofs_extent_item is an in-memory compressor item, not the new on-disk extent-record ABI.

Although a newer-tool extent image existed in the WIP build area, it was not mounted or scored. The required result is therefore:

extent metadata guest result: MKFS-UNAVAILABLE

The independent static review compared src/erofs_fs.h and src/zmap.c against /work/dev-src-linux/fs/erofs/erofs_fs.h and zmap.c, covering record sizes 4/8/16/32, implicit physical bases, explicit-count binary search, physical/logical high words, format bits, partial references, interlaced data, fragments, metabox metadata reads, and malformed explicit zero counts.

Core FreeBSD result matrix

The guest test used repeated mdconfig -a -t vnode -f IMAGE, read-only EROFS mounts, sha256 -q, byte-exact dd/cmp, and a small C helper that printed errno on read failure.

Case	Complete SHA256 / result	Boundary or random proof	Status
DEFLATE full	370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52	full stream completion	PASS
DEFLATE partial	61b17076c2dfae88da7912d00df534b894cf6d27178863c6d8e91f8617ebb91e	offset 122900, 512 bytes: fa381301af1b62fa259addbe7ae427fd54486abc7604ea7619e7a9c47965606d; offset 736700, 1024 bytes: be1d2941b054626376fa58155ce0ef8d6357dd0defcbf9eaa37d19ff6098873c	PASS
DEFLATE corrupt	read_errno=5, failure after 65536 output bytes	target extent only	PASS
MicroLZMA full	370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52	full input consumed	PASS
MicroLZMA partial	5a840803f5372b7be1db70713fed7705bf6e982ca2fd319f722c21f094f6c8dd	offset 65500, 2048 bytes: 49c231f92dde0b0104d7e5b3a01d918dde818c3a6dca05393e2a424d01c214f2; offset 524287, 4097 bytes: b8e80c144eacd1f8863c72eab66272379923f0d738f63576ab8286455e0dde8c	PASS
MicroLZMA corrupt	read_errno=5, zero output bytes	target pcluster only	PASS
ZSTD full	370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52	formal FreeBSD API	PASS
ZSTD partial	5a840803f5372b7be1db70713fed7705bf6e982ca2fd319f722c21f094f6c8dd	same two boundary/random hashes as MicroLZMA	PASS
ZSTD corrupt	read_errno=5, zero output bytes	target pcluster only	PASS
HEAD2 full	7e2f40362554f4460e80ec2f8d91f4e6c04a8e58f2980d637b2d383a7aa3b3f8	offset 65500, 4096 bytes: f0a80e3217f54897eae271b6e570c862abcffad3a4ffd19ffec0444ee06ae721	PASS
HEAD2 corrupt	read_errno=5, zero output bytes	patched HEAD2 pcluster	PASS
interlaced full	7e2f40362554f4460e80ec2f8d91f4e6c04a8e58f2980d637b2d383a7aa3b3f8	offset 16240, 8192 bytes across first compressed/plain transition: b1387900e55e5672944f8299fe66ac9542007f2b0cbed81d5974831d6040cdae	PASS
extent records	static format/control-flow review only	erofs-utils 1.8.6 cannot emit	MKFS-UNAVAILABLE

Fragment-backed metabox results

Case	Actual	Status
positive file	SHA256 4536c1d7121f48829475f29179f54baa57154b4ef817cf0776f81782585d29ad	PASS
shared-prefix xattr	shared-value	PASS
per-file xattr	value-000	PASS
self-loop	mount exit 1; packed inode nid=38 is not a non-recursive regular file: Integrity check failed	PASS
out-of-range	mount exit 1; Integrity check failed	PASS
metabox NID bit 63	mount exit 1; Integrity check failed	PASS
packed NID bit 63	mount exit 1; Integrity check failed	PASS

The vmstat -m EROFS row after the matrix showed zero active allocations:

erofs    0     0 240695 16,32,64,128,256,384,1024,2048,4096,8192,16384,32768,65536

The cumulative allocation count increased as expected; the active allocation and active-byte columns were both zero.

Regression matrix

Regression	Actual	Status
LZ4 legacy full index	SHA256 370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52	PASS
LZ4 compact index	same SHA256	PASS
LZ4 64 KiB big pcluster	same SHA256	PASS
LZ4 all-fragments	SHA256 a3a83e5c524b5ed446a06ce78cf407192a0c80119f15d2bc3489a50515eb49e0	PASS
LZ4 ztailpacking	SHA256 e2aa4a0a0cbcf422f397c7069a38ae0f073781386958e7db0dfa3ff2ca075513	PASS
xattr/metabox	file SHA256 4536c1...; xattr value-000	PASS
single-device chunk	SHA256 de29abbd47ecd9136f64f22f73fcb40bce1718a8865e6282e74953aa1df80c44	PASS
external compressed LZ4 extent	complete SHA256 370eb0...	PASS
external compressed boundary	offset 65500, 4096 bytes: 0d66627577218a39a620e8b28d8c8d64b974b184063c52cb0cceeaf0c297c0db	PASS
local NFSv3/TCP over external compressed extent	complete SHA256 370eb0...	PASS

The external compressed fixture declared slot 1 at unified block 4, stored the real two-block LZ4 pcluster in the external provider, and zeroed the corresponding bytes in the primary image. Primary SHA256 was 5625612426d68624c77dd94b09754da583fc6bd90767e9e9d958e4fc89131a4d; blob SHA256 was 0d71102ce471af0a30d4d61a1fbeef1fc33437c85651d7c7f9c60ebd305ec9d0.

NFS startup-race investigation

The first regression run started rpcbind, mountd, and nfsd and immediately called mount_nfs. The client printed one transient RPCPROG_NFS: RPC: Program not registered, then retried successfully and read the correct complete SHA256. No dmesg line changed.

A separate clean rerun waited for:

rpcinfo -t 127.0.0.1 nfs 3

Readiness succeeded on attempt 2:

program 100003 version 3 ready and waiting

The subsequent NFS mount emitted no RPC warning and produced the same complete SHA256. This proves the first message was a user-space service-registration race, not an EROFS or NFS data-path failure.

dmesg

Core, regression, and clean NFS snapshots were each byte-identical before and after their respective test matrices. All six snapshot files had SHA256:

2aa3500d33a7cfbe0db87854d3427231cda9cf92b7f85e553f957231f7359680

There was no new panic, trap, decompression diagnostic, integrity message, GEOM orphan warning, or NFS kernel message.

Cleanup

Every test used a trap that unmounted the current EROFS/NFS mount, detached the specific md unit, stopped NFS services in client-first order, restored or removed /etc/exports, and unloaded the module by the observed KLD ID.

Final audits after the core, regression, disabled-ZSTD LZ4, and clean NFS runs all reported:

erofs mounts=0
NFS mounts=0
md units=0
erofs modules=0
nfsd/mountd/rpcbind processes=0

Limitations

• MKFS-UNAVAILABLE: erofs-utils 1.8.6 cannot generate the new on-disk extent record format. Extent coverage is independent static ABI/control-flow review only. No newer-tool extent image is counted as a FreeBSD PASS.
• METABOX and the transformed HEAD2/partial-reference fixtures require newer format-aware tooling plus byte-level assertions. Their guest reads are real FreeBSD kernel tests; fixture creation and mutations are documented and checksum-validated.
• No CI, automated runner, benchmark threshold, memory-pressure run, or forced OOM scenario was added. The requested functional manual matrix and active allocation checks were completed.
• Kernel failures: none.