# repo22 compression P0 WIP integration manual report

Date: 2026-08-08 23:06 UTC
Baseline: `78182686e968c659932458cbe7a1e0889397f20b`
Guest: FreeBSD 15.0-RELEASE-p8 amd64, QEMU TCG
Build reference: `/work/dev-freebsd-releng` (`REVISION=15.0`, source branch
`RELEASE-p9`)
Linux behavior reference: `/work/dev-src-linux/fs/erofs`
Host production tool: erofs-utils 1.8.6

## Result

**PASS**, with the extent-record guest case accurately recorded as
**MKFS-UNAVAILABLE** rather than PASS.

The retained ten-file WIP was reviewed and integrated without reset, checkout,
stash, or revert. Both module configurations build and load. DEFLATE, ZSTD,
and MicroLZMA full and partial references read correctly; targeted corruption
returns `EIO`. A fragment-backed compressed metabox now works after packed
inode initialization, while loop, recursive-NID, and range mutations fail
closed. HEAD2 and interlaced images pass real FreeBSD kernel reads. LZ4,
xattr/metabox, chunk, external compressed multi-device, and NFS regressions
also pass.

No CI or test runner was added. Only source, Markdown manual tests, and this
report are intended for the commit.

## Instruction and baseline audit

- `find /work -name AGENTS.md -type f -print` returned no paths. There were no
  applicable `AGENTS.md` files.
- Initial local `HEAD` and `xdm/main` both resolved to
  `78182686e968c659932458cbe7a1e0889397f20b`.
- The exact ten tracked WIP files were present:
  `build.sh`, `src/Makefile`, `src/decompressor.c`, `src/deflate.c`,
  `src/internal.h`, `src/lzma.c`, `src/super.c`, `src/zdata.c`, `src/zmap.c`,
  and `src/zstd.c`.
- Untracked build, fixture, overlay, artifact, and other-repository paths were
  not staged.

## Initial guest module cleanup

Before any build under test was loaded, the guest reported:

```text
Id Refs Address                Size Name
 5    1 0xffffffff82822000     a690 erofs-nozstd.ko
```

`kldstat -v -i 5` proved the path was `./erofs-nozstd.ko` and the contained
module name was `erofs`. The cleanup used the observed KLD ID, not a guessed
filename:

```sh
kldunload -i 5
kldstat
kldstat | grep -i erofs
```

The second `kldstat` contained no EROFS entry and the final grep printed
`none`.

## Source review

### Packed inode and metabox order

- `packed_nid` and `metabox_nid` are decoded and checked before carrier loads.
- A packed NID with the metabox selector bit is rejected before any metadata
  recursion can begin.
- The packed inode is loaded before the metabox inode, matching the Linux
  dependency order and allowing a compressed metabox to terminate in a packed
  fragment.
- The packed inode must be a regular, non-fragment inode. This rejects a packed
  carrier that would recurse back through itself.
- A fragment-backed metabox must have a distinct loaded packed inode, non-zero
  size, a real fragment tail mapping, and a range wholly inside the packed
  inode.
- Fragment recursion checks compare inode NIDs rather than object pointers,
  which also catches separately allocated `erofs_node` objects for the same
  on-disk inode.

### DEFLATE, ZSTD, and MicroLZMA

- DEFLATE uses `inflate(..., Z_SYNC_FLUSH)` until the requested output is full,
  detects no-progress loops, accepts `Z_OK` for partial output, and requires
  `Z_STREAM_END` plus complete input consumption for full output.
- ZSTD uses FreeBSD's formal `<contrib/zstd/lib/zstd.h>` API with
  `ZSTD_createDCtx_advanced`, `ZSTD_DCtx_setParameter`,
  `ZSTD_decompressStream`, `ZSTD_isError`, and `ZSTD_freeDCtx`.
- ZSTD partial decoding stops after the requested output is produced. Full
  decoding requires frame completion and complete input consumption.
- MicroLZMA partial decoding accepts `XZ_OK` or `XZ_STREAM_END` after exact
  requested output. Full decoding requires `XZ_STREAM_END` and
  `buffer.in_pos == srclen`.
- All decoder failures are translated to the filesystem read error `EIO` and
  all allocated decoder/output buffers are released.

### ZSTDIO build gate

- `src/Makefile` consumes `opt_zstdio.h` and adds FreeBSD's zstd compatibility
  include directory for `zstd.c`.
- `build.sh` accepts only `EROFS_ZSTDIO=0` or `1` and creates the corresponding
  option header.
- The disabled translation unit contains only the availability result and a
  local stub; it references no `ZSTD_*` symbol.
- A filesystem advertising ZSTD is rejected during compression-config parsing
  when the module lacks ZSTDIO, before any file read can reach the stub.

### HEAD2, interlaced, and extent records

- HEAD2 selects `z_algorithmtype[1]` and respects the HEAD2 big-pcluster bit.
- Plain records with interlaced advise use the interlaced byte rotation before
  returning data.
- Extent records retain Linux ordering and responsibilities for 4-, 8-, 16-,
  and 32-byte records, partial references, explicit algorithm format, shifted
  and interlaced plain data, and final fragments.
- Explicit extent tables with a non-empty file and zero extent count fail
  closed.
- Final fragment mappings are checked against the loaded packed inode before
  reads.

## Build matrix

Host commands:

```sh
EROFS_ZSTDIO=0 ./build.sh
cp build/erofs.ko /tmp/repo22-erofs-nozstd-probe.ko
nm -u /tmp/repo22-erofs-nozstd-probe.ko

EROFS_ZSTDIO=1 ./build.sh
cp build/erofs.ko /tmp/repo22-erofs-zstd-probe.ko
nm -u /tmp/repo22-erofs-zstd-probe.ko
```

| Build | Size | SHA256 | `ZSTD_*` undefined | `bcmp` undefined | Guest KLD |
|---|---:|---|---|---|---|
| no ZSTDIO | 68744 | `c96f4e1620f005a19d7ed581f98f1894969b38a34d7021f54dbce99919888ef9` | none | none | load/unload PASS |
| ZSTDIO | 69984 | `7f8cb03afb9c76b535709af1a06bf993cbab74bb5899b2f4646a6fd20b00d15c` | five formal API names | none | load/unload PASS |

The enabled module's unresolved ZSTD set was exactly:

```text
ZSTD_DCtx_setParameter
ZSTD_createDCtx_advanced
ZSTD_decompressStream
ZSTD_freeDCtx
ZSTD_isError
```

Both were loaded by path, identified with `kldstat -v -i ID`, and unloaded by
that exact ID.

## ZSTDIO behavior

With the disabled module, a normal LZ4 image still read successfully:

```text
nozstd_lz4_sha256=370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52
nozstd_lz4_dmesg_delta=empty
```

The same module rejected the ZSTD image:

```text
nozstd_mount_rc=1
mount: /dev/md0: erofs: ZSTD compression requires ZSTDIO support: Operation not supported
```

The ZSTDIO module mounted the image and produced the expected complete SHA256
shown below.

## Fixture construction and proof

Fixtures and build outputs remained outside the repository commit.
Authentication used the existing private askpass workflow; its contents and
credentials were not printed or recorded.

### Partial-reference images

`dump.erofs -s` and `dump.erofs --path=PATH -e` established compressor,
layout, logical lengths, physical starts, and pcluster lengths. Byte-level
patching asserted the original inode NID, map-header offset, HEAD record, and
pblk before setting `Z_EROFS_LI_PARTIAL_REF` or redirecting a reused pblk.
CRC32C was recomputed for every superblock mutation.

| Image | SHA256 |
|---|---|
| `deflate-partial-ref.erofs` | `8bc720a1250ef28794d091b6264e76060fbf01d66ea706c59c24c1960fa347ac` |
| `deflate-partial-ref-corrupt.erofs` | `6ff9dff9b3b5aba7da9b4a93f5b068270b4916267d2fe0fd3702c02be4c19fae` |
| `lzma-partial-ref.erofs` | `2f4bd89d2340273dd4052ea73aa9ac43802db431258c118c5dfede7727be9408` |
| `lzma-partial-ref-corrupt.erofs` | `bfbbd304a401c091d63e8b8e42b2b1984d9750760e940aded89acc909551d838` |
| `zstd-partial-ref.erofs` | `7f5ee4f8a20f20d32eaf8780a7081229f7dbf6b896ad23986da36e1a1c580519` |
| `zstd-partial-ref-corrupt.erofs` | `0c6e6b312b879297e2f71f406dcd35236654800e7cd115df0cb9a748297c48c1` |

The LZMA and ZSTD full file was 1048576 bytes. Their partial file was 700000
bytes and reused the complete source pcluster. The DEFLATE fixture contained
multiple real compressed extents, including reused physical extents in the
partial file.

### Fragment-backed compressed metabox

A METABOX-capable inspector proved:

```text
packed_nid=40
metabox_nid=38
metabox inode: regular, compressed-full, size=20480
metabox fragment header=0x800000000004e800
fragmentoff=321536
packed inode: regular, plain, size=342016
321536 + 20480 = 342016
```

Positive image SHA256:
`8a9a62bd203994711b8272192915d811e6c3de23e07ad9607dd63e66cc109bcd`.

The negative images changed one proven field each:

| Negative | Exact mutation | SHA256 |
|---|---|---|
| self-loop | `packed_nid: 40 -> 38` | `e9501ed9d149e2d735d95156669d144e733bf4be620bb69ea8a0c41f996b436c` |
| range | `fragmentoff: 321536 -> 342016` | `3fdcf2a41f50da93a5931edcef0d86ff2f576ecba781833036eaa930d99197d5` |
| metabox recursion | set bit 63 in `metabox_nid` | `b873e0cf4d2892d2d154a5769f494660f47f4429161e7be001b833da6dd0d705` |
| packed recursion | set bit 63 in `packed_nid` | `4fa368d1b1d47bb56b82132e6055d105ed2508b179a8df2a98ec5a728f91c9ac` |

### HEAD2

The HEAD2 fixture patch asserted the original bytes and made only these semantic
changes, plus the resulting CRC32C bytes:

```text
feature_incompat: 0x00000003 -> 0x0000000b
map h_advise:     0x0002 -> 0x0006
first di_advise:  HEAD1 (1) -> HEAD2 (3)
```

Image SHA256:
`fc70cbef0442a86ac2f507aebd7ac7bcfbdfcc3d0b9b3cf2ff5231334583d816`.
The targeted corrupted copy SHA256 was
`f0d9ad645804565c5aca7a92128df3ae9c81717a08ad7e8692489cd2ae606700`.

### Interlaced

This image was generated specifically with installed erofs-utils 1.8.6:

```sh
mkfs.erofs -zlz4 -C4096 -Efragments -T0 \
  interlaced-1.8.6.erofs source
```

`dump.erofs -e` reported 105 real extents, with 4096-byte plain extents
interspersed with compressed extents. Image SHA256:
`d77d86f874361bae86cae9e8f6d05d9b6c68ef04cd5c8779368aef033fa485c3`.

### Extent metadata

Installed `mkfs.erofs -V` reported 1.8.6. Exact source-tree search found no
on-disk `Z_EROFS_ADVISE_EXTENTS`, `z_erofs_extent_recsize`, or
`struct z_erofs_extent {` definition in the 1.8.6 include/lib tree. Its
internal `struct z_erofs_extent_item` is an in-memory compressor item, not the
new on-disk extent-record ABI.

Although a newer-tool extent image existed in the WIP build area, it was not
mounted or scored. The required result is therefore:

```text
extent metadata guest result: MKFS-UNAVAILABLE
```

The independent static review compared `src/erofs_fs.h` and `src/zmap.c`
against `/work/dev-src-linux/fs/erofs/erofs_fs.h` and `zmap.c`, covering record
sizes 4/8/16/32, implicit physical bases, explicit-count binary search,
physical/logical high words, format bits, partial references, interlaced data,
fragments, metabox metadata reads, and malformed explicit zero counts.

## Core FreeBSD result matrix

The guest test used repeated `mdconfig -a -t vnode -f IMAGE`, read-only EROFS
mounts, `sha256 -q`, byte-exact `dd`/`cmp`, and a small C helper that printed
`errno` on read failure.

| Case | Complete SHA256 / result | Boundary or random proof | Status |
|---|---|---|---|
| DEFLATE full | `370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52` | full stream completion | PASS |
| DEFLATE partial | `61b17076c2dfae88da7912d00df534b894cf6d27178863c6d8e91f8617ebb91e` | offset 122900, 512 bytes: `fa381301af1b62fa259addbe7ae427fd54486abc7604ea7619e7a9c47965606d`; offset 736700, 1024 bytes: `be1d2941b054626376fa58155ce0ef8d6357dd0defcbf9eaa37d19ff6098873c` | PASS |
| DEFLATE corrupt | `read_errno=5`, failure after 65536 output bytes | target extent only | PASS |
| MicroLZMA full | `370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52` | full input consumed | PASS |
| MicroLZMA partial | `5a840803f5372b7be1db70713fed7705bf6e982ca2fd319f722c21f094f6c8dd` | offset 65500, 2048 bytes: `49c231f92dde0b0104d7e5b3a01d918dde818c3a6dca05393e2a424d01c214f2`; offset 524287, 4097 bytes: `b8e80c144eacd1f8863c72eab66272379923f0d738f63576ab8286455e0dde8c` | PASS |
| MicroLZMA corrupt | `read_errno=5`, zero output bytes | target pcluster only | PASS |
| ZSTD full | `370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52` | formal FreeBSD API | PASS |
| ZSTD partial | `5a840803f5372b7be1db70713fed7705bf6e982ca2fd319f722c21f094f6c8dd` | same two boundary/random hashes as MicroLZMA | PASS |
| ZSTD corrupt | `read_errno=5`, zero output bytes | target pcluster only | PASS |
| HEAD2 full | `7e2f40362554f4460e80ec2f8d91f4e6c04a8e58f2980d637b2d383a7aa3b3f8` | offset 65500, 4096 bytes: `f0a80e3217f54897eae271b6e570c862abcffad3a4ffd19ffec0444ee06ae721` | PASS |
| HEAD2 corrupt | `read_errno=5`, zero output bytes | patched HEAD2 pcluster | PASS |
| interlaced full | `7e2f40362554f4460e80ec2f8d91f4e6c04a8e58f2980d637b2d383a7aa3b3f8` | offset 16240, 8192 bytes across first compressed/plain transition: `b1387900e55e5672944f8299fe66ac9542007f2b0cbed81d5974831d6040cdae` | PASS |
| extent records | static format/control-flow review only | erofs-utils 1.8.6 cannot emit | MKFS-UNAVAILABLE |

## Fragment-backed metabox results

| Case | Actual | Status |
|---|---|---|
| positive file | SHA256 `4536c1d7121f48829475f29179f54baa57154b4ef817cf0776f81782585d29ad` | PASS |
| shared-prefix xattr | `shared-value` | PASS |
| per-file xattr | `value-000` | PASS |
| self-loop | mount exit 1; `packed inode nid=38 is not a non-recursive regular file: Integrity check failed` | PASS |
| out-of-range | mount exit 1; `Integrity check failed` | PASS |
| metabox NID bit 63 | mount exit 1; `Integrity check failed` | PASS |
| packed NID bit 63 | mount exit 1; `Integrity check failed` | PASS |

The `vmstat -m` EROFS row after the matrix showed zero active allocations:

```text
erofs    0     0 240695 16,32,64,128,256,384,1024,2048,4096,8192,16384,32768,65536
```

The cumulative allocation count increased as expected; the active allocation
and active-byte columns were both zero.

## Regression matrix

| Regression | Actual | Status |
|---|---|---|
| LZ4 legacy full index | SHA256 `370eb0a8df86868c4842ca535ed64670f0277ea2ed47a703f089bbb13ee4ac52` | PASS |
| LZ4 compact index | same SHA256 | PASS |
| LZ4 64 KiB big pcluster | same SHA256 | PASS |
| LZ4 all-fragments | SHA256 `a3a83e5c524b5ed446a06ce78cf407192a0c80119f15d2bc3489a50515eb49e0` | PASS |
| LZ4 ztailpacking | SHA256 `e2aa4a0a0cbcf422f397c7069a38ae0f073781386958e7db0dfa3ff2ca075513` | PASS |
| xattr/metabox | file SHA256 `4536c1...`; xattr `value-000` | PASS |
| single-device chunk | SHA256 `de29abbd47ecd9136f64f22f73fcb40bce1718a8865e6282e74953aa1df80c44` | PASS |
| external compressed LZ4 extent | complete SHA256 `370eb0...` | PASS |
| external compressed boundary | offset 65500, 4096 bytes: `0d66627577218a39a620e8b28d8c8d64b974b184063c52cb0cceeaf0c297c0db` | PASS |
| local NFSv3/TCP over external compressed extent | complete SHA256 `370eb0...` | PASS |

The external compressed fixture declared slot 1 at unified block 4, stored the
real two-block LZ4 pcluster in the external provider, and zeroed the
corresponding bytes in the primary image. Primary SHA256 was
`5625612426d68624c77dd94b09754da583fc6bd90767e9e9d958e4fc89131a4d`;
blob SHA256 was
`0d71102ce471af0a30d4d61a1fbeef1fc33437c85651d7c7f9c60ebd305ec9d0`.

## NFS startup-race investigation

The first regression run started rpcbind, mountd, and nfsd and immediately
called `mount_nfs`. The client printed one transient
`RPCPROG_NFS: RPC: Program not registered`, then retried successfully and read
the correct complete SHA256. No dmesg line changed.

A separate clean rerun waited for:

```sh
rpcinfo -t 127.0.0.1 nfs 3
```

Readiness succeeded on attempt 2:

```text
program 100003 version 3 ready and waiting
```

The subsequent NFS mount emitted no RPC warning and produced the same complete
SHA256. This proves the first message was a user-space service-registration
race, not an EROFS or NFS data-path failure.

## dmesg

Core, regression, and clean NFS snapshots were each byte-identical before and
after their respective test matrices. All six snapshot files had SHA256:

```text
2aa3500d33a7cfbe0db87854d3427231cda9cf92b7f85e553f957231f7359680
```

There was no new panic, trap, decompression diagnostic, integrity message, GEOM
orphan warning, or NFS kernel message.

## Cleanup

Every test used a trap that unmounted the current EROFS/NFS mount, detached the
specific md unit, stopped NFS services in client-first order, restored or
removed `/etc/exports`, and unloaded the module by the observed KLD ID.

Final audits after the core, regression, disabled-ZSTD LZ4, and clean NFS runs
all reported:

```text
erofs mounts=0
NFS mounts=0
md units=0
erofs modules=0
nfsd/mountd/rpcbind processes=0
```

## Limitations

- **MKFS-UNAVAILABLE**: erofs-utils 1.8.6 cannot generate the new on-disk extent
  record format. Extent coverage is independent static ABI/control-flow review
  only. No newer-tool extent image is counted as a FreeBSD PASS.
- METABOX and the transformed HEAD2/partial-reference fixtures require newer
  format-aware tooling plus byte-level assertions. Their guest reads are real
  FreeBSD kernel tests; fixture creation and mutations are documented and
  checksum-validated.
- No CI, automated runner, benchmark threshold, memory-pressure run, or forced
  OOM scenario was added. The requested functional manual matrix and active
  allocation checks were completed.
- Kernel failures: none.