update

2026-08-18 09:20:44 +02:00
commit b826cd721a
522 changed files with 93730 additions and 0 deletions
@@ -0,0 +1,166 @@
+# repo22 Final Review WIP Manual Test Report
+
+Started: 2026-08-09 01:24 UTC
+
+Completed: 2026-08-09 03:33 UTC
+
+Parent commit: `9a3604fba32cfe2ff263d954d80fe588f99db88f`
+
+Guest: FreeBSD 15.0-RELEASE-p8 amd64, QEMU TCG
+
+Linux reference: `/work/dev-src-linux/fs/erofs`
+
+FreeBSD reference: `/work/dev-freebsd-releng`
+
+## Scope
+
+This run reviews and validates four final correctness findings:
+
+- Linux-compatible 48-bit superblock union selection;
+- rejection of extended inode sizes above FreeBSD `OFF_MAX`;
+- bounded reads from multi-GiB explicit extent holes;
+- 64-bit directory block-search indexes.
+
+No CI work, binary fixture, guest overlay, build object, or raw VM artifact is
+part of the intended commit.
+
+## Result
+
+| Test | Result | Evidence |
+|---|---|---|
+| TC150 48-bit fallback root | PASS | Real FreeBSD mount, root read, inode and `df` |
+| TC151 size above `OFF_MAX` | PASS | Six direct failures, rc 1, syscall errno 97 |
+| TC152 bounded extent hole | PASS | Two pread/mmap probes, stable active allocation |
+| TC153 large directory index | SHELVED | Host fixture reproducible; kernel run incomplete |
+
+TC153 is deliberately not marked PASS. See
+`issues/TC153-large-directory-block-index-validation.md`.
+
+## Source Review
+
+Linux `super.c` initializes `blocks_lo` and uses `rb.blocks_hi` only inside
+`48BIT && rootnid_8b`. repo22 now decodes blocks and root NID together under
+that same selector.
+
+FreeBSD exposes signed `off_t` pager and vnode interfaces bounded by
+`OFF_MAX`; repo22 rejects a larger decoded inode before vnode/pager setup.
+
+The extent-hole path now zeroes only the current requested span in
+`z_erofs_do_read()` instead of allocating the complete logical extent.
+
+The directory block search now uses 64-bit bounds and a checked block-offset
+multiplication. The within-block search remains 32-bit, matching the validated
+block-sized domain.
+
+## Build Results
+
+Commands:
+
+```sh
+EROFS_ZSTDIO=0 ./build.sh
+EROFS_ZSTDIO=1 ./build.sh
+nm -u build/erofs.ko
+git diff --check -- repo-community/repo22
+```
+
+| Configuration | Module SHA256 | Result |
+|---|---|---|
+| `EROFS_ZSTDIO=0` | `65bc19d53a2a7f0525bfacadb441dab37ee318f5f5b4c62b5ecd1b5090fe46d2` | PASS |
+| `EROFS_ZSTDIO=1` | `d46ca4dfc858deaf4840fad8b8589b16d4b71b34afa96dc7acf1255890c5cef7` | PASS |
+
+Both freestanding builds completed. Neither module had an unresolved `bcmp`
+reference. The ZSTDIO-enabled module was transferred to the guest, loaded as
+KLD ID 7, and unloaded after testing.
+
+## Fixture Evidence
+
+| Fixture | SHA256 |
+|---|---|
+| `fallback-48bit-root2.erofs` | `bed3be4dfb8499d4e794b03eddd5b9cda95bc8e5571ae7d765c4852ece0d95a3` |
+| `extended-size-bit63.erofs` | `e4a0f550168f1a2911603863d0074d474e61adc787c14c0278c83a060643ee38` |
+| `extent-hole-5g.erofs` | `50014a24493918247e36511ad34a2fe8ab47ae09ea46d7fd62a1bed445a6c65f` |
+| `large-dir-intmax.erofs` | `0f90d3d57adbbbd946e41b225c1f6c464915c6abb0b13478ec9b2a318def1f72` |
+
+The TC153 hash is host generator evidence only.
+
+## TC150
+
+The fixture encoded `rootnid_2b=36`, `rootnid_8b=0`, `blocks_lo=1`, and the
+48-bit incompat bit. The 4 KiB image mounted on `/dev/md0` and produced:
+
+```text
+content=48-bit fallback root
+sha256=d361f537492113ca93cfbf91c06ebc06e2b8b695d8b6e63ad3666013eaa029f0
+root inode=36
+df total=4 one-KiB blocks
+```
+
+This proves the union was not shifted into a high block count. It does not
+replace the large-provider TC010 test.
+
+## TC151
+
+The fixture used extended inode NID 40 at byte 1280 with
+`i_size=0x8000000000000000`. The image mounted, but every access to `big.dat`
+failed before open or pager setup:
+
+| Access | Attempts | Direct rc | `truss` result |
+|---|---:|---:|---|
+| `stat` | 2 | 1, 1 | `fstatat ... ERR#97` |
+| `cat` | 2 | 1, 1 | `openat ... ERR#97` |
+| `mmap_fault` | 2 | 1, 1 | `openat ... ERR#97` |
+
+All six errors were `Integrity check failed`. No file descriptor reached the
+read or mmap phase.
+
+## TC152
+
+The mounted file size was `5368713216` bytes. The native helper probed offset
+`3221225472` with a one-byte pread and one-page private mmap:
+
+```text
+attempt 1: PASS, real 0.03 s
+attempt 2: PASS, real 0.02 s
+```
+
+The `erofs` malloc row was `3` active allocations and `768` active bytes both
+before and after. The cumulative allocation counter moved from 89 to 95, as
+expected for temporary request buffers; active memory did not scale with the
+5 GiB hole.
+
+## TC153
+
+The first 4096-byte fixture was Layout 2 and failed before the target namei
+path. A second 65536-byte base with 400 entries was still Layout 2. The tracked
+generator now converts the directory to Layout 0 by moving its complete data
+to appended blocks and produced a 90112-byte patched image with raw block 16,
+22 image blocks, and the hash listed above.
+
+The corrected fixture was not executed in the FreeBSD kernel during this run.
+TC153 remains SHELVED, and the complete attempt history and acceptance criteria
+are in its issue document.
+
+## Cleanup
+
+The qualified rerun used dmesg line count 123 before and after. It ended with:
+
+```text
+EROFS mounts: 0
+md providers: none
+EROFS modules: 0
+```
+
+An earlier unqualified probe encountered a preloaded differently named EROFS
+KLD and a `truss` exit-status ambiguity. It was discarded. The qualified run
+first unloaded that KLD, loaded the exact module hash above, captured direct
+command exit codes separately from syscall traces, and then cleaned up.
+
+## Deferred Issues
+
+- `issues/TC153-large-directory-block-index-validation.md`
+- `issues/TC010-48bit-statfs-large-provider.md`
+- `issues/extent-metadata-fixture-unavailable.md`
+
+The raw guest transcripts remain untracked under `/work/build`. This report,
+the deterministic generator, and source/test documentation are the tracked
+evidence.
@@ -0,0 +1,271 @@
+#!/bin/sh
+set -eu
+
+script_dir=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+fixture_dir=${FIXTURE_DIR:-"$script_dir/fixture"}
+artifact_dir=${ARTIFACT_DIR:-"$script_dir/artifacts"}
+
+for tool in mkfs.erofs python3 sha256sum; do
+	command -v "$tool" >/dev/null 2>&1 || {
+		echo "missing tool: $tool" >&2
+		exit 1
+	}
+done
+test ! -e "$fixture_dir" || {
+	echo "fixture directory already exists: $fixture_dir" >&2
+	exit 1
+}
+test ! -e "$artifact_dir" || {
+	echo "artifact directory already exists: $artifact_dir" >&2
+	exit 1
+}
+
+mkdir -p "$fixture_dir/fallback" "$fixture_dir/oversize" \
+	"$fixture_dir/extent" "$fixture_dir/large/huge" "$artifact_dir"
+
+FIXTURE_DIR="$fixture_dir" python3 <<'PY'
+from pathlib import Path
+import os
+
+root = Path(os.environ["FIXTURE_DIR"])
+(root / "fallback" / "root.txt").write_text(
+    "48-bit fallback root\n", encoding="ascii")
+(root / "oversize" / "big.dat").write_bytes(b"x")
+(root / "extent" / "hole.dat").write_bytes(b"\0" * (1024 * 1024))
+(root / "large" / "huge" / "anchor.txt").write_text(
+    "large directory anchor\n", encoding="ascii")
+for index in range(400):
+    (root / "large" / "huge" /
+     f"entry-{index:03d}-abcdefghijklmnopqrstuvwxyz.txt").write_text(
+        f"entry {index:03d}\n", encoding="ascii")
+PY
+find "$fixture_dir" -exec touch -h -t 197001010000.00 {} +
+
+build_image()
+{
+	uuid=$1
+	image=$2
+	source=$3
+	shift 3
+	mkfs.erofs -d0 -x-1 -T0 --all-time --all-root --workers=1 \
+		-U "$uuid" "$@" "$artifact_dir/$image" "$fixture_dir/$source"
+}
+
+build_image 22222222-3333-4444-5555-666666666650 \
+	fallback-base.erofs fallback -E force-inode-compact
+build_image 22222222-3333-4444-5555-666666666651 \
+	oversize-base.erofs oversize -E force-inode-extended
+build_image 22222222-3333-4444-5555-666666666652 \
+	extent-base.erofs extent -E legacy-compress,force-inode-extended -z lz4
+build_image 22222222-3333-4444-5555-666666666653 \
+	large-dir-base.erofs large -E force-inode-extended
+
+ARTIFACT_DIR="$artifact_dir" python3 <<'PY'
+from pathlib import Path
+import hashlib
+import os
+import struct
+
+artifact_dir = Path(os.environ["ARTIFACT_DIR"])
+SUPER = 1024
+FEATURE_INCOMPAT = SUPER + 80
+ROOTNID_2B = SUPER + 14
+BLOCKS_LO = SUPER + 36
+META_BLKADDR = SUPER + 40
+ROOTNID_8B = SUPER + 112
+EROFS_FEATURE_INCOMPAT_48BIT = 0x80
+EROFS_INODE_COMPRESSED_FULL = 1
+Z_EROFS_ADVISE_EXTENTS = 0x1
+Z_EROFS_EXTENT_RECSZ_16 = 0x4
+HOLE_SIZE = 5 * 1024 * 1024 * 1024 + 4096
+HUGE_DIR_BLOCKS = (1 << 31) + 1
+
+
+def u16(image, offset):
+    return struct.unpack_from("<H", image, offset)[0]
+
+
+def u32(image, offset):
+    return struct.unpack_from("<I", image, offset)[0]
+
+
+def u64(image, offset):
+    return struct.unpack_from("<Q", image, offset)[0]
+
+
+def put_u16(image, offset, value):
+    struct.pack_into("<H", image, offset, value)
+
+
+def put_u32(image, offset, value):
+    struct.pack_into("<I", image, offset, value)
+
+
+def put_u64(image, offset, value):
+    struct.pack_into("<Q", image, offset, value)
+
+
+def update_superblock_checksum(image):
+    polynomial = 0x82F63B78
+    checksum = 0xFFFFFFFF
+    block_size = 1 << image[SUPER + 12]
+    put_u32(image, SUPER + 4, 0)
+    for byte in image[SUPER:block_size]:
+        checksum ^= byte
+        for _ in range(8):
+            checksum = (checksum >> 1) ^ (
+                polynomial if checksum & 1 else 0)
+    put_u32(image, SUPER + 4, checksum & 0xFFFFFFFF)
+
+
+def inode_offset(image, nid):
+    block_bits = image[SUPER + 12]
+    return (u32(image, META_BLKADDR) << block_bits) + (nid << 5)
+
+
+def inode_info(image, nid):
+    offset = inode_offset(image, nid)
+    inode_format = u16(image, offset)
+    inode_size = 64 if inode_format & 1 else 32
+    size = u64(image, offset + 8) if inode_size == 64 else u32(
+        image, offset + 8)
+    xattr_count = u16(image, offset + 2)
+    xattr_size = 0 if xattr_count == 0 else 12 + 4 * (xattr_count - 1)
+    layout = (inode_format >> 1) & 7
+    return offset, inode_format, inode_size, xattr_size, layout, size
+
+
+def directory_entries(image, nid):
+    offset, _, inode_size, xattr_size, layout, size = inode_info(image, nid)
+    block_size = 1 << image[SUPER + 12]
+    assert 0 < size <= block_size
+    if layout == 2:
+        data = offset + inode_size + xattr_size
+    elif layout == 0:
+        data = u32(image, offset + 16) << image[SUPER + 12]
+    else:
+        raise AssertionError(f"unsupported directory layout {layout}")
+    first_nameoff = u16(image, data + 8)
+    assert first_nameoff >= 12 and first_nameoff % 12 == 0
+    count = first_nameoff // 12
+    entries = []
+    for index in range(count):
+        entry = data + index * 12
+        child_nid = u64(image, entry)
+        nameoff = u16(image, entry + 8)
+        endoff = (u16(image, entry + 20)
+                  if index + 1 < count else size)
+        name = bytes(image[data + nameoff:data + endoff]).split(b"\0", 1)[0]
+        assert name
+        entries.append((name, child_nid))
+    return entries
+
+
+def child_nid(image, parent_nid, name):
+    return next(nid for entry_name, nid in directory_entries(image, parent_nid)
+                if entry_name == name)
+
+
+def convert_inline_directory_to_plain(image, nid):
+    offset, inode_format, inode_size, xattr_size, layout, size = inode_info(
+        image, nid)
+    assert layout == 2 and inode_size == 64 and size > 0
+    block_size = 1 << image[SUPER + 12]
+    tail_size = size % block_size
+    assert tail_size > 0
+    raw_block = u32(image, offset + 16)
+    full_size = size - tail_size
+    inline_offset = offset + inode_size + xattr_size
+    directory_data = bytes(
+        image[raw_block * block_size:raw_block * block_size + full_size] +
+        image[inline_offset:inline_offset + tail_size])
+    assert len(directory_data) == size
+
+    new_raw_block = (len(image) + block_size - 1) // block_size
+    image.extend(b"\0" * (new_raw_block * block_size - len(image)))
+    image.extend(directory_data)
+    image.extend(b"\0" * (-len(image) % block_size))
+    put_u16(image, offset, inode_format & ~(7 << 1))
+    put_u32(image, offset + 16, new_raw_block)
+    put_u32(image, BLOCKS_LO, len(image) // block_size)
+    assert inode_info(image, nid)[4] == 0
+    return new_raw_block, len(image) // block_size
+
+
+def write_image(name, image):
+    update_superblock_checksum(image)
+    (artifact_dir / name).write_bytes(image)
+
+
+evidence = []
+
+fallback = bytearray((artifact_dir / "fallback-base.erofs").read_bytes())
+fallback_root = u16(fallback, ROOTNID_2B)
+fallback_blocks = u32(fallback, BLOCKS_LO)
+assert fallback_root != 0 and u64(fallback, ROOTNID_8B) == 0
+put_u32(fallback, FEATURE_INCOMPAT,
+        u32(fallback, FEATURE_INCOMPAT) | EROFS_FEATURE_INCOMPAT_48BIT)
+put_u64(fallback, ROOTNID_8B, 0)
+write_image("fallback-48bit-root2.erofs", fallback)
+evidence.append(
+    f"fallback rootnid_2b={fallback_root} rootnid_8b=0 "
+    f"blocks_lo={fallback_blocks} union=0x{u16(fallback, ROOTNID_2B):04x}")
+
+oversize = bytearray((artifact_dir / "oversize-base.erofs").read_bytes())
+oversize_root = u16(oversize, ROOTNID_2B)
+oversize_nid = child_nid(oversize, oversize_root, b"big.dat")
+oversize_off, oversize_format, oversize_isize, _, _, _ = inode_info(
+    oversize, oversize_nid)
+assert oversize_format & 1 and oversize_isize == 64
+put_u64(oversize, oversize_off + 8, 1 << 63)
+write_image("extended-size-bit63.erofs", oversize)
+evidence.append(
+    f"oversize nid={oversize_nid} inode_off={oversize_off} "
+    f"i_size=0x{u64(oversize, oversize_off + 8):016x}")
+
+extent = bytearray((artifact_dir / "extent-base.erofs").read_bytes())
+extent_root = u16(extent, ROOTNID_2B)
+extent_nid = child_nid(extent, extent_root, b"hole.dat")
+extent_off, extent_format, extent_isize, extent_xattr, extent_layout, _ = \
+    inode_info(extent, extent_nid)
+assert extent_format & 1 and extent_isize == 64
+assert extent_layout == EROFS_INODE_COMPRESSED_FULL
+header = (extent_off + extent_isize + extent_xattr + 7) & ~7
+record = (header + 8 + 15) & ~15
+root_off = inode_offset(extent, extent_root)
+assert not (header < root_off + 64 and root_off < record + 16)
+put_u64(extent, extent_off + 8, HOLE_SIZE)
+struct.pack_into("<IHH", extent, header, 1,
+                 Z_EROFS_ADVISE_EXTENTS | Z_EROFS_EXTENT_RECSZ_16, 0)
+struct.pack_into("<IIII", extent, record, 0, 0, 0, 0)
+write_image("extent-hole-5g.erofs", extent)
+evidence.append(
+    f"extent-hole nid={extent_nid} inode_off={extent_off} size={HOLE_SIZE} "
+    f"header={header} record={record} extents=1 recsize=16 plen=0 lstart=0")
+
+large = bytearray((artifact_dir / "large-dir-base.erofs").read_bytes())
+large_root = u16(large, ROOTNID_2B)
+large_nid = child_nid(large, large_root, b"huge")
+large_off, large_format, large_isize, _, _, _ = inode_info(large, large_nid)
+assert large_format & 1 and large_isize == 64
+large_raw_block, large_image_blocks = convert_inline_directory_to_plain(
+    large, large_nid)
+large_size = HUGE_DIR_BLOCKS * (1 << large[SUPER + 12])
+put_u64(large, large_off + 8, large_size)
+write_image("large-dir-intmax.erofs", large)
+evidence.append(
+    f"large-dir nid={large_nid} inode_off={large_off} size={large_size} "
+    f"blocks={HUGE_DIR_BLOCKS} last_block={HUGE_DIR_BLOCKS - 1} "
+    f"layout=flat-plain raw_block={large_raw_block} "
+    f"image_blocks={large_image_blocks}")
+
+(artifact_dir / "fixture-evidence.txt").write_text(
+    "\n".join(evidence) + "\n", encoding="ascii")
+with (artifact_dir / "SHA256SUMS").open("w", encoding="ascii") as sums:
+    for path in sorted(artifact_dir.glob("*.erofs")):
+        sums.write(f"{hashlib.sha256(path.read_bytes()).hexdigest()}  "
+                   f"{path.name}\n")
+PY
+
+cat "$artifact_dir/fixture-evidence.txt"
+cat "$artifact_dir/SHA256SUMS"