erofs-freebsd-out-tree/issues/TC153-large-directory-block-index-validation.md

TC153 Large Directory Block Index Validation

Status: RESOLVED - exact-source kernel validation passed

Last updated: 2026-08-09

Priority: Critical validation gap

Implementation status: Source fix present, build-verified, and dynamically validated

Problem

erofs_find_target_block() used signed int values for the binary-search front, back, and midpoint indexes. A directory whose logical size describes 2147483649 4 KiB blocks produces a final block index of 2147483648. Converting that value to int can make the initial back bound negative. The search can then return ENOENT without reading any directory block, and the FreeBSD namecache can retain a false negative result for corrupt media.

The review change uses uint64_t for block-search bounds and midpoints, checks the midpoint multiplication, and handles the mid == 0 lower-bound case. The within-block search remains uint32_t because a validated EROFS directory block can contain only a block-sized number of entries.

Affected Features

• Cold pathname lookup in very large or maliciously sized directories.
• Corruption reporting and EINTEGRITY propagation from directory reads.
• Negative namecache correctness after a failed lookup.
• Linux/FreeBSD behavioral review of EROFS's two-level directory search.

Normal directories are not expected to approach this bound. The practical risk is a crafted image causing an incorrect ENOENT, not an ordinary mkfs image losing entries.

Trigger Conditions

All of the following are needed to exercise the original width error:

1. A directory inode is accepted with a logical block count greater than INT_MAX.
2. The directory layout reaches erofs_find_target_block() rather than being rejected by an earlier inline-data bounds check.
3. A cold lookup is issued for a name that is not satisfied by an already cached vnode or negative namecache entry.
4. The backing image is too short for the computed midpoint read, so the fixed code should return EINTEGRITY instead of a synthetic miss.

For 4 KiB blocks, TC153 encodes a size of 8796093026304 bytes, 2147483649 blocks, and a final index of 2147483648.

Current Analysis

The source change is mechanically narrow and has passed both repo22 build configurations. The checked multiplication protects a future block-size or index change even though the current OFF_MAX inode limit already bounds the product. The mid == 0 guard prevents unsigned subtraction from wrapping when the search moves below the first block.

Dynamic proof must show more than a large st_size. It must prove that the lookup enters the block binary search. An image rejected in erofs_read_inode() is not evidence for this fix, and an ENOENT observed after a warm negative cache is ambiguous.

Attempts and Results

Attempt 1: Small FLAT_INLINE image

• Artifact directory: /work/build/repo22-review-artifact.LtqNxH
• Base image: large-dir-base.erofs, 4096 bytes, SHA256 973d2a1c90ac1bf776ed76a1fc6a7e88b2156fac3bf04ddbae19cfafebbd562e
• Patched image: large-dir-intmax.erofs, 4096 bytes, SHA256 ce5cad3eff34ea50ca89eedf93d8ef90e6cea96858a138685fae45f39084de1f
• Recorded inode: NID 40 at byte 1280.

The directory was Layout 2 (FLAT_INLINE). Patching only i_size made its declared inline tail range invalid, so inode validation rejected it before erofs_find_target_block(). This attempt is invalid as TC153 dynamic evidence and must never be marked PASS.

Attempt 2: More entries to force block data

• Artifact directory: /work/build/repo22-review-artifact2.FmMTup
• Generated base: large-dir-base.erofs, 65536 bytes, SHA256 50eb28351788ab1801408aca0a6fca6352755f5ca56efe6e2bb3e2fe08305663
• Directory: NID 40, 21052 bytes, still Layout 2.

The generation process was interrupted before producing a patched image or a FreeBSD result. A later deterministic rerun reproduced the same base hash and showed why the approach failed: erofs-utils 1.8.6 tail-packs directories even when they span multiple data blocks. Its noinline_data option does not turn directory tail packing off.

Attempt 3: Follow-up agents

Multiple follow-up agents were assigned the fixture and guest validation. They either stopped while preparing the second image or returned no executed commands, files, hashes, or guest output. These attempts added no evidence and are recorded to prevent their elapsed time from being mistaken for progress.

Attempt 4: Sparse-provider design

A guest-side sparse vnode provider was considered so a valid directory could declare the complete multi-terabyte range. It was not executed. Such a scheme must prove the GEOM media size, avoid materializing terabytes, preserve valid initial directory blocks, and still force a cold read at the binary-search midpoint. No result exists for this approach.

Attempt 5: Reproducible FLAT_PLAIN conversion

The tracked generator now copies all 21052 bytes of the generated directory to appended blocks, changes only that inode to Layout 0 (FLAT_PLAIN), updates the image block count and CRC32C, and then applies the large logical size.

Host reproduction on 2026-08-09 produced:

• large-dir-base.erofs: 65536 bytes, SHA256 50eb28351788ab1801408aca0a6fca6352755f5ca56efe6e2bb3e2fe08305663
• large-dir-intmax.erofs: 90112 bytes, SHA256 0f90d3d57adbbbd946e41b225c1f6c464915c6abb0b13478ec9b2a318def1f72
• Patched inode: NID 40 at byte 1280, Layout 0, raw block 16.
• Declared image blocks: 22.

This resolved the fixture-construction problem. The qualified kernel run is recorded below.

Attempt 6: Qualified sparse GEOM provider

The G3 takeover generated a checksum-valid sparse prefix from Attempt 5 and performed one targeted FreeBSD 15 run against the exact source requested by the assignment.

• Source commit: 9ae22009f23a65320730072a780998e80aa9b728.
• KLD SHA256: 19ad086bd2508cbb4c57b1a51f38c93057d438b84fbcfa2e637ff2519f5bc590.
• Sparse-prefix SHA256: f9337f83b1f568f6d904331a7749e6362a691055cd5af95b59bc89772f04e3b0.
• Prefix qualification: NID 40, inode offset 1280, extended Layout 0, start block 16, 2147483649 directory blocks, final block index 2147483648, valid superblock checksum.
• Sparse provider: 8796093091840 logical bytes, 448 allocated 512-byte sectors, and the first 90112 bytes retained the prefix SHA256.
• GEOM diskinfo size: 8796093091840 bytes.
• Mounted /huge: size 8796093026304, NID 40.

The first two cold stat operations for /huge/missing each returned EINTEGRITY (97). Root readdir then returned three complete entries and validated all kernel/libc restart cookies. A third lookup after that readdir again returned EINTEGRITY. No lookup returned ENOENT, so no false negative namecache entry masked the corruption.

Unmount, md detach, sparse-provider removal, and KLD unload all succeeded. The only new dmesg lines in the complete G3 run were expected SIGBUS child exits from the unrelated assigned mmap tests; TC153 added no diagnostic.

Historical Remaining Validation

Attempt 6 completed steps 1-5 and 7 below. The assignment mandated the exact fixed source, so the optional pre-fix comparison in step 6 was not run.

1. Regenerate the images with tests/results/manual/2026-08-09T0124Z-final-review/prepare-fixtures.sh and verify the hashes and Layout 0 assertion.
2. Load the exact module under test in a clean FreeBSD 15 guest.
3. Mount the 90112-byte image and confirm /huge reports 8796093026304 bytes.
4. With a cold parent cache, run two lookups of /huge/missing and capture direct command exit codes plus truss errno.
5. Require EINTEGRITY on every fixed-module lookup, including after root readdir. Confirm no negative cache changes the result.
6. Repeat with the pre-fix module to demonstrate the erroneous ENOENT path.
7. Record dmesg, mount/md/KLD cleanup, module and fixture hashes in a dated manual report.

Original Acceptance Criteria

TC153 can move from SHELVED to PASS only after the fixed FreeBSD KLD returns EINTEGRITY repeatedly from the valid Layout 0 fixture, the pre-fix behavior is distinguished, no trap or panic occurs, and complete cleanup evidence is recorded. Host fixture generation and static source review alone are insufficient.

Resolution

TC153 is PASS because the exact fixed FreeBSD KLD returned EINTEGRITY repeatedly from the valid Layout 0 fixture before and after root readdir, no trap or panic occurred, and complete hash/provider/cleanup evidence is recorded in tests/results/manual/2026-08-09T1059Z-g3/manual-test-report.md. Host fixture generation and static source review alone remain insufficient.