erofs-freebsd-out-tree/tests/TC133-nfs-export-stress.md

# Test Case: FreeBSD NFSv3 READDIRPLUS and Export Stress

**Test ID**: TC133-nfs-export-stress
**Category**: NFS Export
**Priority**: High
**Regression**: directory cookies, EOF, concurrency, and export stability

## Objective

Stress the FreeBSD NFSv3 export with a 10,000+ entry directory, small negotiated
readdir sizes, repeated cookie pagination, multiple concurrent traversals, and
parallel reads/stat operations. Verify no duplicates, omissions, premature EOF,
stale handles, panic, or resource leak.

## Preconditions

- Complete TC131 server setup.
- A fixture with at least 12,050 deterministically named files in `bigdir`, a
  `concurrent` file set, and a larger sequential-read file.
- Four client mount points are available.

## Procedure

1. Mount four NFSv3 client views with READDIRPLUS. Request small readdir sizes
   on three mounts and record the effective values:

   ```sh
   mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=512 \
     127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-512
   mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=1024 \
     127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-1024
   mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=4096 \
     127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-4096
   mount_nfs -o nfsv3,tcp,rdirplus \
     127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-default
   nfsstat -m
   ```

   FreeBSD may clamp very small requests to its client minimum. Record both the
   requested and effective values; a clamp is an environment characteristic,
   not a test failure.

2. Traverse each mount and verify exact names, counts, and uniqueness:

   ```sh
   jot -w file-%05d 12050 0 > expected.names
   for M in 512 1024 4096 default; do
     find /mnt/repo22-nfs-$M/bigdir -type f -maxdepth 1 | \
       sed 's#.*/##' > names.$M
     test "$(wc -l < names.$M)" -eq 12050
     sort names.$M | uniq -d > duplicates.$M
     test ! -s duplicates.$M
     sort names.$M > sorted.$M
     cmp expected.names sorted.$M
   done
   sha256 expected.names sorted.*
   ```

3. Verify `.` and `..` are each returned exactly once and EOF is reached only
   after all real entries:

   ```sh
   ls -a1 /mnt/repo22-nfs-default/bigdir > dot-list
   test "$(wc -l < dot-list)" -eq 12052
   test "$(grep -cx '\.' dot-list)" -eq 1
   test "$(grep -cx '\.\.' dot-list)" -eq 1
   ```

4. Force cold cookie pagination by repeatedly unmounting/remounting the small
   client and comparing every complete listing:

   ```sh
   for round in 1 2 3 4 5; do
     umount /mnt/repo22-nfs-512
     mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=512 \
       127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-512
     find /mnt/repo22-nfs-512/bigdir -type f -maxdepth 1 | \
       sed 's#.*/##' | sort > cold-$round.sorted
     cmp expected.names cold-$round.sorted
   done
   ```

5. Run at least eight traversal workers, three rounds each, distributed across
   the four mounts. Save every PID and wait for every PID individually; a bare
   final `wait` does not prove that all background jobs succeeded:

   ```sh
   pids=""
   worker=0
   for round in 1 2 3; do
     for M in 512 1024 4096 default; do
       worker=$((worker + 1))
       (find /mnt/repo22-nfs-$M/bigdir -type f -maxdepth 1 | \
          sed 's#.*/##' | sort | cmp expected.names -) \
          > walk-$worker.log 2>&1 &
       pids="$pids $!"
     done
   done
   status=0
   for pid in $pids; do
     wait "$pid" || status=1
   done
   test "$status" -eq 0
   ```

6. Run parallel data and metadata operations:

   ```sh
   pids=""
   for M in 512 1024 4096 default; do
     (find /mnt/repo22-nfs-$M/concurrent -type f -maxdepth 1 -print0 | \
       xargs -0 -n 1 -P 8 cat > /dev/null) > cat-$M.log 2>&1 &
     pids="$pids $!"
     (find /mnt/repo22-nfs-$M/concurrent -type f -maxdepth 1 -print0 | \
       xargs -0 -n 1 -P 8 stat -f '%i %z' > /dev/null) \
       > stat-$M.log 2>&1 &
     pids="$pids $!"
   done
   status=0
   for pid in $pids; do
     wait "$pid" || status=1
   done
   test "$status" -eq 0
   ```

7. Record throughput as information only:

   ```sh
   /usr/bin/time -p dd if=/mnt/repo22-nfs-default/throughput.dat \
     of=/dev/null bs=1m
   ```

8. Record NFS and kernel state:

   ```sh
   nfsstat -c
   nfsstat -s
   vmstat -H
   dmesg
   ```

   READDIRPLUS must be non-zero. Timed-out RPCs, retries, successful write RPCs,
   stale-handle messages, traps, and panics must be zero.

## Expected Results

- Every listing contains exactly 12,050 unique expected names.
- Cold remount and concurrent listings have identical SHA256 values.
- Cookie pagination has no duplicate or missing entry and no premature EOF.
- Dot-omitted EROFS directories expose one `.` and one `..` through NFS.
- Concurrent reads and stats complete without stale handles or deadlock.
- READDIRPLUS activity is visible in client/server statistics.
- Throughput is reported without a fixed pass/fail threshold.

## Cleanup

Unmount every NFS client while nfsd is still running, then reload an empty
export list, stop services, unmount EROFS, detach md, and unload the module:

```sh
for M in 512 1024 4096 default; do
  umount /mnt/repo22-nfs-$M
done
: > /etc/exports
service mountd onereload
service nfsd onestop
service mountd onestop
service rpcbind onestop
umount /mnt/repo22-erofs
mdconfig -d -u 42
kldunload erofs
```

Final EROFS/NFS mount counts, md units, loaded EROFS modules, service PIDs, and
unexpected new dmesg lines must all be zero. Do not stop NFS services before
client unmounts; hard localhost NFS mounts can otherwise become uninterruptible.