Test Case: FreeBSD NFSv3 READDIRPLUS and Export Stress

Test ID: TC133-nfs-export-stress Category: NFS Export Priority: High Regression: directory cookies, EOF, concurrency, and export stability

Objective

Stress the FreeBSD NFSv3 export with a 10,000+ entry directory, small negotiated readdir sizes, repeated cookie pagination, multiple concurrent traversals, and parallel reads/stat operations. Verify no duplicates, omissions, premature EOF, stale handles, panic, or resource leak.

Preconditions

Complete TC131 server setup.
A fixture with at least 12,050 deterministically named files in bigdir, a concurrent file set, and a larger sequential-read file.
Four client mount points are available.

Procedure

Mount four NFSv3 client views with READDIRPLUS. Request small readdir sizes on three mounts and record the effective values:

mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=512 \
  127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-512
mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=1024 \
  127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-1024
mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=4096 \
  127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-4096
mount_nfs -o nfsv3,tcp,rdirplus \
  127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-default
nfsstat -m

FreeBSD may clamp very small requests to its client minimum. Record both the requested and effective values; a clamp is an environment characteristic, not a test failure.

Traverse each mount and verify exact names, counts, and uniqueness:

jot -w file-%05d 12050 0 > expected.names
for M in 512 1024 4096 default; do
  find /mnt/repo22-nfs-$M/bigdir -type f -maxdepth 1 | \
    sed 's#.*/##' > names.$M
  test "$(wc -l < names.$M)" -eq 12050
  sort names.$M | uniq -d > duplicates.$M
  test ! -s duplicates.$M
  sort names.$M > sorted.$M
  cmp expected.names sorted.$M
done
sha256 expected.names sorted.*

Verify . and .. are each returned exactly once and EOF is reached only after all real entries:

ls -a1 /mnt/repo22-nfs-default/bigdir > dot-list
test "$(wc -l < dot-list)" -eq 12052
test "$(grep -cx '\.' dot-list)" -eq 1
test "$(grep -cx '\.\.' dot-list)" -eq 1

Force cold cookie pagination by repeatedly unmounting/remounting the small client and comparing every complete listing:

for round in 1 2 3 4 5; do
  umount /mnt/repo22-nfs-512
  mount_nfs -o nfsv3,tcp,rdirplus,readdirsize=512 \
    127.0.0.1:/mnt/repo22-erofs /mnt/repo22-nfs-512
  find /mnt/repo22-nfs-512/bigdir -type f -maxdepth 1 | \
    sed 's#.*/##' | sort > cold-$round.sorted
  cmp expected.names cold-$round.sorted
done

Run at least eight traversal workers, three rounds each, distributed across the four mounts. Save every PID and wait for every PID individually; a bare final wait does not prove that all background jobs succeeded:

pids=""
worker=0
for round in 1 2 3; do
  for M in 512 1024 4096 default; do
    worker=$((worker + 1))
    (find /mnt/repo22-nfs-$M/bigdir -type f -maxdepth 1 | \
       sed 's#.*/##' | sort | cmp expected.names -) \
       > walk-$worker.log 2>&1 &
    pids="$pids $!"
  done
done
status=0
for pid in $pids; do
  wait "$pid" || status=1
done
test "$status" -eq 0

Run parallel data and metadata operations:

pids=""
for M in 512 1024 4096 default; do
  (find /mnt/repo22-nfs-$M/concurrent -type f -maxdepth 1 -print0 | \
    xargs -0 -n 1 -P 8 cat > /dev/null) > cat-$M.log 2>&1 &
  pids="$pids $!"
  (find /mnt/repo22-nfs-$M/concurrent -type f -maxdepth 1 -print0 | \
    xargs -0 -n 1 -P 8 stat -f '%i %z' > /dev/null) \
    > stat-$M.log 2>&1 &
  pids="$pids $!"
done
status=0
for pid in $pids; do
  wait "$pid" || status=1
done
test "$status" -eq 0

Record throughput as information only:

/usr/bin/time -p dd if=/mnt/repo22-nfs-default/throughput.dat \
  of=/dev/null bs=1m

Record NFS and kernel state:
```
nfsstat -c
nfsstat -s
vmstat -H
dmesg
```
READDIRPLUS must be non-zero. Timed-out RPCs, retries, successful write RPCs, stale-handle messages, traps, and panics must be zero.

Expected Results

Every listing contains exactly 12,050 unique expected names.
Cold remount and concurrent listings have identical SHA256 values.
Cookie pagination has no duplicate or missing entry and no premature EOF.
Dot-omitted EROFS directories expose one . and one .. through NFS.
Concurrent reads and stats complete without stale handles or deadlock.
READDIRPLUS activity is visible in client/server statistics.
Throughput is reported without a fixed pass/fail threshold.

Cleanup

Unmount every NFS client while nfsd is still running, then reload an empty export list, stop services, unmount EROFS, detach md, and unload the module:

for M in 512 1024 4096 default; do
  umount /mnt/repo22-nfs-$M
done
: > /etc/exports
service mountd onereload
service nfsd onestop
service mountd onestop
service rpcbind onestop
umount /mnt/repo22-erofs
mdconfig -d -u 42
kldunload erofs

Final EROFS/NFS mount counts, md units, loaded EROFS modules, service PIDs, and unexpected new dmesg lines must all be zero. Do not stop NFS services before client unmounts; hard localhost NFS mounts can otherwise become uninterruptible.

5.4 KiB Raw Blame History