L2ARC: do not feed a device while its rebuild is pending

l2arc_add_vdev() leaves the device at l2ad_start with l2ad_first set,
marks it l2ad_rebuild, and starts its feed thread.  The rebuild runs
later from the spa async thread and only then restores the geometry the
on-disk header describes.

A feed in that window writes at l2ad_start believing the device is
empty, and its headers go on the head of l2ad_buflist while the rebuild
appends the restored ones at the tail.  l2arc_evict() walks that list
from the tail and stops at the first header outside the range it is
clearing, so once list position stops tracking device offset the walk
exits early, headers the hand is about to overwrite are never destroyed,
and the writes that replace them charge the space twice.  vs_alloc
passes vs_space and zpool(8) reports 16.0E free.

l2arc_dev_invalid() rejected a device with l2ad_rebuild or l2ad_trim_all
set until b525525b4 made the feed threads per-device; neither flag has
had a reader since.  Restore the test, after SCL_L2ARC is taken so it
also orders against vdev_reopen() setting the flag.

External-issue: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250323

Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Ameer Hamza <[email protected]>
Signed-off-by: Nick Price <[email protected]>
Closes #19075

module/zfs/arc.c

@@ -10250,6 +10250,11 @@ l2arc_feed_thread(void *arg)
 		if (!spa_config_tryenter(spa, SCL_L2ARC, dev, RW_READER))
 			continue;
 
+		if (dev->l2ad_rebuild || dev->l2ad_trim_all) {
+			spa_config_exit(spa, SCL_L2ARC, dev);
+			continue;
+		}
+
 		/*
 		 * Avoid contributing to memory pressure.
 		 */