ZIO: Order a batch arrival against the hold it releases

zio_batch_arrive() publishes onto zb_arrived and then drops its hold;
whoever drops the last one walks that list with a plain load.  Neither
side carries a barrier, so on arm64, where FreeBSD's atomics are
relaxed, a thread can see zb_holds reach zero and still read a stale
zb_arrived.  The missed member never leaves ZIO_VDEV_IO_DONE, its parent
never completes, and the txg sync thread waits forever in zio_wait().

Linux and FreeBSD/x86 are unaffected; their atomics are ordered.  A
Cortex-A76 litmus test observes the reordering in roughly 4% of the
cases where the reader wins the batch.

Reported by: Dave Cottlehuber <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Reviewed-by: Alexander Motin <[email protected]>
Signed-off-by: Nick Price <[email protected]>
Closes #19152

module/zfs/zio.c

@@ -2360,6 +2360,9 @@ zio_batch_run(zio_batch_t *zb)
 {
 	zio_t *list = NULL, *zio, *next;
 
+	/* Pairs with zio_batch_arrive(). */
+	membar_consumer();
+
 	for (zio = zb->zb_arrived; zio != NULL; zio = next) {
 		next = zio->io_exec_next;
 		zio->io_batch = NULL;
@@ -2450,6 +2453,9 @@ zio_batch_arrive(zio_t *zio)
 		zio->io_exec_next = head;
 	} while (atomic_cas_ptr(&zb->zb_arrived, head, zio) != head);
 
+	/* Publish the arrival before dropping the hold that runs the batch. */
+	membar_producer();
+
 	if (atomic_dec_64_nv(&zb->zb_holds) == 0) {
 		zio_taskq_dispatch_func(zio, ZIO_TASKQ_INTERRUPT, B_FALSE,
 		    zio_batch_execute);