Fix race condition in raidz expansion startup

When starting a raidz expansion, we first add the new child to the 
raidz device, then pause existing trim and initialization operations, 
and then set up all the expansion metadata. However, if the system 
crashes between the first step and the third step, the pool will have 
problems importing; the raidz device will believe that it is the new 
larger size, and will attempt to read based on that geometry, rather 
than the correct (unexpanded) geometry.

We move the third step immediately after the first one, before 
releasing the spa config. This way, the raidz expansion will be set 
up in one atomic operation, rather than in multiple steps that can 
fail. To prevent any interactions between the initialize and trim 
code, the raidz expansion thread is not started until after the 
initialize and trim processes have been stopped.

Sponsored-by: Klara, Inc.
Sponsored-by: Fastmail Pty Ltd
Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Paul Dagnelie <[email protected]>
Closes #18920

module/zfs/spa.c

@@ -8574,6 +8574,12 @@ spa_vdev_attach(spa_t *spa, uint64_t guid, nvlist_t *nvroot, int replacing,
 	dtl_max_txg = txg + TXG_CONCURRENT_STATES;
 
 	if (raidz) {
+		dmu_tx_t *tx = dmu_tx_create_assigned(spa->spa_dsl_pool,
+		    txg);
+		dsl_sync_task_nowait(spa->spa_dsl_pool, vdev_raidz_attach_sync,
+		    newvd, tx);
+		dmu_tx_commit(tx);
+
 		/*
 		 * Wait for the youngest allocations and frees to sync,
 		 * and then wait for the deferral of those frees to finish.
@@ -8591,12 +8597,7 @@ spa_vdev_attach(spa_t *spa, uint64_t guid, nvlist_t *nvroot, int replacing,
 
 		vdev_dirty_leaves(tvd, VDD_DTL, dtl_max_txg);
 		vdev_config_dirty(tvd);
-
-		dmu_tx_t *tx = dmu_tx_create_assigned(spa->spa_dsl_pool,
-		    dtl_max_txg);
-		dsl_sync_task_nowait(spa->spa_dsl_pool, vdev_raidz_attach_sync,
-		    newvd, tx);
-		dmu_tx_commit(tx);
+		zthr_wakeup(spa->spa_raidz_expand_zthr);
 	} else {
 		vdev_dtl_dirty(newvd, DTL_MISSING, TXG_INITIAL,
 		    dtl_max_txg - TXG_INITIAL);

module/zfs/vdev_raidz.c

@@ -5238,7 +5238,6 @@ vdev_raidz_attach_sync(void *arg, dmu_tx_t *tx)
 	vdrz->vn_vre.vre_offset = 0;
 	vdrz->vn_vre.vre_failed_offset = UINT64_MAX;
 	spa->spa_raidz_expand = &vdrz->vn_vre;
-	zthr_wakeup(spa->spa_raidz_expand_zthr);
 
 	/*
 	 * Dirty the config so that ZPOOL_CONFIG_RAIDZ_EXPANDING will get