Fix race between device removal completion and pool export

vdev_remove_complete() finalizes a device removal in two phases under
the spa lock framework. Between the two phases it called
spa_vdev_exit(), which drops both the config locks (SCL_ALL) and
spa_namespace_lock and blocks on a txg sync. By that point
vdev_remove_replace_with_indirect() has already set svr->svr_thread =
NULL, and that is the only thing the export path (spa_export_common()
-> spa_async_suspend() -> spa_vdev_remove_suspend()) waits on. Once the
namespace lock is dropped, a concurrent export or destroy can acquire
it and set spa->spa_export_thread. When the removal thread re-enters
for its second phase via spa_vdev_enter(), it trips the
ASSERT0P(spa->spa_export_thread) assertion.

Hold spa_namespace_lock across both phases instead of dropping and
re-taking it: the intermediate spa_vdev_exit() becomes
spa_vdev_config_exit(), which drops only SCL_ALL and syncs the txg
while keeping the namespace lock held, and the second spa_vdev_enter()
becomes spa_vdev_config_enter(). Because the namespace lock is never
dropped between the phases, a concurrent export blocks in
spa_namespace_enter() and cannot set spa_export_thread until removal
finalization is done. This mirrors the multi-phase locking pattern
already used by the attach/detach and split paths in spa.c.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Prakash Surya <[email protected]>
Closes #18657

module/zfs/vdev_removal.c

@@ -1432,14 +1432,17 @@ vdev_remove_complete(spa_t *spa)
 	vdev_remove_replace_with_indirect(vd, txg);
 
 	/*
-	 * We now release the locks, allowing spa_sync to run and finish the
+	 * Release the config lock and allow spa_sync to run and finish the
 	 * removal via vdev_remove_complete_sync in syncing context.
 	 *
+	 * Retain the namespace lock to prevent the pool from being exported
+	 * or destroyed before completion of the device removal.
+	 *
 	 * Note that we hold on to the vdev_t that has been replaced.  Since
 	 * it isn't part of the vdev tree any longer, it can't be concurrently
 	 * manipulated, even while we don't have the config lock.
 	 */
-	(void) spa_vdev_exit(spa, NULL, txg, 0);
+	(void) spa_vdev_config_exit(spa, NULL, txg, 0, FTAG);
 
 	/*
 	 * Top ZAP should have been transferred to the indirect vdev in
@@ -1452,7 +1455,8 @@ vdev_remove_complete(spa_t *spa)
 	 */
 	ASSERT0(vd->vdev_leaf_zap);
 
-	txg = spa_vdev_enter(spa);
+	/* Update the vdev labels and dirty the config. */
+	txg = spa_vdev_config_enter(spa);
 	(void) vdev_label_init(vd, 0, VDEV_LABEL_REMOVE);
 	/*
 	 * Request to update the config and the config cachefile.