Wait for the initialize and trim threads to exit

An initializing thread marks the vdev VDEV_INITIALIZE_COMPLETE, drops
vdev_initialize_lock, syncs out the new state with txg_wait_synced()
and only then clears vdev_initialize_thread.  spa_vdev_activity_in_
progress_impl() looks at the state alone:

    boolean_t in_progress = (activity == ZPOOL_WAIT_INITIALIZE) ?
        (vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE) :
        (vd->vdev_trim_state == VDEV_TRIM_ACTIVE);

so "zpool wait -t initialize" and "zpool initialize -w" return while
the thread is still in that txg wait.  A command issued right after
them then hits the vdev_initialize_thread != NULL checks in
spa_vdev_initialize_impl() and fails with EBUSY, both for uninit and
for starting another initialization:

    cannot initialize '/var/tmp/zpool_disk2.dat': currently initializing

ZTS zpool_initialize_multiple_pools fails on this: it runs "zpool
initialize -u -a" right after "zpool initialize -w -a".  The window is
one txg sync wide, so a loaded machine hits it and an idle one usually
does not.

Wait for the thread as well as for the state, and notify the waiters
once the thread has cleared itself, so that the wait covers the whole
operation the way its callers already assume.  vdev_trim_thread() ends
the same way and shares the predicate, so it gets the same treatment.

Verified in a VM by adding a delay(2 * hz) where the txg sync happens,
to imitate the loaded machine: uninit right after the wait fails on
every one of five runs before this change, and on none after it.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: George Melikov <[email protected]>
Closes #19042

module/zfs/spa.c

@@ -11753,9 +11753,18 @@ spa_vdev_activity_in_progress_impl(vdev_t *vd, zpool_wait_activity_t activity)
 	mutex_enter(lock);
 	mutex_enter(&spa->spa_activities_lock);
 
+	/*
+	 * A thread that has finished still has to sync out the new state
+	 * before it exits, and until it does the vdev cannot be initialized
+	 * or trimmed again.  Wait for the thread itself, not just the state,
+	 * so that a command issued after the wait returns does not fail with
+	 * EBUSY.
+	 */
 	boolean_t in_progress = (activity == ZPOOL_WAIT_INITIALIZE) ?
-	    (vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE) :
-	    (vd->vdev_trim_state == VDEV_TRIM_ACTIVE);
+	    (vd->vdev_initialize_state == VDEV_INITIALIZE_ACTIVE ||
+	    vd->vdev_initialize_thread != NULL) :
+	    (vd->vdev_trim_state == VDEV_TRIM_ACTIVE ||
+	    vd->vdev_trim_thread != NULL);
 	mutex_exit(lock);
 
 	if (in_progress)

module/zfs/vdev_initialize.c

@@ -617,6 +617,7 @@ vdev_initialize_thread(void *arg)
 
 	vd->vdev_initialize_thread = NULL;
 	cv_broadcast(&vd->vdev_initialize_cv);
+	spa_notify_waiters(spa);
 	mutex_exit(&vd->vdev_initialize_lock);
 
 	thread_exit();

module/zfs/vdev_trim.c

@@ -985,6 +985,7 @@ vdev_trim_thread(void *arg)
 
 	vd->vdev_trim_thread = NULL;
 	cv_broadcast(&vd->vdev_trim_cv);
+	spa_notify_waiters(spa);
 	mutex_exit(&vd->vdev_trim_lock);
 
 	thread_exit();