mmp: do not require writes to mirror legs the config marks absent

The MMP uberblock claim requires one good write per configured leaf of
each top-level vdev.  For a mirror it required two writes
unconditionally (MIN(MAX(children, 1), 2)), so a mirror with a leg that
is persistently offline, faulted, or removed could produce only one good
write and the activity-check claim failed with EIO.  A degraded mirror
could therefore not be imported with multihost=on, blocking HA failover.

Count only the legs the pool config still expects to be present, and
require a write to every one of them.  A leg taken out of service is
recorded persistently in the config and is seen the same way by every
host, so it is not required.  A leg merely unreachable from the
importing host keeps none of those states and stays required, so a host
that can see only some of the legs of an otherwise healthy mirror still
fails the claim and cannot split the pool.

The previous cap of two writes was a compromise made because requiring
every child was too strict for wide mirrors, in particular where a leg
is left offline for long periods as part of a backup strategy.
Consulting the config covers that case directly, so the cap is no longer
needed: on a three-way mirror with one leg unreachable and not marked
absent, a cap of two would accept the claim while another host holding
the third leg could accept it as well.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Michael Heller <[email protected]>
Closes #18855

module/zfs/mmp.c

@@ -560,8 +560,8 @@ mmp_claim_uberblock_sync_done(zio_t *zio)
 
 /*
  * Write the uberblock to the first label of all leaves of the specified vdev.
- * Two writes required for each mirror, one for a singleton, and parity+1 for
- * raidz or draid vdevs.
+ * One write per mirror leg the config still expects present, one for a
+ * singleton, and parity+1 for raidz or draid vdevs.
  */
 static void
 mmp_claim_uberblock_sync(zio_t *zio, uint64_t *good_writes,
@@ -580,8 +580,32 @@ mmp_claim_uberblock_sync(zio_t *zio, uint64_t *good_writes,
 			if (nparity) {
 				*req_writes += nparity + 1;
 			} else {
-				*req_writes +=
-				    MIN(MAX(cvd->vdev_children, 1), 2);
+				/*
+				 * Mirror: any single leg is enough for a
+				 * remote host to see the claim, so require a
+				 * write to every leg the pool config still
+				 * expects to be present.  A leg taken out of
+				 * service is recorded persistently in the
+				 * config (offline, faulted, or removed) and
+				 * is seen the same way by every host, so it
+				 * is not required and a degraded mirror can
+				 * still be claimed.  A leg merely unreachable
+				 * from this host has none of those states and
+				 * stays required, so a host which can see
+				 * only some of the legs of an otherwise
+				 * healthy mirror still fails the claim and
+				 * cannot split the pool.
+				 */
+				uint64_t present = 0;
+				for (uint64_t l = 0; l < cvd->vdev_children;
+				    l++) {
+					vdev_t *lvd = cvd->vdev_child[l];
+					if (!lvd->vdev_offline &&
+					    !lvd->vdev_faulted &&
+					    !lvd->vdev_removed)
+						present++;
+				}
+				*req_writes += MAX(present, 1);
 			}
 		}