dmu_objset_open_impl: unregister prop callbacks on error

If dsl_prop_register() fails partway, the error path frees the objset
without removing the callbacks already registered, leaving records
that point at freed memory.  Call dsl_prop_unregister_all() first.

Reviewed-by: Tony Hutter <[email protected]>
Reviewed-by: Alexander Motin <[email protected]>
Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: Martin Minkus <[email protected]>
Closes #18903

module/zfs/dmu_objset.c

@@ -628,6 +628,7 @@ dmu_objset_open_impl(spa_t *spa, dsl_dataset_t *ds, blkptr_t *bp,
 			}
 		}
 		if (err != 0) {
+			dsl_prop_unregister_all(ds, os);
 			arc_buf_destroy(os->os_phys_buf, &os->os_phys_buf);
 			kmem_free(os, sizeof (objset_t));
 			return (err);