Skip interior dnode slots when receiving DRR_FREEOBJECTS

receive_freeobjects() looks up every object of the record with
dmu_object_info() and fails the receive on any error but ENOENT.  An
interior slot of a multi-slot dnode answers EEXIST, and that is the
errno zfs receive then reports as "destination already exists".  This
is how ZTS send_realloc_files fails in CI:

    cannot restore to testpool/newfs@snap3: destination already exists

The sender emits such a record whenever a multi-slot dnode was freed
and a smaller one was later allocated on one of its interior slots: it
frees the leading slots, claims the new object and frees the trailing
slots as three separate records, e.g. for an 8-slot dnode at 10:

    FREEOBJECTS firstobj = 10 numobjs = 3
    OBJECT object = 13 ... dn_slots = 1
    FREEOBJECTS firstobj = 14 numobjs = 4

The first record frees the whole dnode, but its interior slots stay
marked as such until that free syncs.  Until 90cdd9aef ("Batch object
reallocation syncs in zfs receive") the claim of 13 then waited for
the sync, so by the time the second FREEOBJECTS was looked at, slot 14
was already free.  Now the claim is deferred instead, and the record
is not parked behind it, since the deferred range only covers 13: the
lookup of 14 answers EEXIST and the receive fails.

Treat EEXIST like ENOENT there.  The sender never names a live
interior slot, so a FREEOBJECTS that starts on one only ever follows
the free of that dnode's head, which is what releases the slot, and
there is nothing left to do for it.  The in-block scan of
dmu_object_next() and the slot expansion loop of receive_object()
already skip EEXIST the same way; drr_firstobj was the only object
that reached dmu_object_info() unfiltered.

Add ZTS send_realloc_dnode_interior, which builds exactly this layout
and fails with the error above before this change.  With the change
it passes, as do 10 iterations of send_realloc_files; with the change
reverted and zfs_recv_defer_batch_size=0 the layout is received fine,
which confirms the batching as the trigger.

Reviewed-by: Brian Behlendorf <[email protected]>
Signed-off-by: George Melikov <[email protected]>
Closes #19062

module/zfs/dmu_recv.c

@@ -3608,8 +3608,17 @@ receive_freeobjects(struct receive_writer_arg *rwa,
 		dmu_object_info_t doi;
 		int err;
 
+		/*
+		 * EEXIST means obj is an interior slot of a multi-slot
+		 * dnode.  There is nothing to free here: the sender never
+		 * names a live interior slot, so the head of that dnode was
+		 * freed already, either by this record or by a deferred
+		 * claim, and the slot goes with it once that free syncs.
+		 * The in-block scan of dmu_object_next() skips such slots
+		 * the same way; only drr_firstobj arrives here unfiltered.
+		 */
 		err = dmu_object_info(rwa->os, obj, &doi);
-		if (err == ENOENT)
+		if (err == ENOENT || err == EEXIST)
 			continue;
 		else if (err != 0)
 			return (err);

tests/runfiles/common.run

@@ -1020,7 +1020,8 @@ tests = ['recv_dedup', 'recv_dedup_encrypted_zvol', 'rsend_001_pos',
     'send_encrypted_holds',
     'send_encrypted_props', 'send_encrypted_truncated_files',
     'send_freeobjects', 'send_realloc_files', 'send_realloc_encrypted_files',
-    'send_realloc_dnode_nblkptr', 'send_realloc_dnode_mixed',
+    'send_realloc_dnode_nblkptr', 'send_realloc_dnode_interior',
+    'send_realloc_dnode_mixed',
     'send_realloc_dnode_resume',
     'send_spill_block', 'send_split_large_block', 'send_holds',
     'send_hole_birth', 'send_mixed_raw',

tests/zfs-tests/tests/Makefile.am

@@ -2215,6 +2215,7 @@ nobase_dist_datadir_zfs_tests_tests_SCRIPTS += \
 	functional/rsend/send_raw_ashift.ksh \
 	functional/rsend/send_raw_spill_block.ksh \
 	functional/rsend/send_raw_large_blocks.ksh \
+	functional/rsend/send_realloc_dnode_interior.ksh \
 	functional/rsend/send_realloc_dnode_mixed.ksh \
 	functional/rsend/send_realloc_dnode_nblkptr.ksh \
 	functional/rsend/send_realloc_dnode_resume.ksh \

tests/zfs-tests/tests/functional/rsend/send_realloc_dnode_interior.ksh

@@ -0,0 +1,116 @@
+#!/bin/ksh
+# SPDX-License-Identifier: CDDL-1.0
+
+#
+# This file and its contents are supplied under the terms of the
+# Common Development and Distribution License ("CDDL"), version 1.0.
+# You may only use this file in accordance with the terms of version
+# 1.0 of the CDDL.
+#
+# A full copy of the text of the CDDL should have accompanied this
+# source.  A copy of the CDDL is also available via the Internet at
+# https://opensource.org/license/CDDL-1.0.
+#
+
+#
+# Copyright (c) 2026 George Melikov
+#
+
+. $STF_SUITE/include/libtest.shlib
+. $STF_SUITE/tests/functional/rsend/rsend.kshlib
+
+#
+# Description:
+# Verify incremental receive handles a FREEOBJECTS record that starts on an
+# interior slot of a multi-slot dnode the same stream freed just before.
+#
+# When a multi-slot dnode is freed on the sender and a smaller one is later
+# allocated on one of its interior slots, the stream frees the leading
+# slots, claims the new object and then frees the trailing slots as three
+# separate records.  The receiver frees the whole dnode on the first one,
+# but its interior slots stay marked as such until that free syncs, so the
+# claim is deferred and the trailing FREEOBJECTS lands on an interior slot.
+#
+# Strategy:
+# 1. Create files with 4k dnodes and send the full stream
+# 2. Remove one of them and, with dnodesize=legacy, allocate objects on
+#    the slots it occupied, keeping only those on its interior slots
+# 3. Verify the incremental stream can be received and the result matches
+#    the source
+#
+
+verify_runnable "both"
+
+log_assert "Verify incremental receive handles frees over an interior slot"
+
+function cleanup
+{
+	rm -f $BACKDIR/fs-full
+	rm -f $BACKDIR/fs-incr
+
+	datasetexists $POOL/fs && destroy_dataset $POOL/fs -rR
+	datasetexists $POOL/newfs && destroy_dataset $POOL/newfs -rR
+}
+
+log_onexit cleanup
+
+# 1. Create files with 4k dnodes and send the full stream
+log_must zfs create -o dnodesize=4k -o xattr=sa $POOL/fs
+
+typeset -i i
+for (( i = 0; i < 3; i++ )); do
+	log_must touch /$POOL/fs/old.$i
+done
+log_must sync_pool $POOL
+typeset -i freed=$(get_objnum /$POOL/fs/old.1)
+
+log_must zfs snapshot $POOL/fs@a
+log_must eval "zfs send $POOL/fs@a > $BACKDIR/fs-full"
+log_must eval "zfs recv $POOL/newfs < $BACKDIR/fs-full"
+
+# 2. Remove one of them and, with dnodesize=legacy, allocate objects on
+#    the slots it occupied, keeping only those on its interior slots
+log_must zfs set dnodesize=legacy $POOL/fs
+log_must rm /$POOL/fs/old.1
+log_must sync_pool $POOL
+
+#
+# Object allocation continues from where it left off for as long as the
+# objset stays open, so reopen it to make it start over from the lowest
+# free slots.  The first allocation may take the head slot of the freed
+# dnode and other CPUs allocate from other chunks of the object space, so
+# keep the files that landed on its interior slots and retry if none did.
+#
+typeset -i attempt inside=0
+for (( attempt = 0; attempt < 5 && inside == 0; attempt++ )); do
+	log_must zpool export $POOL
+	log_must zpool import $POOL
+
+	for (( i = 0; i < 8; i++ )); do
+		log_must touch /$POOL/fs/new.$i
+	done
+	log_must sync_pool $POOL
+
+	for (( i = 0; i < 8; i++ )); do
+		typeset -i obj=$(get_objnum /$POOL/fs/new.$i)
+		if (( obj > freed && obj < freed + 8 )); then
+			log_note "new.$i is object $obj, inside $freed"
+			(( inside += 1 ))
+		else
+			log_must rm /$POOL/fs/new.$i
+		fi
+	done
+	log_must sync_pool $POOL
+done
+(( inside > 0 )) || log_fail "No object landed on an interior slot of $freed"
+
+log_must zfs snapshot $POOL/fs@b
+
+# 3. Verify the incremental stream can be received and the result matches
+#    the source
+log_must eval "zfs send -i $POOL/fs@a $POOL/fs@b > $BACKDIR/fs-incr"
+log_must eval "zfs recv $POOL/newfs < $BACKDIR/fs-incr"
+
+log_must directory_diff /$POOL/fs /$POOL/newfs
+
+log_pass "Verify incremental receive handles frees over an interior slot"