clnt_vc.c: Fix handling of broken TCP connections

After more than, I don't know, maybe 10k operations: mount, copy,
remove, verify and unmount cycles, one cp command hung in
close() / ncl_flush and never recovered. The machine and the mount
continued to work normally through a new connection, but the writes
using the old connection stayed frozen.

I did not understand exactly what happened.  I traced what appears to
be the issue in the code. My current understanding is that
clnt_vc_soupcall() saw the EOF and woke the caller waiting for RPC
replies, but one caller remained blocked in sosend().
That thread continued holding a reference to the old client, preventing
it from being completely cleaned up.

The attached patch calls socantsendmore() when EOF is received, which
should wake the blocked sender and let the normal reconnect code replace
the connection.

Reviewed by:	rmacklem
MFC after:	1 week
Sponsored by:	VersatusHPC

sys/rpc/clnt_vc.c

@@ -964,11 +964,8 @@ clnt_vc_soupcall(struct socket *so, void *arg, int waitflag)
 	mtx_unlock(&ct->ct_lock);
 
 	/*
-	 * If another thread is already here, it must be in
-	 * soreceive(), so just return to avoid races with it.
-	 * ct_upcallrefs is protected by the socket receive buffer lock
-	 * which is held in this function, except when
-	 * soreceive() is called.
+	 * ct_upcallrefs prevents another upcall from processing the
+	 * socket while the receive buffer lock is dropped.
 	 */
 	if (ct->ct_upcallrefs > 0)
 		return (SU_OK);
@@ -1009,6 +1006,11 @@ clnt_vc_soupcall(struct socket *so, void *arg, int waitflag)
 			 * to read from the stream.
 			 */
 			error = ECONNRESET;
+
+			/* Avoid reversing the so_snd -> so_rcv lock order. */
+			SOCK_RECVBUF_UNLOCK(so);
+			socantsendmore(so);
+			SOCK_RECVBUF_LOCK(so);
 		}
 
 		/*