pty: preserve O_NONBLOCK when opening /dev/ptmx

ptmx_fdopen() masks the supplied file flags to FREAD | FWRITE before
passing them to pts_alloc(). This discards FNONBLOCK even when the
caller explicitly requests O_NONBLOCK.

As a result, open("/dev/ptmx", O_RDWR | O_NOCTTY | O_NONBLOCK) succeeds,
but the returned file is blocking. F_GETFL reports O_NONBLOCK cleared,
and a subsequent read can block instead of returning EAGAIN when no
data is available.

This causes Linux xfce4-terminal/VTE running under Linuxulator to
freeze. VTE opens the PTY master with O_NONBLOCK and relies on
nonblocking reads while draining terminal output. Once the available
data has been consumed, the main UI thread blocks in ptsdev_read(),
leaving the window unresponsive even though the child shell has
started successfully.

Include FNONBLOCK in the flags passed to pts_alloc(). Blocking opens
retain their existing behavior. No change to posix_openpt(2) flag
validation is required.

The flag loss also reproduces with a native FreeBSD program, so the
problem is in the shared /dev/ptmx implementation rather than Linux
syscall flag translation.

Validation:
- Before the fix, a native FreeBSD reproducer observes O_NONBLOCK
  cleared after opening /dev/ptmx with that flag.
- Explicitly setting O_NONBLOCK with F_SETFL works, confirming that
  nonblocking PTY operation itself is supported.
- After the fix, F_GETFL preserves O_NONBLOCK when requested and
  leaves it cleared for a blocking open.
- xfce4-terminal in the Linux jail displays the shell prompt and
  accepts input.

Reviewed by:	kib
MFC after:	1 week
Differential revision:	https://reviews.freebsd.org/D60700
This commit is contained in:
Vladimir Grebenshchikov authored and Konstantin Belousov committed 2026-10-16 16:34:20 +03:00
1 parent 38479a4832
commit 495bc1924e
1 file changed
+1 -1
+1 -1
View File
@@ -129,7 +129,7 @@ ptmx_fdopen(struct cdev *dev __unused, int fflags, struct thread *td,
struct file *fp)
{
return (pts_alloc(fflags & (FREAD|FWRITE), td, fp));
return (pts_alloc(fflags & (FREAD | FWRITE | FNONBLOCK), td, fp));
}
static struct cdevsw ptmx_cdevsw = {