Check that once fsync(2), fdatasync(2), or a write(2) with O_SYNC or
O_DSYNC returns, the new data, the file size and every block pointer
leading to the data are on disk. The cases append a block, fill a hole,
append the first block that needs an indirect block and append below an
existing indirect block. Two more append after another write left a
new direct block unsynced, which with soft updates cuts the size and the
later pointers of an inode written meanwhile. With O_SYNC and O_DSYNC
on soft updates, those also check that the open already made the
unsynced block durable.
Each case runs on UFS without soft updates, with soft updates and with
SU+J.
The file systems live on md(4) and the checks read the inode, the
indirect block and the data block straight from the device with libufs
right after the synchronous call returns. There is no cache in front
of the disk device, so this shows what would survive a crash at that
moment without having to crash anything.
Sponsored by: Sippy Software, Inc.
Reviewed by: pho
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60134
MFC after: 1 month
With soft updates, ffs_syncvnode() writes the inode between its passes
while buffers are still dirty. That write can go out with new block
pointers rolled back, for instance a new indirect block whose contents
are only written in a later pass, yet ffs_update() clears IN_SIZEMOD
and IN_IBLKDATA because the write was synchronous. A data-only sync
then decides at the end that the inode need not be written, and
fdatasync(2) or an O_DSYNC write that allocated the first indirect
block returned with the on-disk inode not pointing to it.
Note on entry whether a data-only sync has to write the inode, and base
the final write, done once all dependencies are flushed, on that alone:
nothing else can set IN_SIZEMOD or IN_IBLKDATA meanwhile, as the vnode
is locked. A full sync always writes the inode last and was not
affected.
Sponsored by: Sippy Software, Inc.
Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60134
MFC after: 1 month
For a synchronous write, ffs_write() writes the data buffers with
bwrite() and then the inode with ffs_update(vp, 1). With soft updates,
a new block pointer can only reach the disk after the dependencies of
the block, such as the cylinder group map that allocates it, are
written. Until then initiate_write_inodeblock_ufs[12]() and
initiate_write_indirdep() roll the pointer, and for an extending write
the size, back in the buffer being written. Nothing in the write path
flushes those dependencies, so the inode went out with the old size and
pointers, ffs_update() cleared IN_SIZEMOD and IN_IBLKDATA, and a crash
before the syncer rewrote the inode lost data that O_SYNC or O_DSYNC
promised was on stable storage. The same held for a new block below an
indirect block, which ffs_balloc() writes synchronously but with the
pointer rolled back.
When soft updates are enabled and the write allocated blocks, finish it
with ffs_syncvnode(), as fsync(2) and fdatasync(2) do: it flushes the
dependencies, rewrites the rolled back indirect blocks and writes the
inode, or for O_DSYNC only writes the inode when IN_SIZEMOD or
IN_IBLKDATA say it is needed. The flags cannot tell whether the write
allocated anything, because allocating a block below an existing
indirect block sets neither, so compare the inode's block count before
and after the write instead. A write that allocated nothing, such as
an overwrite of existing blocks, adds no dependencies of its own and
keeps writing only the inode, so that it does not pay for a sync of the
whole vnode. ffs_syncvnode() only drops the vnode lock when it has to
resolve directory dependencies, which ffs_write() never sees.
ffs_syncvnode() also writes whatever other descriptors left dirty on
the vnode, so the first synchronous write after a lot of buffered
writes would pay for all of them. To put that cost on the open
instead, where it is less unexpected for a descriptor opened for
synchronous writes, add ffs_open() and, with soft updates, sync the
vnode there when a regular file is opened for writing with O_SYNC or
O_DSYNC, the same way the writes would: a full sync for O_SYNC, a
data-only one for O_DSYNC. It calls ffs_syncvnode() rather than
VOP_FSYNC(), whose softdep_fsync() may drop the vnode lock in the middle
of the open. The sync on open only moves work that the writes would
otherwise do, so it is skipped rather than waited for when writes are
suspended, and its errors do not fail the open. Without soft updates
the writes sync nothing but their own blocks and the inode, so there is
nothing to move and the open does not sync.
Sponsored by: Sippy Software, Inc.
Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60134
MFC after: 1 month
A data-only ffs_syncvnode() skips dirty indirect blocks on file systems
without soft updates, and with soft updates stops making passes once
only indirect blocks are left dirty. Indirect blocks only get dirty
when the block pointers in them change, and those pointers are what
makes a newly allocated block reachable. So when fdatasync() follows
a write that extended the file or filled a hole past the direct blocks,
the data and the inode size reached the disk but the pointer to the new
block did not, and a crash lost the "synced" data. With soft updates
the indirect block was usually written once, with the new pointer
rolled back, and never rewritten.
POSIX requires fdatasync() to write the metadata needed to retrieve the
data, which 52488b5148 already acknowledged for the inode, so treat
dirty indirect blocks the same way for a data-only sync as for a full
one. This restores the loop to what it was before 2f514f92cf. Plain
overwrites, the case data-only sync is meant to speed up, do not dirty
indirect blocks and still write neither them nor the inode.
Fixes: 2f514f92cf ("Implement VOP_FDATASYNC() for UFS.")
Sponsored by: Sippy Software, Inc.
Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60134
MFC after: 1 month
ffs_syncvnode() in the DATA_ONLY case and ffs_write() for IO_DATASYNC
decide whether the inode has to be written by testing IN_SIZEMOD and
IN_IBLKDATA against ip->i_flags, the chflags(2) word, instead of the
in-core state ip->i_flag. IN_SIZEMOD and IN_IBLKDATA alias UF_READONLY
and UF_ARCHIVE, so for ordinary files the test was always false: after
an fdatasync() or an O_DSYNC write that extended a file or filled a
hole, the data blocks were on disk but the inode size and block pointers
were not, and a crash before the syncer got to the inode lost the
"synced" data. Files with uarchive or urdonly set got a synchronous
inode write on every call instead.
journal_mount() has the same mixup and sets IN_MODIFIED in i_flags,
which leaves UF_OPAQUE set on the in-core .sujournal inode. The
synchronous ffs_update() that follows writes the inode regardless, so
only the stray flag is fixed here.
Fixes: 7428630b75 ("UFS: write inode block for fdatasync(2) if pointers in inode where allocated")
Fixes: 52488b5148 ("Further evaluation of the POSIX spec for fdatasync()...")
Fixes: e7347be9e3 ("ffs: Support O_DSYNC.")
Fixes: 113db2dddb ("Merge soft-updates journaling from projects/suj/head into head.")
Sponsored by: Sippy Software, Inc.
Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60134
MFC after: 1 month
On a soft updates file system, keep a directory at UFS_LINK_MAX with
workers removing and recreating subdirectories in it, so that mkdir(2)
keeps going through ufs_sync_nlink(), and force-unmount, check and
remount the file system in a loop. The workers work relative to the
directory: a lookup through the mount point would wait for the unmount
on the covered vnode lock and never race with it. Before the fix, the
first forced unmount deadlocked with mkdir(2) sleeping in vfs_busy().
With dtrace=1, count the vfs_busy() calls made during an unmount by
caller, and the mkdir(2) calls that synced and restarted with ERELOOKUP
during an unmount, which shows that the race was hit.
Sponsored by: Sippy Software, Inc.
Reviewed by: pho
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60204
MFC after: 1 month
With soft updates, mkdir(2), link(2) and rename(2) into a directory
whose i_nlink is at UFS_LINK_MAX while i_effnlink is below it call
ufs_sync_nlink1() to sync the file system and restart. It busied the
mount with vfs_busy(mp, 0), which sleeps for as long as an unmount is in
progress. The caller is still between vn_start_write() and
vn_finished_write() there, and the unmount, once it has set
MNTK_UNMOUNT, goes on to suspend writes in VFS_UNMOUNT() and waits in
vfs_write_suspend() for that write to end. Neither could make progress,
and every later vfs_busy() of the mount, such as from getfsstat(2) or a
lookup crossing the mount point, hung behind them.
The window is wide: after setting MNTK_UNMOUNT, the unmount waits for
the busy references to drain, which includes another thread already
syncing in ufs_sync_nlink1(), and for vfs_periodic(). A forced unmount
of a file system with a directory at the link limit under mkdir/rmdir
load hits it within seconds.
This is the deadlock that 29d03af1e4 fixed for VOP_RENAME(), and the
same reasoning applies: the caller's vn_start_write() already keeps the
unmount from getting past the write suspension, before which
ffs_unmount() tears nothing down, so the mount point does not need to
be busied for VFS_SYNC(). The extra vfs_ref() taken by the callers is
not needed either: a successful vn_start_write() keeps the reference
that VOP_GETWRITEMOUNT() took until vn_finished_write() drops it. With
both gone, ufs_sync_nlink1() was only VFS_SYNC() and ERELOOKUP, so
remove it and call VFS_SYNC() directly from ufs_sync_nlink() and the
two places in ufs_rename().
Fixes: 8db679af66 ("UFS: make mkdir() and link() reliable when using SU and reaching nlink limit")
Fixes: bc6d0d72f4 ("UFS rename: make it reliable when using SU and reaching nlink limit")
Sponsored by: Sippy Software, Inc.
Reviewed by: kib
Tested by: pho
Differential Revision: https://reviews.freebsd.org/D60204
MFC after: 1 month
flush_pagedep_deps() drops the soft updates lock to obtain the vnode of
a new directory. If its MKDIR_BODY dependency completes in that window,
the lookup of the directory's first block in flush_newblk_dep() can find
a stale allocindir left behind by a previous owner of the same block,
relocated by ffs_reallocblks() and freed, and panic.
The test grows interleaved files past UFS_NDADDR to keep clusters being
relocated, while other workers create subdirectories in a parent with
IN_ENDOFF set, so that ffs_vput_pair() syncs it, and fsync() them to
complete the mkdir dependencies. The file system layout and the way
the writers put their blocks on disk are arranged so that a new
directory takes over a freed block whose dependency is still retained;
the details are in the script. With dtrace=1, the test also counts how
often this precondition is met, which works on a fixed kernel too.
PR: 297976
Reviewed by: kib, pho
Tested by: pho
Sponsored by: Sippy Software, Inc.
Differential revision: https://reviews.freebsd.org/D59356
flush_pagedep_deps() drops the soft updates lock while obtaining the
vnode of a newly created directory with get_parent_vp(). The
MKDIR_BODY dependency may complete during this interval, invalidating
the diradd selected before the lock was dropped.
Once the directory's allocdirect is retired, the lookup of its first
block by block number in flush_newblk_dep() can find an older
dependency for a previous use of the same physical block. The newblk
hash is not unique by block number: when ffs_reallocblks() relocates a
cluster, the completed allocindirs of the old blocks stay on the
indirdep's ir_completehd until the indirect block pointer in the inode
is written, while the old blocks are already free and may be allocated
to a new directory. flush_newblk_dep() then finds a D_ALLOCINDIR where
it expects a D_ALLOCDIRECT and panics with "flush_newblk_dep: Bad
newblk".
Retain the vnode returned by get_parent_vp(), reacquire the soft
updates lock, and restart dependency selection. Use the retained vnode
only when the newly selected MKDIR_BODY dependency refers to the same
inode.
Enter flush_newblk_dep() with the soft updates lock held and pass it
the associated diradd. Recheck MKDIR_BODY after every operation that
may drop the lock, so that completion during the helper's retry loop
cannot result in another lookup of the retired allocation.
Keep the vnode handling in flush_pagedep_deps(), which owns the retained
vnode and the list of unfinished diradds, and move the dependency
selection into flush_pagedep_deps1(). The latter never acquires or
releases a vnode: it returns EJUSTRETURN with the inode number when it
needs one, or with 0 when the retained vnode must be released first,
and every call restarts the selection, so a vnode is only used for a
diradd found while it was held. This also merges the two
get_parent_vp() call sites. Every restart goes through that return:
obtaining or releasing a vnode, and continuing after jwait() or
getdirtybuf() had to wait with the softdep lock dropped. The diradds
deferred on the unfinished list are put back on the pagedep's list
before the lock is dropped for a restart, so that none are kept off it
while other threads can run. Assert that flush_newblk_dep() is called
with the vnode exclusively locked.
This preserves the original newblk dependency ordering and retains the
existing assertion for an active MKDIR_BODY dependency that resolves to
an unexpected dependency type.
PR: 297976
Reviewed by: kib
Tested by: pho
Sponsored by: Sippy Software, Inc.
Differential revision: https://reviews.freebsd.org/D59356
Add mountpoint(1), a simple utility to tell whether the file pointed
to by the argument is a mount point. It prints whether it is, unless
-q is given, and exits 0 if it is, 1 if it is not, and 2 on error.
The answer comes from the kernel with a single stat(2): the root vnode
of a mounted file system is reported with SFBSD_MNTPOINT in
st_bsdflags. Unlike comparing realpath(3) of the argument with that
of statfs(2)'s f_mntonname, this works for arbitrarily deep
hierarchies, and after chroot(2) or inside a jail. A chroot or jail
root is reported as a mount point only if it is one.
The argument does not have to be a directory: file systems such as
nullfs(5) can be mounted over regular files and sockets, and stat(2)
reports those mount points as well.
Since all that is needed is one stat(2) call, make mountpoint a
hardlink to stat(1), the same way readlink(1) is, and document it in
stat.1.
Add ATF tests, including chroot(8) and jail(8) roots that are and are
not mount points, nullfs roots, file systems mounted inside the root,
both from the host and from within the jail, and a nullfs mount of a
file over a file.
Suggested by: kib
Differential revision: https://reviews.freebsd.org/D59906
Sponsored by: Sippy Software, Inc.
MFC after: 2 weeks
gnop6.sh recorded the checkfs result with "checkfs ... || s=1 && s=0",
which the shell parses as "(checkfs || s=1) && s=0", so the test passed
even when the file system check failed.
gnop7.sh only sets s when unmount or fsck_ffs fails and ends with
"exit $s". On a clean run s is unset, and the script exits with the
status of the preceding "[ $notloaded ] && gnop unload", which is 1
whenever geom_nop was already loaded, for instance by an earlier test.
fdatasync.sh and fdatasync2.sh ran "df -i $RUNDIR" before creating the
directory. With a RUNDIR that does not exist yet, df failed, the test
printed "[: -lt: unexpected operator" and the free inode check was
skipped.
Reviewed by: pho
Differential Revision: https://reviews.freebsd.org/D60136
Sponsored by: Sippy Software, Inc.
MFC after: 2 weeks
When ufs_rename() moves a directory to a new parent and ufs_dirrewrite()
fails to rewrite its ".." entry, it reports "bad dir ... rename: missing
.. entry" whatever the error. ufs_dirrewrite() never finds a missing
"..", though: it fails with EIDRM when the ".." entry names another inode
than the expected one, and otherwise only when the directory block cannot
be read or written. The latter happens for every directory rename in
flight when the device goes away under a forcibly unmounted file system,
and the log then fills with reports of directories that are intact on
disk.
Report the EIDRM case with ufs_dirbad() in ufs_dirrewrite() itself, so
that all of its callers get the same diagnostic, and drop the report
from ufs_rename(). Errors from the lower layers are not reported, as
usual for an I/O initiator.
Reviewed by: kib
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D60137
Those are FreeBSD-specific member and flag still better to be
documented.
Reviewed by: kib
Sponsored by: Sippy Software, Inc.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D59915
After reading both manual pages, our TCP_MAXUNACKTIME is fairly
similar to the TCP_USER_TIMEOUT, the only considerable difference
is ours is in seconds and linux's in milliseconds.
Round up linux's in setsockopt(2) to a next whole second and
clamp ours getter to UINT_MAX ms.
Reviewed by: tuexen, glebius
Differential Revision: https://reviews.freebsd.org/D56168
MFC after: 2 weeks
Sponsored by: Sippy Software, Inc.
Fix potential crash in the ng_nat module when attaching directly
to the layer 2 (ethernet) while calculating TCP checksum.
The issue is due to in_delayed_cksum() expecting to access IP
header at the offset 0 from the mbuf start, while if we are
attached to the L2 directly, the IP header at going to be at the
certain offset.
Reviewed by: markj, tuexen
Approved by: tuexen
Sponsored by: Sippy Software, Inc.
Differential Revision: https://reviews.freebsd.org/D49677
MFC After: 2 weeks
The need for such a variant comes from the fact that we need to
re-calculate checksum aftet ng_nat(4) transformations while getting
mbufs from the layer 2 (ethernet) directly.
Reviewed by: markj, tuexen
Approved by: tuexen
Sponsored by: Sippy Software, Inc.
Differential Revision: https://reviews.freebsd.org/D49677
MFC After: 2 weeks
Fix list-old-dirs to not generate empty line at the end of the
output if OLD_DIRS happens to have a space at the end of the
last word of the output before the final \n. Then that space
is turned into \n and we end up with a blank line.
Futhermore this gets converted into a "/" i.e. root fs when
calling check-old-dirs.
This is the regression since a8267ecc3d.
Reviewed by: emaste
Approved by: emaste
Sponsored by: Sippy Software, Inc.
Differential Revision: https://reviews.freebsd.org/D52153
MFC After: 3 days
For some reason return value of the __je_bootstrap_malloc()
is not checked and then de-referenced few lines below, causing
a SEGV if an early allocation fails.
MFC after: 1 month
- Make less ambiguous;
- extend to provide more context for post-mortem.
Reviewed by: markj
Differential Revision: https://reviews.freebsd.org/D43776
MFC after: 2 weeks
Loss of the trailing space in the multi-line format string has
resulted in column name being emitted as "FAILSLEEP", instead of
two columns "FAIL" and "SLEEP".
Split out termcap.small generation into its own Makefile under
etc/termcap, so it's properly executed by the underlying command:
make 'SUBDIR_OVERRIDE=etc' everything
Reported by: gbe
MFC after: 1 month
need to do synchronization by hand when termcap is updated.
Reviewed by: uqs, bapt, imp
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D36333
This is consistent with what other uid-morphing utilities
do, i.e. jexec(1), su(1) etc.
MFC after: 2 weeks
Reviewed by: gbe
Differential Revision: https://reviews.freebsd.org/D36148
Consider IP_MF flag when checking length of the UDP packet to
match the declared value.
Sponsored by: Sippy Software, Inc.
Differential Revision: https://reviews.freebsd.org/D32363
MFC after: 2 weeks
The change makes block caching algorithm to work better for remote
media on low-BW/high-delay links.
This cuts boot time over IP KVMs noticeably, since the initialization
stage reads bunch of small 4th (and now lua) files that are not in
the same cache stripe (usually), thus wasting lot of bandwidth and
increasing latency even further.
The original regression came in 2017 with revision 87ed2b7f5. We've
seen increase of time it takes for the loader to get to the kernel
loading from under a minute to 10-15 minutes in many cases.
Reviewed by: tsoome
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D31623
define procname or have a PID file. This might be useful for cases,
such as mounting local FS, when there is no running daemon
still some other persistent state in the system which status
can be checked.
It is still possible to have a status method before this by having
extra_commands="status", but it's not obvious and might give
an script writer some extra legwork to figure out how and why
the straight method is not working.
Reviewed by: cy
MFC after: 1 week
Relnotes: yes
Differential Revision: https://reviews.freebsd.org/D31614
i386 and the rest of supported architectures by defining KERNLOAD in the
vmparam.h and getting rid of magic constant in the linker script, which albeit
documented via comment but isn't programmatically accessible at a compile time.
Use KERNLOAD to eliminate another (matching) magic constant 100 lines down
inside unremarkable TU "copy.c" 3 levels deep in the EFI loader tree.
Reviewed by: markj
Approved by: markj
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D27355
received.
The default system log rotation mechanism (newsyslog(8)) requires ability to send
signal to a daemon in order to properly complete rotation of the logs in an "atomic"
manner without having to making a copy and truncating original file. Unfortunately
our built-in mechanism to convert "dumb" programs into daemons has no way to handle
this rotation properly. This change adds this ability, to be enabled by supplying -H
option in addition to the -o option.
Reviewed by: markj, rpokala (manpages)
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D26526
That mode is useful to call gstat from other app, however kinda useless
since gstat won't exit and stay running forever when its parent process
has long gone.
MFC after: 2 weeks
instead of sprinkling them out over many disjoint files. This is a follow-up
to achieve the same goal in an incomplete rev.348521.
Approved by: imp
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D20520
the followup stopgap change, because I don't think it's a correct. I still
need to figure out where to stick it in. In cannot be in Makefile.inc1
and it cannot be in etc/Makefile from the looks of it to avoid
chicken-and-egg problem.
install -> ${INSTALL}
mtree -> ${MTREE_CMD}
services_mkdb -> ${SERVICES_MKDB_CMD}
cap_mkdb -> ${CAP_MKDB_CMD}
pwd_mkdb -> ${PWD_MKDB_CMD}
kldxref -> ${KLDXREF_CMD}
If you do custom FreeBSD builds you may want to override those
in some cases.
Sponsored by: Sippy Software, Inc.
has been in the PR system for 5 months and then on reviews for another 5.
Nobody came with any cases where it fails, while many people cried for
it to be commited & merged.
PR: 209468
Submitted by: Prasad B M <[email protected]>
Reported by: Steven Peterson <[email protected]>
Approved by: scottl
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D18408
to go to the FS image we are making cannot be read (e.g. EPERM).
Current behaviour when we issue waring but still proceeed and
return success is definitely not correct: masking out error
condition as well as making a slighly inconsistent FS where
attempt to access the file in question ends up in EBADF. See
linked DR for details.
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D18584
MTU if we've set it once and there were no changes on the DHCP server
side since the last refresh. This is consistent I believe with how dhclient
handles other settings like IP address, mask etc.
Approved by: cem, eugen
Differential Revision: https://reviews.freebsd.org/D18546
or the likes. Add new control message types: setdlt and getdlt to switch
from default DLT_RAW (no encapsulation) to DLT_EN10MB (ethernet).
Approved by: glebius
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D18535
handling slightly out-of-bound requests properly (r340187).
Perform range check here rather then rely on g_delete_data() to DTRT.
The g_delete_data() would always return success for requests
starting just the next byte after providers media boundary.
MFC after: 4 weeks
with macro based around memcmp(). The latter is expected to be some
8 times faster on a modern 64-bit architectures.
In practice, throughput of doing conv=sparse from /dev/zero to /dev/null
went up some 5-fold here from 1.9GB/sec to 9.7GB/sec with this change
(bs=128k).
MFC after: 2 weeks
the malloc()/free() as well as having potential of softening the handling
in case error is detected down to a mere warning as compared to hard panic
in free().
Submitted by: tsoome
Differential Revision: https://reviews.freebsd.org/D18299
is that both file name and current directory is recorded, however
file name sometimes already contains absolute path. In which case
prefixing it with directory name results in an invalid pathname.
Only append directory name if the file name does not start with '/'.
This seems to DTRT.
Approved by: emaste
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D18290
i/o into last_sector+N is handled differently for N==1 and N>1 cases to
accomodate that, so some other approach would be needed to fix DIOCGDELETE
ioctl(2).
fully beyond the end of providers media. The only exception is made
for the zero length transfers which are allowed to be just on the
boundary. Previously, any requests starting on the boundary (i.e. next
byte after the last one) have been allowed to go through.
No response from: freebsd-geom@, phk
MFC after: 1 month
and convert any messages of types SCM_BINTIME, SCM_TIMESTAMP,
SCM_REALTIME and SCM_MONOTONIC from 64-bit to its 32-bit
representation. Otherwise we either run out of user-supplied
buffer to copy those out resulting in the MSG_CTRUNC or simply
return values that the userland 32-bit code is not going
to parse correctly. This fixes at least two regression tests
failing to function properly in 32-bit compat mode:
tools/regression/sockets/udp_pingpong
tools/regression/sockets/unix_cmsg
PR: kern/222039
MFC after: 30 days
While I am here, declare struct md_ioctl32 as packed which allows
us to stop playing tricks with sizeof(md_ioctl32)+y as well as
simplifies md_pad handling. Both were necessary because of different
alignment preferences on amd64 vs i386.
MFC after: 4 weeks
This feature comes from the fact that we rely memory-backed md(4)
in our build process heavily. However, if the build goes haywire
the allocated resources (i.e. swap and memory-backed md(4)'s) need
to be purged. It is extremely useful to have ability to attach
arbitrary labels to each of the virtual disks so that they can
be identified and GC'ed if neecessary.
MFC after: 4 weeks
Differential Revision: https://reviews.freebsd.org/D10457
file. That logic has grown quite significantly now;
o add a special handling for the snapshot images. Those have some
extra headers at the end of the image and we don't need those
in the output image really.
MFC after: 6 weeks
the r316874: don't call shutdown(2) on all sockets, but only net ones, which seems
to be the behaviour existed before that refactoring. Also don't call listen(2)
in datagram sockets and fix misplaced debug messages while I am here.
Reported by: peter
don't bother to select/recv on that socket. This prevents syslogd(8)
from spinning endlessly when started with the -s option (default).
Reported by: peter
by the shutdown(2) system call. This ability has been lost as part of the svn
revision 285910.
Reviewed by: ed, rwatson, glebius, hiren
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D10351
The problem is that the statfs(2) system call used to determine the relevant
mount point returns path within real root in the f_mntonname, causing
nmount(2) system call to fail with ENOENT.
Use a bit of heuristics to skip over few starting path elements when it
happens until we hit an actual mount point.
For this to work properly the whole mount should be accessible within the
chroot, it's going to still fail if chroot only has access to a part of the
mounted fs.
Reviewed by: mckusick
Approved by: mckusick
MFC after: 2 weeks
sources to return timestamps when SO_TIMESTAMP is enabled. Two additional
clock sources are:
o nanosecond resolution realtime clock (equivalent of CLOCK_REALTIME);
o nanosecond resolution monotonic clock (equivalent of CLOCK_MONOTONIC).
In addition to this, this option provides unified interface to get bintime
(equivalent of using SO_BINTIME), except it also supported with IPv6 where
SO_BINTIME has never been supported. The long term plan is to depreciate
SO_BINTIME and move everything to using SO_TS_CLOCK.
Idea for this enhancement has been briefly discussed on the Net session
during dev summit in Ottawa last June and the general input was positive.
This change is believed to benefit network benchmarks/profiling as well
as other scenarios where precise time of arrival measurement is necessary.
There are two regression test cases as part of this commit: one extends unix
domain test code (unix_cmsg) to test new SCM_XXX types and another one
implementis totally new test case which exchanges UDP packets between two
processes using both conventional methods (i.e. calling clock_gettime(2)
before recv(2) and after send(2)), as well as using setsockopt()+recv() in
receive path. The resulting delays are checked for sanity for all supported
clock types.
Reviewed by: adrian, gnn
Differential Revision: https://reviews.freebsd.org/D9171
and tw_so_options defined here which is supposed to be a copy of the
former (short vs u_short respectively).
Switch tw_so_options to be "signed short" to match the type of the field
it's inherited from.
away in the past from the current time. This should be plenty for the
scheduler to do its job. It provides assurance that the timestamp
returned is actually a valid one, not just some random garbage.
used. We can do it programmatically, but that would make code convoluted
and more complex. I have two more of those types coming for the CLOCK_REALTIME
and CLOCK_MONOTONIC. This seems like an elegant and scallable approach.
file into smaller pieces that are hopefully easier to understand
and extend. This is to pave the ground for adding few more
socket timestamp formats that I am working on here.
No functional changes (I hope).
superblock, allowing provider to be bit bigger, i.e. have some
extra padding after the FS image. That in some cases might be
a side-effect of using CLOOP format which enforces certain block
size and trying to compress image that is not exactly the number
of those blocks in size. The UFS itself does not have any issues
mounting such padded file systems, so it's what GEOM_LABEL should
do.
Submitted by: @mizhka_gmail.com
Differential Revision: https://reviews.freebsd.org/D6208
necessary because CLOOP format lacks explicit EOF or length, so that
in the presence of padding or when the CLOOP is put onto a larger
partition upper level provider size may be larger. Bound amount
of extra data that we might touch to the max length of the compressed
block and detect zero-padding in the last cluster, which when
sector is all-zero might cause us to emit bogus I/O error after
decompression of that fails. To not make code any more complicated
that it needs to be deal with it in lazy-manner, i.e. when we
first access that specific cluster.
This change also fixes stupid mistake in the LZMA code, inherited
from geom_lzma, which does not share length of the output buffer
buffer with the decompression routine, so that in the presence
of corrupted or purposedly tailored data may easily cause heap
overflow and kernel memory corruption.
Beef up validation of the CLOOP TOC by checking that lengths of
all but the last compressed clusters match upper limit set by
the decompressor and improve some error diagnostic output while
I am here.
2.Add kern.geom.uzip.attach_to tunable to artifically limit
attaching uzip to certain devices in the dev tree only.
For example the following only makes us attaching to the
GPT labels:
kern.geom.uzip.attach_to="gpt/*"
3.Add kern.geom.uzip.noattach_to, which does opposite to the (2)
above, i.e. prevents geom_uzip from tasting / attaching to
providers matching some pattern. By default we don't attach
to our own kind, i.e. kern.geom.uzip.noattach_to="*.uzip".
It saves us quite some CPU cycles, esp on low-end embedded
systems.
Approved by: re (gjb)
Differential Revision: https://reviews.freebsd.org/D7013
o Split the compression across several worker threads. By default, "several"
matches number of CPUs, capped at 24 for sanity when running on a very big
hardwares. Provide option to set that number manually;
o Fix bug inherited from the mkulzma (R.I.P) which degraded already slow LZMA
compression even further by calling function to release compression state
after processing each block.
It is neither documented as required nor actually required by the LZMA
library. This caused spree of system calls to release memory and then map
it again for every block. LZMA compression is more than 2x faster after this
change alone;
o Record time it takes to do compression and report throughput achieved.
o Add simple first-level 256 entry hash table for de-dup code, so it's not
becoming a bottleneck at big files.
is defined explicitly. It's kinda pointless and results in extra step in
boot sequence which is not really needed, i.e.:
md0: Embedded image 1331200 bytes at 0x8038b7b4
Trying to mount root from ufs:/dev/md0 []...
Mounting from ufs:/dev/md0 failed with error 22.
Trying to mount root from ufs:md0.uzip []...
warning: no time-of-day clock registered, system time will not be set accurately
start_init: trying /sbin/init
and geom_uncompress(4):
1. mkuzip(8):
- Proper support for eliminating all-zero blocks when compressing an
image. This feature is already supported by the geom_uzip(4) module
and CLOOP format in general, so it's just a matter of making mkuzip(8)
match. It should be noted, however that this feature while it sounds
great, results in very slight improvement in the overall compression
ratio, since compressing default 16k all-zero block produces only 39
bytes compressed output block, which is 99.8% compression ratio. With
typical average compression ratio of amd64 binaries and data being
around 60-70% the difference between 99.8% and 100.0% is not that
great further diluted by the ratio of number of zero blocks in the
uncompressed image to the overall number of blocks being less than
0.5 (typically). However, this may be important from performance
standpoint, so that kernel are not spinning its wheels decompressing
those empty blocks every time this zero region is read. It could also
be important when you create huge image mostly filled with zero
blocks for testing purposes.
- New feature allowing to de-duplicate output image. It turns out that
if you twist CLOOP format a bit you can do that as well. And unlike
zero-blocks elimination, this gives a noticeable improvement in the
overall compression ratio, reducing output image by something like
3-4% on my test UFS2 3GB image consisting of full FreeBSD base system
plus some of the packages (openjdk, apache etc), about 2.3GB worth of
file data (800+MB compressed). The only caveat is that images created
with this feature "on" would not work on older versions of FeeBSDxi
kernel, hence it's turned off by default.
- provide options to control both features and document them in manual
page.
- merge in all relevant LZMA compression support from the mkulzma(8),
add new option to select between both.
- switch license from ad-hoc beerware into standard 2-clause BSD.
2. geom_uzip(4):
- implement support for de-duplicated images;
- optimize some code paths to handle "all-zero" blocks without reading
any compressed data;
- beef up manual page to explain that geom_uzip(4) is not limited only
to md(4) images. The compressed data can be written to the block
device and accessed directly via magic of GEOM(4) and devfs(4),
including to mount root fs from a compressed drive.
- convert debug log code from being compiled in conditionally into
being present all the time and provide two sysctls to turn it on or
off. Due to intended use of the module, it can be used in
environments where there may not be a luxury to put new kernel with
debug code enabled. Having those options handy allows debug issues
without as much problem by just having access to serial console or
network shell access to a box/appliance. The resulting additional
CPU cycles are just few int comparisons and branches, and those are
minuscule when compared to data decompression which is the main
feature of the module.
- hopefully improve robustness and resiliency of the geom_uzip(4) by
performing some of the data validation / range checking on the TOC
entries and rejecting to attach to an image if those checks fail.
- merge in all relevant LZMA decompression support from the
geom_uncompress(4), enable automatically when appropriate format is
indicated in the header.
- move compilation work into its own worker thread so that it does not
clog g_up. This allows multiple instances work in parallel utilizing
smp cores.
- document new knobs in the manual page.
Reviewed by: adrian
MFC after: 1 month
Differential Revision: https://reviews.freebsd.org/D5333
and manual page were.
For whatever reason it listed myself as a primary author, which is
just not true.
Also, majority of the manpage is copied verbatim from the geom_uzip(4),
contributed by ceri, with only minor adjustments from loos, so put ceri
back into the copyright secrion where he belongs and reflect that in the
AUTHORS section.
For what it's worth, I think this one should be deleted and LZMA
support just folded back into geom_uzip(4) / mkuzip(4) whete it really
belongs.
MFC after: 1 month
documented and easy to miss.
At the same time, it's pretty important for anyone who is trying to use
SEEK_HOLE/SEEK_DATA in real app. Try to bridge that gap by making that
description more pronounced and also document how it affects failure codes.
Reviewed by: kib
Differential Revision: https://reviews.freebsd.org/D5162
Maxim Sobolev