Colin Percival
0f7b8f79f6
We had ENA_RX_BUDGET = 256 in order to allow up to 256 received packets to be processed before we do other cleanups (handling tx packets and, critically, refilling the rx buffer ring). Since the ring holds 1024 buffers by default, this was fine for normal packets: We refill the ring when it falls below 7/8 full, and even with a large burst of incoming packets allowing it to fall by another 1/4 before we consider refilling the ring still leaves it at 7/8 - 1/4 = 5/8 full. With jumbos, the story is different: A 9k jumbo (as is used by default within the EC2 network) consumes 3 descriptors, so a single rx cleanup pass can consume 3/4 of the default-sized rx ring; if the rx buffer ring wasn't completely full before a packet burst arrives, this puts us perilously close to running out of rx buffers. This precise failure mode has been observed on some EC2 instance types within a Cluster Placement Group, resulting in the nominal 10 Gbps single-flow throughput between instances dropping to ~100 Mbps as a result of repeated rx overruns causing packet loss and ultimately retransmission timeouts. To correct this, switch from processing up to ENA_RX_BUDGET (256) packets to processing up to ENA_RX_DESC_BUDGET (256) descriptors (or slightly more, if we hit the limit in the middle of a packet). This ensures that, even with jumbos, we refill the ring before processing most of a ring worth of descriptors, and returns the throughput to expected levels. Note that theoretically up to ENA_PKT_MAX_BUFS (19) descriptors can be used for a single packet, in which case even 54 packets would exhaust the default rx buffer ring; it's not clear if this ever occurs in practice, but this fix will address that case as well. Reviewed by: akiyano Sponsored by: Amazon MFC after: 6 days Differential Revision: https://reviews.freebsd.org/D56479
FreeBSD Source:
This is the top level of the FreeBSD source directory.
FreeBSD is an operating system used to power modern servers, desktops, and embedded platforms. A large community has continually developed it for more than thirty years. Its advanced networking, security, and storage features have made FreeBSD the platform of choice for many of the busiest web sites and most pervasive embedded networking and storage devices.
For copyright information, please see the file COPYRIGHT in this directory. Additional copyright information also exists for some sources in this tree - please see the specific source directories for more information.
The Makefile in this directory supports a number of targets for building components (or all) of the FreeBSD source tree. See build(7), config(8), FreeBSD handbook on building userland, and Handbook for kernels for more information, including setting make(1) variables.
For information on the CPU architectures and platforms supported by FreeBSD, see the FreeBSD website's Platforms page.
For official FreeBSD bootable images, see the release page.
Source Roadmap:
| Directory | Description |
|---|---|
| bin | System/user commands. |
| cddl | Source code for third-party software under the Common Development and Distribution License. |
| contrib | Source code for third-party software. |
| crypto | Source code for cryptographic libraries and commands (see crypto/README). |
| etc | Template files for /etc. |
| gnu | Source code for third-party software under the GNU General Public License (GPL) or Lesser General Public License (LGPL). Please see gnu/COPYING and gnu/COPYING.LIB for more information. |
| include | System include files. |
| kerberos5 | Build system for Kerberos 5 (Heimdal). |
| krb5 | Build system for Kerberos 5 (MIT). |
| lib | System libraries. |
| libexec | System commands intended to be executed by other commands or daemons. |
| release | Makefiles and scripts used for building releases and VM images. |
| rescue | Build system for statically linked /rescue commands. |
| sbin | System commands. |
| secure | Build system for cryptographic libraries and commands (excluding Kerberos). |
| share | Shared resources. |
| stand | Boot loader sources. |
| sys | Kernel sources (see sys/README.md). |
| targets | Support for experimental DIRDEPS_BUILD |
| tests | Tests which can be run by Kyua. See tests/README for additional information. |
| tools | Ancillary utilities and tests (not included in the build). |
| usr.bin | User commands. |
| usr.sbin | System administration commands. |
For information on synchronizing your source tree with one or more of the FreeBSD Project's development branches, please see FreeBSD Handbook.