Partitions are trivially aligned to sector boundaries, so asking e.g.
for a partition to be aligned to a 64-byte boundary is pointless but
harmless. Treat it as such, with a warning (in case the request was
a mistake, e.g. requesting 1-byte instead of 1M-byte alignment) rather
than as an error.
Reviewed by: markj
Suggested by: markj
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60819
If VM_PARTALIGN is set, pass %${VM_PARTALIGN} to mkimg in the
specification of swap, config, root, and ESP partitions. (Boot
partitions are not aligned, both because they're small and in
case firmware makes assumptions about their locations.)
Reviewed by: lwhsu, imp
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60454
This adds a "%align" field to partition specifications, resulting in
the start of a partition being shifted to the next aligned position.
When both :offset and %align are used, a relative offset is treated as
a minimum value, while an unaligned absolute offset is an error.
Reviewed by: imp (earlier version)
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60450
When handling PART_KIND_FILE, the "error" variable is set to reflect
any errors in parsing the specified offset; but that value is then
overwritten by the return value of image_copyin.
Error out immediately instead. Note that it is not necessary to change
the code in PART_KIND_SIZE which parses the size, since in that case we
do not overwrite the value in "error".
Fixes: 8f3c71c85e ("mkimg: Add support for offset if the source is an image")
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60422
We don't need to pick a repository URL when all we're doing is enabling
one which was specified in /etc/pkg/.
Reviewed by: dch, imp
Fixes: c73ae67348 ("release/tools: use same pkg settings ...")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D60868
Add code for using got(1), matching how git(1) is currently used:
1. Determining the git branch,
2. Determining the checked-out hash,
3. Determining the commit count,
4. Determining the timestamp of the most recent commit,
5. Determining if the tree is "dirty",
6. Generating lists of files for the src and src-sys packages, and
7. Excluding .got directories from src.txz and ports.txz.
The release engineer hopes to use this functionality in release builds,
since got's minimalism makes it a far more manageable build tool than
git.
Reviewed by: naddy
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60606
Once enabled on all of the branches, this will reduce bandwidth
consumption from uploading weekly snapshot builds from ~500 GB to
~100 GB, as well as significantly speeding up the process.
MFC after: 2 months
Sponsored by: Amazon
Starting with version 1.5.0, bsdec2-image-upload supports uploading
stream-optimized VMDK files. If we build images in that format, we
need to upload them appropriately.
MFC after: 2 weeks
Sponsored by: Amazon
This is the 'stream-optimized' version of the VMDK format.
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60154
Also, adjust bootstrap tools code in Makefile.inc1 to reflect this; in
addition to rebuilding mkimg if it is out of date, we now need to build
libz, which was not previously needed by mkimg.
Reviewed by: kevans
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60153
FreeBSD VM/cloud images tend to be highly compressible: The generic VM
images compress roughly 3.5:1, and cloud images typically even more
since they have significant unused space in their virtual disks. This
generally doesn't matter for users who can download compressed release
images and extract them locally, but for EC2 in particular relying on
uncompressed image formats is painful: We now upload over 500 GB/week
to AWS.
This commit adds the "stream-optimized" version of the VMDK format,
which compresses each 64 kB "grain" individually (and omits grains
which are all zeroes). Compared to uncompressed formats, this can
produce much smaller images; experiments indicate that using this
format for EC2 image uploads will reduce the weekly traffic to under
100 GB/week.
Co-authored-by: Claude Opus 5.5
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60151
This will allow it to be reused in upcoming work.
Suggested by: Claude Opus 5.5
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60150
The image_file_map function adjusts the provided file offset to be
page-aligned, with a resulting increase in the size of the mapped
region; the increased size needs to be used when unmapping as well.
Reported by: Claude Opus 5.5
Fixes: baf4abfc39 ("Allow building mkimg as cross-tool")
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60148
If realloc fails, clean up and return ENOMEM; don't just copy data
into NULL.
Reviewed by: emaste
Reported by: Claude Opus 5.5
MFC after: 2 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60147
When creating VM images, we filter the METALOG file created by pkg(8)
when installing non-base packages, rejecting any lines which correspond
to files which don't exist; this solves a problem which arose when a
package was installed and then deinstalled (or upgraded) later in the
image-building process.
Unfortunately [ -e ... ] follows symlinks and is not basedir-aware, so
an absolute symlink which is valid *inside* the image is omitted from
the image if it points to something which isn't present in the build
host system.
Replace [ -e ... ] with [ -e ... ] || [ -L ... ] so that symlinks are
included even if dangling.
While I'm here, add quoting in case future paths become problematic.
Sponsored by: Amazon
MFC after: 3 days
We don't have any packages to install by default when the instance
first boots, so (as with the "small" flavour) don't enable the
firstboot_pkgs script. If someone wants to launch a desktop with
packages autoinstalled at first boot, they can enable the script at
the same time as they provide the list of packages.
Sponsored by: Amazon
Using a backtick inside a backtick-expansion isn't a good idea; switch
instead to using $( ... ).
Fixes: a971034951 ("EC2: Add desktop flavour")
Sponsored by: Amazon
Desktop AMIs have xrdp enabled and boot to a KDE desktop; they are
as compatible as possible with EC2 Windows AMIs, setting a random
password and printing it to the console in encrypted format to be
retrieved using the EC2 GetPasswordData API.
Two rc.d scripts are included in this commit which will not exist
in the long term: ec2_addpass will become part of the ec2-scripts
package, and ec2_desktop_extras will go away once its functionality
is included elsewhere.
MFC After: 1 month
Relnotes: yes
Sponsored by: Amazon
By default, makefs uses the host environment's user and group databases
when creating filesystems. This causes makefs to fail when trying to
create files owned by users or groups which don't exist in the host
environment, for example when creating a VM with packages pre-installed
which added their own users/groups.
Pass "-N ${DESTDIR}/etc" to makefs to point it at the user and group
databases from the image being created.
MFC after: 1 week
Sponsored by: Amazon
The strtoul function sets errno on error, but does not clear it on
success; when using strtoul and checking errno (as one should) for
ERANGE / EINVAL afterwards, it's important to zero errno first.
While here, remove a dead store.
Reviewed by: phk
Fixes: 4fe8c1b67b ("Add error and range checking ... ")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D59270
Enable the new growfs_postboot mechanism. Note that this also implies
disabling automatic allocation of swap space on the root disk, since we
cannot grow the root filesystem if swap space is allocated after it.
This will not be MFCed since it is a significant behavioural change.
Sponsored by: Amazon
Relnotes: yes
In VM and cloud environments it is often possible to enlarge virtual
disks; this can be useful, for example, if a system is launched with a
small root disk and it later becomes clear that more space is needed.
On kernels which support run-time resizing of disks (for NVMe, this was
added in November 2025; some other disk types have supported this for
longer) a SIZECHANGE notification is sent to userland via devd.
Add a "nostart" rc.d script (runnable manually but not automatically at
boot time) and a devd script which invokes it when a notification
arrives. The rc.d script enlarges the "final partition" on partitioned
geoms, or the UFS filesystem or zpool device when triggered on a disk
containing either of those.
Reviewed by: imp, ziaee
MFC after: 2 weeks
Relnotes: Disk partitions and filesystems can be enlarged
automatically when disks grow by setting
growfs_postboot_enable=YES in /etc/rc.conf.
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D58582
This was a good idea, but we don't build metapackages in the kmods
repo so it ends up breaking the release build. I might resurrect this
change if/when the kmods repo includes the wifi-firmware-kmod
metapackage.
This reverts commit bda8028146.
MFC after: 1 second
This controller exhibits QUIRK_EMPTY_NAMESPACE_CHANGED_LOG behaviour.
A bug report has been filed with the vendor.
Reviewed by: imp
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D58232
Add a QUIRK_EMPTY_NAMESPACE_CHANGED_LOG quirk which indicates that the
nvme controller may not properly populate the namespace-changed log
page. If we receive a NVME_LOG_CHANGED_NAMESPACE page for a device
with this quirk and the page is empty, probe all of the namespaces
rather than none of them.
Reviewed by: imp
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D58231
We reach ndaasync with the CAM device lock held, so we must pass
M_NOWAIT to disk_* rather than M_WAITOK.
Reviewed by: imp
Fixes: 628d7a3270 ("nda: AC_GETDEV_CHANGED calls media chanaged for sectorsize change")
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D58230
We used to pass CONFIGURE_ARGS to the make command which builds pkg,
but ports/ports-mgmt/pkg/Makefile has its own CONFIGURE_ARGS and the
version we were providing at the command line didn't contain the
--mandir setting which was added to the port with pkg 2.8.0. This
broke release builds.
Instead of passing --prefix=${LOCALBASE} via CONFIGURE_ARGS, pass
PREFIX=${LOCALBASE}; the port Makefile passes that value through to
its configure script. We also used to pass a --host parameter, but
that seems to have become unnecessary at some point in the past decade.
MFC after: 1 day
Sponsored by: Amazon
When compiled without 'options RSS', the ena driver created taskqueues
using taskqueue_start_threads_cpuset passing a mask value of NULL,
both in the ena_setup_tx_resources path (for enqueues) and in the
ena_create_io_queues path (for the completion-processing).
In the default configuration, on most EC2 instances, this results in
taskqueues running in the right NUMA domain, but only by accident; in
non-default configurations (e.g. with with multiple EBS volumes
attached and associated NVMe taskqueues) the taskqueues may land in
the wrong NUMA domain even on instance types where the one-EBS-one-ENA
case produces the desired results.
Set (struct ena_que)->domain and use that to inform the choice of CPU
sets. On a c8gn.48xlarge EC2 instance this doubles throughput on a
32-TCP-stream benchmark.
Reviewed by: akiyano
MFC after: 7 days
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D57918
The EC2 "base" flavour installs the devel/py-awscli package at boot
time by default; we don't do this in the "small" flavour, so the
default behaviour was to update the FreeBSD-ports repository and
then do nothing with it.
Turn off firstboot_pkgs by default; if someone is using the "small"
flavour of AMIs and wants to install packages at instance launch
time, they simply need to add 'firstboot_pkgs_enable="YES"' to
/etc/rc.conf (which they must already be editing via user-data, in
order to provide the list of packages they want installed).
Sponsored by: Amazon
MFC after: 3 days
MFC to: stable/15
Relnotes: EC2 "small" images now have firstboot_pkgs_enable="NO".
Update a couple more cloudware images which I forgot about earlier.
Reviewed by: ziaee
Fixes: 464a351267 ("Cloud releases: Switch to firstboot_pkg_upgrade")
Differential Revision: https://reviews.freebsd.org/D57006
We had ENA_RX_BUDGET = 256 in order to allow up to 256 received
packets to be processed before we do other cleanups (handling tx
packets and, critically, refilling the rx buffer ring). Since the
ring holds 1024 buffers by default, this was fine for normal packets:
We refill the ring when it falls below 7/8 full, and even with a large
burst of incoming packets allowing it to fall by another 1/4 before we
consider refilling the ring still leaves it at 7/8 - 1/4 = 5/8 full.
With jumbos, the story is different: A 9k jumbo (as is used by default
within the EC2 network) consumes 3 descriptors, so a single rx cleanup
pass can consume 3/4 of the default-sized rx ring; if the rx buffer
ring wasn't completely full before a packet burst arrives, this puts
us perilously close to running out of rx buffers.
This precise failure mode has been observed on some EC2 instance types
within a Cluster Placement Group, resulting in the nominal 10 Gbps
single-flow throughput between instances dropping to ~100 Mbps as a
result of repeated rx overruns causing packet loss and ultimately
retransmission timeouts.
To correct this, switch from processing up to ENA_RX_BUDGET (256)
packets to processing up to ENA_RX_DESC_BUDGET (256) descriptors (or
slightly more, if we hit the limit in the middle of a packet). This
ensures that, even with jumbos, we refill the ring before processing
most of a ring worth of descriptors, and returns the throughput to
expected levels.
Note that theoretically up to ENA_PKT_MAX_BUFS (19) descriptors can be
used for a single packet, in which case even 54 packets would exhaust
the default rx buffer ring; it's not clear if this ever occurs in
practice, but this fix will address that case as well.
Reviewed by: akiyano
Sponsored by: Amazon
MFC after: 6 days
Differential Revision: https://reviews.freebsd.org/D56479
The ena_[rt]x_cleanup functions are limited internally to a maximum
number of packets; this ensures that TX doesn't starve RX (or vice
versa) and also attempts to ensure that we get a chance to refill
the RX buffer ring before the device runs out of buffers and starts
dropping packets.
Historically these functions have returned the number of packets which
they processed which ena_cleanup compares to their respective budgets
to decide whether to reinvoke them. This is unnecessary complication;
since the precise number of packets processed is never used, adjust
the APIs of those functions to return a bool indicating if they want
to be reinvoked (aka if they hit their limits).
Since ena_tx_cleanup now only uses work_done if diagnostics are
enabled (ena_log_io macros to nothing otherwise) eliminate that
variable and pass its value (ENA_TX_BUDGET - budget) to ena_log_io
directly.
No functional change intended; this will simplify a future commit.
Reviewed by: akiyano
Sponsored by: Amazon
MFC after: 6 days
Differential Revision: https://reviews.freebsd.org/D56478
Hyper-V advertises support for the Extended Destination ID standard via
bit 2 of the value returned in the EAX register when the hypervisor
stack properties are queried via CPUID.
This is based on a commit to the Linux kernel, as there does not seem
to be any other documentation of this feature.
Reviewed by: Souradeep Chakrabarti
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55432
Bhyve advertises support for the Extended Destination ID standard via
bit 0 (aka CPUID_BHYVE_FEAT_EXT_DEST_ID) of the value returned in the
EAX register when Bhyve features are queried via CPUID.
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55431
The values CPUID_BHYVE_FEATURES and CPUID_BHYVE_FEAT_EXT_DEST_ID are
useful for guests, not just hosts; so they belong in a header file in
sys/x86/include rather than simply in the .c file implementing the
bhyve host side.
The original addition of these defines took place without adding a
copyright statement, but since I'm moving them into a new file I've
added the original author's standard copyright (Amazon).
MFC after: 3 weeks
Fixes: 313a68ea20 ("bhyve: Add CPUID_BHYVE_FEATURES leaf")
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55430
Xen advertises support for the Extended Destination ID standard via
bit 5 (aka XEN_HVM_CPUID_EXT_DEST_ID) of the value returned in the
EAX register when Xen features are queried via CPUID.
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55429
KVM advertises support for the Extended Destination ID standard via
bit 15 of the value returned in the EAX register when KVM features
are queried via CPUID.
Tested on: EC2 r8i.96xlarge
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55427
If APIC Extended Destination ID support is enabled, use it in APIC RTEs
by allowing APIC IDs up to 2^15 - 1 and encoding the high bits into
Intel "reserved" bits per the standard.
Reviewed by: kib
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55889
If APIC Extended Destination ID support is enabled, use it in MSIs by
allowing APIC IDs up to 2^15 - 1 and encoding the high bits into
Intel "reserved" bits per the standard.
Tested on: EC2 r8i.96xlarge
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55426
Without an IOMMU, the APIC standard only allows 8 bits of Destination
ID for MSI messages, limiting us to 256 CPUs. While IOMMUs can allow
for more than 256 CPUs to be supported, they are not necessarily
desirable in virtualized environments.
The Extended Destination ID standard authored by David Woodhouse uses
7 "Reserved" bits for the high bits of a 15-bit Extended Destination
ID in order to address this: http://david.woodhou.se/ExtDestId.pdf
Add a loader tunable machdep.apic_ext_dest_id to control the use of
this feature; the default value (-1) means "autodetect" while 0 and
1 mean disabled and enabled respectively.
Code to detect host support in Xen, Hyper-V, KVM, and Bhyve will come
in future commits, as will the code to use this setting in msi_map and
ioapic_program_intpin.
Tested on: EC2 r8i.96xlarge
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55890
I/O APIC Redirection Table Entries use 8 bits to encode the Destination
ID. Attempting to route an IRQ to a higher APIC ID would result in it
being silently routed to the value reduced modulo 256, causing a panic
if the IRQ fired since the receiving CPU would not expect that IRQ.
Instead, print a warning and mark the interrupt as invalid, resulting
in it being forcibly masked.
Reviewed by: kib
Tested on: EC2 r8i.96xlarge
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55857
As of this commit, io_valid is always set to 1; but a future commit
will set it to 0, at which point IOART_INTMSET will be set to forcibly
disable interrupt sources regardless of whether they are requested to
be "masked".
Reviewed by: kib
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D56006
If an interrupt arrives at a CPU which isn't expecting that particular
vector, intr_lookup_source will return an isrc of NULL and we'll panic
when intr_execute_handlers increments *isrc->is_count.
Place a KASSERT a few nanoseconds earlier in order to leave some more
breadcrumbs for the next person to trip over this behaviour.
Tested on: EC2 r8i.96xlarge
MFC after: 3 weeks
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55851
This reverts commit 6a13aeac3c.
The "bogus dependencies being installed" issue was traced down to pkg
confusion surrounding libclang_rt.asan-i386.so linking to a 32-bit
libgcc_s.so.1, and a more minimal workaround of "install clibs-lib32"
has been applied, so this hack is no longer required.
MFC after: 3 days
Sponsored by: Amazon
The FreeBSD-clang package on amd64 contains libclang_rt.asan-i386.so,
which links to the 32-bit version of libgcc_s.so.1. It is not clear
if that file belongs in FreeBSD-clang or if it should be placed into
a different package.
For unknown reasons, pkg *sometimes* recognizes this and decides that
it needs to install a package to supply libgcc_s.so.1:32.
In particular, when we initially install the VM with 'pkg install
[long list of FreeBSD-* packages]', pkg does not recognize that it
wants this; but the *next* time 'pkg install' runs, it decides that
it needs libgcc_s.so.1:32 -- even though that particular library is
not needed by the particular package we're trying to install -- and
goes looking for it... and ends up picking gcc12-devel as a provider.
Later in the EC2 image building process, we run 'pkg autoremove' (which
was added during 15.0 to get rid of the "bogus dependencies" which were
added while installing other packages) and pkg is once again confused:
It decides that gcc12-devel is an automatic port which is not required
by anything else, but upon determining that it should be removed it
then determines that FreeBSD-clang depends upon a port which is being
removed and deletes FreeBSD-clang as well.
It's not clear what is breaking in pkg, but as a temporary workaround
add the FreeBSD-clibs-lib32 package to the "small" and "builder" EC2
image flavours in order to satisfy the libgcc_s.so.1 demand.
This commit should be reverted if libclang_rt.asan-i386.so moves out
of FreeBSD-clang, (i.e. if it becomes possible to install a 32-bit-free
amd64 system) or when the pkg issue is resolved (i.e. pkg consistently
recognizes the shared library dependency and installs clibs-lib32
automatically as a FreeBSD-clang dependency).
MFC after: 3 days
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55978
The boot loader menu is disabled by default in EC2, but if it is ever
turned on, the default (unicode) output breaks EC2's web interface to
the serial console.
Set loader_menu_frame="ascii" instead.
MFC after: 3 days
Sponsored by: Amazon
Having these packages available on release media may help users who
need to sneakernet other packages (e.g. firmware) from systems running
Linux or Windows.
Suggested by: vladlen, ziaee
MFC after: 3 days
When interrupt vectors are first allocated, they get assigned to
CPU #0; at SI_SUB_SMP / SI_ORDER_SECOND (aka once we have multiple
CPUs), the intr_irq_shuffle SYSINIT clears their CPU sets with the
effect of forcing them to be assigned to new CPUs later.
In case where interrupt vectors were allocated *but not yet bound*
this code did not run, with the effect that those interrupts would
remain pinned to CPU #0 forever. This affected the ena(4) driver,
which allocates interrupts for I/O when the device is attached but
doesn't set them up until the interface is brought up much later in
the boot process (and, crucially, long after intr_irq_shuffle runs).
Adjust intr_irq_shuffle to clear the CPU set for an interrupt source
even if it currently has no handlers, so that it will be properly
assigned to a CPU when it is used later.
Reviewed by: andrew, mhorne
MFC after: 1 month
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55284
Running `pkg -d` in pkg-stage.sh results in multiple GB of network
traffic being written into the log files, which is less than helpful
when it comes to tracking down build failures. Remove the -d flag.
MFC after: 5 days
X-MFC-note: The code in 15 has diverged from 14, but the flag is
there, just in a different place.
The packages for X.Y-RELEASE are a snapshot of the quarterly branch
for stable/X, i.e. typically built on X.(Y-1)-RELEASE. (The case of
Y=0 is an exception for obvious reasons.) This works for most ports
but not for kernel modules, which may need to be built on the release
in question; this is why we have a separate "kmods" repository.
Make sure that we put the packages from the kmods repository onto the
mini-pkg-repo on the DVD rather than shipping unusable firmware.
Reviewed by: bz
MFC after: 1 week
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D52581
When building VM and Cloud images with a packaged base system, install
the pkg package; this makes it possible for the base system to be
updated without installing pkg from the FreeBSD-ports repository.
Sponsored by: Amazon
For unclear reasons, we used to print two warnings about not having a
DTB: First,
printf("No valid device tree blob found!\n");
and immediately later a second,
printf("WARNING! Trying to fire up the kernel, but no "
"device tree blob found!\n");
A previous commit silenced the second warning in the event that the
acpi.revision environment variable was set, on the theory that systems
which have ACPI might not need a DTB; this commit silences the first
of the two warnings under the same condition.
Reviewed by: andrew
Fixes: 3989a3d817 ("loader: Only warn when we have neither FDT nor ACPI")
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D54772
Record releases in the SSM Parameter Store not just as e.g.
/aws/service/freebsd/arm64/small/ufs/15.0/RELEASE
but also as .../RELEASE/latest, e.g.
/aws/service/freebsd/arm64/small/ufs/15.0/RELEASE/latest
The latter parameters will be updated when new AMIs are built with
security and errata updates appplied, and have been backfilled for
15.0.
MFC after: 3 days
MFC to: stable/15 (14.x will not get security-update AMIs)
Sponsored by: Amazon
Rather than fetching packages directly from the CDN which currently
backs pkgbase.freebsd.org, requests will go to pkg.freebsd.org mirrors
and be 302ed to the correct servers. This adds ~70 seconds to the
process of installing or upgrading a pkgbase system; it also orphans
systems with 15.0-{PRERELEASE,ALPHA*,BETA*} installed since they are
expecting to see pkgbase files signed with the pkg keys, not the new
pkgbase signing keys.
Reviewed by: dch, philip
MFC after: immediately (for 15.0-RELEASE)
With hat: re
Requested by: clusteradm, core
Differential Revision: https://reviews.freebsd.org/D53964
The patch level is not part of the branch per se and should not be
used in constructing the FreeBSD-base.conf file used by bsdinstall.
MFC after: 1 day
Contrary to the claim made in a previous commit, removing KDE and
adding all of vim and emacs results in an image which does not fit
into 4.7 GB; to be specific, it lands at 4.722 GB rather than the
claimed 4.689 GB. (This descrepancy resulted from doing test DVD
image builds using an out-of-date tree, and became visible when the
15.0-RC3 images were built.)
Limit the emacs packages shipped on the DVD to the "nox" flavor;
this brings the disk image down to 4.407 GB, aka under the 4.7 GB
limit for standard DVDs.
Fixes: 6cc6beb4c8 ("release: Remove KDE from dvd1.iso")
MFC after: 1 day (for 15.0-RC4)
Prior to this commit, we were shipping 2155 MB of packages (from the
ports tree, not counting pkgbase) on dvd1.iso. Due to the amount of
space required by shipping pkgbase packages *and* distribution sets
on the DVD images, we only have 1696 MB available if we want to fit
into the 4.7 GB limit for DVDs. Many users have indicated that this
is indeed important.
It is practically impossible to hit this target without removing KDE;
while KDE and its dependencies narrowly fit (1550 MB), we exceed the
limit as soon as we include either of freebsd-doc-all or gnome. While
we would pick KDE over GNOME (surveys regularly indicate that KDE is
the more widely used of the two), we believe that documentation is the
most important thing to include.
Since removing KDE leaves a bit of extra space, add editors/emacs and
editors/vim. This takes the 15.0 amd64 dvd1.iso up to 4.689 GB. [1]
Requested by: adamw [1]
MFC after: immediately (for 15.0-RC3)
Differential Revision: https://reviews.freebsd.org/D53800
The packages for 15.0-RELEASE built without the bug fix needed to make
files created via @sample get properly listed in METALOG. Fix the
cloudware which contain @sample-using packages by adding the necessary
files to METALOG manually.
This should be reverted after the next full package build, and live on
only in releng/15.0.
Reviewed by: markj
MFC after: immediately (15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53797
We ship these in order to comply with GCE Marketplace rules about
providing source code and licenses for all the software we ship as
part of images.
Reviewed by: markj
MFC after: immediately (15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53796
These were forgotten during the METALOGization process earlier.
Reviewed by: markj
MFC after: immediately (for 15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53795
Paths all need to start with "./" because that's what newfs wants.
Fixes: e0c41af925 ("vmimage.subr: Enable FreeBSD-base repo if pkgbase")
MFC after: immediately
When installing "extra" packages (aka those built from the ports tree),
we record everything being installed in METALOG.pkg; the contents of
that file is appended to METALOG before we generate the filesystem.
There are two cases when files recorded in METALOG.pkg will no longer
exist by the time we create the final disk image:
1. If a pkg bug results in false dependencies being installed which
are later removed by "pkg autoremove", and
2. If the pkg we build and install from /usr/ports is older than the
pkg on pkg.freebsd.org, and pkg gets upgraded automatically as part of
installing extra packages.
The ultimate issue in both cases is that there's no mechanism for
removing entries from METALOG when we run 'pkg delete'.
Address this build breakage by checking, line by line, if filesystem
objects mentioned in METALOG.pkg exist before appending them to METALOG.
Fixes: 6a13aeac3c ("vmimage.subr: pkg autoremove after pkg install")
MFC after: immediately (needed for 15.0-RC1)
Running 'pkg autoremove' without -y results in VM image builds failing
when (bogusly installed) packages are removed.
Fixes: 6a13aeac3c ("vmimage.subr: pkg autoremove after pkg install")
MFC after: immediately (needed for 15.0-RC1)
When creating a VM image using pkgbase, create a configuration file in
/usr/local/etc/pkg/repos/FreeBSD.conf which enables the FreeBSD-base
repository. (This repository is defined in /etc/pkg/FreeBSD.conf as
being disabled by default.)
Reported by: Mark Millard
MFC after: immediately (needed for 15.0-RC1)
When performing a pkgbase install, create a configuration file in
/usr/local/etc/pkg/repos/FreeBSD.conf which enables the FreeBSD-base
repository. (This repository is defined in /etc/pkg/FreeBSD.conf as
being disabled by default.)
Reported by: Mark Millard
Reviewed by: markj
MFC after: immediately (needed for 15.0-RC1)
Differential Revision: https://reviews.freebsd.org/D53777
We were doing this in vm_extra_install_packages but VM images without
any extra packages installed would not get this installed. This
results in a pkgbase system which thinks it doesn't have any packages
installed (even though all the files are right there).
Add a "metalog_add_data ./var/db/pkg/local.sqlite" call to the pkgbase
install code path, and make the call from vm_extra_install_packages
conditional on !PKGBASE.
Reported by: Michael Dexter
MFC after: immediately (needed for 15.0-RC1)
We ingest Makefile.gce even when we're not trying to create GCE images
so we don't want to .error here. Instead, set GCE_ARCH to a dummy
value which should make the problem clear to anyone who attempts to
create GCE images on an unsupported architecture.
Reported by: Jenkins
Fixes: 0a8ecca4e3 ("GCE: Specify the architecture of images")
On -CURRENT and -STABLE this gets bits from pkg.freebsd.org; but we get
base system bits from pkgbase.freebsd.org for BETA/RC/RELEASE.
Note that this repository is disabled by default, but can still be used
by explicitly specifying it, e.g. "pkg upgrade -r FreeBSD-base".
With hat: re
MFC after: 8 hours (needed in 15.0-RC1)
When installing e.g. 15.0-RC1, we want to get files from the 15.0-RC1
pkgbase repository; but running 'pkg upgrade' after installation should
get the latest bits build from releng/15.0.
With hat: re
MFC after: 8 hours (needed in 15.0-RC1)
For BETA/RC/RELEASE builds, fetch files from the appropriate repository
on pkgbase.freebsd.org, using the appropriate signing keys. Note that
there is a separate repository for each BETA and RC; this ensures that
someone installing from e.g. a 15.0-RC1 ISO will get 15.0-RC1 bits and
not whatever happens to be the most recent build from releng/15.0.
With hat: re
MFC after: 8 hours (needed in 15.0-RC1)
These keys were generated by cperciva@ using the AWS Key Management
Service. They will not be used for signing anything other than
FreeBSD 15.x pkgbase repositories.
Keys will be generated for FreeBSD 16 at a later date, and likely in
a different way.
With hat: re
MFC after: 8 hours (needed in 15.0-RC1)
Differential Revision: https://reviews.freebsd.org/D53768
The keys used for pkgbase signing are going to be placed in
/usr/share/keys/pkgbase-N where N is the FreeBSD major version
number; as such it's not sufficient to copy /usr/share/keys/pkg
into the install chroot, but instead we need /usr/share/keys/*.
With hat: re
MFC after: 8 hours (needed in 15.0-RC1)
Differential Revision: https://reviews.freebsd.org/D53753
We need to specify the correct image names -- *.vhdf, not *.vhd -- in
order for them to upload.
15.0 candidate
Reviewed by: lwhsu
MFC after: 2 days
Differential Revision: https://reviews.freebsd.org/D53684
We have NO_ROOT here, so we need WITHOUT_QEMU to avoid problems.
15.0 candidate.
Reviewed by: emaste, markj
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53637
We want to fetch distfiles, regardless of whether they contain known
vulnerabilities or we're building images for a different version of
FreeBSD.
Reviewed by: ivy
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53569
In order to comply with the require that GCE images must include their
source code, we fetch distfiles for all of the packages installed into
GCE images. This fails for obvious reasons for packages with an origin
of base/*; filter those out to generate the list to fetch.
Reviewed by: ivy
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53568
GCE images are required by Google to include their source code; we do
this by extracting {src,ports}.txz into the images, from the (legacy)
distribution sets.
Make sure those distribution sets actually exist.
Reviewed by: ivy
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53567
A bug in pkg, which somehow only surfaced as a consequence of pkgbase,
results in pkg install sometimes pulling in false dependencies. This
problem might be limited to cases when the lib32 pkgbase packages are
not installed. In the case of EC2 "small" images, installing the
ebsnvme-id package results in binutils, gcc12-devel, gmp, indexinfo,
liblz4, mpc, mpfr, and zstd packages being installed.
These false dependencies are however not recorded as dependencies --
at some level pkg does understand that they're not needed -- so running
pkg autoremove immediately after pkg install cleans them up.
Note: This does not remove lines from METALOG corresponding to these
packages, and makefs emits an error when it attempts to create the
filesystem but cannot find the files listed in METALOG -- but makefs
does seem to complete normally despite the error messages.
This change should be reverted once the pkg issue has been located and
fixed.
Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53543
In order to make sure that man pages are all installed before we run
makewhatis to generate mandoc.db files, we have long placed etc at the
end of the list of subdirectories being recursed into by the build.
In order to support installworld -jN, a .WAIT was more recently added
here.
With the recent adoption by the release engineering team of parallel
*release* builds (aka 'make release -jN') it is now also necessary to
add the same .WAIT before recursing for the 'distribute' target, as we
otherwise end up with distribution sets containing incomplete mandoc.db
files.
Reviewed by: bdrewery
PR: 289683
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53533
If build logic breaks such that we can't figure out which branch we're
on, we should error out rather than shipping a default (and probably
wrong) /etc/pkg/FreeBSD.conf
Reviewed by: ivy
MFC after: 5 minutes
Suggested by: ivy
Differential Revision: https://reviews.freebsd.org/D53487
Building VMs as non-root requires no-QEMU code paths (installing
packages from outside the VM image rather than inside it) and vice
versa; we have a check for broken combinations.
Unfortunately that check was breaking
make -C src/usr.sbin/pkg NO_ROOT=YES -V PKGCONFBRANCH
because that code reaches into src/release to determine the branch
name (which is then used to determine which /etc/pkg/FreeBSD.conf to
install).
Wrap the no-root/no-qemu check in an .if to only run when we've
asked for VM and/or CLOUD building to be enabled.
Reviewed by: ivy
MFC after: 5 minutes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53486
We're correctly recording all of the packages in the dvd METALOG file,
but if we don't record ./packages/repos/FreeBSD_install_cdrom.conf then
users won't be able to install them very easily.
Reviewed by: markj
Reported by: Lars Tunkrans
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53199
We only need to check for unMETALOGed directories and sort the METALOG
file if we're using it, i.e. if we're doing a NO_ROOT build. This
non-NO_ROOT builds by no longer bogusly writing to /METALOG*.
We only need to add databases (spwd.db etc) to METALOG if we're doing
a pkgbase-enabled NO_ROOT build; but we should always do this before
creating the filesystem, not only if we installed extra packages (in
vm_extra_install_packages, where that code was erroneously placed).
This fixes non-cloud VM images, which in 15.0-BETA2 shipped without
password databases.
Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53194
Theoretically METALOG should include everything which needs to go
into disk images; unfortunately there are still a few bugs which
are resulting in directories not being listed -- and if METALOG
has files in unrecorded directories, the directories end up being
created with 000 permissions.
Oddly enough, systems where / has 000 permissions are not very
usable.
As a temporary hack, compare the staging tree against METALOG and
add entries for any unrecorded directories. This will hopefully
be reverted before 15.0-RELEASE.
Reviewed by: bapt, emaste, ivy
Sponsored by: https://www.patreon.com/cperciva
MFC after: 5 minutes
Differential Revision: https://reviews.freebsd.org/D53153
Colin Percival