If VM_PARTALIGN is set, pass %${VM_PARTALIGN} to mkimg in the
specification of swap, config, root, and ESP partitions. (Boot
partitions are not aligned, both because they're small and in
case firmware makes assumptions about their locations.)
Reviewed by: lwhsu, imp
MFC after: 1 week
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D60454
We don't need to pick a repository URL when all we're doing is enabling
one which was specified in /etc/pkg/.
Reviewed by: dch, imp
Fixes: c73ae67348 ("release/tools: use same pkg settings ...")
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D60868
When creating VM images, we filter the METALOG file created by pkg(8)
when installing non-base packages, rejecting any lines which correspond
to files which don't exist; this solves a problem which arose when a
package was installed and then deinstalled (or upgraded) later in the
image-building process.
Unfortunately [ -e ... ] follows symlinks and is not basedir-aware, so
an absolute symlink which is valid *inside* the image is omitted from
the image if it points to something which isn't present in the build
host system.
Replace [ -e ... ] with [ -e ... ] || [ -L ... ] so that symlinks are
included even if dangling.
While I'm here, add quoting in case future paths become problematic.
Sponsored by: Amazon
MFC after: 3 days
We don't have any packages to install by default when the instance
first boots, so (as with the "small" flavour) don't enable the
firstboot_pkgs script. If someone wants to launch a desktop with
packages autoinstalled at first boot, they can enable the script at
the same time as they provide the list of packages.
Sponsored by: Amazon
Using a backtick inside a backtick-expansion isn't a good idea; switch
instead to using $( ... ).
Fixes: a971034951 ("EC2: Add desktop flavour")
Sponsored by: Amazon
Desktop AMIs have xrdp enabled and boot to a KDE desktop; they are
as compatible as possible with EC2 Windows AMIs, setting a random
password and printing it to the console in encrypted format to be
retrieved using the EC2 GetPasswordData API.
Two rc.d scripts are included in this commit which will not exist
in the long term: ec2_addpass will become part of the ec2-scripts
package, and ec2_desktop_extras will go away once its functionality
is included elsewhere.
MFC After: 1 month
Relnotes: yes
Sponsored by: Amazon
By default, makefs uses the host environment's user and group databases
when creating filesystems. This causes makefs to fail when trying to
create files owned by users or groups which don't exist in the host
environment, for example when creating a VM with packages pre-installed
which added their own users/groups.
Pass "-N ${DESTDIR}/etc" to makefs to point it at the user and group
databases from the image being created.
MFC after: 1 week
Sponsored by: Amazon
In 16.0/15.1, the PAM modules were split from FreeBSD-runtime into a
new FreeBSD-pam package. FreeBSD-runtime does not install FreeBSD-pam,
which means if a user starts from runtime, then installs sshd, sshd will
fail to authenticate users because of missing PAM modules.
Since FreeBSD-pam is relatively small (about 230kB on amd64), and is
already part of FreeBSD-set-minimal, add it to the runtime image as
well. Users who absolutely don't want this can still build their
own images without it.
MFC after: 1 week
Reviewed by: dfr
Reported by: Michael Johnson <[email protected]>
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D59194
Enable the new growfs_postboot mechanism. Note that this also implies
disabling automatic allocation of swap space on the root disk, since we
cannot grow the root filesystem if swap space is allocated after it.
This will not be MFCed since it is a significant behavioural change.
Sponsored by: Amazon
Relnotes: yes
In some cases having a src tree in a VM image is convenient
for development or debugging. Add a WITH_SRC variable,
which, when set, will cause the vm-release target to include
FreeBSD-set-src in the list of packages installed in an image.
Signed-off-by: Krzysztof Galazka <[email protected]>
Sponsored by: Intel Corporation
Reviewed by: cperciva
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D57143
The EC2 "base" flavour installs the devel/py-awscli package at boot
time by default; we don't do this in the "small" flavour, so the
default behaviour was to update the FreeBSD-ports repository and
then do nothing with it.
Turn off firstboot_pkgs by default; if someone is using the "small"
flavour of AMIs and wants to install packages at instance launch
time, they simply need to add 'firstboot_pkgs_enable="YES"' to
/etc/rc.conf (which they must already be editing via user-data, in
order to provide the list of packages they want installed).
Sponsored by: Amazon
MFC after: 3 days
MFC to: stable/15
Relnotes: EC2 "small" images now have firstboot_pkgs_enable="NO".
Update a couple more cloudware images which I forgot about earlier.
Reviewed by: ziaee
Fixes: 464a351267 ("Cloud releases: Switch to firstboot_pkg_upgrade")
Differential Revision: https://reviews.freebsd.org/D57006
Cloud images are deployed with base system packages. Introduce a
firstboot package auto updater to patch the base system on first boot.
MFC after: 1 hour
MFC to: stable/15
Reviewed by: cperciva
Sponsored by: Google Cloud
Differential Revision: https://reviews.freebsd.org/D56890
Oracle's previous support is no longer available to the project.
Repeated attempts to find a sponsor within Oracle's cloud business
have not been successful.
The last published official images are from 15.0-RELEASE.
https://marketplace.oracle.com/app/freebsd-release
Relnotes: yes
Sponsored by: SkunkWerks, GmbH
Differential Revision: https://reviews.freebsd.org/D56360
MFC after: 3 days
This reverts commit 6a13aeac3c.
The "bogus dependencies being installed" issue was traced down to pkg
confusion surrounding libclang_rt.asan-i386.so linking to a 32-bit
libgcc_s.so.1, and a more minimal workaround of "install clibs-lib32"
has been applied, so this hack is no longer required.
MFC after: 3 days
Sponsored by: Amazon
The FreeBSD-clang package on amd64 contains libclang_rt.asan-i386.so,
which links to the 32-bit version of libgcc_s.so.1. It is not clear
if that file belongs in FreeBSD-clang or if it should be placed into
a different package.
For unknown reasons, pkg *sometimes* recognizes this and decides that
it needs to install a package to supply libgcc_s.so.1:32.
In particular, when we initially install the VM with 'pkg install
[long list of FreeBSD-* packages]', pkg does not recognize that it
wants this; but the *next* time 'pkg install' runs, it decides that
it needs libgcc_s.so.1:32 -- even though that particular library is
not needed by the particular package we're trying to install -- and
goes looking for it... and ends up picking gcc12-devel as a provider.
Later in the EC2 image building process, we run 'pkg autoremove' (which
was added during 15.0 to get rid of the "bogus dependencies" which were
added while installing other packages) and pkg is once again confused:
It decides that gcc12-devel is an automatic port which is not required
by anything else, but upon determining that it should be removed it
then determines that FreeBSD-clang depends upon a port which is being
removed and deletes FreeBSD-clang as well.
It's not clear what is breaking in pkg, but as a temporary workaround
add the FreeBSD-clibs-lib32 package to the "small" and "builder" EC2
image flavours in order to satisfy the libgcc_s.so.1 demand.
This commit should be reverted if libclang_rt.asan-i386.so moves out
of FreeBSD-clang, (i.e. if it becomes possible to install a 32-bit-free
amd64 system) or when the pkg issue is resolved (i.e. pkg consistently
recognizes the shared library dependency and installs clibs-lib32
automatically as a FreeBSD-clang dependency).
MFC after: 3 days
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55978
The boot loader menu is disabled by default in EC2, but if it is ever
turned on, the default (unicode) output breaks EC2's web interface to
the serial console.
Set loader_menu_frame="ascii" instead.
MFC after: 3 days
Sponsored by: Amazon
This matches non-pkgbase behavior as installworld/installkernel skip the
debug bits if these knobs are set.
MFC after: 1 week
Sponsored by: Chelsio Communications
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55572
We always use NO_ROOT for release artifact builds, so remove the
alternate code paths.
For the first step we set NO_ROOT unconditionally in cases that invoke
submakes, and turn NO_ROOT being unset into an error in lover-level
targets so that we can catch potential out-of-tree build scripts (or
missed in-tree cases) that expect to run not-NO_ROOT builds. The second
step will be to remove those entirely.
Reviewed by: cperciva
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D54179
Instructions in /etc/pkg/FreeBSD.conf and elsewhere recommend putting
changes in /usr/local/etc/pkg/repos/FreeBSD.conf so bring OCI containers
into line as well.
Reviewed by: dfr, ivy
Differential Revision: https://reviews.freebsd.org/D54090
MFC after: 5 days
Sponsored by: SkunkWerks, GmbH
When building VM and Cloud images with a packaged base system, install
the pkg package; this makes it possible for the base system to be
updated without installing pkg from the FreeBSD-ports repository.
Sponsored by: Amazon
The default shell for root has been changed to sh(1) followup changing
in release images sh(1) the shell for the "freebsd" user.
MFC After: 1 week
Reviewed by: manu, emaste (re)
Approved by: manu, emaste (re)
Differential Revision: https://reviews.freebsd.org/D54602
This is required for MSI support on GCE ARM64 instances which is
prerequisite to gve(4) not panicking at boot, and nvme(4) also has
a real sad time without interrupts. Tested on a variety of c4a VMs.
This is meant to be a temporary hack; long term fix would be to
check for the hypervisor and quirk gve(4) device with
PCI_QUIRK_ENABLE_MSI_VM.
PR: kern/292081
MFC after: 1 week
Removes hw.vtnet.mq_disable=1.
This workaround was originally introduced nearly a decade ago to
address stability issues on KVM that have long since been resolved
in both the FreeBSD driver and the GCE hypervisor. Removing this
allows network interrupts to scale across multiple vCPUs.
Tested on n2-highcpu-16 VM with 15.0-RELEASE and confirmed multiple
queue pairs active and interrupts handling across cores.
PR: kern/292081
MFC after: 1 day
The sed command was missing the ${DESTDIR} prefix, meaning it was
attempting to modify the build host's /etc/rc.d/growfs instead of
the target image's script. Tested in an arm64 builder that builds
as non-root.
PR: kern/292081
MFC after: 1 day
The packages for 15.0-RELEASE built without the bug fix needed to make
files created via @sample get properly listed in METALOG. Fix the
cloudware which contain @sample-using packages by adding the necessary
files to METALOG manually.
This should be reverted after the next full package build, and live on
only in releng/15.0.
Reviewed by: markj
MFC after: immediately (15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53797
We ship these in order to comply with GCE Marketplace rules about
providing source code and licenses for all the software we ship as
part of images.
Reviewed by: markj
MFC after: immediately (15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53796
These were forgotten during the METALOGization process earlier.
Reviewed by: markj
MFC after: immediately (for 15.0-RC2)
Differential Revision: https://reviews.freebsd.org/D53795
Paths all need to start with "./" because that's what newfs wants.
Fixes: e0c41af925 ("vmimage.subr: Enable FreeBSD-base repo if pkgbase")
MFC after: immediately
When installing "extra" packages (aka those built from the ports tree),
we record everything being installed in METALOG.pkg; the contents of
that file is appended to METALOG before we generate the filesystem.
There are two cases when files recorded in METALOG.pkg will no longer
exist by the time we create the final disk image:
1. If a pkg bug results in false dependencies being installed which
are later removed by "pkg autoremove", and
2. If the pkg we build and install from /usr/ports is older than the
pkg on pkg.freebsd.org, and pkg gets upgraded automatically as part of
installing extra packages.
The ultimate issue in both cases is that there's no mechanism for
removing entries from METALOG when we run 'pkg delete'.
Address this build breakage by checking, line by line, if filesystem
objects mentioned in METALOG.pkg exist before appending them to METALOG.
Fixes: 6a13aeac3c ("vmimage.subr: pkg autoremove after pkg install")
MFC after: immediately (needed for 15.0-RC1)
Running 'pkg autoremove' without -y results in VM image builds failing
when (bogusly installed) packages are removed.
Fixes: 6a13aeac3c ("vmimage.subr: pkg autoremove after pkg install")
MFC after: immediately (needed for 15.0-RC1)
When creating a VM image using pkgbase, create a configuration file in
/usr/local/etc/pkg/repos/FreeBSD.conf which enables the FreeBSD-base
repository. (This repository is defined in /etc/pkg/FreeBSD.conf as
being disabled by default.)
Reported by: Mark Millard
MFC after: immediately (needed for 15.0-RC1)
We were doing this in vm_extra_install_packages but VM images without
any extra packages installed would not get this installed. This
results in a pkgbase system which thinks it doesn't have any packages
installed (even though all the files are right there).
Add a "metalog_add_data ./var/db/pkg/local.sqlite" call to the pkgbase
install code path, and make the call from vm_extra_install_packages
conditional on !PKGBASE.
Reported by: Michael Dexter
MFC after: immediately (needed for 15.0-RC1)
We want to fetch distfiles, regardless of whether they contain known
vulnerabilities or we're building images for a different version of
FreeBSD.
Reviewed by: ivy
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53569
In order to comply with the require that GCE images must include their
source code, we fetch distfiles for all of the packages installed into
GCE images. This fails for obvious reasons for packages with an origin
of base/*; filter those out to generate the list to fetch.
Reviewed by: ivy
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53568
A bug in pkg, which somehow only surfaced as a consequence of pkgbase,
results in pkg install sometimes pulling in false dependencies. This
problem might be limited to cases when the lib32 pkgbase packages are
not installed. In the case of EC2 "small" images, installing the
ebsnvme-id package results in binutils, gcc12-devel, gmp, indexinfo,
liblz4, mpc, mpfr, and zstd packages being installed.
These false dependencies are however not recorded as dependencies --
at some level pkg does understand that they're not needed -- so running
pkg autoremove immediately after pkg install cleans them up.
Note: This does not remove lines from METALOG corresponding to these
packages, and makefs emits an error when it attempts to create the
filesystem but cannot find the files listed in METALOG -- but makefs
does seem to complete normally despite the error messages.
This change should be reverted once the pkg issue has been located and
fixed.
Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53543
This defaults to plain "pkg", but being able to override it is useful
when testing pkg itself.
Reviewed by: cperciva
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D53307
We only need to check for unMETALOGed directories and sort the METALOG
file if we're using it, i.e. if we're doing a NO_ROOT build. This
non-NO_ROOT builds by no longer bogusly writing to /METALOG*.
We only need to add databases (spwd.db etc) to METALOG if we're doing
a pkgbase-enabled NO_ROOT build; but we should always do this before
creating the filesystem, not only if we installed extra packages (in
vm_extra_install_packages, where that code was erroneously placed).
This fixes non-cloud VM images, which in 15.0-BETA2 shipped without
password databases.
Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53194
Theoretically METALOG should include everything which needs to go
into disk images; unfortunately there are still a few bugs which
are resulting in directories not being listed -- and if METALOG
has files in unrecorded directories, the directories end up being
created with 000 permissions.
Oddly enough, systems where / has 000 permissions are not very
usable.
As a temporary hack, compare the staging tree against METALOG and
add entries for any unrecorded directories. This will hopefully
be reverted before 15.0-RELEASE.
Reviewed by: bapt, emaste, ivy
Sponsored by: https://www.patreon.com/cperciva
MFC after: 5 minutes
Differential Revision: https://reviews.freebsd.org/D53153
When creating VM images from pkgbase, the METALOG may not be in order;
in particular, files may be listed before the directories which contain
them. This causes makefs to create directories with 000 permissions.
Interestingly, such VM images boot just fine, since root ignores those
permissions; the first sign of trouble was sshd refusing logins with an
error message which said absolutely nothing about /etc/ having
incorrect permissions or being unable to read files inside it.
Immediately prior to running makefs, sort the METALOG file. While
we're here, uniquify as well; this does not guarantee that we do not
have duplicate paths, but if there are duplicate paths with different
settings something else has gone wrong and we don't really have any
good way of solving the problem anyway.
Reviewed by: ivy
Hint from: imp
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53046
The file /usr/local/etc/ssl/cert.pem is not present on "small" and
"builder" AMIs, so we don't need to add it to METALOG.
Fixes: 2b0ffc0ee4 ("EC2: metalog_add missing files from packages")
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
The files
/usr/local/etc/dhclient.conf
/usr/local/etc/ssl/cert.pem
are created in EC2 images by installed packages, but are not being
registered in METALOG. The reason for this is not immediately clear,
but may be related to the handling of "sample" files.
Reviewed by: dch
MFC after: 5 minutes
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53035
Colin Percival