Once enabled on all of the branches, this will reduce bandwidth
consumption from uploading weekly snapshot builds from ~500 GB to
~100 GB, as well as significantly speeding up the process.
MFC after: 2 months
Sponsored by: Amazon
Starting with version 1.5.0, bsdec2-image-upload supports uploading
stream-optimized VMDK files. If we build images in that format, we
need to upload them appropriately.
MFC after: 2 weeks
Sponsored by: Amazon
When creating VM images, we filter the METALOG file created by pkg(8)
when installing non-base packages, rejecting any lines which correspond
to files which don't exist; this solves a problem which arose when a
package was installed and then deinstalled (or upgraded) later in the
image-building process.
Unfortunately [ -e ... ] follows symlinks and is not basedir-aware, so
an absolute symlink which is valid *inside* the image is omitted from
the image if it points to something which isn't present in the build
host system.
Replace [ -e ... ] with [ -e ... ] || [ -L ... ] so that symlinks are
included even if dangling.
While I'm here, add quoting in case future paths become problematic.
Sponsored by: Amazon
MFC after: 3 days
We don't have any packages to install by default when the instance
first boots, so (as with the "small" flavour) don't enable the
firstboot_pkgs script. If someone wants to launch a desktop with
packages autoinstalled at first boot, they can enable the script at
the same time as they provide the list of packages.
Sponsored by: Amazon
Using a backtick inside a backtick-expansion isn't a good idea; switch
instead to using $( ... ).
Fixes: a971034951 ("EC2: Add desktop flavour")
Sponsored by: Amazon
Desktop AMIs have xrdp enabled and boot to a KDE desktop; they are
as compatible as possible with EC2 Windows AMIs, setting a random
password and printing it to the console in encrypted format to be
retrieved using the EC2 GetPasswordData API.
Two rc.d scripts are included in this commit which will not exist
in the long term: ec2_addpass will become part of the ec2-scripts
package, and ec2_desktop_extras will go away once its functionality
is included elsewhere.
MFC After: 1 month
Relnotes: yes
Sponsored by: Amazon
By default, makefs uses the host environment's user and group databases
when creating filesystems. This causes makefs to fail when trying to
create files owned by users or groups which don't exist in the host
environment, for example when creating a VM with packages pre-installed
which added their own users/groups.
Pass "-N ${DESTDIR}/etc" to makefs to point it at the user and group
databases from the image being created.
MFC after: 1 week
Sponsored by: Amazon
In 16.0/15.1, the PAM modules were split from FreeBSD-runtime into a
new FreeBSD-pam package. FreeBSD-runtime does not install FreeBSD-pam,
which means if a user starts from runtime, then installs sshd, sshd will
fail to authenticate users because of missing PAM modules.
Since FreeBSD-pam is relatively small (about 230kB on amd64), and is
already part of FreeBSD-set-minimal, add it to the runtime image as
well. Users who absolutely don't want this can still build their
own images without it.
MFC after: 1 week
Reviewed by: dfr
Reported by: Michael Johnson <[email protected]>
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D59194
This mirrors the changes applied to kernel debug symbol sets in commit
9a354a41be.
Reviewed by: emaste
Sponsored by: AFRL, DARPA
Differential Revision: https://reviews.freebsd.org/D59058
Enable the new growfs_postboot mechanism. Note that this also implies
disabling automatic allocation of swap space on the root disk, since we
cannot grow the root filesystem if swap space is allocated after it.
This will not be MFCed since it is a significant behavioural change.
Sponsored by: Amazon
Relnotes: yes
Increase EFI partition size to begin rootfs at 64mb. I believe this was
my original intention. I have a microSD card with 8mb block size which
emits an advisory in verbose dmesg about the misaligned partition.
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Google Cloud recommends migrating from gsutil to gcloud storage CLI.
Update gce-do-upload target to use `gcloud storage buckets create` and
`gcloud storage cp` instead of `gsutil mb` and `gsutil cp` commands.
PR: conf/297016
Reviewed by: lwhsu
MFC after: 3 days
Differential Revision: https://reviews.freebsd.org/D58464
This was a good idea, but we don't build metapackages in the kmods
repo so it ends up breaking the release build. I might resurrect this
change if/when the kmods repo includes the wifi-firmware-kmod
metapackage.
This reverts commit bda8028146.
MFC after: 1 second
We used to pass CONFIGURE_ARGS to the make command which builds pkg,
but ports/ports-mgmt/pkg/Makefile has its own CONFIGURE_ARGS and the
version we were providing at the command line didn't contain the
--mandir setting which was added to the port with pkg 2.8.0. This
broke release builds.
Instead of passing --prefix=${LOCALBASE} via CONFIGURE_ARGS, pass
PREFIX=${LOCALBASE}; the port Makefile passes that value through to
its configure script. We also used to pass a --host parameter, but
that seems to have become unnecessary at some point in the past decade.
MFC after: 1 day
Sponsored by: Amazon
In some cases having a src tree in a VM image is convenient
for development or debugging. Add a WITH_SRC variable,
which, when set, will cause the vm-release target to include
FreeBSD-set-src in the list of packages installed in an image.
Signed-off-by: Krzysztof Galazka <[email protected]>
Sponsored by: Intel Corporation
Reviewed by: cperciva
MFC after: 2 weeks
Differential Revision: https://reviews.freebsd.org/D57143
Update the GCE image creation process to automatically apply the
'public-image=true' label when publishing new images. This aligns
with standard labeling expectations for images hosted in public
projects.
MFC after: 3 days
The EC2 "base" flavour installs the devel/py-awscli package at boot
time by default; we don't do this in the "small" flavour, so the
default behaviour was to update the FreeBSD-ports repository and
then do nothing with it.
Turn off firstboot_pkgs by default; if someone is using the "small"
flavour of AMIs and wants to install packages at instance launch
time, they simply need to add 'firstboot_pkgs_enable="YES"' to
/etc/rc.conf (which they must already be editing via user-data, in
order to provide the list of packages they want installed).
Sponsored by: Amazon
MFC after: 3 days
MFC to: stable/15
Relnotes: EC2 "small" images now have firstboot_pkgs_enable="NO".
Update a couple more cloudware images which I forgot about earlier.
Reviewed by: ziaee
Fixes: 464a351267 ("Cloud releases: Switch to firstboot_pkg_upgrade")
Differential Revision: https://reviews.freebsd.org/D57006
Cloud images are deployed with base system packages. Introduce a
firstboot package auto updater to patch the base system on first boot.
MFC after: 1 hour
MFC to: stable/15
Reviewed by: cperciva
Sponsored by: Google Cloud
Differential Revision: https://reviews.freebsd.org/D56890
Commit d1c176fedf made create-sets.sh exit when it encounters an
error, instead of creating an empty repository. However, this turns
out to cause some issues:
1. A package not having any sets is considered an error, but during
the release build, we stuff a 'pkg' package into the repository
which doesn't have any sets, which causes a failure. Avoid this
by simply ignoring the pkg package.
2. No error was printed in this case, which made the problem hard
to diagnose. Add an explicit error message.
3. A similar problem occurred running on a repository which already
contained sets, which is not usually done during the build, but
is not necessarly an inappropriate thing to do. Fix this one by
ignoring set packages when looking for sets.
While here, fix another issue that might cause packages to be wrongly
skipped if the path to the repository contains a '-' character, since
we didn't strip the path before testing the package name.
PR: 294966
Fixes: d1c176fedf ("packages: Make create-sets.sh more robust")
MFC after: 2 weeks
Reported by: Alastair Hogge <[email protected]>
Reviewed by: emaste
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D56792
Use ${PKG_CMD} rather than bare 'pkg' to fix the build when pkg is
not in the tools path. Provide a default in case it's not set for
some reason (e.g., running the script by hand).
Since set -- $(...) does not trigger an exit from set -e if the
command fails, this failure was silent and resulted in sets not
being built correctly if we failed to run pkg. Use a temporary
variable, which does trigger set -e, to fail correctly.
MFC after: 2 weeks
Reviewed by: sjg
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D56676
This makes it less likely we will silently generate broken artifacts.
Reviewed by: ivy
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D56671
Invoke ln with -n and -f. In normal use it doesn't matter, but during
development this might be run in a partially populated leftover tree.
Reviewed by: ivy
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D52883
This should fix the creation of the new packages introduced by the
import of pkgconf into the base system.
Fixes: b8352da33f ("pkgconf: import into the base system")
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D56589
Instead of driving the world package build from Makefile.inc1,
use a subdir build where each package has a subdirectory under
packages/ using the new <bsd.pkg.mk>.
Convert some metadata that was previously in the UCL files (e.g.
sets and dependencies) to Makefile variables.
Build the packages under objdir (not repodir), and use the new
stagepackages target to copy them to repodir when creating the
repository.
Determine an explicit list of packages to build in packages/Makefile
based on enabled src.conf options, and add logic to abort the build
if we attempt to build an empty package. This inverts the previous
logic in Makefile.inc1 which would simply skip empty packages.
There are a few advantages to doing it this way:
* The package build works more like the rest of the build system,
so it's more accessible to developers.
* We can customise the packages we build based on src.conf options,
e.g. skipping a package entirely, or adjusting its dependencies
based on what it actually requires.
* We have a specific list of packages that we want to build, and an
unexpectedly missing package results in a build error, instead of
silently producing a broken repository.
* It's possible to build (and in the future, install) an individual
package without having to rebuild the entire repository.
This doesn't apply to the dtb, kernel-* or src-* packages; those
have their own build systems in Makefile.inc1 and will be converted
later.
MFC after: 4 weeks (stable/15 only)
Reviewed by: jlduran, sjg, brooks
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D56087
This should fix the creation of the new packages introduced by the
import of pkgconf into the base system.
Fixes: b8352da33f ("pkgconf: import into the base system")
PR: 294724
Reviewed by: ivy, ngie
Approved by: ngie
Sponsored by: The FreeBSD Foundation
Differential revision: https://reviews.freebsd.org/D56589
The FreeBSD-clang package contains a 32-bit shared object at
/usr/lib/clang/19/lib/freebsd/libclang_rt.asan-i386.so
This is expected, since clang uses this object when compiling for i386
targets with asan enabled.
What is not expected is that the FreeBSD-clang package currently depends
on 32-bit libc packages due to pkg's shared library analysis, making it
impossible to install pkgbase on x86_64 without any lib32 packages.
This commit leverages a new pkg feature implemented in [1], but could
be landed before a pkg version including that feature is released
without any ill effects. Unknown keys in package manifests are ignored.
[1]: https://github.com/freebsd/pkg/pull/2594
Reviewed by: ivy
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D54792
Oracle's previous support is no longer available to the project.
Repeated attempts to find a sponsor within Oracle's cloud business
have not been successful.
The last published official images are from 15.0-RELEASE.
https://marketplace.oracle.com/app/freebsd-release
Relnotes: yes
Sponsored by: SkunkWerks, GmbH
Differential Revision: https://reviews.freebsd.org/D56360
MFC after: 3 days
To avoid breaking the build due to incompatible changes in flua or lua
libraries on the host, use the bootstrap flua for the package build.
MFC after: 2 weeks
Reviewed by: kevans, emaste
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D56271
In libucl 0.9.3, macros and includes are disabled by default when
creating a new UCL parser. This breaks the package build, which
relies on includes. Fix this by explicitly passing zero flags
to ucl.parser().
MFC after: 3 days
Fixes: abda442d92 ("contrib/libucl: Import libucl 0.9.3")
Reviewed by: kevans, bapt
Reported by: [email protected]
Sponsored by: https://www.patreon.com/bsdivy
Differential Revision: https://reviews.freebsd.org/D56266
These were modified to avoid triggering a libucl bug which is now fixed.
MFC after: 1 week
Reviewed by: ivy, kevans
Differential Revision: https://reviews.freebsd.org/D52824
/etc/rc.d/var_run uses mtree, which is in the devel set and isn't
installed as part of either minimal or optional, so add a manual
dependency.
Reviewed by: des, bapt, emaste
Differential Revision: https://reviews.freebsd.org/D54540
Sponsored by: https://www.patreon.com/bsdivy
pkg(8), via its daily periodic script, requires xz. We don't have
a way to encode dependencies from ports on base packages right now,
so instead move xz to the minimal set so it's always installed.
This isn't an ideal solution, but it's justified in this case since
pkg(8) is always installed, so having its dependencies always
installed is acceptable.
(Following discusson on the diff, new versions of pkg have now been
changed to use zstd instead of xz, but we still think xz is useful
enough to keep in minimal.)
MFC after: 1 week (stable/15 only)
Reviewed by: des, bapt, emaste
Differential Revision: https://reviews.freebsd.org/D55630
Sponsored by: https://www.patreon.com/bsdivy
The chroot_cleanup routine handles any cleanup needed post-chroot_setup,
etc. This consists of purely tearing down `${CHROOTDIR}/dev` today, but
might involve additional steps, as needed for custom functions. This
allows end-users to override the various chroot functions without having
to modify code in main() or replicate the unmount procedure in an
equivalent routine setup via the trap builtin.
This change modifies the /dev unmount process to use `umount -f` instead
of `umount`. The latter can result in failures if resources are still
mounted or are running post-build, whereas the former will clean up any
resources still in use by processes running in the chroot at time of
build. Moreover, the `chroot_cleanup` routine is now called when the
script is killed with `SIGINT` and `SIGTERM`, as well as at `EXIT`,
better ensuring that the script's resources are cleaned up in relatively common
scenarios that can be detected/handled.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D55450 (as part of a larger change)
This helps others attempting to customize the behavior of `release.sh`.
MFC after: 1 week
Differential Revision: https://reviews.freebsd.org/D55450 (as part of a larger change)
This reverts commit 6a13aeac3c.
The "bogus dependencies being installed" issue was traced down to pkg
confusion surrounding libclang_rt.asan-i386.so linking to a 32-bit
libgcc_s.so.1, and a more minimal workaround of "install clibs-lib32"
has been applied, so this hack is no longer required.
MFC after: 3 days
Sponsored by: Amazon
The FreeBSD-clang package on amd64 contains libclang_rt.asan-i386.so,
which links to the 32-bit version of libgcc_s.so.1. It is not clear
if that file belongs in FreeBSD-clang or if it should be placed into
a different package.
For unknown reasons, pkg *sometimes* recognizes this and decides that
it needs to install a package to supply libgcc_s.so.1:32.
In particular, when we initially install the VM with 'pkg install
[long list of FreeBSD-* packages]', pkg does not recognize that it
wants this; but the *next* time 'pkg install' runs, it decides that
it needs libgcc_s.so.1:32 -- even though that particular library is
not needed by the particular package we're trying to install -- and
goes looking for it... and ends up picking gcc12-devel as a provider.
Later in the EC2 image building process, we run 'pkg autoremove' (which
was added during 15.0 to get rid of the "bogus dependencies" which were
added while installing other packages) and pkg is once again confused:
It decides that gcc12-devel is an automatic port which is not required
by anything else, but upon determining that it should be removed it
then determines that FreeBSD-clang depends upon a port which is being
removed and deletes FreeBSD-clang as well.
It's not clear what is breaking in pkg, but as a temporary workaround
add the FreeBSD-clibs-lib32 package to the "small" and "builder" EC2
image flavours in order to satisfy the libgcc_s.so.1 demand.
This commit should be reverted if libclang_rt.asan-i386.so moves out
of FreeBSD-clang, (i.e. if it becomes possible to install a 32-bit-free
amd64 system) or when the pkg issue is resolved (i.e. pkg consistently
recognizes the shared library dependency and installs clibs-lib32
automatically as a FreeBSD-clang dependency).
MFC after: 3 days
Sponsored by: Amazon
Differential Revision: https://reviews.freebsd.org/D55978
The boot loader menu is disabled by default in EC2, but if it is ever
turned on, the default (unicode) output breaks EC2's web interface to
the serial console.
Set loader_menu_frame="ascii" instead.
MFC after: 3 days
Sponsored by: Amazon
This matches non-pkgbase behavior as installworld/installkernel skip the
debug bits if these knobs are set.
MFC after: 1 week
Sponsored by: Chelsio Communications
Reviewed by: cperciva
Differential Revision: https://reviews.freebsd.org/D55572
We added the option to build BSD diff3 in commit 2201f7c49f ("Build
BSD diff3 if GNU diff3 is disabled.") and made it the default in
4d5c434ed1 ("diff3: use bsd diff3 by default") after resolving
deficiencies relative to GNU diff3. Thus, we can now remove the GNU
diff3 build infrastructure and source.
Reviewed by: bapt, ziaee
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D46878
Having these packages available on release media may help users who
need to sneakernet other packages (e.g. firmware) from systems running
Linux or Windows.
Suggested by: vladlen, ziaee
MFC after: 3 days
Running `pkg -d` in pkg-stage.sh results in multiple GB of network
traffic being written into the log files, which is less than helpful
when it comes to tracking down build failures. Remove the -d flag.
MFC after: 5 days
X-MFC-note: The code in 15 has diverged from 14, but the flag is
there, just in a different place.
Colin Percival