Don't set the config_path_pretty variable and pass it through various
functions until it ends up being used in check_option. This was just the
directory name of the kernel config and the kernel config filename (or
in case of pmbootstrap kconfig check --file, just the same as the
config path).
Instead we can just print the filename of the kernel config, for example
"config-postmarketos-qcom-sdm845.aarch64". It is shorter and already
obvious to which package it belongs.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230312151325.1968-3-ollieparanoid@postmarketos.org%3E
When looking at the table for possible output modes, it only makes sense
to pass stdin to "interactive" and "tui". The output mode "stdout" is
for non-interactive commands.
This fixes apk going interactive (asking for confirmation) when running
pmbootstrap with --details-to-stdout and building a package that depends
on postmarketos-base.
Fixes: issue 2208
Tested-by: Clayton Craft <clayton@craftyguy.net>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230301204112.4351-1-ollieparanoid@postmarketos.org%3E
pmOS cannot boot without these two. There was recently someone in the
unofficial postmarketOS Telegram group who used some minimal config and
couldn't figure out why the initramfs couldn't start up, and these
options being disabled turned out to be the culprit. As such, let's
ensure these are enabled to make life easier for people setting up new
kernel configurations.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230128203716.152281-1-newbyte@postmarketos.org%3E
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.
Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.
This speeds up multiple iterations of building the same package
significantly. I'm using it for:
$ pmbootstrap build postmarketos-mkinitfs --src=...
I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
Instead of letting the mount -t binfmt_misc … command fail, and simply
telling the user that the command failed: rely on the following check
that prints a better error message on error. If it fails at this point,
the kernel option isn't enabled. Add two comments explaining why
check=False is used in this function while at it.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230123064516.1607-1-ollieparanoid@postmarketos.org%3E
Recent changes to qemu and Alpine packaging now require using the
virtio-vga-gl device and installing -gl packages to get virglrenderer
support.
Without this, wlroots fails to get an EGL context (among other problems
you'd expect by not having a useful GPU around...)
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221210185021.3546-1-clayton@craftyguy.net%3E
Running abuild on the host directly creates directories in the
aport where it gets built. Interrupting abuild results in those
working-dirs not getting deleted.
We don't want to copy those entries to our builder.
It's only really noticeable if pmbootstrap tries to copy a broken
symlink in src/ and thus fails.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221207205201.22139-1-jane400@bingo-ev.de%3E
You can either say you want all scripts, or give a list of script names,
not both. Add it this way and not with an add_mutually_exclusive_group,
as I'll add a add_mutually_exclusive_group in the next patch to only
specify --all or --fast, but having --fast with script names is fine.
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-2-ollieparanoid@postmarketos.org%3E
Currently when any device does not conform to the options they declare,
we fail the whole kconfig check.
Now that we start requiring more options, especially with
pmb:kconfigcheck-community it makes sense to relax these restrictions so
we're more free to edit kconfig options and don't have to adjust all
testing devices that may or may not be properly maintained.
As a side effect this patch makes it practically impossible to make
kconfig check actually fail for any testing device which might not be
optimal. If these use cases appear in the future we will want to adjust
pmbootstrap to allow for that.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221105074432.13804-2-luca@z3ntu.xyz%3E
Adjust to avbtool now being part of android-tools in alpine edge.
Instead of trying to install both (which fails on edge), take the
dependencies from a new pmaports.cfg variable
supported_fastboot_depends, which only contains android-tools in
pmaports.git master branch.
Related: https://postmarketos.org/pmaports.cfg
The typical workflow for upgrading cross/gcc-* is:
$ cd pmaports/cross
$ pmbootstrap aportgen gcc-*
Currently this is failing because the APKBUILD has been updated in
Alpine for gcc, but gcc has not been built for all arches yet. This
shouldn't prevent us from generating the proper updated APKBUILDs in
pmaports so just print a note and don't fail here.
I'm committing this directly to master as this currently breaks
test/test_aportgen.py::test_aportgen.
On qemu-system-riscv64 the -nic option doesn't seem to work correctly.
qemu-system-riscv64: warning: requested NIC (anonymous, model virtio-net-pci) was not created (not supported by this machine?)
Using -netdev and -device provides the same functionality and also works
on riscv64.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221029114536.100268-1-luca@z3ntu.xyz%3E
Commit d8f2f20186 removed
the requirement of selecting MEMCG_SWAP for Kernels >= 6.1.
However, it did not fully account for the behavior change,
as MEMCG_SWAP depends on both MEMCG and SWAP.
Signed-off-by: Alexander Martinz <amartinz@shiftphones.com>
Add a new command that makes running CI scripts easy. The user goes to
the git repository of choice, which has CI scripts written in a certain
format, and then runs 'pmbootstrap ci' to get an interactive selection
of which of the available scripts to run (or "all"). Specifying one or
multiple scripts on the command-line is also possible, e.g.
$ pmbootstrap ci flake8
$ pmbootstrap ci shellcheck flake8 pytest
$ pmbootstrap ci --all
pmbootstrap then either runs the selected scripts in a chroot (and
installs dependencies as defined at the beginning of the CI scripts), or
natively (with checks inside the scripts for having dependencies
installed). Running natively is needed for .ci/pytest.sh in this
pmbootstrap.git repository, as pmbootstrap can't run inside pmbootstrap.
Running natively or in chroot is defined in an "# Options: " comment
inside the script file.
Documentation for this command and how script files look like:
https://postmarketos.org/pmb-ci
When running the testsuite, most logging gets written to a separate
log_testsuite.txt file. Check if it exists, and if so, instruct tail to
print its output as well. This allows immediatelly figuring out what the
testsuite is doing without manually attaching to log_testsuite.txt
(which I often did while running the testsuite).
At the moment the "envkernel.sh hasn't run, assuming the kernel was
cross compiled on host and using current dir as source" code path
triggers even when using envkernel.sh, which works somewhat but
requires sourcing envkernel.sh again after each invocation of
"pmbootstrap build --envkernel ...".
The reason is that os.path.ismount() does not work for bind mounts
(see https://bugs.python.org/issue29707). There is a workaround for
that already in pmbootstrap but it is not used here for some reason.
Enable pmbootstrap to work on riscv64 packages, and at the same time
also expand the alpine_to_hostspec list to match upstream (adding
riscv32 and loongarch*).
While introducing multiple pmb:kconfigcheck-* APKBUILD options we missed
adding them to this array leading to potential apkbuild-lint failures.
This hasn't been noticed much because the check in apkbuild-lint is
relatively buggy, see https://gitlab.alpinelinux.org/Leo/atools/-/issues/48
Add support for packaging a kernel that was compiled outside of
envkernel. The envkernel.sh wrapper is great for someone new to kernel
development, but it makes it difficult to do things like "make
dt_binding_check". This lets you avoid the wrapper and build on your
host machine but still use pmbootstrap to package your kernel for easier
testing.
The plan was to drop this workaround, once alpine releases a new gcc
version, and we regenerate our cross compilers against that version
_and_ drop options="!tracedeps", as latest pmbootstrap aportgen would
do. However !tracedeps wasn't dropped in the latest upgrade. This was
too easy to miss, I'll add a comment to the gcc-cross APKBUILDs so we
see it next time. Let's just update the workaround for this version.
Device packages shouldn't depend on it anymore, in order to end up with
a smaller installation for ui=none this package only gets pulled in by
postmarketos-base if mesa is installed.
Related: pmaports MR 3478
Set the env var, so python programs running inside pmbootstrap chroot
don't buffer their output and only end up printing everything when they
are done. I've seen this with meson. This is bad for usability because
we don't see output, but also a problem because pmbootstrap kills
commands if they don't print any output for some time (default: 15 min).
Install isl25 for the specific gcc versions we currently have in
pmaports master that are missing it; this is easier than building new
packages and going out of sync with Alpine's versioning. This can be
reverted as soon as there is a newer gcc.
Related: pmaports issue 1732
abuild traces shared library dependencies for binaries in built
packages, and adds all of them to depends. This is needed for the cross
gcc builds, as otherwise it will not add the isl dependency (currently
isl25).
With the crossdirect compilation method we mount the cross-gcc from the
native chroot in the foreign arch chroot. Due to the missing dependency,
isl was not installed in the native chroot, and so when attempting to
use that native cross gcc from the foreign arch chroot, it would try to
load the foreign arch isl instead of the native one. Resulting in lots
of the following errors (and of course not a working gcc):
Error relocating /usr/lib/libisl.so.23: unsupported relocation type 1026
I had probably added '!tracedeps' because abuild complains about it
otherwise... but as noted in [1] these complaints are safe to ignore.
Keep !strip for now even though it's not in Alpine's APKBUILD. I've
tried to build without and the build fails with:
strip: Unable to recognise the format of the input file `./usr/aarch64-alpine-linux-musl/lib/libgcc_s.so.1'
[1]: 0415ebf6f9
Related: pmaports issue 1732
Add the subpackage, because g++-$arch depends on it since
12.1.1_git20220630-r6. This fix was already applied to pmaports, thanks
to ungeskriptet! Adjust the aportgen code to make sure this is added in
the future as well.
Related: pmaports MR 3464
Related: f1044b1dae
Do not build the foreign arch libgcc. We are using the one provided by
Alpine's binary repository instead, and when building it here and trying
to use it, we actually run into a conflict.
I tried setting BOOTSTRAP="nolibc" first since there is already logic to
then disable libgcc in the APKBUILD. But that also disables g++ which we
do want to build, so we can't use this.
Fixes: issue 2168
Remove it from depends, as the package has been removed in Alpine and
since quite some time we shouldn't have depended on it anyway.
Related: issue 2167
Add the new option that will be mandatory for all devices in
community/main category. This is just a combination of anbox + iwd +
nftables + containers + zram + netboot.
While the existing options could be removed we're keeping it for now
given that also some devices with downstream kernel might find some
options useful.
Do not only parse triggers, but also emit them in the packages dicts in
pmb.parse.apkbuild. This is the first half to fix pmaports issue 1690,
the other is in pmaports MR 3386.
Set CTARGET and CTARGET_ARCH on top of the generated APKBUILD and use
the existing code in the APKBUILD for build(), package(), makedepends,
subpackages. This is now possible, because the pmbootstrap parser for
APKBUILDs has been improved a lot since this was originally written and
because the Alpine APKBUILD does everything that was hacked in here,
like removing man, info directories and bfd-plugins if doing a cross
build.
This is now very similar to bootstrapping binutils for another
architecture via scripts/bootstrap.sh in aports.git.
Most importantly this sets --disable-gprofng, which fails to build
against musl and is the reason why our cross/binutils-* packages
generated from binutils 2.39.x didn't build anymore.
Related: https://git.alpinelinux.org/aports/tree/main/binutils/APKBUILD
Related: https://sourceware.org/bugzilla/show_bug.cgi?id=29477
Even add_cmd has exited with code 99, it will be overwritten by clean_cmd
later. Exit with code returned by add_cmd to raise a runtime error when
an error occurs.
Don't pass stdin to commands that aren't supposed to be used
interactively (output: log, background, pipe).
This fixes an inconsistency between building packages in CI on gitlab
and building them via bpo on sourcehut or locally. In gitlab, apparently
there is no stdin for the entire build job and so unanswered kernel
config prompts will just use the default. In local builds and on
sourcehut stdin is available and so it just hangs at the prompt until
pmbootstrap kills the build job due to no output being written.
I considered adding an additional check to pmaports to ensure that there
are no unanswered kernel config prompts just in case users run abuild
manually on the kernel APKBUILD with stdin available. But I think
forcing the users to answer all the prompts even if it's not really
needed just creates additional work / makes the workflow worse without
real benefit.
Related: https://builds.sr.ht/~postmarketos/job/824373#task-pmbootstrap_build-432
Fixes: pmaports issue 1225
Here's an example of the properties this could generate:
deviceinfo_header_version="2"
deviceinfo_append_dtb="false"
deviceinfo_flash_offset_dtb="0x0bc08000"
This includes setting header_version="2" as well as dtb_offset
according to the input boot.img when header v2 is detected.
Also adds the following previously missed deviceinfo_attributes:
* "header_version"
* "bootimg_custom_args"
And fixes failing tests now that header_version is always parsed from
boot.img files.
In case a user removes all contents of the work path then pmbootstrap
init will fail and show
WARNING: Your work folder version needs to be migrated (from version 0 to 6)!
Later the migration would fail with the error
ERROR: We have split the aports repository from the pmbootstrap repository (#383). Please run 'pmbootstrap init' again to clone it.
This is due to the existing check not accounting for e.g. log.txt being
written in the work path before we get to this check. Now change it so
we always create the version file if it doesn't exist yet.
Test plan:
$ grep work ~/.config/pmbootstrap.cfg
work = /tmp/pmbootstrap-work
$ rm -rf /tmp/pmbootstrap-work
$ mkdir /tmp/pmbootstrap-work
$ pmbootstrap init
This was previously attempted to be fixed in !1975
Since pmaports@50ee94d8 the kernel filename won't contain the flavor
information, but this is not correctly synchronized with the Odin
export. This fixes Odin export.
When using a kernel from Alpine the kernel configuration cannot be
found in pmaports. We cannot check the kernel config for missing
options in that case, but that's no reason to break the flasher
entirely.
Check if the pkgnames are sane in install_run_apk, right before running
apk. This makes sure that we really run it on all arguments that are
supposed to be packages / files and not options to apk.
Previously pmbootstrap would only show the packages that are about to be
installed. In case all packages were already installed, this would lead
to weird empty install messages:
(rootfs_asus-me176c) install
Show all packages that we want to install, even if they are already
installed in the given chroot.
(rootfs_asus-me176c) install device-asus-me176c
Previously to this patch, pmbootstrap would pass a full dependency tree
to "apk add". It would use a virtual package to ensure only the right
packages get added to /etc/apk/world. For example:
apk add -u --virtual .pmbootstrap postmarketos-base device-asus-me176c \
postmarketos-ui-sxmo-de-sway device-asus-me176c-nonfree-firmware w3m \
sfeed clickclack firefox-esr font-noto font-noto-emoji gnome-icon-theme \
imv megapixels mobile-config-firefox ttyescape postmarketos-base-nofde \
eudev openssh postmarketos-mkinitfs postmarketos-mvcfg postmarketos-keys \
...
Instead of doing that, only pass the packages we want to install and let
apk figure out the dependencies. Most of the time we can even avoid
using the virtual package now.
== Remaining edge case: locally built packages
apk will only upgrade a package with the same pkgver + pkgrel but a
different build date if the full path to an apk file gets passed as
argument. So if the user built a package locally that will be installed,
or one of its dependencies then we still need to use a virtual package
and possibly pass a dependency to apk. Replace
replace_aports_packages_with_path() with packages_get_locally_built_apks()
to get a list of such packages and adjust install() and
install_run_apk() to use it.
Make the code easier to read by moving split_to_add_del() to a separate
function and do some related refactoring. A future patch will use it
twice in install().
Move "arch = ..." to the top of the function while at it, since it's
needed later in the function in 2 places and is not needed for figuring
out packages_with_depends, to_add and to_del.
Remove "# Add depends to packages" because it's obvious from the
packages_with_depends variable name, and getting to_add/to_del is a
different action that stood under the same comment.
Split out the part that builds the apk commands and runs them out from
install() to a new function install_run_apk(). This makes install()
easier to read.
Since the previous commit that adds install_build(), all packages either
end up in to_add or to_del. Move the check for empty packages to the top
of the function, and directly check the packages variable.
I think it's worth keeping this check because it's shorter to add this
check once here than having it a few times in other place where we may
have or may not have something to install. And so we can avoid printing
an empty "install" message with no packages.
install_is_necessary used to do the following things:
1. Error out if there's no binary package but pmb was invoked as
"pmbootstrap install" and build_pkgs_on_install is disabled.
2. Build the package if necessary.
3. Return if a package "needs to be installed" (Boolean or Float).
The only caller of the function is pmb.chroot.apk.install. It would not
add the package to the long "apk add" command if according to 3. it does
not need to be installed.
When I implemented this a few years ago, I probably thought it would be
useful to not unnecessarily pass packages to apk. But this actually
makes it more complicated and doesn't have a benefit, apk is perfectly
capable of recognizing which packages it had already installed.
Replace the function with a much simpler pmb.chroot.apk.install_build,
which only does 1. and 2. Change the order of the package, arch
arguments to match called functions pmb.parse.apkindex.package and
pmb.build.package.
Put underscores between the words for consistency with other variable
names. Since the whole function is about dealing with packages, remove
the "packages_" prefix to make them shorter.
packages_toadd -> to_add
packages_todel -> to_del