pmbootstrap netboot command exposes the generated vendor-codename.img
rootfs through nbd interface so that device can mount it and boot
postmarketOS without having any storage medium at all.
Co-authored-by: Luca Weiss <luca@z3ntu.xyz>
When installing dependencies for a package, conflicting (!) dependencies
are now deleted (with `apk del pkg`) whereas before a constraint for
their _absence_ was added (with `apk add !pkg`). Doing it the new way
around prevents creating deadlocks because a `!pkg` constraint will
prevent pkg from ever being installed without an explicit `apk del`
call.
Fixes: #2092
Replace "args.cache" with a global variable in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Replace "args.arch_native" with the direct function call in order to
avoid passing "args" to all functions. This is a step to get rid of this
args-passed-to-all-functions pattern in pmbootstrap.
Follow-up to 09794ef832 -
initfs file does not have flavor now, too.
Without this 'pmbootstrap initfs extract' fails with:
(rootfs_samsung-klte) % cp /boot/initramfs-postmarketos-qcom-msm8974 /tmp/initfs-extracted/_initfs.gz
cp: cannot stat '/boot/initramfs-postmarketos-qcom-msm8974': No such file or directory
Fix the issue of having the postmarketOS binary repository key deleted
from the chroots after "abuild undeps" removes the postmarketos-keys
package. This happens if e.g. building two device packages in a row (as
they depend on postmarketos-base, which depends on postmarketos-keys in
current pmaports.git master).
I've also considered installing the postmarketos-keys next to
alpine-base in new chroots. But this would introduce a bootstrap
problem, since you can't install the postmarketos-keys package unless
it already exists in the repository. We'd run into that when building
the next release.
Move code to install Alpine and postmarketOS keys into its own function,
so it can be called outside of pmb.chroot.init too (next patch).
Describe why this is needed in the first place and, while at it, tweak
the function to only copy the key if it does not exist in the target
directory.
kernel is named /boot/vmlinuz now, looking at the filename will no
longer tell us what flavor it is. This now will look at
/usr/share/kernel, which has always contained the kernel 'flavor', and
since we currently only install 1 kernel these days, guarding this with
pmaports.cfg should be unnecessary. In the worst case (if there are
multiple kernel 'flavors' installed), it'll just grab the first one and
return it.
The new mkinitfs does not have options for kernel flavor or version
A new pmaports.cfg variable, "supported_mkinitfs_without_flavors" is used to
determine how to invoke mkinitfs
gitlab runners seem to be registering binfmt now (?), so this fixes an
issue where register() might return immediately because the OS (or ??)
took care of registration, but the qemu-<arch> binary wasn't added to
the chroot.
It seems like the gitlab runners will now automatically register archs
when binfmt is mounted (I'm not completely sure, it's really
hard/annoying to confirm using gitlab CI jobs and no direct access to
the runner)
In any case, checking if it's already registered fixes a problem where
CI fails with "File exists" when it tries to register the arch.
Remove /in-pmbootstrap inside chroots in "pmbootstrap shutdown" instead
of having it at a specific part of "pmbootstrap install".
Reasoning:
* With current approach, it didn't get removed in the on-device
installer chroot.
* This is less error prone than calling it multiple times in
"pmbootstrap install"
Touch the file /in-pmbootstrap in chroots so that we can avoid
performing automated actions that should only happen on a real device
(like flashing the kernel).
Support branches, so pmbootstrap won't fail if v20.05 is selected:
ERROR: You have an outdated version of the 'apk' package manager installed
(your version: 2.10.5-r1, expected at least: 2.12.1-r0).
Move the logic for this check to pmb.helpers.apk.check_outdated and
adjust the test.
This fixes the CI failure in test_crossdirect_rust, which uses the
stable channel. (My bad for not creating this patch earlier, while at
the same time explaining in the creating pmbootstrap release instructions,
that this minimum apk version should be adjusted.)
Remove "_static" from the variable name, as this version isn't just
used to compare apk-tools-static's version (used to set up chroot), but
also for regular apk-tools before entering chroots.
Replace the "kill_as_root" argument with a much simpler "sudo" argument
and remove the now obsolete check for the output mode of "kill_as_root".
"kill_as_root" would only get set to True if both conditions are met:
a) command is running with sudo
b) command is running with an output mode ("log" or "stdout") where
pmb.helpers.run_core would kill it if it does not output anything
before a timeout is reached
The new "sudo" argument just indicates if the command is running with
sudo (a), regardless of the output mode (b).
Do not attempt to upgrade packages in the rootfs chroot when running
"pmbootstrap install".
This was responsible for placing every single package in /etc/apk/world
(which should only hold the packages explicitly installed), because the
upgrade function was literally implemented as getting a list of
installed packages and explicitly running pmb.chroot.apk.install on each
of them. The intention was to rebuild these packages if they were outdated,
I guess I didn't realize that this makes /etc/apk/world unusable when I
introduced this three years ago in 51bdc243 ("Properly rebuild/install
packages when something changed").
Remove pmb.chroot.apk.upgrade altogether, because:
1) pmb.install.install builds and upgrades outdated pmaports
2) pmb.install.install is the only user of pmb.chroot.apk.upgrade
3) 'pmbootstrap init' is warning that the chroots do not get upgraded
automatically, so let's not go against that expectation. users who
want an updated rootfs chroot can simply run zap and install again.
Replace it with a call to pmb.helpers.repo.update, because we still need
to update the APKINDEX files before attempting to build/install the
generated list of packages.
Some Mediatek devices have a special 512-byte header around the zImage
which must be generated so the device boots.
Support for that exists for a while in postmarketOS but detection was
missing. Add that.
Add a question at the end of "pmbootstrap init", to ask if the user
wants to build outdated packages during "pmbootstrap install". Store the
result in the new pmbootstrap.cfg key "build_pkgs_on_install". I've put it at
the end, because it is a rather complicated question compared to the rest.
This is useful to speed up the installation for casual users who can now
avoid compiling packages. But also for the official images where we only
want to ship the official binary packages and not build anything
on-the-fly.
Add initial support for the on-device installer in pmbootstrap. Let
pmbootstrap create a regular split image, then prepare a new installer
rootfs and copy the previously generated rootfs image into the installer
rootfs. Put the installer rootfs into a new image, with reserved space.
There is more to do from here, such as disabling the generation of the
user account when using --ondev. But this requires support in
postmarketos-ondev first, so let's build that iteratively.
Related: https://wiki.postmarketos.org/wiki/On-device_installer
Related: https://gitlab.com/postmarketOS/postmarketos-ondev/-/issues
Migrate to workdir version 5 and move already built packages into the edge
channel subdir, for example:
$WORK/packages/x86_64/hello-world-1-r5.apk
to:
$WORK/packages/edge/x86_64/hello-world-1-r5.apk
The build.postmarketos.org code has already been adjusted to find built
packages in either directory structure.
Do not go through the pmb.chroot.init() code path when running
pmb.install.losetup.umount() inside pmb.chroot.shutdown(). This is not
necessary, as pmb.install.losetup.umount() only gets called if the
chroot is already initialized and /dev/loop-control is mounted inside
the chroot.
Not going through this code path is important for the upcoming workdir
migration patch. Without this fix, it will fail with the following if
running "pmbootstrap install" before the work migration:
ERROR: Could not figure out on which release channel the 'native' chroot is
Whenever initializing new chroots, save the date in $WORK/workdir.cfg.
Add pmb.config.workdir.chroots_outdated() to check if it's time to zap
the chroots or not (since we don't update them automatically). Mark them
as outdated after two days.
This will be the first check in "pmbootstrap status" (future patches).
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
Remove in favor of similar, but more visible "--no-install" option for
"pmbootstrap flasher" and "pmbootstrap export".
It seems that the option wasn't really used by anybody, so let's just
remove it without a deprication period. If dear reader thinks otherwise
after this is merged, and is not happy with "--no-install", please state
your reasoning in a new issue, so we can discuss it.
Related: !1863
Rust packaging is new and still a bit weird in Alpine and postmarketOS.
As of writing, we only have one package (squeekboard), and use cargo to
download the source of all dependencies at build time (several git
repositories!) and compile it. Usually, this is a no-go, but at least
until this is resolved properly, let's cache the downloads as suggested
in: https://doc.rust-lang.org/cargo/guide/cargo-home.html
Related: #1861
When initializing the building chroots, some symlinks are created to
/mnt/pmbootstrap* dirs. With upcoming rust-related symlinks, they will
point to subdirs in /mnt/pmbootstrap-rust, such as
/mnt/pmbootstrap-rust/registry/cache. Create this directory structure if
it does not exist.
Because of an APKINDEX subpkg parsing bug, pmbootstrap currently assumes
that "ktp-common-internals" provides "make". It fails with a cryptic
error when trying to build a package without an existing postmarketOS
binary packages mirror (like done in the build.postmarketos.org
testsuite):
File "/home/user/code/pmbootstrap/pmb/chroot/apk.py", line 174, in replace_aports_packages_with_path
package + "-" + data_repo["version"] + ".apk")
TypeError: 'NoneType' object is not subscriptable
Replace this error with a more meaningful message, that points to the
issue analyzing this bug.
Related: https://gitlab.com/postmarketOS/build.postmarketos.org/issues/61
Fix configure crash when building mozjs60 by mounting tmpfs as /dev/shm
in chroots. This is an important fix, because of this dependency chain,
that is currently broken in armhf and armv7 (see pmaports#244):
postmarketos-base -> networkmanager -> polkit -> mozjs60
Launch native cross compilers inside foreign chroot. Enable by default,
but allow disabling with --no-crossdirect for now. This option and the
distcc-sshd related code will be removed in the future.
Obscure feature: it was possible to specify a local path as
--mirror-pmOS. It would then get mounted to /mnt/postmarketos-mirror
inside the chroot, and be specified as such in the generated
/etc/apk/repositories file.
I had used this once for some testing scripts, but I am sure nobody is
using this anymore. The same can be achieved with running a local http
server anyway:
<https://wiki.postmarketos.org/wiki/Installing_packages_on_a_running_phone>
Removing this makes it easier to support multiple postmarketOS mirrors
(next commit).
Mark Hargreaves
Johannes Marbach
Oliver Smith
Anjandev Momi
BO41
Alexey Min
Newbyte
Clayton Craft
Maxim Karasev
Caleb Connolly
Anri Dellal
Luca Weiss
Daniele Debernardi
Minecrell
Oliver Smith
Zhuowei Zhang