This is needed for sphinx autoprogram since that expects an
argparse.ArgumentParser, and arguments() returns some argparse
"Namespace" obj. Useful for sphinx/autoprogram and maybe other things
later that want to get at pmb's full args.
A common pattern in APKBUILDs, is to introduce custom variables prefixed
with underscores that get then used in makedepends and other variables.
For example:
_wlrootsmakedepends="
eudev-dev
# ...
"
makedepends="
# ...
$_wlrootsmakedepends
"
Adjust the APKBUILD parser code, so it parses all top-level variables
and can use them further below when referenced inside other variables.
Before returning the parsed APKBUILD data, remove all variables that are
not in pmbootstrap's list of known APKBUILD parsing attributes (so the
result is the same).
I've compared "pmbootstrap apkbuild_parse" (which parses all APKBUILDs
in the currently checked out pmaports dir), before and after this
change, and the result is the same except for having more variables
successfully replaced.
- Performance Note-
This new implementation is actually faster than the previous one,
because we don't need to iterate through all known keys on each line of
the APKBUILDs. On my machine, average of 3 runs, parsing all APKBUILDs
of current pmaports master takes about half as long as with the previous
implementation.
$ time pmbootstrap -q apkbuild_parse >/dev/null
-> old code: 0.954
-> new code: 0.483
Make the handling of the custom NonBugError and BuildFailedError
exceptions more consistent with the handling of other exceptions, by
printing "ERROR: " infront of the actual error text. Then we don't need
to duplicate that where we raise the errors. pmbootstrap prints "ERROR"
in red.
Reimplement "pmbootstrap status" to be just a simple and useful status
overview. The previous version ran a bunch of checks every time, and
would fail on these even if pmaports was used for normal development:
* "non-official" branch checked out in pmaports
* pmaports.git is not clean
The information about aports.git was also considered not so useful upon
revisiting this command, since it is only used for "pmbootstrap
aportgen". Most users don't need this, and if the user runs this
command, it will tell if aports.git is outdated.
All of the above made the previous version unpleasant to use and I
suspect most people stopped using the command after trying it out a few
times and seeing the irrelevant but loud NOK complaints.
New version:
$ pmbootstrap status
Channel: edge (pmaports: master_staging_systemd)
Device: qemu-amd64 (x86_64, kernel: virt)
UI: console
systemd: no (default for selected UI)
Old version (without --details it only shows NOK checks):
$ pmbootstrap status --details
[00:55:20] *** CONFIG ***
[00:55:20] Device: qemu-amd64 (x86_64, "QEMU amd64")
[00:55:20] Kernel: virt
[00:55:20] User Interface: console
[00:55:20]
[00:55:20] *** GIT REPOS ***
[00:55:20] Path: /home/user/.local/var/pmbootstrap/cache_git
[00:55:20] - aports_upstream (master)
[00:55:20] - pmaports (master)
[00:55:20]
[00:55:20] *** CHECKS ***
[00:55:20] [OK ] Chroots zapped recently (or non-existing)
[00:55:20] [OK ] aports_upstream: on official channel branch
[00:55:20] [OK ] aports_upstream: workdir is clean
[00:55:20] [OK ] aports_upstream: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] aports_upstream: up to date with remote branch
[00:55:20] [OK ] aports_upstream: remote information updated recently (via git fetch/pull)
[00:55:20] [OK ] pmaports: on official channel branch
[00:55:20] [OK ] pmaports: workdir is clean
[00:55:20] [OK ] pmaports: tracking proper remote branch 'origin/master'
[00:55:20] [OK ] pmaports: up to date with remote branch
[00:55:20] [OK ] pmaports: remote information updated recently (via git fetch/pull)
[00:55:20]
[00:55:20] NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
[00:55:20] DONE!
This works around an issue where some armhf apps are compiled with
instructions that are not compatible with aarch64, for example apk uses
MCR, and that does not exist on aarch64 causing apk to fail in a chroot
with SIGILL. It's impossible to build armhf images on aarch64 hosts, and
this also fixes a crash when running `pmbootstrap zap -a`, since pmb zap
will init an armhf chroot and blow up with apk generates a SIGILL.
For reference, here's how I arrived at the conclusion that apk (and gdb,
and probably others...) are executing invalid instructions on aarch64
when not using binfmt+qemu emulation:
$ uname -m
aarch64
$ file sbin/apk.static
sbin/apk.static: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), statically linked, BuildID[sha1]=997a8ef97b17cb4951a6220b7807a66fed67bc10, stripped
$ gdb sbin/apk.static
(gdb) r
Starting program: /run/user/10000/foo/sbin/apk.static
Program received signal SIGILL, Illegal instruction.
0x0012843c in ?? ()
(gdb) x /1i $pc
=> 0x12843c: mcr 15, 0, r0, cr7, cr10, {5}
I'm not sure why pmb.parse.arch is being imported here given that it's
the module we already are in. Replace it with pmb.config which this
module actually needs.
Instead of assuming the architecture of the foreign device to be what
the user selected in pmbootstrap init, actually query the architecture
from the device and use that.
This does mean that one extra ssh connection is necessary, which does
slow down the procedure somewhat. However, I think that is worth the
user experience improvement this brings.
Also, the deduction process can be skipped by manually specifying
--arch in the sideload invocation should it fail, or if one really wants
to skip that extra ssh roundtrip.
Related: https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2317
Sometimes it's useful to map an arbitrary machine type to an Alpine
architecture. alpine_native only lets you get the Alpine architecture
mapped to the machine type of the system pmbootstrap is running on. As
such, break out this functionality into a new function that takes the
machine type as a parameter.
The intent of 9a74848f19 was to not print
a stacktrace when this happened. However, I still opted to raise an
exception, so you still get a stacktrace, just with a more helpful
message. There's no need to give the user a big stacktrace here, so
raise a NonBugError instead of a RuntimeError.
Implement a function to verify the type of QCDT image by comparing the
first four bytes of the file. This is represented in a deviceinfo
variable, used by boot-deploy for dt.img generation.
Often I find myself confused and subsequently frustrated by the
behaviour of --fork-alpine as usually I apply whatever changes I want to
the aports package before forking. This is due to that in the proper
aports repo I can leverage the full power of git and apply patches or
revert commits, whereas after the package has been forked to pmaports
the context git needs to do this is lost.
To fix that, this commit introduces --fork-alpine-retain-branch which
works the same way as --fork-alpine except it doesn't change the aports
branch to match the current channel.
If there are multiple sections to a subpackage declaration, the middle one
(item 1 in the sequence in this case) is the custom function name which we
should use instead of the deduced one.
For example:
$pkgname-something-separate:something_separate:noarch
Here, the subpackage is called $pkgname-something-separate, but function names
cannot contain hyphens, so it's instead given the custom name
something_separate where the hyphen is replaced by an underscore. At the end,
the architecture of the subpackage is also overriden as noarch instead of
whatever architecture the main package has.
However, it is also possible to only override architecture of a subpackage and
not the function name.
In such cases, we get this:
$pkgname-something::noarch
We still have a "middle section" (item 1 in the sequence), but it will be
empty. As such, we not only need to check whether there are more than 1
subpackage part, but also whether that extra subpackage part is an empty string
or not.
This fixes an issue where pmbootstrap would not end up just setting the
properties of subpackages with a set arch but no custom function as
None instead of giving them proper depends, install, pkgdesc, et cetera
properties.
Due to that logging.verbose is "monkeypatched" into logging and not
present by default in the module, Mypy isn't able to pick up that it
exists. As such, just disable the check altogether in this file for
now.
Repurposing argcomplete as a boolean telling whether the module
argcomplete was found results in a variable with two distinct purposes
and types, which is bad practice and confuses Mypy. As such, don't
assign False to argcomplete if the module is not found and instead just
check for whether it exists in sys.modules to determine whether it was
imported successfully.
I zap chroots a lot, since I've found that it often "fixes" a lot of
weird issues that come about if you have stale chroots laying around.
So a common pattern I do is "pmb zap && pmb install ...". Having an
option to pmb install let's me simplify this.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231214234051.4673-1-clayton@craftyguy.net%3E
Fix that APKBUILDs mentioning other packages with an operator could not
be found. As we are building the initial branch of v23.12, this
currently happens with postmarketos-mkinitfs: it depends on
devicepkg-utils>=0.2.0 and currently pmbootstrap will only remove the
>=0.2.0 when looking for the pkgname in the APKINDEX of binary packages
(which is why it works on master). But it does not yet do that when
looking for the pkgname in pmaports.
Move the code for stripping the operator to a common place and use it
for getting packages from pmaports too.
Change the order of operators while at it, try to find <= before =, as
otherwise it would cut off example<=1.2.3 as "example<" instead of
"example".
Reviewed-by: Caleb Connolly <kc@postmarketos.org> (via chat)
If a package had _pmb_select set and an appropriate provider configured
in pmb's cfg, it was not being used when providers were being resolved.
With this change, those are now being considered. The order is also
important, we want a selected provider to be chosen before it tries to
make a decision based on provider_priority later in step #6.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231122081214.11872-2-clayton@craftyguy.net%3E
Currently, pmbootstrap checks if either the kernel or the ramdisk in a boot.img
contains the MediaTek header, and if one does, it assumes both do. It hardcodes
the label KERNEL for the kernel and ROOTFS for the ramdisk.
My Amazon Echo Dot (gen 2) has a boot.img where only the kernel has the header,
but not the ramdisk. These changes (as well as those in my new boot-deploy MR)
account for that situation (and any possible label an image has) by splitting
bootimg_mtk_mkimage into two variables for the kernel and the ramdisk labels.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/patches/46351
It can be quite useful to check the configs for a list of packages, so
adjust the argument parsing code to support that.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20231028085901.35205-1-luca@z3ntu.xyz%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
A long time ago we renamed the flash_system action into flash_rootfs.
Since we still kept some variables around, it's finally time to clean
that up.
Keep backwards compatibility for now since we cannot update pmaports at
the same time since the new deviceinfo names won't be supported in older
pmbootstrap versions.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230605220329.14328-3-luca@z3ntu.xyz%3E
Make sure that we disable legacy gadgets like USB_ETH. Our initramfs
uses configfs gadgets instead, currently RNDIS for USB networking.
In the future this can be expanded to more options like mass storage,
MIDI or whatever we're going to integrate into the OS that can be
configured by the user.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230423163438.746054-1-luca@z3ntu.xyz%3E
Python passes all arguments by reference. For mutable objects such as
a list used here, changing the object will change the original one. If
components_list is not set, this means the default value gets modified.
This lead to kernels getting checked with the wrong required components.
For example, when checking a kernel from community first with the
default component_list, it would get extended with all options needed
for a kernel in community. When checking another kernel from the
testing category, also with the default component_list, it would now
check for the community options in the testing kernel.
Related: https://stackoverflow.com/a/986145
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230402124338.64886-1-ollieparanoid@postmarketos.org%3E
Clayton Craft
Oliver Smith
Newbyte
methanal
Andras Sebok
Luca Weiss
Ben Westover
Lauren N. Liberda
Pablo Correa Gómez
Anton Bambura