Download all sources and verify their checksums.
This will be used in pmaports.git CI, if ci:skip-build is set in the
commit message (currently it just skips the build, and we don't test if
the source checksums are valid or not).
Launch native cross compilers inside foreign chroot. Enable by default,
but allow disabling with --no-crossdirect for now. This option and the
distcc-sshd related code will be removed in the future.
Aborts the build if any dependencies would have to be build first. This
is useful for build.postmarketos.org, because we want to build exactly
one package in one build job. If dependencies would need to be built, we
made a mistake earlier, and not aborting the build makes it harder to
find that orginal mistake.
Ignore APKBUILDs that have "!pmb:kconfigcheck" in their options by
default in "pmbootstrap kconfig check", but print a note that they have
been skipped. Check all kernels with "pmbootstrap kconfig check -f".
This is necessary, because the Librem 5 devboard kernel's config does
not have CONFIG_DM_CRYPT enabled in their config, and we check for
that. As the device is still under heavy development, we will make our
lives easier by just using the upstream kernel config without any
changes and ignoring it in our check by default.
Provides a quick way to incrementally compile a kernel and push it to
device.
Example usage.
Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX
Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Test with flake8: *.py
./test/check_checksums.py:13:13: E117 over-indented
./pmb/config/init.py:97:8: F632 use ==/!= to compare str, bytes, and int literals
./pmb/parse/arguments.py:229:13: E117 over-indented
Allow changing the kernel partition for fastboot and heimdall in
deviceinfo and on the fly while doing "pmbootstrap flasher
flash_kernel". Also allow changing the partition for
"... flash_rootfs" with fastboot (this was only possible with
heimdall so far).
Introduce two new deviceinfo variables:
* flash_fastboot_partition_kernel
* flash_fastboot_partition_system
This is useful for devices with dual partitioning that have boot_a
and boot_b.
Properly replace ~ with $HOME in all paths. Fix tab completion when
passing a path starting with ~ to pmaports:
$ pmbootstrap --aports ~/src/pmaports/ build linux-<TAB>
Make tab completion work again for packages, if the default pmaports
dir is used. In the last "args" code refactor, the variable replacing
code was moved into its own function. We did not call it in the
packagecompleter() yet, so it could not replace the $WORK variable in
the default pmaports path.
Show all config keys that can be queried and set in the
'pmbootstrap config -h' output and make tab completion work for the
key names.
I've set "metavar" and placed the variables in the helpstring. That
way, argparse will not generate a huge "positional arguments" string
that blows up the layout of the help output:
[{ccache_size,device,extra_packages,hostname,jobs,kernel,keymap,...
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories
This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
-m is the Alpine mirror, -mp is the postmarketOS mirror. Use "URL" as
metavar and add help text that explains how to disable the postmarketOS
mirror (so all pmaports get built locally).
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
With this patch, "pmbootstrap flasher" will fail with "the following
arguments are required: action_flasher". Without it, it just prints
"Done" and quits.
Overview:
Since Alpine updated to distcc 3.3 last week, pmbootstrap wasn't able to use
distcc for cross compilation anymore. It always falled back to running the
compiler in QEMU (which works, but is a lot slower). The reason for that is,
that distcc requires all compilers that are being used in a whitelist now.
This partially fixes CVE-2004-2687 in distccd, which allowed trivial remote
code execution by any process connecting to the distccd server. We only run
distccd on localhost, but still this can be used for privilege escalation of
sandboxed processes running on the host system (not part of pmbootstrap
chroots).
Because the CVE is only partially fixed (see the comment in
`pmb/chroot/distccd.py` for details), we make sure that only the building
chroots can talk to the distcc server by running distcc over ssh.
Details:
* Completely refactored `pmb/chroot/distccd.py` to run distcc over ssh
* Store the running distcc server's arguments as JSON now, not as INI
* Make debugging distcc issues easy:
* Set DISTCC_BACKOFF_PERIOD=0, so the distcc client will not ignore the
server after errors happened (this masks the original error!)
* New pmbootstrap parameters:
* `--distcc-nofallback`: avoids falling back to compiling with QEMU and not
throwing an error
* `--ccache-disable`: avoid ccache (when the compiler output is cached,
distcc does not get used)
* `--verbose` prints verbose output of the distcc too
* New test case, that uses the new pmbootstrap parameters to force
compilation through distcc, and shows the output of distcc and distccd in
verbose mode on error (as well as the log of sshd)
Follow-up to !1373, where `pmbootstrap flasher flash_system` was
replaced with `pmbootstrap flasher flash_rootfs`. We still had used
terms like "system partition" in a lot of places.
This commit replaces it everywhere, so it's clear that we're talking
about the pmOS rootfs (which may or may not be installed to Android's
system partition).
We are analyzing the `boot.img` with `file` before we send it to
`unpackbootimg`. File does not recognize all kinds of `boot.img` files,
which `unpackbootimg` can extract, so we need a way to skip this check.
Details:
* Add `-f` parameter, continues extraction with a warning if the file
seems to be invalid
* Tell the user that `-f` can be used if the `boot.img` is invalid and
it's not specified
* Consistent spelling of `boot.img` instead of `bootimg` in messages
Fixes#1608
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
(with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
aports ([skip ci] because it would rebuild all kernels and run out of
time). Alpine has dropped this as well, and it wouldn't work with the
new code anyway.
* Usage: pmbootstrap install --split
* Make obvious that export is the next step when split images are created
* Fix note for missing rootfs image on export
* Change wording from "system image" to "rootfs image"
* The idea was to show the note only when the rootfs image was not
generated yet. But this was broken, because the path we checked for
was missing the chroot path prefix (which is added now).
* Also don't display the message, when the split image files exist
Changes:
* `helpers/envkernel.sh`:
* installs everything needed for kernel compilation in the native
chroot
* mounts the kernel source to `/mnt/linux` inside the chroot
* creates `/mnt/linux/.output` and chowns it to the `pmos` user, that
folder will be used for the kernel build output
* sets up aliases for `make`, `pmbootstrap`, `pmbroot`, `kernelroot`
* new action `pmbootstrap work_migrate`: does the interactive work
folder migration if necessary, otherwise it doesn't output anything
* when calling this first, we can safely use all other commands
non-interactively without showing the output
Benefits:
* Fast setup (especially for people who are new to kernel
compilation
* No need to figure out distribution specific package names
(cross compilers!)
* No need to do a test build just to verify that the right
packages are installed
* Less error prone
* The right dependencies are always installed
* `ARCH` and `CROSS_COMPILE` variables always get set automatically
and based on `deviceinfo_arch`
* If the build environment is broken for some reason, just zap and
start over
* Easy to reproduce problems
Notes:
* `make menuconfig` works as well
* Sourcing was tested with `zsh`, `bash` and `fish`, it should be easy to
extend for other shells
* Change `pmbootstrap flasher flash_system` command to
`pmbootstrap flasher flash_rootfs`
* The old command still works, but all references have been changed to
the new command
* Remove obsolete `pmbootstrap flasher export` (that was changed to
`pmbootstrap export` a few months ago)
* Update `README.md` and ZSH auto completion
* Change the description of the generated rootfs image (not talking
about a system image anymore, mention that it has subpartitions)
* Better description of `pmbootstrap flasher flash_rootfs --partition`
* pmbootstrap newapkbuild: Properly parse arguments
The `pmbootstrap newapkbuild` action wraps Alpine's `newapkbuild`. We
used to directly pass all arguments to `newapkbuild` without verifying
in Python whether they make sense or not. However, as `newpakbuild`
doesn't do strict sanity checks on the arguments, it is easy to end up
with unexpected behavior when using the command for the first time.
For example, `newapkbuild` allows either specifying a PKGNAME or SRCURL
as last parameter, and also allows setting a PKGNAME with the `-n`
parameter. It only makes sense to use that option when passing a
SRCURL.
With this commit, we duplicate the optins that should be passed through
to `newapkbuild` and use argparse to fully sanitize the options and
display a help page (`pmbootstrap newapkbuild -h`) that is consistent
with the other help pages.
Details:
* The `-f` (force) flag does not get passed through anymore. Instead we
use it in Python to skip asking if an existing aport should be
overwritten (the aports are outside of the chroot, so `newapkbuild`
can't handle it in a way that makes sense for pmbootstrap).
* Output of `newapkbuild` gets redirected to the log file now, as we
don't need it to display a help page.
* Don't verify the pkgver while creating the new APKBUILD. When passing
a SRCURL, the pkgver gets extracted from the end of the URL and may
not have a valid format yet (but we want the APKBUILD anyway).
* Stored options passed through in `pmb/config/__init__.py` and use it
in both `pmb/parse/arguments.py` and `pmb/helpers/frontend.py`.
* Only allow `-n` with SRCURL
* The postmarketOS aports folder gets specified with `--folder` now.
That way the generated help page is much closer to the original one
from `newapkbuild`. The default is `main`.
* Made the package type flags (CMake, autotools, ...) exclusive so only
one of them can be specified
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.
After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).
The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:
pmbootstrap chroot --add=git --user -- \
git -C /mnt/pmbootstrap-git/aports_upstream pull
Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
the absolute path to the upstream aport on disk, after checking the
version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
APKINDEX if necessary and returns the absolute path to the APKINDEX.
This code was basically present already, but not as function, so now
we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
the official gcc APKBUILD before we modify it (the current APKBUILD
parser is not meant to be perfect, as this would require a full shell
parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
`testcases_fast.sh`. Previously it was disabled due to traffic
concerns (cloning the aports repo, but then again we do a full KDE
plasma mobile installation in Travis now, so that shouldn't matter
too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
from the disabled-by-default list (left over from timestamp based
rebuilds), there were no more test cases disabled by default. I've
changed it, so now the qemu_running_processes test case is disabled,
and added an `--all` parameter to the script to disable no test
cases. Travis runs with the `--all` parameter while it's useful to
do a quick local test without `--all` in roughly 2 minutes instead of
10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
default Alpine mirror (so the aportgen testcase runs through).
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.
That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).
Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
list of blocks, instead of the dict with removed duplicates with
lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
what they are doing now.
```
short long (old) long (new)
-p --packages --pkgs-local
-m --mismatch-bins --pkgs-local-mismatch
-o, --old-bins --pkgs-online-mismatch
```
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.
### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
* Default is updating all arches where the APKBUILD files exist
* Add existing_only parameter
* Return True when files have been downloaded
* Properly print which arches will be updated
* Print update reason only in verbose log
* Add and improve comments
* pmb.parse.arguments(), update action:
* Add --non-existing parameter
* Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
* Inform about --non-existing if no APKBUILDs have been updated
* The APKINDEX parser used to return a dictionary with one package for
a given package name. This works for the installed packages database,
because there can only be one provider for a package. But when
parsing packages from binary repositories, we need to support
multiple providers for one package. It is now possible to get a
dictionary with either multiple providers, or just a single provider
for each package.
* Dependency parsing logic has been adjusted, to support multiple
providers. For multiple providers, the one with the same package
name as the package we are looking up is prefered. If there is none
(eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
that will be installed anyway, and after that packages that are
already installed. When all else fails, it just picks the first one
and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
that pmbootstrap should build if neccessary, and install in the
chroot. This can be used to quickly test the depencency resolution
of pmbootstrap without doing a full "pmbootstrap install".
Fixes#1122.
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
we can write into the source folder if necessary, for better
compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
usually), we extend the APKBUILD with overrides to make it use
mountpoint's source instead of downloading the package's source
from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
Follow up to #1162.
* `pmb.build.buildinfo()`: Used to record the build environment. It is
flawed because it scans the repo APKINDEX files instead of using the
actually installed packages list. When it was implemented we were not
able to do the latter. After this is removed, `pmb.parse.depends` can
be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
used exclusively by `pmb.build.buildinfo()`, to learn about which
files have been changed in the local repository folder after a
package was built. The idea was, that we could find subpackages that
way. But this information is present in the installed package list as
well, which is a much cleaner approach.
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
have an aport or where the aport has another version. This also
works with subpackages now,
we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"
zap in general:
* Show the amount of cleared up space after the deletion instead
of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
(we need to call it twice during zap now to get the space
calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
for debugging) to list the packages/chroots that would get
deleted
* Roughly output the command that would get executed to delete
files, so it's obvious what's going on in --dry mode. (% rm ...)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:
pmbootstrap build --force hello-world
Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close#1167, close#1156, close#1023 and
close#985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
Preparation for #1122.
* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
an exception when two entries in the apkindex provided the same package.
Turns out this is *not* invalid after all, two packages can provide the same
soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
which was a dead end. This code uses the broken strict feature.
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
(also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
Luca Weiss
Oliver Smith
Zhuowei Zhang
Robert Yang
Grant Miller
Ion Agorria
steamport
Daniele Debernardi
Pavel Machek