Overview:
In order to execute foreign arch binaries on the host system, we are
using the Linux kernel's binfmt_misc feature in combination with
static builds of QEMU. Before this patch, the statically compiled
QEMU binaries were taken from Debian (mostly because I did not realize
that Alpine ships them as well). Now we can use the ones from the aport.
Benefits:
This allows us to easily update and patch the QEMU executables, we
don't need to be in sync with Debian's versions anymore.
Alpine's package is more modular, so we can save some download,
install, zap time, as well as disk space: setting up an armhf chroot
with pmbootstrap took ~102 MB before, now it's ~18 MB.
Detailed changes:
* Remove `cross/qemu-user-static-repack` aport
* Add `data/qemu-user-binfmt.txt` with the binfmt_misc flags for ELF
binaries of various arches (extracted from Debian's packaging)
* When parsing that file, don't write verbose messages to
`pmbootstrap log` anymore, only to the verbose log (can be enabled
with `pmbootstrap -v`)
* Rename `pmb.parse.arch.alpine_to_debian()` to ...`alpine_to_qemu()`
* Rename `arch_debian` to `arch_qemu`
This commit adds a test case, which makes sure that the KDE framework
and plasma framework version are always the same.
Additional changes:
* APKBUILD parser parses the URL now (that's the best way I found to
categorize the KDE aports in frameworks and other)
* Changed single quotes to double quotes in KDE APKBUILDs, so the
parser doesn't include the single quotes in the parsed result
* Added the test case to the gitlab CI config
With this patch, "pmbootstrap flasher" will fail with "the following
arguments are required: action_flasher". Without it, it just prints
"Done" and quits.
Overview:
Since Alpine updated to distcc 3.3 last week, pmbootstrap wasn't able to use
distcc for cross compilation anymore. It always falled back to running the
compiler in QEMU (which works, but is a lot slower). The reason for that is,
that distcc requires all compilers that are being used in a whitelist now.
This partially fixes CVE-2004-2687 in distccd, which allowed trivial remote
code execution by any process connecting to the distccd server. We only run
distccd on localhost, but still this can be used for privilege escalation of
sandboxed processes running on the host system (not part of pmbootstrap
chroots).
Because the CVE is only partially fixed (see the comment in
`pmb/chroot/distccd.py` for details), we make sure that only the building
chroots can talk to the distcc server by running distcc over ssh.
Details:
* Completely refactored `pmb/chroot/distccd.py` to run distcc over ssh
* Store the running distcc server's arguments as JSON now, not as INI
* Make debugging distcc issues easy:
* Set DISTCC_BACKOFF_PERIOD=0, so the distcc client will not ignore the
server after errors happened (this masks the original error!)
* New pmbootstrap parameters:
* `--distcc-nofallback`: avoids falling back to compiling with QEMU and not
throwing an error
* `--ccache-disable`: avoid ccache (when the compiler output is cached,
distcc does not get used)
* `--verbose` prints verbose output of the distcc too
* New test case, that uses the new pmbootstrap parameters to force
compilation through distcc, and shows the output of distcc and distccd in
verbose mode on error (as well as the log of sshd)
All firmware packages go into subpackages nowadays, so we have them
optional. The `test/test_aports.py` file has a test case, that makes
sure that there are no firmware packages in the `depends` of a device
package. So far we have only checked for `firmware-*` and
`linux-firmware`, this commit makes it check for `linux-firmware*` as
well.
In addition, the URL displayed in the "Please use kernel subpackages"
message has been fixed (points to devicepkg instead of deviceinfo now).
Alpine's `abuild` will uninstall all dependencies by default, when a
package build fails.
Leaving this configuration unchanged leads to unexpected behavior with
pmbootstrap: when executing `pmbootstrap build --strict` and pressing
`^C` during the build, pmbootstrap will stop, but an `apk` process
will be started in the background to remove the dependency packages.
Running `pmbootstrap shutdown` at this time will not work, because the
`apk` process is still running.
With this commit, dependencies don't get cleaned up from the chroots.
In gitlab, there is no extra CI job running for commits in a merge
request (MR). This means, we can't run different code in a MR against
"master".
So instead of only checking whether all devices are booting if there's
a MR against master, always perform this check.
I've edited the message slightly, so it's clear that it's only required
to have the devices booting when making a merge request against master.
Follow-up to !1373, where `pmbootstrap flasher flash_system` was
replaced with `pmbootstrap flasher flash_rootfs`. We still had used
terms like "system partition" in a lot of places.
This commit replaces it everywhere, so it's clear that we're talking
about the pmOS rootfs (which may or may not be installed to Android's
system partition).
The test suite needed a `pmbootstrap shutdown` after running through,
before it could successfully run again.
Explanation:
This was caused by `test/test_pkgrel_bump.py`, which creates a
temporary work folder with every subfolder ("chroot_native",
"cache_apk_x86_64", ...) linked to the original work folder except for
the "packages" folder. At the end of the test case,
`pmbootstrap shutdown` gets executed and is expected to umount
everything as usual. But it does not umount anything because of the
symlinks, so `work/chroot_native/mnt/pmbootstrap-packages` points to
the fake packages folder of that test case, even after it is finished.
As a result, any test case that tries to access the packages folder in
the native chroot, will fail until `pmbootstrap shutdown` gets called.
Detailed Changes:
* Umount all folders inside the work folder, even if these are symlinks
* Remove obsolete reference to "disable timestamp based rebuilds" in a
comment in `test/test_pkgrel_bump.py`
* Run `pmbootstrap work_migrate` and `pmbootstrap shutdown` at the
beginning of `test/testcases_fast.sh`, in case the pkgrel_bump test
case was aborted before it could properly shutdown and to make it
more robust in general (user may have changed the mountpoints, work
folder may need to be migrated)
This makes debugging much easier, because the relevant log messages
before the crash can always be found at the bottom of the log files,
and not somewhere in the middle.
Prior to this commit, it was possible to type in packages with a
trailing comma in `pmbootstrap init` when asked for extra packages.
This leads to problems during `pmbootstrap install`, so now we disallow
it. Fixes#1540.
If the test case couldn't connect to the VM with SSH, it used to fail
with a Python error of accessing the 'missing' variable without having
it defined.
With this commit, it prints out 'Could not connect to the VM via SSH'
instead.
This commit adds two new ignores to the flake8 call:
* W504 line break after binary operator
* W605 invalid escape sequence 'x'
Without these, newer flake8 versions, such as the one shipped by Arch
Linux, don't run through on the code base (which leads to not running
it at all locally, only in CI).
Additional changes:
* Don't give the exception instance of subprocess.CalledProcessError a
name, so it does not trigger F841
* Dont' advise to run "autopep8" anymore, because that will likely
touch unrelated code.
For reference, here is the list of error codes:
https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes
It is unexpected for quite a lot of people, that the chroot folders are
still mounted when a pmbootstrap command has finished. With this commit,
it will let the user know explicitly:
> NOTE: chroot is still active (use 'pmbootstrap shutdown' as necessary)
Close#1524
Marti Raudsepp
Bart Ribbers
Martijn Braam
Martin Rys
Oliver Smith
Kieran Elmes
NotKit
ryang
Grant Miller
Magnus Gulin
joshas
Xinayder
Ashymad
Michael Orishich