The APKBUILD reference for "provides" [1] is not worded clearly; but
after reading it over and over again, my understanding is:
* package with provides='foo=1.2' will be automatically installed if
user requests installing "foo"
* package with provides='foo' (without version) will NOT get
automatically installed if user requests installing "foo"
For pmbootstrap, this means, that we must not attempt to build a package
where the pkgname mentioned in provides matches what we are currently
resolving, unless there is an equals sign in the provides entry.
Fixes: #1862, pmaports#404
[1] https://wiki.alpinelinux.org/wiki/APKBUILD_Reference#provides says:
"List of package names (and optionally version info) this package
provides.
If package with a version is provided (provides='foo=1.2') apk will
consider it as an alternate name and it will automatically consider the
package for installation by the alternate name, and conflict with other
packages having the same name, or provides.
If version is not provided (provides='foo'), apk will consider it as
virtual package name. Several package with same non-versioned provides
can be installed simultaneously. However, none of them will be installed
by default when requested by the virtual name - instead, error message
is given and user is asked to choose which package providing the virtual
name should be installed."
When running pmbootstrap init, first select device vendor, then device
codename. Also fixed tests for new behavior and added some new ones for
new scenarios.
Packages ending in -dev: just assume that the originating aport has the
same pkgname, except for the -dev at the end. Otherwise we may end up
with the wrong package.
For example, if something depends on plasma-framework-dev, and
plasma-framework is in Alpine, but plasma is in pmaports, then the
regular guess_main() algorithm below would pick plasma instead of
plasma-framework.
Fixes: build.postmarketos.org#52
Remove the recursive check, as it caused an infinite loop. It took a
very long time to complete anyway, even when it worked. The reasoning
for having the recursive check in the first place was, that we would
have device packages with arch=noarch set in the APKBUILD, but which
could only be built for a certain architecture (the device
architecture). Nowadays using arch=noarch in device packages is
forbidden, and we enforce that rule [1]. So the recursive arch check
isn't necessary anymore.
[1] ac6c0a2997
This allows for example for me to call the kconfig check function on the
.config file in my Linux tree: $ pmbootstrap kconfig check --file
.config and it reports me which kconfig options I need to enable.
Download all sources and verify their checksums.
This will be used in pmaports.git CI, if ci:skip-build is set in the
commit message (currently it just skips the build, and we don't test if
the source checksums are valid or not).
Do not crash during "pmbootstrap zap" and other actions when the user
needs to type in the sudo password before running "du" to calculate the
work folder size. We're trying to parse the "du" output as integer, but
of course the 'Sorry, try again' message from sudo is not a valid
integer.
$ pmbootstrap --details-to-stdout zap
(008707) [14:00:57] Shutdown complete
(008707) [14:00:57] Calculate work folder size
(008707) [14:00:57] % sudo du --summarize --block-size=1 /home/luca/.local/var/pmbootstrap
[sudo] password for luca:
[sudo] password for luca: Sorry, try again.
12966293504 /home/luca/.local/var/pmbootstrap
(008707) [14:01:03] ERROR: invalid literal for int() with base 10: 'Sorry, try again.\n12966293504'
It would be a good idea to separate the sudo output from the process
output in general, see #1677 for a related proposal.
When the pmaport arch does not match, but there's a matching binary
package, return the binary package instead of the pmaport.
This is needed to fix "pmbootstrap repo_missing" for x86_64
networkmanager-qt, which indirectly depends on mesa (and mesa needs to
be taken from the binary packages for x86_64, not from the pmaports,
see #1741).
Split the part that actually checks the arch against the arches list
into pmb.helpers.pmaports.check_arches(arches, arch), and call it
from pmb.helpers.package.check_arch(args, pkgname, arch, binary).
This will be used in a follow up commit, where we have already resolved
the package data and only need to check the architecture.
In pmb.helpers.repo.alpine_apkindex_path(), default to
arch = args.arch_native *before* calling pmb.helpers.repo.update().
Because otherwise update() defaults to all arches instead of
args.arch_native.
This caused the APKINDEX files for all arches to get downloaded, as
pmb.chroot.apk_static.init() calls alpine_apkindex_path() without
setting arch = args.arch_native explicitly.
Ignore APKBUILDs that have "!pmb:kconfigcheck" in their options by
default in "pmbootstrap kconfig check", but print a note that they have
been skipped. Check all kernels with "pmbootstrap kconfig check -f".
This is necessary, because the Librem 5 devboard kernel's config does
not have CONFIG_DM_CRYPT enabled in their config, and we check for
that. As the device is still under heavy development, we will make our
lives easier by just using the upstream kernel config without any
changes and ignoring it in our check by default.
"breeze-icons" depends on "qt5-qtbase-dev", but
"pmbootstrap repo_missing" should return "qt5-qtbase" instead.
This patch fixes it, as one can see with:
$ pmbootstrap repo_missing --built breeze-icons --overview
Provides a quick way to incrementally compile a kernel and push it to
device.
Example usage.
Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX
Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
This drops the --apparent-size parameter when calculating the size
required for rootfs, which seemed to return a size that was too small
for some devices. This also includes specifying the number of inodes to
support when formatting rootfs.
Fixes#1717
Rename pmb.helpers.devices.list() to
pmb.helpers.devices.list_codenames(). Python already has a list()
function, so we name our function to make the calls to the codenames
listing function inside pmb/helpers/devices.py less confusing.
This fixes list_apkbuilds which is dead code currently but is useful
for external software using pmbootstrap. It also adds list_deviceinfos
which does the same thing but for the deviceinfo data.
The plan is to add an API later, where both functions can be used.
Properly replace ~ with $HOME in all paths. Fix tab completion when
passing a path starting with ~ to pmaports:
$ pmbootstrap --aports ~/src/pmaports/ build linux-<TAB>
Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.
This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.
Change the cache format from args.cache["offline_msg_shown"] to
args.cache["pmb.helpers.repo.update"]["offline_msg_shown"]. This is in
preparation for saving more data in the cache of
pmb.helpers.repo.update in the next commit.
Do not fail anymore when attempting to start a new binary repository
build without any existing binary packages:
pmbootstrap -mp="" repo_missing
Find subpackages defined with subpackage functions:
subpackages="dev:mydevfunc"
Find provides defined with specific versions:
provides="mkbootimg=1.0.0"
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories
This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
Obscure feature: it was possible to specify a local path as
--mirror-pmOS. It would then get mounted to /mnt/postmarketos-mirror
inside the chroot, and be specified as such in the generated
/etc/apk/repositories file.
I had used this once for some testing scripts, but I am sure nobody is
using this anymore. The same can be achieved with running a local http
server anyway:
<https://wiki.postmarketos.org/wiki/Installing_packages_on_a_running_phone>
Removing this makes it easier to support multiple postmarketOS mirrors
(next commit).
In pmb.helpers.package.get(), we are differentiating between packages
that do not exist at all, and packages that do not exist for the
specified architecture. Make sure to actuallyy download the foreign arch
APKINDEX files, before trying to find the packages in there. Otherwise
the "could not find aport, and could not find this package in any
APKINDEX" package would appear.
We ran into this when testing on sr.ht, because pmbootstrap runs on a
fresh install every time, where no old APKINDEX files are present
(which would work around the bug).
This was meant to be part of the previous commit already, and was
tested, but it was not submitted correctly. So here it is as separate
commit straight to master. It is trivial anyway.
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.
Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
When the timeout occurs it is important to ensure clean up of child
processes. Killing only the direct process created by a command can
leave child processes running.
For example a pmbootstrap.py install will run apk add. This run command
creates multiple processes as follows:
(cmd line arguments snipped for readability)
$ ps -e -o pid,ppid,pgid,cmd
PID PPID PGID CMD
31738 23247 31738 python3 ./pmbootstrap.py -t 15 install --no-fde
31746 31738 31738 sudo env -i /bin/sh -c ... ;apk --no-progress add
31747 31746 31738 /bin/sh -c ... ;apk --no-progress add
31748 31747 31738 apk --no-progress add
The root process of the run command is PID 31746. We want to kill
the child processes too. Otherwise only running kill -9 31746 will leave
the processes 31747 and 31748 running.
* Travis and Coveralls badges
* aports: instead of <https://github.com/postmarketOS>, use
<https://postmarketos.org>
* References to full URLs to issues and pull requests replaced with
a hash and the number
* grsec check: simplify error message, remove link to github issue
(nobody is using that anymore anyway)
Alpine ships `mkbootimg` with the `android-tools` package now. This
conflicts with the `mkbootimg` fork from osm0sis (see #441).
Changes:
* Rename `mkbootimg` to `mkbootimg-osm0sis` (aport and binary name)
* `mkbootimg-osm0sis`: provides `mkbootimg` now (so we don't need to
change all the device aports), update version to 2018.05.10
* Adjust our `mkinitfs` script to call `mkbootimg-osm0sis`
* Better aport description
* pkgrel_bump testcase: don't fail on deleted aport
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
(with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
aports ([skip ci] because it would rebuild all kernels and run out of
time). Alpine has dropped this as well, and it wouldn't work with the
new code anyway.
We require binfmt_misc to run programs of foreign architectures (e.g.
armhf) with QEMU. This is set up by default in most distributions, but
in some (e.g. Alpine, Void) it needs to be configured manually (see
the troubleshooting page in the wiki).
We have a check in place, which points to that troubleshooting wiki
page. However, the check was flawed, because we assumed the binfmt_misc
folder would not exist.
Thanks to @fxkrait for making the fix and for testing it!
Changes:
* `helpers/envkernel.sh`:
* installs everything needed for kernel compilation in the native
chroot
* mounts the kernel source to `/mnt/linux` inside the chroot
* creates `/mnt/linux/.output` and chowns it to the `pmos` user, that
folder will be used for the kernel build output
* sets up aliases for `make`, `pmbootstrap`, `pmbroot`, `kernelroot`
* new action `pmbootstrap work_migrate`: does the interactive work
folder migration if necessary, otherwise it doesn't output anything
* when calling this first, we can safely use all other commands
non-interactively without showing the output
Benefits:
* Fast setup (especially for people who are new to kernel
compilation
* No need to figure out distribution specific package names
(cross compilers!)
* No need to do a test build just to verify that the right
packages are installed
* Less error prone
* The right dependencies are always installed
* `ARCH` and `CROSS_COMPILE` variables always get set automatically
and based on `deviceinfo_arch`
* If the build environment is broken for some reason, just zap and
start over
* Easy to reproduce problems
Notes:
* `make menuconfig` works as well
* Sourcing was tested with `zsh`, `bash` and `fish`, it should be easy to
extend for other shells
`pmbootstrap pkgrel_bump --auto` automatically increases the pkgrel for
packages linking against libraries, which don't exist anymore (because
the soname has been changed). The feature is explained in detail in
The previous implementation did not detect soname breakage, when a
subpackage linked against a certain library, but the main package
did not (e.g. `qt5-qtbase-mysql` and `qt5-qtbase`). This was, because
we iterated over the aports/* to find the packages to be checked.
To fix this, we are iterating over the packages found in the APKINDEX
files instead (of both the locally compiled packages and the downloaded
index from the pmOS mirror).
Details:
* `pmb/helpers/pkgrel_bump.py`:
* Rewrite `auto_apkindex_package()` to act upon a given parsed
`aport` and `apk` (from the index) instead of finding the `apk`
dict by itself (we need it earlier anyway).
* Rewrite `auto()` to iterate over APKINDEX files instead of aports
* Skip packages already found, so the `pkgrel` does not get
increased multiple times when the same package was found in
multipe index files.
* Put the package name at the beginning of the log messages to make
them more readable
* testdata: Create a new `testsubpkg` aport, where only the subpackage
links against `testlib`
* Adjust testing code to test everything with `testsubpkg` as well.
NOTE: This makes the command a bit slower, but we could improve
performance again by smart caching of `pmb.parse.apkindex.depends()`.
This could come in a future PR, the important part here is that the
command is bug-free again with this fix.
Oliver Smith
Alexey Min
Luca Weiss
Idan Horo
Oliver Smith
Robert Yang
Clayton Craft
Martijn Braam
NotKit
steamport