After initializing the build environment, the cache_distfiles folder
currently is writable by everyone (which is not ideal, fix following
soon). The aportgen code for `busybox-static-*` and `musl-*` copies
the foreign arch `.apk` file to the distfiles, but it executes this
action as regular user and not as root. This only works as long as
build initialization ran before (which may not be the case on Travis
and expecting this to run before is a bug in general).
With this commit, the copy action gets executed as root, so it works
in any case. I'm commiting this directly (without a PR), because it
is a super simple fix and it unblocks our continuous integration.
Local testing:
$ pmbootstrap -y zap
$ sudo rm -r ~/.local/var/pmbootstrap/cache_distfiles
$ pmbootstrap aportgen musl-armhf
* Qualcomm MSM modem: 'rmtfs' support packages
* qcom_rmtfs: Server that talks to modem over IPC to allow it
read/write data for its persistent storage. This is needed for it to
boot, as well as periodically during usage. Added a patch that
it expects the storage path symlinks in /etc instead of /boot.
* qrtr: IPC library for rmtfs
* libqipcrtr4msmipc: adapter library to make qrtr work on kernels with
AF_MSM_IPC support. AF_QIPCRTR is the mainline equivalent since Linux
~4.7.
* msmipc-dev: Header files for qrtr and libqipcrtr4msmipc.
Thanks to Bjorn Andersson <https://github.com/andersson> for rmtfs and
qrtr.
* libsmdpkt_wrapper: adapter lib for QMI clients
The SMD serial packet driver in Qualcomm kernels has, AFAICT, a bug
in poll(); this works around it so that qmicli et al can work.
* i9195: firmware (modem only right now)
* add ofono (with patch for MSM devices)
Based on Alpine's package.
* i9195: add modem support
* move all modem related packages to aports/modem
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.
After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).
The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:
pmbootstrap chroot --add=git --user -- \
git -C /mnt/pmbootstrap-git/aports_upstream pull
Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
the absolute path to the upstream aport on disk, after checking the
version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
APKINDEX if necessary and returns the absolute path to the APKINDEX.
This code was basically present already, but not as function, so now
we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
the official gcc APKBUILD before we modify it (the current APKBUILD
parser is not meant to be perfect, as this would require a full shell
parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
`testcases_fast.sh`. Previously it was disabled due to traffic
concerns (cloning the aports repo, but then again we do a full KDE
plasma mobile installation in Travis now, so that shouldn't matter
too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
from the disabled-by-default list (left over from timestamp based
rebuilds), there were no more test cases disabled by default. I've
changed it, so now the qemu_running_processes test case is disabled,
and added an `--all` parameter to the script to disable no test
cases. Travis runs with the `--all` parameter while it's useful to
do a quick local test without `--all` in roughly 2 minutes instead of
10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
default Alpine mirror (so the aportgen testcase runs through).
## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.
This is not a security issue, as we don't pass strings from untrusted
input to this function.
## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`
In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`
## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`
So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.
Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.
## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.
## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
list and environment variables as dict, and creates a properly escaped
flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
executed (in addition to the simplified form we always write down for
readability)
* `test_shell_escape.py`: verify that the command passed by reference
has not been modified, add a new test for strings with spaces, add
tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
so running ncurses applications like `menuconfig` and `nano` works out of
the box
Since PR #1247 we are using the virtual package `.pmbootstrap` to mark
packages as not explcitly installed. In case `pmbootstrap` doesn't
finish the installation because of a bug, the `.pmbootstrap` virtual
package may not get uninstalled.
As virtual packages don't have the "timestamp" attribute set in the
package database (which indicates when the package was built), the
APKINDEX parser fails to parse them.
Changes:
* pmb.parse.apkindex.parse_next_block(): don't require the "timestamp"
attribute to be set (but the arch attribute instead, which is always
present)
* pmb.parse.apkindex.parse(): when a block does not have a `timestamp`
attribute, skip it, because it must be a virtual package.
* add test cases for both functions with a package database that
contains a virtual package.
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.
That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).
Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
list of blocks, instead of the dict with removed duplicates with
lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
what they are doing now.
```
short long (old) long (new)
-p --packages --pkgs-local
-m --mismatch-bins --pkgs-local-mismatch
-o, --old-bins --pkgs-online-mismatch
```
Use case: `mkbootimg` provides the `unpackbootimg` package. When
running `pmb.chroot.apk.install(args,"unpackbootimg")`, it was not
able to properly build the package.
Reproducing the error:
```
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/mkbootimg*
pmbootstrap index
pmbootstrap --mirror-pmOS="" chroot --add=unpackbootimg
```
Or alternatively (simpler but less illustrative):
```
pmbootstrap build unpackbootimg --force
```
Specifying alternate subpackages in the provides variable of the main
package means that the main package itself is providing an alternate for
the subpackages (e.g. gtk+2.0-maemo provides an alternate gtk+2.0-dev).
The provides variable must be set for each subpackage. Fixes#1284.
* Add charging-sdl package
* Include charging-sdl into the initramfs-extra
* [initramfs] Detect charging mode and use triggerhappy to start
charging-sdl when the power key is pressed
The `suffix` argument was not specified in chroot commands executed in
`pmb.build._package.override_source()`. Because of that, it was not
possible to use "build --src" when compiling in a non-native chroot,
for example:
```
$ pmbootstrap build hello-world --arch=armhf
...
(native) % rm /tmp/APKBUILD.append
rm: can't remove '/tmp/APKBUILD.append': No such file or directory
```
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.
### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
* Default is updating all arches where the APKBUILD files exist
* Add existing_only parameter
* Return True when files have been downloaded
* Properly print which arches will be updated
* Print update reason only in verbose log
* Add and improve comments
* pmb.parse.arguments(), update action:
* Add --non-existing parameter
* Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
* Inform about --non-existing if no APKBUILDs have been updated
Since it works for linux-postmarketos-stable, I've simply copied over
the config, and ran "pmbootstrap menuconfig" on it again. Then it
worked again in Qemu.
* Add NetworkManager and PulseAudio applets to Plasma
* This crashes plasma mobile on armhf (where it is not usable due
to performance problems anyway), except on mainlined kernels
with hardware accelerated graphics. This is mentioned in the
pkgdesc of postmarketos-ui-plasma-mobile now.
In case the user does not specify for which arch packages should be
built with `pmbootstrap build`, we detect it automatically.
Previous logic was, that if the APKBUILD's arch is "all" or "noarch",
then prefer the native arch, and otherwise use the first one in the
list of available arches.
New behavior is, that we also check if the list of possible arches
contains the native arch (and if that fails, the device arch). If that
is the case, we return the native/device arch instead of the first one
in the list.
### Use case
The arch from `gcc-armhf` and similar packages (as generated by
`pmbootstrap aportgen`) used to be "all", but is nowadays a specific
list of arches. This means, that after updating the `gcc-armhf` and
`gcc-aarch64` packages, and calling `pmbootstrap build gcc-armhf`,
it will try to build `gcc-armhf` for `aarch64` instead of the native
architecture, because that is the first one listed.
And since compiling to `aarch64` requires `gcc-aarch64`, it will build
that for the native architecture first.
So you're asking for `gcc-armhf` and it compiles `gcc-aarch64`, which
is very confusing (see #1272).
When building device packages, the postmarketos-mkinitfs package gets
installed as dependency of postmarketos-base. It must not try to
create an initramfs at this point, when there is not deviceinfo file.
We build the initramfs during the installation, so it's fine.
Added [skip ci] because linux kernels and KDE updates are currently
getting built for the binary repository (so Travis couldn't finish
anyway).
* Add nonfree_firmware subpackage to all devices, that depend on
nonfree firmware.
* Some packages were depending on `linux-firmware`, but without having
Wifi working. Removed that dependency as it was probably added by
accident. If it was really necessary, chosing the appropriate
split linux-firmware package (e.g. linux-firmware-brcm) is better
anyway (that has been changed recently in Alpine and is possible
now).
* Add a test case that makes sure we don't have firmware depends
without subpackages in device aports anymore.
pmbootstrap does dependency resolving on its own, and passes the list
of resolved packages to apk when we want it to install something. The
reason was outlined in #129:
> fixing #120: packages do not get updated in "pmbootstrap install"
> after they have been rebuilt. For this to work, we specify all
> packages explicitly for abuild, instead of letting abuild do the
> resolving.
This new PR fixes#1212 (which noted that all of these dependencies
were explicitly marked for installation) by doing the following:
1. All packages and dependencies get attached to the virtual package
.pmbootstrap instead of world
2. We install the packages (without depends) explcitly
3. .pmbootstrap gets removed, which means that all packages from 1.
stay installed, but are no longer marked as explicitly installed.
They will get removed automatically, when the depending packages get
removed.
In addition, the mechanism for replacing the package of locally built
packages with their full path, was broken and has been fixed in this
commit. This is necessary to update packages of the same version with
apk.
* device-*: add postmarketos-base to depends
* aportgen: add postmarketos-base to depends
* Add test case
* postmarketos-base: Don't depend on devicepkg
* msm-fb-refresher: Enable service in post-install
We don't use 'arch="all"' in our kernels anymore (that does not make sense,
since each arch needs its own kernel config). This patch fixes the menuconfig
code to work with multiple values in the "arch" field.
It lacked some hardware config, such as the USB switch.
With this change, I was able to SSH over the USB network.
I based this off of LineageOS 14.1's config file from its
build root. I think we were missing the secondary config
file
(TARGET_KERNEL_VARIANT_CONFIG := msm8930_serrano_eur_lte_defconfig
in BoardConfig.mk).