## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.
This is not a security issue, as we don't pass strings from untrusted
input to this function.
## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`
In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`
## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`
So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.
Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.
## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.
## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
list and environment variables as dict, and creates a properly escaped
flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
executed (in addition to the simplified form we always write down for
readability)
* `test_shell_escape.py`: verify that the command passed by reference
has not been modified, add a new test for strings with spaces, add
tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
so running ncurses applications like `menuconfig` and `nano` works out of
the box
Since PR #1247 we are using the virtual package `.pmbootstrap` to mark
packages as not explcitly installed. In case `pmbootstrap` doesn't
finish the installation because of a bug, the `.pmbootstrap` virtual
package may not get uninstalled.
As virtual packages don't have the "timestamp" attribute set in the
package database (which indicates when the package was built), the
APKINDEX parser fails to parse them.
Changes:
* pmb.parse.apkindex.parse_next_block(): don't require the "timestamp"
attribute to be set (but the arch attribute instead, which is always
present)
* pmb.parse.apkindex.parse(): when a block does not have a `timestamp`
attribute, skip it, because it must be a virtual package.
* add test cases for both functions with a package database that
contains a virtual package.
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.
That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).
Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
list of blocks, instead of the dict with removed duplicates with
lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
what they are doing now.
```
short long (old) long (new)
-p --packages --pkgs-local
-m --mismatch-bins --pkgs-local-mismatch
-o, --old-bins --pkgs-online-mismatch
```
Use case: `mkbootimg` provides the `unpackbootimg` package. When
running `pmb.chroot.apk.install(args,"unpackbootimg")`, it was not
able to properly build the package.
Reproducing the error:
```
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/mkbootimg*
pmbootstrap index
pmbootstrap --mirror-pmOS="" chroot --add=unpackbootimg
```
Or alternatively (simpler but less illustrative):
```
pmbootstrap build unpackbootimg --force
```
Specifying alternate subpackages in the provides variable of the main
package means that the main package itself is providing an alternate for
the subpackages (e.g. gtk+2.0-maemo provides an alternate gtk+2.0-dev).
The provides variable must be set for each subpackage. Fixes#1284.
* Add charging-sdl package
* Include charging-sdl into the initramfs-extra
* [initramfs] Detect charging mode and use triggerhappy to start
charging-sdl when the power key is pressed
The `suffix` argument was not specified in chroot commands executed in
`pmb.build._package.override_source()`. Because of that, it was not
possible to use "build --src" when compiling in a non-native chroot,
for example:
```
$ pmbootstrap build hello-world --arch=armhf
...
(native) % rm /tmp/APKBUILD.append
rm: can't remove '/tmp/APKBUILD.append': No such file or directory
```
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.
### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
* Default is updating all arches where the APKBUILD files exist
* Add existing_only parameter
* Return True when files have been downloaded
* Properly print which arches will be updated
* Print update reason only in verbose log
* Add and improve comments
* pmb.parse.arguments(), update action:
* Add --non-existing parameter
* Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
* Inform about --non-existing if no APKBUILDs have been updated
Since it works for linux-postmarketos-stable, I've simply copied over
the config, and ran "pmbootstrap menuconfig" on it again. Then it
worked again in Qemu.
* Add NetworkManager and PulseAudio applets to Plasma
* This crashes plasma mobile on armhf (where it is not usable due
to performance problems anyway), except on mainlined kernels
with hardware accelerated graphics. This is mentioned in the
pkgdesc of postmarketos-ui-plasma-mobile now.
In case the user does not specify for which arch packages should be
built with `pmbootstrap build`, we detect it automatically.
Previous logic was, that if the APKBUILD's arch is "all" or "noarch",
then prefer the native arch, and otherwise use the first one in the
list of available arches.
New behavior is, that we also check if the list of possible arches
contains the native arch (and if that fails, the device arch). If that
is the case, we return the native/device arch instead of the first one
in the list.
### Use case
The arch from `gcc-armhf` and similar packages (as generated by
`pmbootstrap aportgen`) used to be "all", but is nowadays a specific
list of arches. This means, that after updating the `gcc-armhf` and
`gcc-aarch64` packages, and calling `pmbootstrap build gcc-armhf`,
it will try to build `gcc-armhf` for `aarch64` instead of the native
architecture, because that is the first one listed.
And since compiling to `aarch64` requires `gcc-aarch64`, it will build
that for the native architecture first.
So you're asking for `gcc-armhf` and it compiles `gcc-aarch64`, which
is very confusing (see #1272).
When building device packages, the postmarketos-mkinitfs package gets
installed as dependency of postmarketos-base. It must not try to
create an initramfs at this point, when there is not deviceinfo file.
We build the initramfs during the installation, so it's fine.
Added [skip ci] because linux kernels and KDE updates are currently
getting built for the binary repository (so Travis couldn't finish
anyway).
* Add nonfree_firmware subpackage to all devices, that depend on
nonfree firmware.
* Some packages were depending on `linux-firmware`, but without having
Wifi working. Removed that dependency as it was probably added by
accident. If it was really necessary, chosing the appropriate
split linux-firmware package (e.g. linux-firmware-brcm) is better
anyway (that has been changed recently in Alpine and is possible
now).
* Add a test case that makes sure we don't have firmware depends
without subpackages in device aports anymore.
pmbootstrap does dependency resolving on its own, and passes the list
of resolved packages to apk when we want it to install something. The
reason was outlined in #129:
> fixing #120: packages do not get updated in "pmbootstrap install"
> after they have been rebuilt. For this to work, we specify all
> packages explicitly for abuild, instead of letting abuild do the
> resolving.
This new PR fixes#1212 (which noted that all of these dependencies
were explicitly marked for installation) by doing the following:
1. All packages and dependencies get attached to the virtual package
.pmbootstrap instead of world
2. We install the packages (without depends) explcitly
3. .pmbootstrap gets removed, which means that all packages from 1.
stay installed, but are no longer marked as explicitly installed.
They will get removed automatically, when the depending packages get
removed.
In addition, the mechanism for replacing the package of locally built
packages with their full path, was broken and has been fixed in this
commit. This is necessary to update packages of the same version with
apk.
* device-*: add postmarketos-base to depends
* aportgen: add postmarketos-base to depends
* Add test case
* postmarketos-base: Don't depend on devicepkg
* msm-fb-refresher: Enable service in post-install
We don't use 'arch="all"' in our kernels anymore (that does not make sense,
since each arch needs its own kernel config). This patch fixes the menuconfig
code to work with multiple values in the "arch" field.
It lacked some hardware config, such as the USB switch.
With this change, I was able to SSH over the USB network.
I based this off of LineageOS 14.1's config file from its
build root. I think we were missing the secondary config
file
(TARGET_KERNEL_VARIANT_CONFIG := msm8930_serrano_eur_lte_defconfig
in BoardConfig.mk).
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.
Changes:
* New config file options nonfree_firmware and nonfree_userland, which
we ask for during "pmbootstrap init" if there are non-free components
for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
packages are called $pkgname-nonfree-firmware and
$pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
subpackages. Parsing that is implemented in
pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
the regular APKBUILD parsing, as this would need a change in the
output format, and it is a lot *less* code if done like in this
commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
pmb/install/install.py to _install.py: needed to call the function in
the usual way (e.g. pmb.parse.apkbuild()) but still being able to
test the individual functions from these files in the test suite.
We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
packages that will be installed, based on the user's choice in
"pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
refactored test/test_questions.py to have multiple functions for
testing the various questions / question types from
"pmbootstrap init" instead of having it all in one big function.
This allows to use another aport folder for testing the new
non-free related questions in init.