Commit Graph

118 Commits

Author SHA1 Message Date
Oliver Smith 1f8832c4cc install apk keys and binfmt data with setup.py 2018-09-17 10:06:57 +00:00
Oliver Smith 3ce00de710 Move aports into own repository (pmaports) 2018-09-05 05:57:38 +00:00
Oliver Smith f10ac0eadd deviceinfo: prepend manufacturer to device name 2018-08-23 00:44:46 +00:00
Oliver Smith 7cb471cb42 aports/kde: test case for framework versions
This commit adds a test case, which makes sure that the KDE framework
and plasma framework version are always the same.

Additional changes:
* APKBUILD parser parses the URL now (that's the best way I found to
  categorize the KDE aports in frameworks and other)
* Changed single quotes to double quotes in KDE APKBUILDs, so the
  parser doesn't include the single quotes in the parsed result
* Added the test case to the gitlab CI config
2018-08-02 20:33:20 +00:00
Oliver Smith 4844719b1d pmb: adjust to distcc 3.3 and wrap it with sshd
Overview:
Since Alpine updated to distcc 3.3 last week, pmbootstrap wasn't able to use
distcc for cross compilation anymore. It always falled back to running the
compiler in QEMU (which works, but is a lot slower). The reason for that is,
that distcc requires all compilers that are being used in a whitelist now.

This partially fixes CVE-2004-2687 in distccd, which allowed trivial remote
code execution by any process connecting to the distccd server. We only run
distccd on localhost, but still this can be used for privilege escalation of
sandboxed processes running on the host system (not part of pmbootstrap
chroots).

Because the CVE is only partially fixed (see the comment in
`pmb/chroot/distccd.py` for details), we make sure that only the building
chroots can talk to the distcc server by running distcc over ssh.

Details:
* Completely refactored `pmb/chroot/distccd.py` to run distcc over ssh
  * Store the running distcc server's arguments as JSON now, not as INI
* Make debugging distcc issues easy:
  * Set DISTCC_BACKOFF_PERIOD=0, so the distcc client will not ignore the
    server after errors happened (this masks the original error!)
  * New pmbootstrap parameters:
    * `--distcc-nofallback`: avoids falling back to compiling with QEMU and not
	   throwing an error
	* `--ccache-disable`: avoid ccache (when the compiler output is cached,
	  distcc does not get used)
  * `--verbose` prints verbose output of the distcc too
  * New test case, that uses the new pmbootstrap parameters to force
	compilation through distcc, and shows the output of distcc and distccd in
	verbose mode on error (as well as the log of sshd)
2018-07-26 09:47:23 +00:00
Oliver Smith d648794f7a test/aports: detect linux-firmware-* in depends
All firmware packages go into subpackages nowadays, so we have them
optional. The `test/test_aports.py` file has a test case, that makes
sure that there are no firmware packages in the `depends` of a device
package. So far we have only checked for `firmware-*` and
`linux-firmware`, this commit makes it check for `linux-firmware*` as
well.

In addition, the URL displayed in the "Please use kernel subpackages"
message has been fixed (points to devicepkg instead of deviceinfo now).
2018-07-25 20:13:37 +00:00
Oliver Smith 0e2c7507cd
Wiki CI: require all devices to be booting
In gitlab, there is no extra CI job running for commits in a merge
request (MR). This means, we can't run different code in a MR against
"master".

So instead of only checking whether all devices are booting if there's
a MR against master, always perform this check.

I've edited the message slightly, so it's clear that it's only required
to have the devices booting when making a merge request against master.
2018-07-22 23:19:02 +02:00
Oliver Smith 0d7802d7ef
s/system partition/rootfs: fix remaining mentions
Follow-up to !1373, where `pmbootstrap flasher flash_system` was
replaced with `pmbootstrap flasher flash_rootfs`. We still had used
terms like "system partition" in a lot of places.

This commit replaces it everywhere, so it's clear that we're talking
about the pmOS rootfs (which may or may not be installed to Android's
system partition).
2018-07-15 23:41:31 +02:00
Oliver Smith cca5c9aa30 pmb: fix test suite not running through twice
The test suite needed a `pmbootstrap shutdown` after running through,
before it could successfully run again.

Explanation:
This was caused by `test/test_pkgrel_bump.py`, which creates a
temporary work folder with every subfolder ("chroot_native",
"cache_apk_x86_64", ...) linked to the original work folder except for
the "packages" folder. At the end of the test case,
`pmbootstrap shutdown` gets executed and is expected to umount
everything as usual. But it does not umount anything because of the
symlinks, so `work/chroot_native/mnt/pmbootstrap-packages` points to
the fake packages folder of that test case, even after it is finished.

As a result, any test case that tries to access the packages folder in
the native chroot, will fail until `pmbootstrap shutdown` gets called.

Detailed Changes:
* Umount all folders inside the work folder, even if these are symlinks
* Remove obsolete reference to "disable timestamp based rebuilds" in a
  comment in `test/test_pkgrel_bump.py`
* Run `pmbootstrap work_migrate` and `pmbootstrap shutdown` at the
  beginning of `test/testcases_fast.sh`, in case the pkgrel_bump test
  case was aborted before it could properly shutdown and to make it
  more robust in general (user may have changed the mountpoints, work
  folder may need to be migrated)
2018-07-15 20:52:41 +00:00
Oliver Smith 0adc6a1f4b Testsuite: fail after first error
This makes debugging much easier, because the relevant log messages
before the crash can always be found at the bottom of the log files,
and not somewhere in the middle.
2018-07-15 20:37:02 +00:00
Oliver Smith 8268dc0e3d pmbootstrap: kill process if silent for 5 minutes (rewrite logging) 2018-07-14 01:13:28 +00:00
Oliver Smith 796668d7df More obvious test failure messages for QEMU test
If the test case couldn't connect to the VM with SSH, it used to fail
with a Python error of accessing the 'missing' variable without having
it defined.

With this commit, it prints out 'Could not connect to the VM via SSH'
instead.
2018-07-11 19:37:06 +00:00
Oliver Smith 74e103d1f2 static code analysis: make it pass flake8 3.5.0
This commit adds two new ignores to the flake8 call:
* W504 line break after binary operator
* W605 invalid escape sequence 'x'

Without these, newer flake8 versions, such as the one shipped by Arch
Linux, don't run through on the code base (which leads to not running
it at all locally, only in CI).

Additional changes:
* Don't give the exception instance of subprocess.CalledProcessError a
  name, so it does not trigger F841
* Dont' advise to run "autopep8" anymore, because that will likely
  touch unrelated code.

For reference, here is the list of error codes:
https://pycodestyle.readthedocs.io/en/latest/intro.html#error-codes
2018-07-11 18:58:29 +00:00
clayton craft 023fbf53dc gitlab ci: restructure 'tests' stage 2018-07-08 23:06:00 +00:00
Oliver Smith d4f4ea8488
pmb: add -f (force) parameter to bootimg_analyze
We are analyzing the `boot.img` with `file` before we send it to
`unpackbootimg`. File does not recognize all kinds of `boot.img` files,
which `unpackbootimg` can extract, so we need a way to skip this check.

Details:
* Add `-f` parameter, continues extraction with a warning if the file
  seems to be invalid
* Tell the user that `-f` can be used if the `boot.img` is invalid and
  it's not specified
* Consistent spelling of `boot.img` instead of `bootimg` in messages

Fixes #1608
2018-07-06 21:57:18 +02:00
Oliver Smith 0854aa1dc4
Remove github.com references
* Travis and Coveralls badges
* aports: instead of <https://github.com/postmarketOS>, use
  <https://postmarketos.org>
* References to full URLs to issues and pull requests replaced with
  a hash and the number
* grsec check: simplify error message, remove link to github issue
  (nobody is using that anymore anyway)
2018-06-30 09:00:48 +02:00
Clayton Craft 256914c1a0
Implement support for gitlab CI
This implements support for the gitlab CI system. Wiki, static analysis,
and build tests are implemented.
2018-06-29 23:16:41 -07:00
steamport 30df0725ca menuconfig: add gconfig/xconfig support (#1509)
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
  (with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
  pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
  aports ([skip ci] because it would rebuild all kernels and run out of
  time). Alpine has dropped this as well, and it wouldn't work with the
  new code anyway.
2018-06-09 06:52:24 +00:00
George Hopkins 5edabd4d17 pmbootstrap: handle checkdepends (#1533) 2018-06-04 19:07:52 +00:00
Daniele Debernardi dfde37c132 Add Raspberry Pi device (#1501) 2018-05-29 19:33:40 +00:00
Oliver Smith 47cb5a7cf1
upstream compatibility test: check against x86_64
armhf tends to be outdated, just like it is right now. The Alpine
aports have binutils-2.30-r2, and the x86_64 version has that version
as well. But the armhf version is still at -r1, which caused the test
case to fail.
2018-05-20 22:04:44 +02:00
Oliver Smith 9c83cdcea1
Display wiki link on dependency error (#1410) 2018-04-23 20:51:06 +00:00
Oliver Smith a77a0dcc32
Add envkernel.sh: instant mainlining environment (#1424)
Changes:
* `helpers/envkernel.sh`:
  * installs everything needed for kernel compilation in the native
    chroot
  * mounts the kernel source to `/mnt/linux` inside the chroot
  * creates `/mnt/linux/.output` and chowns it to the `pmos` user, that
    folder will be used for the kernel build output
  * sets up aliases for `make`, `pmbootstrap`, `pmbroot`, `kernelroot`
* new action `pmbootstrap work_migrate`: does the interactive work
  folder migration if necessary, otherwise it doesn't output anything
  * when calling this first, we can safely use all other commands
    non-interactively without showing the output

Benefits:
* Fast setup (especially for people who are new to kernel
  compilation
  * No need to figure out distribution specific package names
    (cross compilers!)
  * No need to do a test build just to verify that the right
    packages are installed
* Less error prone
  * The right dependencies are always installed
  * `ARCH` and `CROSS_COMPILE` variables always get set automatically
    and based on `deviceinfo_arch`
  * If the build environment is broken for some reason, just zap and
    start over
* Easy to reproduce problems

Notes:
* `make menuconfig` works as well
* Sourcing was tested with `zsh`, `bash` and `fish`, it should be easy to
  extend for other shells
2018-04-19 21:27:38 +00:00
Oliver Smith b1b5acd79d
Fix test/test_build_package.py
Tested with Travis on an extra branch before pushing to master this
time.
2018-04-19 02:29:48 +02:00
Oliver Smith ca20ead505
"pkgrel_bump --auto": Handle subpackages properly (#1388)
`pmbootstrap pkgrel_bump --auto` automatically increases the pkgrel for
packages linking against libraries, which don't exist anymore (because
the soname has been changed). The feature is explained in detail in

The previous implementation did not detect soname breakage, when a
subpackage linked against a certain library, but the main package
did not (e.g. `qt5-qtbase-mysql` and `qt5-qtbase`). This was, because
we iterated over the aports/* to find the packages to be checked.

To fix this, we are iterating over the packages found in the APKINDEX
files instead (of both the locally compiled packages and the downloaded
index from the pmOS mirror).

Details:
* `pmb/helpers/pkgrel_bump.py`:
  * Rewrite `auto_apkindex_package()` to act upon a given parsed
    `aport` and `apk` (from the index) instead of finding the `apk`
    dict by itself (we need it earlier anyway).
  * Rewrite `auto()` to iterate over APKINDEX files instead of aports
    * Skip packages already found, so the `pkgrel` does not get
      increased multiple times when the same package was found in
      multipe index files.
* Put the package name at the beginning of the log messages to make
  them more readable
* testdata: Create a new `testsubpkg` aport, where only the subpackage
  links against `testlib`
* Adjust testing code to test everything with `testsubpkg` as well.

NOTE: This makes the command a bit slower, but we could improve
performance again by smart caching of `pmb.parse.apkindex.depends()`.
This could come in a future PR, the important part here is that the
command is bug-free again with this fix.
2018-04-15 21:34:40 +00:00
BK d79e13d226 Add helpers/envsetup.sh (#1390)
Usage:
$ source helpers/envsetup.sh
It sets up the aliases `pmbootstrap` and `pmbroot`.
The latter changes the directory to the pmbootstrap folder.
2018-04-08 12:58:10 +00:00
Oliver Smith b66b5dcc34
pmbootstrap init: kernel selection / remove linux-pmos-lts (#1363)
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
  for now. The kernel isn't used right now, and we save lots of
  maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
  `"downstream", "mainline", "stable"` (downstream is always
  `linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
  has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
  the `deviceinfo_dtb` file is missing (because we expect it to be
  missing for downstream kernels)
* device-sony-amami:
  * add kernel subpackages for downstream, mainline
  * set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
  * don't mix specifying kernels in depends *and* subpackages
  * 1 kernel in depends is maximum
  * kernel subpackages must have a valid name
  * Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
  `--ignore-depends` can be used with device packages to avoid building
  the kernel. The idea was to make the transition easier after a change
  we did months ago, and now the kernel doesn't always get built before
  building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
  * Add autoinstall=True to kernel_flavors_installed(). When the flag
    is set, the function makes sure that at least one kernel for the
    device is installed.
  * Remove kernel_flavor_autodetect() function, wherever it was used,
    it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
  kernel_flavors_installed() takes care of that now.
2018-04-03 23:50:09 +00:00
Oliver Smith 03f74565fb
Travis: Check for device documentation (#1369)
Add a simple script, that checks whether all device's codenames can be
found in the device overview in the wiki. This way we're always sure
that every device is documented in the wiki's device table.
2018-03-28 21:28:26 +00:00
Daniele Debernardi 147082ec58 pmbootstrap init: Ask for hostname, default: device name (#1327)
* Save "" (empty string) in the user's config as hostname if the user
  let it default to the name of the device. That way, when the device
  gets changed, the user won't get the old device's name as hostname
  by accident.
* Add a test case
2018-03-17 18:41:41 +00:00
Oliver Smith 2a096e6a56
Qemu test case: wait dynamically until booted (#1330)
We have a test case, that performs full installations with with XFCE4
and Plasma Mobile. After each installation is through, we run
"pmbootstrap qemu" and connect to the VM via SSH to check for specific
running processes.

Before this commit, we assumed that the boot takes 90 seconds at most
and then the processes should exist. But sometimes the boot is faster
(so Travis waits just doing nothing) and sometimes it is slower (which
causes the test case to fail), all depending on the current workload
of the Travis CI servers. Now we try to find the processes we're
interested in each second for 3 minutes until we either succeed or
reach the time limit.
2018-03-14 21:24:13 +00:00
Oliver Smith f75ac13a4f
Travis CI: folded output/default tracebacks (#1331)
* Moved the `script` and `after_failure` sections of `.travis.yml`
  to extra files in a new `.travis/` folder
* Copy paste `.travis/common.sh` from Alpine's aport. They have neat
  formatting functions in there for folding
* Add a folding block around each script that gets called
* Use native tracebacks in pytest. These are the same as we see when
  pmbootstrap crashes usually, instead of printing out the entire
  function that causes an error. I think this makes the error output
  easier to read.
2018-03-14 20:32:47 +00:00
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00
Oliver Smith 3666388619
Properly escape commands in pmb.chroot.user() (#1316)
## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.

This is not a security issue, as we don't pass strings from untrusted
input to this function.

## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`

In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`

## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`

So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.

Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.

## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.

## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
  list and environment variables as dict, and creates a properly escaped
  flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
  executed (in addition to the simplified form we always write down for
  readability)
* `test_shell_escape.py`: verify that the command passed by reference
  has not been modified, add a new test for strings with spaces, add
  tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
  variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
  so running ncurses applications like `menuconfig` and `nano` works out of
  the box
2018-03-10 22:58:39 +00:00
Oliver Smith af4b817c21
Skip virtual packages when parsing APKINDEX (#1278)
Since PR #1247 we are using the virtual package `.pmbootstrap` to mark
packages as not explcitly installed. In case `pmbootstrap` doesn't
finish the installation because of a bug, the `.pmbootstrap` virtual
package may not get uninstalled.

As virtual packages don't have the "timestamp" attribute set in the
package database (which indicates when the package was built), the
APKINDEX parser fails to parse them.

Changes:
* pmb.parse.apkindex.parse_next_block(): don't require the "timestamp"
  attribute to be set (but the arch attribute instead, which is always
  present)
* pmb.parse.apkindex.parse(): when a block does not have a `timestamp`
  attribute, skip it, because it must be a virtual package.
* add test cases for both functions with a package database that
  contains a virtual package.
2018-03-10 13:15:30 +00:00
Daniele Debernardi 94306b51ee deviceinfo: remove external_disk_install and external_disk, use external_storage instead (#1301)
* deviceinfo: remove external_disk_install and external_disk, use external_storage instead
* Complain when external_disk* is used
2018-03-07 22:35:02 +00:00
Oliver Smith af6913ab06
Fix build --src for foreign-arch chroots (#1282)
The `suffix` argument was not specified in chroot commands executed in
`pmb.build._package.override_source()`. Because of that, it was not
possible to use "build --src" when compiling in a non-native chroot,
for example:

```
$ pmbootstrap build hello-world --arch=armhf
...
(native) % rm /tmp/APKBUILD.append
rm: can't remove '/tmp/APKBUILD.append': No such file or directory
```
2018-03-05 21:04:28 +00:00
Oliver Smith 99127111a1
Only download APKINDEX for relevant architectures (#1201)
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.

### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
  * Default is updating all arches where the APKBUILD files exist
  * Add existing_only parameter
  * Return True when files have been downloaded
  * Properly print which arches will be updated
  * Print update reason only in verbose log
  * Add and improve comments
* pmb.parse.arguments(), update action:
  * Add --non-existing parameter
  * Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
  * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
Oliver Smith ab2fde4fef
Make nonfree depends optional part 2: device pkgs (#1268)
* Add nonfree_firmware subpackage to all devices, that depend on
  nonfree firmware.
* Some packages were depending on `linux-firmware`, but without having
  Wifi working. Removed that dependency as it was probably added by
  accident. If it was really necessary, chosing the appropriate
  split linux-firmware package (e.g. linux-firmware-brcm) is better
  anyway (that has been changed recently in Alpine and is possible
  now).
* Add a test case that makes sure we don't have firmware depends
  without subpackages in device aports anymore.
2018-02-26 22:11:44 +00:00
Oliver Smith 948b5af09b
device packages: depend on postmarketos-base (#1258)
* device-*: add postmarketos-base to depends
* aportgen: add postmarketos-base to depends
* Add test case
* postmarketos-base: Don't depend on devicepkg
* msm-fb-refresher: Enable service in post-install
2018-02-25 18:40:54 +00:00
Oliver Smith ad5a0d4294
Make proprietary drivers optional (1/2): pmbootstrap changes (#1254)
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.

Changes:
* New config file options nonfree_firmware and nonfree_userland, which
  we ask for during "pmbootstrap init" if there are non-free components
  for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
  packages are called $pkgname-nonfree-firmware and
  $pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
  subpackages. Parsing that is implemented in
  pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
  the regular APKBUILD parsing, as this would need a change in the
  output format, and it is a lot *less* code if done like in this
  commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
  pmb/install/install.py to _install.py: needed to call the function in
  the usual way (e.g. pmb.parse.apkbuild()) but still being able to
  test the individual functions from these files in the test suite.
  We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
  packages that will be installed, based on the user's choice in
  "pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
  refactored test/test_questions.py to have multiple functions for
  testing the various questions / question types from
  "pmbootstrap init" instead of having it all in one big function.
  This allows to use another aport folder for testing the new
  non-free related questions in init.
2018-02-24 21:49:10 +00:00
Oliver Smith db5e69630e
Index parser: support multiple package providers (#1202)
* The APKINDEX parser used to return a dictionary with one package for
  a given package name. This works for the installed packages database,
  because there can only be one provider for a package. But when
  parsing packages from binary repositories, we need to support
  multiple providers for one package. It is now possible to get a
  dictionary with either multiple providers, or just a single provider
  for each package.
* Dependency parsing logic has been adjusted, to support multiple
  providers. For multiple providers, the one with the same package
  name as the package we are looking up is prefered. If there is none
  (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
  that will be installed anyway, and after that packages that are
  already installed. When all else fails, it just picks the first one
  and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
  pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
  that pmbootstrap should build if neccessary, and install in the
  chroot. This can be used to quickly test the depencency resolution
  of pmbootstrap without doing a full "pmbootstrap install".

Fixes #1122.
2018-02-20 19:52:28 +00:00
Oliver Smith 0f371e426f
pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
2018-02-19 22:04:01 +00:00
Oliver Smith ceac21766f
check_checksums.py: Non-zero exit on build error
Fixes #1124.
2018-02-14 00:12:46 +01:00
Oliver Smith 3fe75ddb56
Fix screwed up progress bars in Travis output (#1213)
Right now, they appear on screen when using --details-to-stdout. This
does not work well with Travis CI and screws up the log.

Disabling the progress bars in abuild works just like Alpine does it in
their Travis CI script: Exporting SUDO_APK as
"abuild-apk --no-progress" instead of "abuild-apk".

test_check_checksums.py: Run "pmbootstrap build_init" before building
any packages, so it is a bit less verbose (downloading the APKINDEX
files etc.). Later we run the build init code again (because we use
--strict while building the packages), but then the APKINDEX files
are already present. So overall the log is a bit shorter before the
building starts. (It is still logged to the logfile, which gets
printed on error anyway.)
2018-02-09 18:43:58 +00:00
Oliver Smith 5e85d72ca0
Testsuite: Run UIs in Qemu and check running processes and more (#982)
* Testsuite: Run UIs in Qemu and check running processes (and other changes)

* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
  it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
  built before running tests (otherwise tests would hang without any output
  while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
  plasma-mobile currently, easy to extend), runs it via Qemu and checks the
  running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
  our source tree, so we don't need to clone their git repo anymore. Now it
  is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.

* Various fixes
* Build device-packages for the device arch and don't raise an
  exception, but print a note if --ignore-depends is not specified
  and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
  check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
  completion
* Exit with 1 when showing usage in check_checksums.py
2018-02-02 00:16:29 +00:00
Oliver Smith e8c27795a8
Remove rest of 'pmbootstrap challenge' left overs (#1173)
Follow up to #1162.

* `pmb.build.buildinfo()`: Used to record the build environment. It is
  flawed because it scans the repo APKINDEX files instead of using the
  actually installed packages list. When it was implemented we were not
  able to do the latter. After this is removed, `pmb.parse.depends` can
  be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
  used exclusively by `pmb.build.buildinfo()`, to learn about which
  files have been changed in the local repository folder after a
  package was built. The idea was, that we could find subpackages that
  way. But this information is present in the installed package list as
  well, which is a much cleaner approach.
2018-02-01 22:03:21 +00:00
Oliver Smith 3c59126bc1
Remove timestamp based rebuilds (#1174)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:

    pmbootstrap build --force hello-world

Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close #1167, close #1156, close #1023 and
close #985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
2018-01-28 23:27:33 +00:00
Oliver Smith 566da1718d
version.py: fix validation of 6.0_0002 (#1160) 2018-01-25 18:23:55 +00:00
Oliver Smith 0479031f7e
Remove lots of legacy 'challenge' code (#1162)
Preparation for #1122.

* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
  an exception when two entries in the apkindex provided the same package.
  Turns out this is *not* invalid after all, two packages can provide the same
  soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
  listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
  which was a dead end. This code uses the broken strict feature.
2018-01-25 18:08:39 +00:00
Oliver Smith 34d46b2fe3
static_code_analysis.sh: Find CHANGEMEs in APKBUILDs (#1161) 2018-01-24 21:31:07 +00:00