Commit Graph

337 Commits

Author SHA1 Message Date
Daniele Debernardi b7b1fe9ef7 flasher: install depends of the method specified in the arguments (#1393) 2018-04-04 23:48:39 +00:00
Oliver Smith b66b5dcc34
pmbootstrap init: kernel selection / remove linux-pmos-lts (#1363)
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
  for now. The kernel isn't used right now, and we save lots of
  maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
  `"downstream", "mainline", "stable"` (downstream is always
  `linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
  has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
  the `deviceinfo_dtb` file is missing (because we expect it to be
  missing for downstream kernels)
* device-sony-amami:
  * add kernel subpackages for downstream, mainline
  * set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
  * don't mix specifying kernels in depends *and* subpackages
  * 1 kernel in depends is maximum
  * kernel subpackages must have a valid name
  * Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
  `--ignore-depends` can be used with device packages to avoid building
  the kernel. The idea was to make the transition easier after a change
  we did months ago, and now the kernel doesn't always get built before
  building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
  * Add autoinstall=True to kernel_flavors_installed(). When the flag
    is set, the function makes sure that at least one kernel for the
    device is installed.
  * Remove kernel_flavor_autodetect() function, wherever it was used,
    it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
  kernel_flavors_installed() takes care of that now.
2018-04-03 23:50:09 +00:00
Oliver Smith 4d8afc4aa5
Fix: /var/cache/distfiles writable by everyone (#1329)
As noted in commit 255c715624
`/var/cache/distfiles` is writable by everyone. It is supposed to be
writable only by `root` and by the `abuild` group (in which we put the
`pmos` user already for building packages).

Changes:
* `pmb.build.init()`: make `/var/cache/distfiles` writable only by
  members of the `abuild` group (and root)
* Increase workfolder version to 2
* Add migration code that fixes the permissions for existing work
  folders
* Refactor the migration code a bit to make this possible
2018-03-30 21:46:31 +00:00
Oliver Smith 6bb8444fef
"...flasher flash_rootfs" instead of "..._system" (#1373)
* Change `pmbootstrap flasher flash_system` command to
  `pmbootstrap flasher flash_rootfs`
* The old command still works, but all references have been changed to
  the new command
* Remove obsolete `pmbootstrap flasher export` (that was changed to
  `pmbootstrap export` a few months ago)
* Update `README.md` and ZSH auto completion
* Change the description of the generated rootfs image (not talking
  about a system image anymore, mention that it has subpartitions)
* Better description of `pmbootstrap flasher flash_rootfs --partition`
2018-03-30 01:11:20 +00:00
Lionel Duboeuf b0c05436a9 pmbootstrap install: print 'flash_system' before 'flash_kernel' (#1372) 2018-03-30 00:52:38 +00:00
Oliver Smith 491ee6b1c1
Disable metadata_csum when generating ext4 rootfs (#1367)
Some downstream kernels don't support this, and this recently became
the default in e2fsprogs.
2018-03-28 18:00:18 +00:00
Oliver Smith 33a09ea1c7
Fix dependency resolver being stuck after reboot (#1359)
When the native arch (e.g. `x86_64`) `APKINDEX` files are outdated, and
`pmbootstrap` gets instructed to build a linux package for a foreign
arch, then the `APKINDEX` cache did not get used anymore for the
current session. This means that every lookup of a package in an
`APKINDEX` caused the whole `APKINDEX` file to get parsed again instead
of using the cached version. This slowed it down so much that it felt
like `pmbootstrap` was looping forever.

How this happens in detail:
* Whenever pmbootstrap parses an `APKINDEX`, it fills up the
  `args.cache["apkindex"]` dict with the parsed information and the
  last modified date of the file.
* `pmbootstrap` checks the last modified date of the `APKINDEX` files
  and updates them if they are older than 4 hours.
* When the bug appeared, then the cache was already filled up, then an
  update happened and then `pmbootstrap` tried to read from the cache.
* So when reading from the `APKINDEX`, the cache gets ignored because
  the last modified date is different.
* Up to this commit, the cache does not get deleted and filled up
  again!

How to test:
Try these commands once without this commit, and then with this commit
applied:
```
$ sudo touch -m -t 201801010000 \
    ~/.local/var/pmbootstrap/cache_apk_x86_64/APKINDEX.*
$ pmbootstrap -v build linux-postmarketos-mainline --arch=armhf
```

Without the patch, you can see in `pmbootstrap log` that it is
resolving the dependencies properly, but very slowly. With the patch
the resolving happens almost instantly.
2018-03-24 16:31:31 +00:00
Oliver Smith 1ed51f83f1
Package resolving: Don't fail on fuzzy versions (#1355)
When parsing the depends of entries in the APKINDEX file, we ignore
all operators (<, =, >). (This is enough for our use case, was we only
do the dependency resolving to check which packages need to be built
and `apk` does the dependency resolving again before installing
anything).

We did not ignore the ~ character for fuzzy version compares, this is
fixed with this commit.
2018-03-24 16:29:32 +00:00
Daniele Debernardi 147082ec58 pmbootstrap init: Ask for hostname, default: device name (#1327)
* Save "" (empty string) in the user's config as hostname if the user
  let it default to the name of the device. That way, when the device
  gets changed, the user won't get the old device's name as hostname
  by accident.
* Add a test case
2018-03-17 18:41:41 +00:00
Oliver Smith 5ea00e0862
pmbootstrap newapkbuild: Properly parse arguments (#1320)
* pmbootstrap newapkbuild: Properly parse arguments

The `pmbootstrap newapkbuild` action wraps Alpine's `newapkbuild`. We
used to directly pass all arguments to `newapkbuild` without verifying
in Python whether they make sense or not. However, as `newpakbuild`
doesn't do strict sanity checks on the arguments, it is easy to end up
with unexpected behavior when using the command for the first time.

For example, `newapkbuild` allows either specifying a PKGNAME or SRCURL
as last parameter, and also allows setting a PKGNAME with the `-n`
parameter. It only makes sense to use that option when passing a
SRCURL.

With this commit, we duplicate the optins that should be passed through
to `newapkbuild` and use argparse to fully sanitize the options and
display a help page (`pmbootstrap newapkbuild -h`) that is consistent
with the other help pages.

Details:
* The `-f` (force) flag does not get passed through anymore. Instead we
  use it in Python to skip asking if an existing aport should be
  overwritten (the aports are outside of the chroot, so `newapkbuild`
  can't handle it in a way that makes sense for pmbootstrap).
* Output of `newapkbuild` gets redirected to the log file now, as we
  don't need it to display a help page.
* Don't verify the pkgver while creating the new APKBUILD. When passing
  a SRCURL, the pkgver gets extracted from the end of the URL and may
  not have a valid format yet (but we want the APKBUILD anyway).
* Stored options passed through in `pmb/config/__init__.py` and use it
  in both `pmb/parse/arguments.py` and `pmb/helpers/frontend.py`.
* Only allow `-n` with SRCURL
* The postmarketOS aports folder gets specified with `--folder` now.
  That way the generated help page is much closer to the original one
  from `newapkbuild`. The default is `main`.
* Made the package type flags (CMake, autotools, ...) exclusive so only
  one of them can be specified
2018-03-15 21:42:34 +00:00
Oliver Smith 255c715624
Fix aportgen without initializing buildenv
After initializing the build environment, the cache_distfiles folder
currently is writable by everyone (which is not ideal, fix following
soon). The aportgen code for `busybox-static-*` and `musl-*` copies
the foreign arch `.apk` file to the distfiles, but it executes this
action as regular user and not as root. This only works as long as
build initialization ran before (which may not be the case on Travis
and expecting this to run before is a bug in general).

With this commit, the copy action gets executed as root, so it works
in any case. I'm commiting this directly (without a PR), because it
is a super simple fix and it unblocks our continuous integration.

Local testing:
$ pmbootstrap -y zap
$ sudo rm -r ~/.local/var/pmbootstrap/cache_distfiles
$ pmbootstrap aportgen musl-armhf
2018-03-12 17:17:32 +01:00
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00
Oliver Smith 3666388619
Properly escape commands in pmb.chroot.user() (#1316)
## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.

This is not a security issue, as we don't pass strings from untrusted
input to this function.

## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`

In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`

## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`

So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.

Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.

## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.

## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
  list and environment variables as dict, and creates a properly escaped
  flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
  executed (in addition to the simplified form we always write down for
  readability)
* `test_shell_escape.py`: verify that the command passed by reference
  has not been modified, add a new test for strings with spaces, add
  tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
  variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
  so running ncurses applications like `menuconfig` and `nano` works out of
  the box
2018-03-10 22:58:39 +00:00
Oliver Smith dc624aecea
Mount tmpfs in chroot's /dev folders (#1317) 2018-03-10 14:24:16 +00:00
Oliver Smith af4b817c21
Skip virtual packages when parsing APKINDEX (#1278)
Since PR #1247 we are using the virtual package `.pmbootstrap` to mark
packages as not explcitly installed. In case `pmbootstrap` doesn't
finish the installation because of a bug, the `.pmbootstrap` virtual
package may not get uninstalled.

As virtual packages don't have the "timestamp" attribute set in the
package database (which indicates when the package was built), the
APKINDEX parser fails to parse them.

Changes:
* pmb.parse.apkindex.parse_next_block(): don't require the "timestamp"
  attribute to be set (but the arch attribute instead, which is always
  present)
* pmb.parse.apkindex.parse(): when a block does not have a `timestamp`
  attribute, skip it, because it must be a virtual package.
* add test cases for both functions with a package database that
  contains a virtual package.
2018-03-10 13:15:30 +00:00
Oliver Smith e72afc53fd
pmbootstrap zap -m: delete outdated packages too (#1306)
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.

That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).

Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
  list of blocks, instead of the dict with removed duplicates with
  lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
  what they are doing now.
  ```
  short  long (old)         long (new)
  -p     --packages         --pkgs-local
  -m     --mismatch-bins    --pkgs-local-mismatch
  -o,    --old-bins         --pkgs-online-mismatch
  ```
2018-03-10 12:08:02 +00:00
Oliver Smith 3510a4868f
Fix building packages by provides name (#1303)
Use case: `mkbootimg` provides the `unpackbootimg` package. When
running `pmb.chroot.apk.install(args,"unpackbootimg")`, it was not
able to properly build the package.

Reproducing the error:
```
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/mkbootimg*
pmbootstrap index
pmbootstrap --mirror-pmOS="" chroot --add=unpackbootimg
```

Or alternatively (simpler but less illustrative):
```
pmbootstrap build unpackbootimg --force
```
2018-03-08 21:30:55 +00:00
Daniele Debernardi 94306b51ee deviceinfo: remove external_disk_install and external_disk, use external_storage instead (#1301)
* deviceinfo: remove external_disk_install and external_disk, use external_storage instead
* Complain when external_disk* is used
2018-03-07 22:35:02 +00:00
Oliver Smith af6913ab06
Fix build --src for foreign-arch chroots (#1282)
The `suffix` argument was not specified in chroot commands executed in
`pmb.build._package.override_source()`. Because of that, it was not
possible to use "build --src" when compiling in a non-native chroot,
for example:

```
$ pmbootstrap build hello-world --arch=armhf
...
(native) % rm /tmp/APKBUILD.append
rm: can't remove '/tmp/APKBUILD.append': No such file or directory
```
2018-03-05 21:04:28 +00:00
Oliver Smith ac2a0b2749 convert dict.keys() to list() (#1290) 2018-03-04 21:10:27 +01:00
Oliver Smith 99127111a1
Only download APKINDEX for relevant architectures (#1201)
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.

### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
  * Default is updating all arches where the APKBUILD files exist
  * Add existing_only parameter
  * Return True when files have been downloaded
  * Properly print which arches will be updated
  * Print update reason only in verbose log
  * Add and improve comments
* pmb.parse.arguments(), update action:
  * Add --non-existing parameter
  * Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
  * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
Oliver Smith 0f15951023
Always prefer native/device arch when building (#1277)
In case the user does not specify for which arch packages should be
built with `pmbootstrap build`, we detect it automatically.

Previous logic was, that if the APKBUILD's arch is "all" or "noarch",
then prefer the native arch, and otherwise use the first one in the
list of available arches.

New behavior is, that we also check if the list of possible arches
contains the native arch (and if that fails, the device arch). If that
is the case, we return the native/device arch instead of the first one
in the list.

### Use case

The arch from `gcc-armhf` and similar packages (as generated by
`pmbootstrap aportgen`) used to be "all", but is nowadays a specific
list of arches. This means, that after updating the `gcc-armhf` and
`gcc-aarch64` packages, and calling `pmbootstrap build gcc-armhf`,
it will try to build `gcc-armhf` for `aarch64` instead of the native
architecture, because that is the first one listed.
And since compiling to `aarch64` requires `gcc-aarch64`, it will build
that for the native architecture first.

So you're asking for `gcc-armhf` and it compiles `gcc-aarch64`, which
is very confusing (see #1272).
2018-03-01 20:04:51 +00:00
Daniele Debernardi e694e00d85 Use default partitions in odin export if not specified in deviceinfo (#1271) 2018-02-28 01:28:36 +00:00
Oliver Smith 937686d164
Don't mark all packages as explicitly installed (#1247)
pmbootstrap does dependency resolving on its own, and passes the list
of resolved packages to apk when we want it to install something. The
reason was outlined in #129:

> fixing #120: packages do not get updated in "pmbootstrap install"
> after they have been rebuilt. For this to work, we specify all
> packages explicitly for abuild, instead of letting abuild do the
> resolving.

This new PR fixes #1212 (which noted that all of these dependencies
were explicitly marked for installation) by doing the following:
1. All packages and dependencies get attached to the virtual package
  .pmbootstrap instead of world
2. We install the packages (without depends) explcitly
3. .pmbootstrap gets removed, which means that all packages from 1.
  stay installed, but are no longer marked as explicitly installed.
  They will get removed automatically, when the depending packages get
  removed.

In addition, the mechanism for replacing the package of locally built
packages with their full path, was broken and has been fixed in this
commit. This is necessary to update packages of the same version with
apk.
2018-02-25 20:49:47 +00:00
Luca Weiss 6b9e7ecf75 qemu: allow the user to specify the kernel flavor (#1256) 2018-02-25 19:20:22 +00:00
Oliver Smith 948b5af09b
device packages: depend on postmarketos-base (#1258)
* device-*: add postmarketos-base to depends
* aportgen: add postmarketos-base to depends
* Add test case
* postmarketos-base: Don't depend on devicepkg
* msm-fb-refresher: Enable service in post-install
2018-02-25 18:40:54 +00:00
Attila Szöllősi e4875e54fa Fix menuconfig arch check (#1259)
We don't use 'arch="all"' in our kernels anymore (that does not make sense,
since each arch needs its own kernel config). This patch fixes the menuconfig
code to work with multiple values in the "arch" field.
2018-02-25 17:44:01 +00:00
Oliver Smith ad5a0d4294
Make proprietary drivers optional (1/2): pmbootstrap changes (#1254)
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.

Changes:
* New config file options nonfree_firmware and nonfree_userland, which
  we ask for during "pmbootstrap init" if there are non-free components
  for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
  packages are called $pkgname-nonfree-firmware and
  $pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
  subpackages. Parsing that is implemented in
  pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
  the regular APKBUILD parsing, as this would need a change in the
  output format, and it is a lot *less* code if done like in this
  commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
  pmb/install/install.py to _install.py: needed to call the function in
  the usual way (e.g. pmb.parse.apkbuild()) but still being able to
  test the individual functions from these files in the test suite.
  We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
  packages that will be installed, based on the user's choice in
  "pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
  refactored test/test_questions.py to have multiple functions for
  testing the various questions / question types from
  "pmbootstrap init" instead of having it all in one big function.
  This allows to use another aport folder for testing the new
  non-free related questions in init.
2018-02-24 21:49:10 +00:00
Oliver Smith 8f5ffe0589
APKINDEX not found: Only show in verbose log (#1250)
The message appears 20 or so times in a row when setting up a new
chroot. Right now we log it into the debug log (which is what
"pmbootstrap log" and "pmbootstrap --details-to-stdout" show). This
is annoying for normal pmbootstrap usage, and really clutters up the
Travis output.

This commit changes the log level to verbose, so it only shows up
when "-v" is passed to pmbootstrap.
2018-02-24 08:03:34 +00:00
Oliver Smith 90ca7b6351
zap_mismatch_bins: Adjust to new apkindex parsing
Without this fix, `pmbootstrap zap -m` fails with:

  File "/home/user/code/pmbootstrap/pmb/__init__.py", line 61, in main
    getattr(frontend, args.action)(args)
  File "/home/user/code/pmbootstrap/pmb/helpers/frontend.py", line 322, in zap
    distfiles=args.distfiles)
  File "/home/user/code/pmbootstrap/pmb/chroot/zap.py", line 54, in zap
    zap_mismatch_bins(args, confirm, dry)
  File "/home/user/code/pmbootstrap/pmb/chroot/zap.py", line 110, in zap_mismatch_bins
    if pkgname != bin_data["pkgname"]:
KeyError: 'pkgname'
2018-02-20 21:41:03 +00:00
Oliver Smith 2a9c1da46c
Bump version to 0.7.0 (previous one got tagged) 2018-02-20 19:53:50 +00:00
Oliver Smith db5e69630e
Index parser: support multiple package providers (#1202)
* The APKINDEX parser used to return a dictionary with one package for
  a given package name. This works for the installed packages database,
  because there can only be one provider for a package. But when
  parsing packages from binary repositories, we need to support
  multiple providers for one package. It is now possible to get a
  dictionary with either multiple providers, or just a single provider
  for each package.
* Dependency parsing logic has been adjusted, to support multiple
  providers. For multiple providers, the one with the same package
  name as the package we are looking up is prefered. If there is none
  (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
  that will be installed anyway, and after that packages that are
  already installed. When all else fails, it just picks the first one
  and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
  pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
  that pmbootstrap should build if neccessary, and install in the
  chroot. This can be used to quickly test the depencency resolution
  of pmbootstrap without doing a full "pmbootstrap install".

Fixes #1122.
2018-02-20 19:52:28 +00:00
Oliver Smith 0f371e426f
pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
2018-02-19 22:04:01 +00:00
Alexander 57b45a4798 Add a check for binfmt_misc (#1236)
Closes #1223.
2018-02-17 20:41:54 +00:00
Alexander 3d35b45835 Check kernel config options based on the architecture (#1233)
The 'necessary_kconfig_options' dictionary in pmb/config/__init__.py
now has the different architectures (space separated) as the keys and
the dictionary, which matches kernel config options and their
expected value, as its value.
For that purpose, the 'check' function in pmb/parse/kconfig.py was
modified, so that it takes the architecture from the kconfig filename
and uses it to find the needed kernel config options.
Closes #1218.
2018-02-17 13:37:35 +00:00
Oliver Smith 57afbda3b9
Bump version to 0.6.0 2018-02-11 12:36:47 +01:00
Lucas Ramage 42018f8014 mkbootimg: replace Python version with osm0sis' C version (#1193)
Closes #441. Adjust bootimg_analyze code:
* Install mkbootimg (which now provides unpackbootimg) instead of
  unpackbootimg. In theory, pmbootstrap should recognize this
  automatically, however right now it does not yet handle this case.
* The file names of the extracted files have changed.
2018-02-11 11:41:41 +00:00
Oliver Smith 3fe75ddb56
Fix screwed up progress bars in Travis output (#1213)
Right now, they appear on screen when using --details-to-stdout. This
does not work well with Travis CI and screws up the log.

Disabling the progress bars in abuild works just like Alpine does it in
their Travis CI script: Exporting SUDO_APK as
"abuild-apk --no-progress" instead of "abuild-apk".

test_check_checksums.py: Run "pmbootstrap build_init" before building
any packages, so it is a bit less verbose (downloading the APKINDEX
files etc.). Later we run the build init code again (because we use
--strict while building the packages), but then the APKINDEX files
are already present. So overall the log is a bit shorter before the
building starts. (It is still logged to the logfile, which gets
printed on error anyway.)
2018-02-09 18:43:58 +00:00
Oliver Smith 5e85d72ca0
Testsuite: Run UIs in Qemu and check running processes and more (#982)
* Testsuite: Run UIs in Qemu and check running processes (and other changes)

* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
  it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
  built before running tests (otherwise tests would hang without any output
  while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
  plasma-mobile currently, easy to extend), runs it via Qemu and checks the
  running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
  our source tree, so we don't need to clone their git repo anymore. Now it
  is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.

* Various fixes
* Build device-packages for the device arch and don't raise an
  exception, but print a note if --ignore-depends is not specified
  and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
  check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
  completion
* Exit with 1 when showing usage in check_checksums.py
2018-02-02 00:16:29 +00:00
Oliver Smith 5eb75afc08
Fix invalid /etc/apk/repositories entry on device (#1189)
It used to have an entry for /mnt/pmbootstrap-packages, which only
makes sense while working on the chroot with pmbootstrap. After the
installation on the device, there's no repo in that path.
Furthermore, empty lines were added to the recovery installer script
for readability (thanks @ata2001!)
2018-02-01 22:44:20 +00:00
Oliver Smith e8c27795a8
Remove rest of 'pmbootstrap challenge' left overs (#1173)
Follow up to #1162.

* `pmb.build.buildinfo()`: Used to record the build environment. It is
  flawed because it scans the repo APKINDEX files instead of using the
  actually installed packages list. When it was implemented we were not
  able to do the latter. After this is removed, `pmb.parse.depends` can
  be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
  used exclusively by `pmb.build.buildinfo()`, to learn about which
  files have been changed in the local repository folder after a
  package was built. The idea was, that we could find subpackages that
  way. But this information is present in the installed package list as
  well, which is a much cleaner approach.
2018-02-01 22:03:21 +00:00
Oliver Smith a5d2b2b064
Fix: Native arch APKINDEX downloaded twice (#1190)
Nowadays pmb.config.build_device_architectures holds the native
architecture as well, so we don't need to explicitly download the
native architecture APKINDEX.
2018-02-01 21:27:17 +00:00
Daniele Debernardi c925791387 Fix typo in check_arch function (#1195)
Added a missing space to separate words in the log message
2018-02-01 21:02:04 +00:00
Oliver Smith 77701ac484
mkinitfs: Fail on missing depends / better usability in general (#1133)
* Fail if mkbootimg/uboot-tools are not installed, but creating a
  boot.img file / u-boot legacy image was requested via deviceinfo
  (fixes #312)
* Fail if /boot/dt.img is missing, but we have a qcdt device
* Fail if the dtb file specified in deviceinfo does not exist
* Fail if mkbootimg etc. exit with error code
* Don't try to add the ext4 module into the initramfs. We always
  compile it into the kernel. Instead, kconfig_check makes sure it
  is enabled now. (fixes #1037)
* Add a note that modprobe warnings can be ignored mostly
2018-01-31 19:39:09 +00:00
Oliver Smith 0f5056f6b9
Fix pmbootstrap zap -m / various zap improvements (#1166)
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
  where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
  have an aport or where the aport has another version. This also
  works with subpackages now,
  we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"

zap in general:
* Show the amount of cleared up space after the deletion instead
  of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
  (we need to call it twice during zap now to get the space
  calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
  for debugging) to list the packages/chroots that would get
  deleted
* Roughly output the command that would get executed to delete
  files, so it's obvious what's going on in --dry mode. (% rm ...)
2018-01-31 19:34:02 +00:00
Oliver Smith 3c59126bc1
Remove timestamp based rebuilds (#1174)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:

    pmbootstrap build --force hello-world

Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close #1167, close #1156, close #1023 and
close #985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
2018-01-28 23:27:33 +00:00
Pavel Machek bdeec7a255 Optional --rsync parameter for pmbootstrap install only copies the diff (#1151)
Should reduce wear of sd card. Example usage:
pmbootstrap install --sdcard=/dev/mmcblk0 --no-fde --rsync
More rsync output with pmbootstrap -v.
2018-01-28 23:25:21 +00:00
Oliver Smith cd7280e1ee
Bump version (last version has been tagged) 2018-01-27 18:53:27 +01:00
Oliver Smith 566da1718d
version.py: fix validation of 6.0_0002 (#1160) 2018-01-25 18:23:55 +00:00
Oliver Smith 0479031f7e
Remove lots of legacy 'challenge' code (#1162)
Preparation for #1122.

* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
  an exception when two entries in the apkindex provided the same package.
  Turns out this is *not* invalid after all, two packages can provide the same
  soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
  listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
  which was a dead end. This code uses the broken strict feature.
2018-01-25 18:08:39 +00:00