Commit Graph

119 Commits

Author SHA1 Message Date
Oliver Smith 5102d48064 Make APKINDEX downloads returning 404 non-fatal (!1726)
Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.

This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.
2018-12-11 06:56:53 +00:00
Oliver Smith fa7860c8f6 pmb.helpers.repo.update: refactor cache code (!1726)
Change the cache format from args.cache["offline_msg_shown"] to
args.cache["pmb.helpers.repo.update"]["offline_msg_shown"]. This is in
preparation for saving more data in the cache of
pmb.helpers.repo.update in the next commit.
2018-12-11 06:56:53 +00:00
Oliver Smith a874baa037 Cosmetic: pmb.helpers.http: describe parameters (!1726)
Describe all parameters of pmb.helpers.download().
2018-12-11 06:56:53 +00:00
Oliver Smith 49212ba317 Cosmetic: pmb.helpers.repo.update(): fix comment (!1726)
APKBUILD was written where it should have been APKINDEX.
2018-12-11 06:56:53 +00:00
Oliver Smith d6f7571cb9
repo_missing: properly find subpackages, provides (!1724)
Do not fail anymore when attempting to start a new binary repository
build without any existing binary packages:
	pmbootstrap -mp="" repo_missing

Find subpackages defined with subpackage functions:
	subpackages="dev:mydevfunc"

Find provides defined with specific versions:
	provides="mkbootimg=1.0.0"
2018-12-10 21:00:04 +01:00
Oliver Smith a5a64158e9
allow specifying multiple postmarketOS mirrors (!1718)
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories

This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
2018-12-06 07:31:43 +01:00
Oliver Smith a92e6a89d0
remove local postmarketOS mirror folder support (!1718)
Obscure feature: it was possible to specify a local path as
--mirror-pmOS. It would then get mounted to /mnt/postmarketos-mirror
inside the chroot, and be specified as such in the generated
/etc/apk/repositories file.

I had used this once for some testing scripts, but I am sure nobody is
using this anymore. The same can be achieved with running a local http
server anyway:
<https://wiki.postmarketos.org/wiki/Installing_packages_on_a_running_phone>

Removing this makes it easier to support multiple postmarketOS mirrors
(next commit).
2018-12-06 07:31:38 +01:00
Oliver Smith 1c10dbee50
repo_missing: fix "could not find aport" error
In pmb.helpers.package.get(), we are differentiating between packages
that do not exist at all, and packages that do not exist for the
specified architecture. Make sure to actuallyy download the foreign arch
APKINDEX files, before trying to find the packages in there. Otherwise
the "could not find aport, and could not find this package in any
APKINDEX" package would appear.

We ran into this when testing on sr.ht, because pmbootstrap runs on a
fresh install every time, where no old APKINDEX files are present
(which would work around the bug).

This was meant to be part of the previous commit already, and was
tested, but it was not submitted correctly. So here it is as separate
commit straight to master. It is trivial anyway.
2018-12-03 21:02:18 +01:00
Oliver Smith 933c4d0f0d new action: 'pmbootstrap repo_missing'
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.

This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.

For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.

Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.

Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
  aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
  in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
  distinguish it from the other check_arch function
2018-12-01 21:30:59 +00:00
Oliver Smith a44b80b31d build.find_aport() -> helpers.pmaports.find()
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.

Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
2018-12-01 21:30:59 +00:00
Luca Weiss 9c037831a0
Remove legacy kconfig_check and menuconfig args
Fixes #1690
2018-12-01 01:32:52 +01:00
Oliver Smith e458b1fdbc Add --offline flag
Allow working offline with pmbootstrap, as long as all packages that
are being used have already been downloaded.
2018-10-25 20:37:33 +00:00
Robert Yang 277854e80f Kill the child processes spawned by a run command
When the timeout occurs it is important to ensure clean up of child
processes. Killing only the direct process created by a command can
leave child processes running.

For example a pmbootstrap.py install will run apk add. This run command
creates multiple processes as follows:
(cmd line arguments snipped for readability)

  $ ps -e -o pid,ppid,pgid,cmd
  PID  PPID  PGID CMD
  31738 23247 31738 python3 ./pmbootstrap.py -t 15 install --no-fde
  31746 31738 31738 sudo env -i /bin/sh -c ... ;apk --no-progress add
  31747 31746 31738 /bin/sh -c ... ;apk --no-progress add
  31748 31747 31738 apk --no-progress add

The root process of the run command is PID 31746. We want to kill
the child processes too. Otherwise only running kill -9 31746 will leave
the processes 31747 and 31748 running.
2018-10-02 07:06:46 +00:00
Oliver Smith f68a7a6baa Link to wiki page if first git clone fails 2018-09-26 05:58:11 +00:00
Oliver Smith 3ce00de710 Move aports into own repository (pmaports) 2018-09-05 05:57:38 +00:00
NotKit e22ba8caea Change default user uid to 10000 2018-08-02 20:10:56 +00:00
Oliver Smith 8268dc0e3d pmbootstrap: kill process if silent for 5 minutes (rewrite logging) 2018-07-14 01:13:28 +00:00
Oliver Smith 2b062a1272 pmb: use UTF-8 in pmb.helpers.file.replace()
Without this fix, `pmbootstrap pkgrel_bump` does not work with some
APKBUILDs with special characters.
2018-07-11 18:26:01 +00:00
Oliver Smith ed3a649d00
CI failure check successful, now fix it
(Or in other words, I forgot to add an empty line)
2018-06-30 09:08:02 +02:00
Oliver Smith 0854aa1dc4
Remove github.com references
* Travis and Coveralls badges
* aports: instead of <https://github.com/postmarketOS>, use
  <https://postmarketos.org>
* References to full URLs to issues and pull requests replaced with
  a hash and the number
* grsec check: simplify error message, remove link to github issue
  (nobody is using that anymore anyway)
2018-06-30 09:00:48 +02:00
Oliver Smith 3263bc4cb4
Fix conflict with mkbootimg and android-tools (#1557)
Alpine ships `mkbootimg` with the `android-tools` package now. This
conflicts with the `mkbootimg` fork from osm0sis (see #441).

Changes:
* Rename `mkbootimg` to `mkbootimg-osm0sis` (aport and binary name)
* `mkbootimg-osm0sis`: provides `mkbootimg` now (so we don't need to
  change all the device aports), update version to 2018.05.10
* Adjust our `mkinitfs` script to call `mkbootimg-osm0sis`
* Better aport description
* pkgrel_bump testcase: don't fail on deleted aport
2018-06-17 21:26:34 +00:00
steamport 30df0725ca menuconfig: add gconfig/xconfig support (#1509)
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
  (with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
  pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
  aports ([skip ci] because it would rebuild all kernels and run out of
  time). Alpine has dropped this as well, and it wouldn't work with the
  new code anyway.
2018-06-09 06:52:24 +00:00
Oliver Smith f49da75998
Fix binfmt_misc check (#1449)
We require binfmt_misc to run programs of foreign architectures (e.g.
armhf) with QEMU. This is set up by default in most distributions, but
in some (e.g. Alpine, Void) it needs to be configured manually (see
 the troubleshooting page in the wiki).

We have a check in place, which points to that troubleshooting wiki
page. However, the check was flawed, because we assumed the binfmt_misc
folder would not exist.

Thanks to @fxkrait for making the fix and for testing it!
2018-04-28 23:10:54 +00:00
Oliver Smith a77a0dcc32
Add envkernel.sh: instant mainlining environment (#1424)
Changes:
* `helpers/envkernel.sh`:
  * installs everything needed for kernel compilation in the native
    chroot
  * mounts the kernel source to `/mnt/linux` inside the chroot
  * creates `/mnt/linux/.output` and chowns it to the `pmos` user, that
    folder will be used for the kernel build output
  * sets up aliases for `make`, `pmbootstrap`, `pmbroot`, `kernelroot`
* new action `pmbootstrap work_migrate`: does the interactive work
  folder migration if necessary, otherwise it doesn't output anything
  * when calling this first, we can safely use all other commands
    non-interactively without showing the output

Benefits:
* Fast setup (especially for people who are new to kernel
  compilation
  * No need to figure out distribution specific package names
    (cross compilers!)
  * No need to do a test build just to verify that the right
    packages are installed
* Less error prone
  * The right dependencies are always installed
  * `ARCH` and `CROSS_COMPILE` variables always get set automatically
    and based on `deviceinfo_arch`
  * If the build environment is broken for some reason, just zap and
    start over
* Easy to reproduce problems

Notes:
* `make menuconfig` works as well
* Sourcing was tested with `zsh`, `bash` and `fish`, it should be easy to
  extend for other shells
2018-04-19 21:27:38 +00:00
Oliver Smith ca20ead505
"pkgrel_bump --auto": Handle subpackages properly (#1388)
`pmbootstrap pkgrel_bump --auto` automatically increases the pkgrel for
packages linking against libraries, which don't exist anymore (because
the soname has been changed). The feature is explained in detail in

The previous implementation did not detect soname breakage, when a
subpackage linked against a certain library, but the main package
did not (e.g. `qt5-qtbase-mysql` and `qt5-qtbase`). This was, because
we iterated over the aports/* to find the packages to be checked.

To fix this, we are iterating over the packages found in the APKINDEX
files instead (of both the locally compiled packages and the downloaded
index from the pmOS mirror).

Details:
* `pmb/helpers/pkgrel_bump.py`:
  * Rewrite `auto_apkindex_package()` to act upon a given parsed
    `aport` and `apk` (from the index) instead of finding the `apk`
    dict by itself (we need it earlier anyway).
  * Rewrite `auto()` to iterate over APKINDEX files instead of aports
    * Skip packages already found, so the `pkgrel` does not get
      increased multiple times when the same package was found in
      multipe index files.
* Put the package name at the beginning of the log messages to make
  them more readable
* testdata: Create a new `testsubpkg` aport, where only the subpackage
  links against `testlib`
* Adjust testing code to test everything with `testsubpkg` as well.

NOTE: This makes the command a bit slower, but we could improve
performance again by smart caching of `pmb.parse.apkindex.depends()`.
This could come in a future PR, the important part here is that the
command is bug-free again with this fix.
2018-04-15 21:34:40 +00:00
Oliver Smith b66b5dcc34
pmbootstrap init: kernel selection / remove linux-pmos-lts (#1363)
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
  for now. The kernel isn't used right now, and we save lots of
  maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
  `"downstream", "mainline", "stable"` (downstream is always
  `linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
  has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
  the `deviceinfo_dtb` file is missing (because we expect it to be
  missing for downstream kernels)
* device-sony-amami:
  * add kernel subpackages for downstream, mainline
  * set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
  * don't mix specifying kernels in depends *and* subpackages
  * 1 kernel in depends is maximum
  * kernel subpackages must have a valid name
  * Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
  `--ignore-depends` can be used with device packages to avoid building
  the kernel. The idea was to make the transition easier after a change
  we did months ago, and now the kernel doesn't always get built before
  building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
  * Add autoinstall=True to kernel_flavors_installed(). When the flag
    is set, the function makes sure that at least one kernel for the
    device is installed.
  * Remove kernel_flavor_autodetect() function, wherever it was used,
    it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
  kernel_flavors_installed() takes care of that now.
2018-04-03 23:50:09 +00:00
Oliver Smith 4d8afc4aa5
Fix: /var/cache/distfiles writable by everyone (#1329)
As noted in commit 255c715624
`/var/cache/distfiles` is writable by everyone. It is supposed to be
writable only by `root` and by the `abuild` group (in which we put the
`pmos` user already for building packages).

Changes:
* `pmb.build.init()`: make `/var/cache/distfiles` writable only by
  members of the `abuild` group (and root)
* Increase workfolder version to 2
* Add migration code that fixes the permissions for existing work
  folders
* Refactor the migration code a bit to make this possible
2018-03-30 21:46:31 +00:00
Oliver Smith 6bb8444fef
"...flasher flash_rootfs" instead of "..._system" (#1373)
* Change `pmbootstrap flasher flash_system` command to
  `pmbootstrap flasher flash_rootfs`
* The old command still works, but all references have been changed to
  the new command
* Remove obsolete `pmbootstrap flasher export` (that was changed to
  `pmbootstrap export` a few months ago)
* Update `README.md` and ZSH auto completion
* Change the description of the generated rootfs image (not talking
  about a system image anymore, mention that it has subpartitions)
* Better description of `pmbootstrap flasher flash_rootfs --partition`
2018-03-30 01:11:20 +00:00
Daniele Debernardi 147082ec58 pmbootstrap init: Ask for hostname, default: device name (#1327)
* Save "" (empty string) in the user's config as hostname if the user
  let it default to the name of the device. That way, when the device
  gets changed, the user won't get the old device's name as hostname
  by accident.
* Add a test case
2018-03-17 18:41:41 +00:00
Oliver Smith 5ea00e0862
pmbootstrap newapkbuild: Properly parse arguments (#1320)
* pmbootstrap newapkbuild: Properly parse arguments

The `pmbootstrap newapkbuild` action wraps Alpine's `newapkbuild`. We
used to directly pass all arguments to `newapkbuild` without verifying
in Python whether they make sense or not. However, as `newpakbuild`
doesn't do strict sanity checks on the arguments, it is easy to end up
with unexpected behavior when using the command for the first time.

For example, `newapkbuild` allows either specifying a PKGNAME or SRCURL
as last parameter, and also allows setting a PKGNAME with the `-n`
parameter. It only makes sense to use that option when passing a
SRCURL.

With this commit, we duplicate the optins that should be passed through
to `newapkbuild` and use argparse to fully sanitize the options and
display a help page (`pmbootstrap newapkbuild -h`) that is consistent
with the other help pages.

Details:
* The `-f` (force) flag does not get passed through anymore. Instead we
  use it in Python to skip asking if an existing aport should be
  overwritten (the aports are outside of the chroot, so `newapkbuild`
  can't handle it in a way that makes sense for pmbootstrap).
* Output of `newapkbuild` gets redirected to the log file now, as we
  don't need it to display a help page.
* Don't verify the pkgver while creating the new APKBUILD. When passing
  a SRCURL, the pkgver gets extracted from the end of the URL and may
  not have a valid format yet (but we want the APKBUILD anyway).
* Stored options passed through in `pmb/config/__init__.py` and use it
  in both `pmb/parse/arguments.py` and `pmb/helpers/frontend.py`.
* Only allow `-n` with SRCURL
* The postmarketOS aports folder gets specified with `--folder` now.
  That way the generated help page is much closer to the original one
  from `newapkbuild`. The default is `main`.
* Made the package type flags (CMake, autotools, ...) exclusive so only
  one of them can be specified
2018-03-15 21:42:34 +00:00
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00
Oliver Smith 3666388619
Properly escape commands in pmb.chroot.user() (#1316)
## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.

This is not a security issue, as we don't pass strings from untrusted
input to this function.

## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`

In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`

## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`

So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.

Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.

## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.

## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
  list and environment variables as dict, and creates a properly escaped
  flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
  executed (in addition to the simplified form we always write down for
  readability)
* `test_shell_escape.py`: verify that the command passed by reference
  has not been modified, add a new test for strings with spaces, add
  tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
  variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
  so running ncurses applications like `menuconfig` and `nano` works out of
  the box
2018-03-10 22:58:39 +00:00
Oliver Smith e72afc53fd
pmbootstrap zap -m: delete outdated packages too (#1306)
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.

That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).

Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
  list of blocks, instead of the dict with removed duplicates with
  lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
  what they are doing now.
  ```
  short  long (old)         long (new)
  -p     --packages         --pkgs-local
  -m     --mismatch-bins    --pkgs-local-mismatch
  -o,    --old-bins         --pkgs-online-mismatch
  ```
2018-03-10 12:08:02 +00:00
Oliver Smith 99127111a1
Only download APKINDEX for relevant architectures (#1201)
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.

### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
  * Default is updating all arches where the APKBUILD files exist
  * Add existing_only parameter
  * Return True when files have been downloaded
  * Properly print which arches will be updated
  * Print update reason only in verbose log
  * Add and improve comments
* pmb.parse.arguments(), update action:
  * Add --non-existing parameter
  * Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
  * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
Oliver Smith db5e69630e
Index parser: support multiple package providers (#1202)
* The APKINDEX parser used to return a dictionary with one package for
  a given package name. This works for the installed packages database,
  because there can only be one provider for a package. But when
  parsing packages from binary repositories, we need to support
  multiple providers for one package. It is now possible to get a
  dictionary with either multiple providers, or just a single provider
  for each package.
* Dependency parsing logic has been adjusted, to support multiple
  providers. For multiple providers, the one with the same package
  name as the package we are looking up is prefered. If there is none
  (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
  that will be installed anyway, and after that packages that are
  already installed. When all else fails, it just picks the first one
  and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
  pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
  that pmbootstrap should build if neccessary, and install in the
  chroot. This can be used to quickly test the depencency resolution
  of pmbootstrap without doing a full "pmbootstrap install".

Fixes #1122.
2018-02-20 19:52:28 +00:00
Oliver Smith 0f371e426f
pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
2018-02-19 22:04:01 +00:00
Alexander 57b45a4798 Add a check for binfmt_misc (#1236)
Closes #1223.
2018-02-17 20:41:54 +00:00
Oliver Smith 5e85d72ca0
Testsuite: Run UIs in Qemu and check running processes and more (#982)
* Testsuite: Run UIs in Qemu and check running processes (and other changes)

* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
  it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
  built before running tests (otherwise tests would hang without any output
  while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
  plasma-mobile currently, easy to extend), runs it via Qemu and checks the
  running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
  our source tree, so we don't need to clone their git repo anymore. Now it
  is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.

* Various fixes
* Build device-packages for the device arch and don't raise an
  exception, but print a note if --ignore-depends is not specified
  and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
  check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
  completion
* Exit with 1 when showing usage in check_checksums.py
2018-02-02 00:16:29 +00:00
Oliver Smith e8c27795a8
Remove rest of 'pmbootstrap challenge' left overs (#1173)
Follow up to #1162.

* `pmb.build.buildinfo()`: Used to record the build environment. It is
  flawed because it scans the repo APKINDEX files instead of using the
  actually installed packages list. When it was implemented we were not
  able to do the latter. After this is removed, `pmb.parse.depends` can
  be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
  used exclusively by `pmb.build.buildinfo()`, to learn about which
  files have been changed in the local repository folder after a
  package was built. The idea was, that we could find subpackages that
  way. But this information is present in the installed package list as
  well, which is a much cleaner approach.
2018-02-01 22:03:21 +00:00
Oliver Smith a5d2b2b064
Fix: Native arch APKINDEX downloaded twice (#1190)
Nowadays pmb.config.build_device_architectures holds the native
architecture as well, so we don't need to explicitly download the
native architecture APKINDEX.
2018-02-01 21:27:17 +00:00
Oliver Smith 0f5056f6b9
Fix pmbootstrap zap -m / various zap improvements (#1166)
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
  where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
  have an aport or where the aport has another version. This also
  works with subpackages now,
  we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"

zap in general:
* Show the amount of cleared up space after the deletion instead
  of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
  (we need to call it twice during zap now to get the space
  calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
  for debugging) to list the packages/chroots that would get
  deleted
* Roughly output the command that would get executed to delete
  files, so it's obvious what's going on in --dry mode. (% rm ...)
2018-01-31 19:34:02 +00:00
Oliver Smith 3c59126bc1
Remove timestamp based rebuilds (#1174)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:

    pmbootstrap build --force hello-world

Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close #1167, close #1156, close #1023 and
close #985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
2018-01-28 23:27:33 +00:00
Pavel Machek bdeec7a255 Optional --rsync parameter for pmbootstrap install only copies the diff (#1151)
Should reduce wear of sd card. Example usage:
pmbootstrap install --sdcard=/dev/mmcblk0 --no-fde --rsync
More rsync output with pmbootstrap -v.
2018-01-28 23:25:21 +00:00
Oliver Smith 0479031f7e
Remove lots of legacy 'challenge' code (#1162)
Preparation for #1122.

* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
  an exception when two entries in the apkindex provided the same package.
  Turns out this is *not* invalid after all, two packages can provide the same
  soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
  listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
  which was a dead end. This code uses the broken strict feature.
2018-01-25 18:08:39 +00:00
Oliver Smith 0ae23afa60
Fix #839: Check pkgver after parsing APKBUILD / various small improvements (#854)
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
  (also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
2018-01-18 22:05:27 +00:00
Oliver Smith 9a3ce3ee70
Wrap Alpine's newapkbuild (#894)
Closes #836.
2018-01-15 22:00:11 +00:00
Oliver Smith 1992f37036
Gracefully handle packages breaking because of soname bumps (#1116)
Fixes #893. Changes:
* New action: "pmbootstrap pkgrel_bump"
* pmbootstrap detects missing soname depends when trying to install
  anyting, and suggests "pkgrel_bump --auto" to fix it
* Testcase test_soname_bump.py checks the pmOS binary package repo
  for soname breakage, so we see it when CI runs for new PRs
* libsamsung-ipc: bump pkgrel because of soname bump
2018-01-14 01:26:42 +00:00
drebrez 748f2ce6b1 Show UI selection ordered by name, `none` remains at the beginning (#1092)
Fixes random UI selection screen for Python <3.6.
2018-01-07 03:55:27 +00:00
Oliver Smith 7750c1dd40
Happy new year! (update copyright to 2018) 2018-01-04 04:53:35 +01:00
Oliver Smith 567ac64e26
ccache: Fix for distcc cross-compiling / various improvements (#1026)
* ccache: Fix for distcc cross-compiling / various improvements

* Make ccache work when cross-compiling with distcc (fix #716)
* Allow to configure the ccache size in "pmbootstrap init"
* Moved ccache stats code from pmb/build/other.py to
  pmb/helpers/frontend.py
* Grouped job count, ccache size and timestamp based rebuilds
  together to "build options" and allow to skip them
* Sorted config options that had to be modified anyway
  alphabetically

* Improve comment in arch-bin-masquerade APKBUILD
2017-12-21 16:42:29 +00:00