Commit Graph

94 Commits

Author SHA1 Message Date
Oliver Smith b66b5dcc34
pmbootstrap init: kernel selection / remove linux-pmos-lts (#1363)
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
  for now. The kernel isn't used right now, and we save lots of
  maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
  `"downstream", "mainline", "stable"` (downstream is always
  `linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
  has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
  the `deviceinfo_dtb` file is missing (because we expect it to be
  missing for downstream kernels)
* device-sony-amami:
  * add kernel subpackages for downstream, mainline
  * set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
  * don't mix specifying kernels in depends *and* subpackages
  * 1 kernel in depends is maximum
  * kernel subpackages must have a valid name
  * Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
  `--ignore-depends` can be used with device packages to avoid building
  the kernel. The idea was to make the transition easier after a change
  we did months ago, and now the kernel doesn't always get built before
  building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
  * Add autoinstall=True to kernel_flavors_installed(). When the flag
    is set, the function makes sure that at least one kernel for the
    device is installed.
  * Remove kernel_flavor_autodetect() function, wherever it was used,
    it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
  kernel_flavors_installed() takes care of that now.
2018-04-03 23:50:09 +00:00
Oliver Smith 4d8afc4aa5
Fix: /var/cache/distfiles writable by everyone (#1329)
As noted in commit 255c715624
`/var/cache/distfiles` is writable by everyone. It is supposed to be
writable only by `root` and by the `abuild` group (in which we put the
`pmos` user already for building packages).

Changes:
* `pmb.build.init()`: make `/var/cache/distfiles` writable only by
  members of the `abuild` group (and root)
* Increase workfolder version to 2
* Add migration code that fixes the permissions for existing work
  folders
* Refactor the migration code a bit to make this possible
2018-03-30 21:46:31 +00:00
Oliver Smith 6bb8444fef
"...flasher flash_rootfs" instead of "..._system" (#1373)
* Change `pmbootstrap flasher flash_system` command to
  `pmbootstrap flasher flash_rootfs`
* The old command still works, but all references have been changed to
  the new command
* Remove obsolete `pmbootstrap flasher export` (that was changed to
  `pmbootstrap export` a few months ago)
* Update `README.md` and ZSH auto completion
* Change the description of the generated rootfs image (not talking
  about a system image anymore, mention that it has subpartitions)
* Better description of `pmbootstrap flasher flash_rootfs --partition`
2018-03-30 01:11:20 +00:00
Daniele Debernardi 147082ec58 pmbootstrap init: Ask for hostname, default: device name (#1327)
* Save "" (empty string) in the user's config as hostname if the user
  let it default to the name of the device. That way, when the device
  gets changed, the user won't get the old device's name as hostname
  by accident.
* Add a test case
2018-03-17 18:41:41 +00:00
Oliver Smith 5ea00e0862
pmbootstrap newapkbuild: Properly parse arguments (#1320)
* pmbootstrap newapkbuild: Properly parse arguments

The `pmbootstrap newapkbuild` action wraps Alpine's `newapkbuild`. We
used to directly pass all arguments to `newapkbuild` without verifying
in Python whether they make sense or not. However, as `newpakbuild`
doesn't do strict sanity checks on the arguments, it is easy to end up
with unexpected behavior when using the command for the first time.

For example, `newapkbuild` allows either specifying a PKGNAME or SRCURL
as last parameter, and also allows setting a PKGNAME with the `-n`
parameter. It only makes sense to use that option when passing a
SRCURL.

With this commit, we duplicate the optins that should be passed through
to `newapkbuild` and use argparse to fully sanitize the options and
display a help page (`pmbootstrap newapkbuild -h`) that is consistent
with the other help pages.

Details:
* The `-f` (force) flag does not get passed through anymore. Instead we
  use it in Python to skip asking if an existing aport should be
  overwritten (the aports are outside of the chroot, so `newapkbuild`
  can't handle it in a way that makes sense for pmbootstrap).
* Output of `newapkbuild` gets redirected to the log file now, as we
  don't need it to display a help page.
* Don't verify the pkgver while creating the new APKBUILD. When passing
  a SRCURL, the pkgver gets extracted from the end of the URL and may
  not have a valid format yet (but we want the APKBUILD anyway).
* Stored options passed through in `pmb/config/__init__.py` and use it
  in both `pmb/parse/arguments.py` and `pmb/helpers/frontend.py`.
* Only allow `-n` with SRCURL
* The postmarketOS aports folder gets specified with `--folder` now.
  That way the generated help page is much closer to the original one
  from `newapkbuild`. The default is `main`.
* Made the package type flags (CMake, autotools, ...) exclusive so only
  one of them can be specified
2018-03-15 21:42:34 +00:00
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00
Oliver Smith 3666388619
Properly escape commands in pmb.chroot.user() (#1316)
## Introduction
In #1302 we noticed that `pmb.chroot.user()` does not escape commands
properly: When passing one string with spaces, it would pass them as
two strings to the chroot. The use case is passing a description with
a space inside to `newapkbuild` with `pmboostrap newapkbuild`.

This is not a security issue, as we don't pass strings from untrusted
input to this function.

## Functions for running commands in pmbootstrap
To put the rest of the description in context: We have four high level
functions that run commands:
* `pmb.helpers.run.user()`
* `pmb.helpers.run.root()`
* `pmb.chroot.root()`
* `pmb.chroot.user()`

In addition, one low level function that the others invoke:
* `pmb.helpers.run.core()`

## Flawed test case
The issue described above did not get detected for so long, because we
have a test case in place since day one, which verifies that all of the
functions above escape everything properly:
* `test/test_shell_escape.py`

So the test case ran a given command through all these functions, and
compared the result each time. However, `pmb.chroot.root()`
modified the command variable (passed by reference) and did the
escaping already, which means `pmb.chroot.user()` running directly
afterwards only returns the right output when *not* doing any escaping.

Without questioning the accuracy of the test case, I've escaped
commands and environment variables with `shlex.quote()` *before*
passing them to `pmb.chroot.user()`. In retrospective this does not
make sense at all and is reverted with this commit.

## Environment variables
By coincidence, we have only passed custom environment variables to
`pmb.chroot.user()`, never to the other high level functions. This only
worked, because we did not do any escaping and the passed line gets
executed as shell command:
```
$ MYENV=test echo test2
test 2
```
If it was properly escaped as one shell command:
```
$ 'MYENV=test echo test2'
sh: MYENV=test echo test2: not found
```
So doing that clearly doesn't work anymore. I have added a new `env`
parameter to `pmb.chroot.user()` (and to all other high level functions
for consistency), where environment variables can be passed as a
dictionary. Then the function knows what to do and we end up with
properly escaped commands and environment variables.

## Details
* Add new `env` parameter to all high level command execution functions
* New `pmb.helpers.run.flat_cmd()` function, that takes a command as
  list and environment variables as dict, and creates a properly escaped
  flat string from the input.
* Use that function for proper escaping in all high level exec funcs
* Don't escape commands *before* passing them to `pmb.chroot.user()`
* Describe parameters of the command execution functions
* `pmbootstrap -v` writes the exact command to the log that was
  executed (in addition to the simplified form we always write down for
  readability)
* `test_shell_escape.py`: verify that the command passed by reference
  has not been modified, add a new test for strings with spaces, add
  tests for new function `pmb.helpers.run.flat_cmd()`
* Remove obsolete commend in `pmb.chroot.distccd` about environment
  variables, because we don't use any there anymore
* Add `TERM=xterm` to default environment variables in the chroot,
  so running ncurses applications like `menuconfig` and `nano` works out of
  the box
2018-03-10 22:58:39 +00:00
Oliver Smith e72afc53fd
pmbootstrap zap -m: delete outdated packages too (#1306)
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.

That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).

Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
  list of blocks, instead of the dict with removed duplicates with
  lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
  what they are doing now.
  ```
  short  long (old)         long (new)
  -p     --packages         --pkgs-local
  -m     --mismatch-bins    --pkgs-local-mismatch
  -o,    --old-bins         --pkgs-online-mismatch
  ```
2018-03-10 12:08:02 +00:00
Oliver Smith 99127111a1
Only download APKINDEX for relevant architectures (#1201)
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.

### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
  * Default is updating all arches where the APKBUILD files exist
  * Add existing_only parameter
  * Return True when files have been downloaded
  * Properly print which arches will be updated
  * Print update reason only in verbose log
  * Add and improve comments
* pmb.parse.arguments(), update action:
  * Add --non-existing parameter
  * Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
  * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
Oliver Smith db5e69630e
Index parser: support multiple package providers (#1202)
* The APKINDEX parser used to return a dictionary with one package for
  a given package name. This works for the installed packages database,
  because there can only be one provider for a package. But when
  parsing packages from binary repositories, we need to support
  multiple providers for one package. It is now possible to get a
  dictionary with either multiple providers, or just a single provider
  for each package.
* Dependency parsing logic has been adjusted, to support multiple
  providers. For multiple providers, the one with the same package
  name as the package we are looking up is prefered. If there is none
  (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
  that will be installed anyway, and after that packages that are
  already installed. When all else fails, it just picks the first one
  and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
  pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
  that pmbootstrap should build if neccessary, and install in the
  chroot. This can be used to quickly test the depencency resolution
  of pmbootstrap without doing a full "pmbootstrap install".

Fixes #1122.
2018-02-20 19:52:28 +00:00
Oliver Smith 0f371e426f
pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
2018-02-19 22:04:01 +00:00
Alexander 57b45a4798 Add a check for binfmt_misc (#1236)
Closes #1223.
2018-02-17 20:41:54 +00:00
Oliver Smith 5e85d72ca0
Testsuite: Run UIs in Qemu and check running processes and more (#982)
* Testsuite: Run UIs in Qemu and check running processes (and other changes)

* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
  it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
  built before running tests (otherwise tests would hang without any output
  while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
  plasma-mobile currently, easy to extend), runs it via Qemu and checks the
  running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
  our source tree, so we don't need to clone their git repo anymore. Now it
  is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.

* Various fixes
* Build device-packages for the device arch and don't raise an
  exception, but print a note if --ignore-depends is not specified
  and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
  check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
  completion
* Exit with 1 when showing usage in check_checksums.py
2018-02-02 00:16:29 +00:00
Oliver Smith e8c27795a8
Remove rest of 'pmbootstrap challenge' left overs (#1173)
Follow up to #1162.

* `pmb.build.buildinfo()`: Used to record the build environment. It is
  flawed because it scans the repo APKINDEX files instead of using the
  actually installed packages list. When it was implemented we were not
  able to do the latter. After this is removed, `pmb.parse.depends` can
  be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
  used exclusively by `pmb.build.buildinfo()`, to learn about which
  files have been changed in the local repository folder after a
  package was built. The idea was, that we could find subpackages that
  way. But this information is present in the installed package list as
  well, which is a much cleaner approach.
2018-02-01 22:03:21 +00:00
Oliver Smith a5d2b2b064
Fix: Native arch APKINDEX downloaded twice (#1190)
Nowadays pmb.config.build_device_architectures holds the native
architecture as well, so we don't need to explicitly download the
native architecture APKINDEX.
2018-02-01 21:27:17 +00:00
Oliver Smith 0f5056f6b9
Fix pmbootstrap zap -m / various zap improvements (#1166)
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
  where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
  have an aport or where the aport has another version. This also
  works with subpackages now,
  we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"

zap in general:
* Show the amount of cleared up space after the deletion instead
  of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
  (we need to call it twice during zap now to get the space
  calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
  for debugging) to list the packages/chroots that would get
  deleted
* Roughly output the command that would get executed to delete
  files, so it's obvious what's going on in --dry mode. (% rm ...)
2018-01-31 19:34:02 +00:00
Oliver Smith 3c59126bc1
Remove timestamp based rebuilds (#1174)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:

    pmbootstrap build --force hello-world

Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close #1167, close #1156, close #1023 and
close #985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
2018-01-28 23:27:33 +00:00
Pavel Machek bdeec7a255 Optional --rsync parameter for pmbootstrap install only copies the diff (#1151)
Should reduce wear of sd card. Example usage:
pmbootstrap install --sdcard=/dev/mmcblk0 --no-fde --rsync
More rsync output with pmbootstrap -v.
2018-01-28 23:25:21 +00:00
Oliver Smith 0479031f7e
Remove lots of legacy 'challenge' code (#1162)
Preparation for #1122.

* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
  an exception when two entries in the apkindex provided the same package.
  Turns out this is *not* invalid after all, two packages can provide the same
  soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
  listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
  which was a dead end. This code uses the broken strict feature.
2018-01-25 18:08:39 +00:00
Oliver Smith 0ae23afa60
Fix #839: Check pkgver after parsing APKBUILD / various small improvements (#854)
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
  (also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
2018-01-18 22:05:27 +00:00
Oliver Smith 9a3ce3ee70
Wrap Alpine's newapkbuild (#894)
Closes #836.
2018-01-15 22:00:11 +00:00
Oliver Smith 1992f37036
Gracefully handle packages breaking because of soname bumps (#1116)
Fixes #893. Changes:
* New action: "pmbootstrap pkgrel_bump"
* pmbootstrap detects missing soname depends when trying to install
  anyting, and suggests "pkgrel_bump --auto" to fix it
* Testcase test_soname_bump.py checks the pmOS binary package repo
  for soname breakage, so we see it when CI runs for new PRs
* libsamsung-ipc: bump pkgrel because of soname bump
2018-01-14 01:26:42 +00:00
drebrez 748f2ce6b1 Show UI selection ordered by name, `none` remains at the beginning (#1092)
Fixes random UI selection screen for Python <3.6.
2018-01-07 03:55:27 +00:00
Oliver Smith 7750c1dd40
Happy new year! (update copyright to 2018) 2018-01-04 04:53:35 +01:00
Oliver Smith 567ac64e26
ccache: Fix for distcc cross-compiling / various improvements (#1026)
* ccache: Fix for distcc cross-compiling / various improvements

* Make ccache work when cross-compiling with distcc (fix #716)
* Allow to configure the ccache size in "pmbootstrap init"
* Moved ccache stats code from pmb/build/other.py to
  pmb/helpers/frontend.py
* Grouped job count, ccache size and timestamp based rebuilds
  together to "build options" and allow to skip them
* Sorted config options that had to be modified anyway
  alphabetically

* Improve comment in arch-bin-masquerade APKBUILD
2017-12-21 16:42:29 +00:00
zhuowei 5e7de79b90 pmbootstrap: fix listing of mounts if the file was deleted (#964)
On my system, /proc/mounts sometimes contains a line like

```
udev /media/zhuowei/redhd/pmbootstrap/chroot_native/dev/loop0p2\040(deleted)
 devtmpfs rw,relatime,size=1959476k,nr_inodes=489869,mode=755 0 0
```

The "\040(deleted)" text confuses `pmbootstrap shutdown`. Remove the suffix
if we find it in a /proc/mounts entry. This fixes #545.
2017-12-11 19:56:46 +00:00
Oliver Smith e9ca68dc05
Fix #941: Use the right arch for foreign-arch-only packages (#943)
This is a follow-up to #935.

* fix regression #941: pmbootstrap doesn't automatically pick the
  right architecture for building when none is specified
* remove obsolete --noarch-arch parameter
2017-11-28 19:12:16 +00:00
Oliver Smith d3c77c39ac
Fix #824: Refactor pmb/build/package.py (make depends work like in abuild) (#935)
* Rename pmb/build/package.py to pmb/build/_package.py, so we can
  access the functions it contains in testcases, and still use
  pmb.build.package()
* Refactor the entire file. Instead of one big function that does
  too many things, we have many small ones now, that are tested
  in the testsuite and easier to modify
* Whenever building a package, pmbootstrap does not only build and
  install the "makedepends" (like we did before), now it does the
  same for the "depends". That's required to be compatible with
  abuild. The old behavior can still be used with 'pmbootstrap
  build --ignore-depends'.
* Because of that change, noarch packages can no longer be built in
  the native chroot if we need them for a foreign chroot. A device-
  package depending on a kernel would pull in the same kernel for
  the native architecture otherwise.
* Running 'pmbootstrap build device-...' without '--ignore-depends'
  and without a matching '--arch' displays a note that explains
  this change to the user and tells how to use it instead.
* Noarch packages no longer get symlinked. That was only
  implemented for packages built in the native chroot, and now that
  is not always the case anymore. Symlinking these packages creates
  packages with broken dependencies anyway (e.g.
  device-samsung-i9100 can't be installed in x86_64, because
  linux-samsung-i9100 is armhf only).
* Rename "carch" to "arch" wherever used. Naming it "carch"
  sometimes is confusing with no benefit.
* Add a testcase for the aarch64 qemu workaround (because it failed
  first and I needed to know for sure if it is working again).
* Improved some verbose logging, which helped with development of
  this feature.
* Removed the old "build" test case (which was disabled in
  testcases_fast.sh) as the new "build_package" test case covers its
  functionallity.
* Only build indexes if the packages folder exists for that arch (Travis
  couldn't run a test case otherwise)
2017-11-26 14:32:02 +00:00
Oliver Smith a7b881e4cc
Close #871: Enable binary repository (#887)
* add my own build key
* enable the repo in the config
* update the README file
* Adjust testcase, that validates the keys and enable it in testcases_fast.sh
* Only save/load keys to/from the config file, which we ask for during
  'pmbootstrap init', so the binary repo gets used even if a config file
  already exists (this also removes a workaround, that deletes the work
  folder path from the config dictionary before writing it)
* Download missing APKINDEX.tar.gz files with Python code, before
  attempting to build packages (so we know which ones aleady exist in
  the binary packages repository)
* Consider APKINDEX files older than 4 hours as outdated and download
  them again (also in Python code)
* Provide 'pmbootstrap update' to force-update the APKINDEX files
* Travis: more logging output on failure
* Only allow keys from config_keys to be used by "pmbootstrap config"
2017-11-19 15:04:08 +00:00
drebrez 94e2387af5 Add `pmbootstrap bootimg_analyze` / prompt during new device wizard (#905) 2017-11-19 14:35:23 +00:00
drebrez ba3e2f6d26 Add argument to zap (-o) older versions of packages from all chroots (#841)
- also moved the mismatch clean (-m) before zapping the chroots
- added confirmation questions to -o and -m
2017-11-02 17:33:14 +00:00
Oliver Smith 6627599cf0
pmbootstrap init: Wizard for new port device- and linux-packages (#821)
* pmbootstrap init: Generate new port device- and linux-package
* adds `pmbootstrap aportgen device-*` and
  `pmbootstrap aportgen linux-*`
* ask for confirmation when selecting a non-existing device
* generate the packages directly from init
* refactor aportgen code
* fixed some easy things in the linux- APKBUILD (more to come in
  follow-up PRs!)

Testing:
* Test all questions to the user from pmb.config.init and pmb.aportgen.device
  (except for the timezone question, because we would need to monkeypatch the
  os.path.exists() function, which messes up pytest, so we'd need to refactor
  the timezone function to be more testsuite friendly first)
* Run the device wizard in a testcase a few times and check the output, that
  pmbootstrap.aportgen.device and pmbootstrap.aportgen.linux create by parsing
  the resulting APKBUILDs and deviceinfo and checking its contents.
* Build the generated device package once in the same testcase

Thanks a lot to @drebrez for all the help with this one:
<https://github.com/postmarketOS/pmbootstrap/pull/821>

See also the updated porting guide:
<https://wiki.postmarketos.org/wiki/Porting_to_a_new_device>
2017-10-30 19:56:38 +00:00
drebrez 9c1ef821b3 Add architecture argument for the buildroot chroot (#832)
* Add architecture argument for the buildroot chroot, defaults to device architecture
* Output pmbootstrap log file after failure to debug Travis failure
* Travis: disable timestamp based rebuilds
2017-10-28 00:45:15 +00:00
drebrez 732a94fd45 Unregister all binfmt architectures during `pmbootstrap shutdown` (#822)
* Unregister all binfmt architectures during `pmbootstrap shutdown`
* Remove `-r, --rootfs` argument from `pmbootstrap build_init` command
2017-10-25 22:54:17 +00:00
drebrez 459f97ef1d Support multiple architectures in `pmbootstrap menuconfig` command (#814) 2017-10-24 16:18:42 +00:00
Oliver Smith ed94ab7449 Run fast testcases in Travis CI, remove obsolete testcase (#760)
* Removed obsolete apkindex_files cache testcase (the corresponding
function has been removed in #345 already).
* Fix test_challenge_apk: It failed on Travis, because we're accessing
/etc/abuild.conf, which only exists after initializing the build environment.
It's a random dummy file anyway, so I've replaced it with another file.
* Fix test_folder_size: accept a tolerance in the result
2017-10-23 19:44:08 +00:00
Oliver Smith 92e55ae1cd pmbootstrap init: Show pkgdesc for each postmarketos-ui package (#763)
This way we could give the user a rough idea what will be installed,
and also use this to display a short warning about long compile times
(e.g. until the plasma mobile stuff is upstreamed).
2017-10-16 20:01:21 +00:00
Pablo Castellano b4dd7a89d2 Close #709: Improve user creation (#725)
* Allow to specify a custom username in "pmbootstrap init"
* Build chroots have "pmos" instead of "user" as username now
* Installation user UID is 1000 now (as in all other Linux distributions)
* Adjust autologins
* postmarketos-base: enable wheel group for sudo, removed previous sudoers file
* Implement safe upgrade path:
We save the version of the work folder format now, in $WORK/version.
When this file does not exist, it defaults to 0.
In case it does not match the currently required version
(pmb.config.work_version), then ask the user if it should
automatically be upgraded.
2017-10-12 20:08:10 +00:00
Oliver Smith 1285f74c5f Fix #731: Create symlinks for noarch-subpackages (#740)
* apkindex:
  * Also parse the architecture field
* symlink_noarch_package:
  * Renamed to symlink_noarch_packages
  * Always work on all packages (so we don't need to guess which
    subpackages have been generated after a certain build)
  * Get invoked when running 'pmbootstrap index'
  * Use 'apk index' to generate one index, where the architecture
    does not get rewritten (abuild does that by default, due to
    Alpine's repos not having a 'noarch' folder and diverging from
    that doesn't make things easier for us). That goes super fast,
    and then we know which packages are noarch packages and can
    create the symlinks.
* Made output less verbose:
  * Use -q for 'apk index' when calling it directly (when it gets
    called by abuild we can't control that)
  * Output that the APKINDEXes get reindexed only to the 'pmbootstrap
    log'.
2017-10-11 15:11:25 +00:00
Pablo Castellano c855ee095b Qemu support for the QXL driver and SPICE (#481)
* pmb.helpers.run: support running processes in background
* enable QXL driver support in the linux kernel configurations so
that we can also use SPICE to connect to the VM.

QXL is a paravirtual graphics driver with 2D support

The SPICE project aims to provide a complete open source solution for remote
access to virtual machines in a seamless way.

Both DRM_QXL and DRM_BOCHS are enabled as modules.
According to [1], on Linux guests, the qxl and bochs_drm kernel modules
must be loaded in order to gain a decent performance

* qemu: add new option --spice to connect to VM using a SPICE client

If specified, 'pmbootstrap qemu' will look for some SPICE client in the
user's PATH and run qemu using the QXL driver.

Currently supported spice clients are 'spicy' and 'remote-viewer' but
adding support for more clients can be easily done.

qemu with qxl support will run on port 8077/tcp, which doesn't belong to
any well-known service and represents 'PM' in decimal.

References:
[0] https://www.linux-kvm.org/page/SPICE
[1] https://wiki.archlinux.org/index.php/QEMU#qxl
[2] https://wiki.archlinux.org/index.php/QEMU#SPICE
[3] https://github.com/postmarketOS/pmbootstrap/issues/453 (partially fixed)
2017-09-26 20:52:00 +00:00
Oliver Smith e60eee7dfa Fix #151: git ambiguous argument error (#531)
We check if origin/HEAD is present. In case that reference is
missing, we show a meaningful error message now, with an explanation
on how to add it. Also moved find_out_of_sync_files_tracked() to
pmb.helpers.git
2017-09-25 22:05:29 +00:00
Attila Szöllősi ae6a58b6ed Close #554: kernel config checking (#589)
* Check kernel config
* Allow specifying multiple kernel packages, and also no packages
  which defaults to scanning all kernel configs (it is super fast
  anyway)
* Add the check to Travis CI
* Adjust existing kernel configs, so they pass the kconfig_check.
(We've had to put in a lot of defaults in the aarch64
linux-postmarketos configs, that's why the diff is a bit unclean.)
* Increase modified kernel pkgrels
2017-09-18 21:36:54 +00:00
drebrez 207c200229 zap: add `-d`/`--distfiles` argument to clear downloaded files cache (#576) 2017-09-14 18:28:10 +00:00
Oliver Smith bed1eacbb5 Close #256: Implement strict package building mode (#532)
Contrary to abuild, pmbootstrap only installs makedepends, and
keeps the installed packages around - both hacks save lots of time.

However, they may introduce missing makedepends in the APKBUILDs,
that the authors of the APKBUILDs do not notice because it works
for them.

This PR adss a strict mode, which will always clean the chroots
before building a package, and also remove all installed dependencies
after the package was built. You can use the following syntax to
only zap once, but build many packages at once:
`pmbootstrap build --strict hello-world 0xffff heimdall`

It also builds dependencies properly without leaving makedepends
behind.
2017-09-08 23:50:59 +00:00
Oliver Smith b29cc877a7 Alias: "pmbootstrap export" for "pmbootstrap flasher export" (#417)
* moved export code to pmb/export and split it up
* added deprecation notice to "pmbootstrap flasher export"
* made "pmbootstrap export" work
* adjusted the "pmbootstrap flasher export" hints in the code
2017-09-06 20:14:03 +00:00
Attila Szöllősi c6fe6e134a Flavor instead of device name when creating recovery zip (#491)
* Flavor instead of device name when creating recovery zip.
* Move parse_flavor_arg function
2017-09-02 03:53:58 +00:00
clayton craft a9e5b362dc Implement zapping of apk packages with ver different than aports (466) (#474)
This adds a new option to `zap`: `-m / --mismatch-bins`

When set, any binary apks in the work directory packages folder will be
removed if their version differs from the version in the relevant
APKBUILD in aports.
2017-08-28 20:34:03 +00:00
clayton craft 3942a06a2d Add -y/--assume-yes option to install (#450) (#467) 2017-08-26 15:50:04 +00:00
Pablo Castellano 3954883547 Write custom os-release (closes #324) (#439)
* Write custom os-release (closes #324)
* Return empty string instead of crashing when the directory is not a git repository
* Use https in homepage URL
2017-08-23 16:40:16 +00:00
Oliver Smith 1c13ca4fd9 Fix #363: Omit /home/user when calculating system image size (#389)
* The system image size is now calculated as: root size - home size.
* New function in `pmb/helpers/other.py`: `folder_size()`, with a
  testcase.
* Instead of copying everything to the system image folder, and
  deleting the home folder afterwards, do not copy the home folder
  in the first place.
* Added `pmbootstrap -s` to skip generating the initramfs for faster
  debugging.
* Set the default value in the "are you sure, that your partition has
  at least..." to "y", so we can run `yes '' | pmbootstrap install`
  to make it run through the whole installation process.
* Increase full size to 120%, boot partition gets 15 MB free space now
2017-08-18 19:19:48 +00:00