Commit Graph

56 Commits

Author SHA1 Message Date
Luca Weiss 650ed4925e
Fix spelling mistakes (!1794)
codespell --skip="./.git,./aports*"
2019-06-25 09:20:05 +02:00
Oliver Smith 0431a51932
pmb.build.checksum: rename .check() to .update()
Rename the function to .update(), so it is clear what it does, and it is
clear what the difference to .verify() in the same file is.
2019-05-24 22:41:31 +02:00
Oliver Smith b4c301974e
Add pmbootstrap checksum --verify (!1788)
Download all sources and verify their checksums.

This will be used in pmaports.git CI, if ci:skip-build is set in the
commit message (currently it just skips the build, and we don't test if
the source checksums are valid or not).
2019-05-19 22:07:20 +02:00
Oliver Smith 2634bbea95
kconfig check: allow skip with !pmb:kconfigcheck (!1753)
Ignore APKBUILDs that have "!pmb:kconfigcheck" in their options by
default in "pmbootstrap kconfig check", but print a note that they have
been skipped. Check all kernels with "pmbootstrap kconfig check -f".

This is necessary, because the Librem 5 devboard kernel's config does
not have CONFIG_DM_CRYPT enabled in their config, and we check for
that. As the device is still under heavy development, we will make our
lives easier by just using the upstream kernel config without any
changes and ignoring it in our check by default.
2019-02-21 19:39:25 +01:00
Robert Yang a6db644f00
envkernel: Add build command to create an apk package from envkernel (!1747)
Provides a quick way to incrementally compile a kernel and push it to
device.

Example usage.

Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX

Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel

Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
2019-02-15 16:24:07 +01:00
Oliver Smith f16bdaf0ca
Update copyright to 2019
Happy new year \o/
2019-01-02 09:31:20 +01:00
Oliver Smith 933c4d0f0d new action: 'pmbootstrap repo_missing'
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.

This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.

For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.

Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.

Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
  aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
  in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
  distinguish it from the other check_arch function
2018-12-01 21:30:59 +00:00
Oliver Smith a44b80b31d build.find_aport() -> helpers.pmaports.find()
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.

Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
2018-12-01 21:30:59 +00:00
Luca Weiss 9c037831a0
Remove legacy kconfig_check and menuconfig args
Fixes #1690
2018-12-01 01:32:52 +01:00
Oliver Smith 8268dc0e3d pmbootstrap: kill process if silent for 5 minutes (rewrite logging) 2018-07-14 01:13:28 +00:00
steamport 30df0725ca menuconfig: add gconfig/xconfig support (#1509)
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
  (with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
  pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
  aports ([skip ci] because it would rebuild all kernels and run out of
  time). Alpine has dropped this as well, and it wouldn't work with the
  new code anyway.
2018-06-09 06:52:24 +00:00
Oliver Smith a77a0dcc32
Add envkernel.sh: instant mainlining environment (#1424)
Changes:
* `helpers/envkernel.sh`:
  * installs everything needed for kernel compilation in the native
    chroot
  * mounts the kernel source to `/mnt/linux` inside the chroot
  * creates `/mnt/linux/.output` and chowns it to the `pmos` user, that
    folder will be used for the kernel build output
  * sets up aliases for `make`, `pmbootstrap`, `pmbroot`, `kernelroot`
* new action `pmbootstrap work_migrate`: does the interactive work
  folder migration if necessary, otherwise it doesn't output anything
  * when calling this first, we can safely use all other commands
    non-interactively without showing the output

Benefits:
* Fast setup (especially for people who are new to kernel
  compilation
  * No need to figure out distribution specific package names
    (cross compilers!)
  * No need to do a test build just to verify that the right
    packages are installed
* Less error prone
  * The right dependencies are always installed
  * `ARCH` and `CROSS_COMPILE` variables always get set automatically
    and based on `deviceinfo_arch`
  * If the build environment is broken for some reason, just zap and
    start over
* Easy to reproduce problems

Notes:
* `make menuconfig` works as well
* Sourcing was tested with `zsh`, `bash` and `fish`, it should be easy to
  extend for other shells
2018-04-19 21:27:38 +00:00
Oliver Smith b66b5dcc34
pmbootstrap init: kernel selection / remove linux-pmos-lts (#1363)
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
  for now. The kernel isn't used right now, and we save lots of
  maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
  `"downstream", "mainline", "stable"` (downstream is always
  `linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
  has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
  the `deviceinfo_dtb` file is missing (because we expect it to be
  missing for downstream kernels)
* device-sony-amami:
  * add kernel subpackages for downstream, mainline
  * set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
  * don't mix specifying kernels in depends *and* subpackages
  * 1 kernel in depends is maximum
  * kernel subpackages must have a valid name
  * Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
  `--ignore-depends` can be used with device packages to avoid building
  the kernel. The idea was to make the transition easier after a change
  we did months ago, and now the kernel doesn't always get built before
  building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
  * Add autoinstall=True to kernel_flavors_installed(). When the flag
    is set, the function makes sure that at least one kernel for the
    device is installed.
  * Remove kernel_flavor_autodetect() function, wherever it was used,
    it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
  kernel_flavors_installed() takes care of that now.
2018-04-03 23:50:09 +00:00
Oliver Smith 6bb8444fef
"...flasher flash_rootfs" instead of "..._system" (#1373)
* Change `pmbootstrap flasher flash_system` command to
  `pmbootstrap flasher flash_rootfs`
* The old command still works, but all references have been changed to
  the new command
* Remove obsolete `pmbootstrap flasher export` (that was changed to
  `pmbootstrap export` a few months ago)
* Update `README.md` and ZSH auto completion
* Change the description of the generated rootfs image (not talking
  about a system image anymore, mention that it has subpartitions)
* Better description of `pmbootstrap flasher flash_rootfs --partition`
2018-03-30 01:11:20 +00:00
Oliver Smith 5ea00e0862
pmbootstrap newapkbuild: Properly parse arguments (#1320)
* pmbootstrap newapkbuild: Properly parse arguments

The `pmbootstrap newapkbuild` action wraps Alpine's `newapkbuild`. We
used to directly pass all arguments to `newapkbuild` without verifying
in Python whether they make sense or not. However, as `newpakbuild`
doesn't do strict sanity checks on the arguments, it is easy to end up
with unexpected behavior when using the command for the first time.

For example, `newapkbuild` allows either specifying a PKGNAME or SRCURL
as last parameter, and also allows setting a PKGNAME with the `-n`
parameter. It only makes sense to use that option when passing a
SRCURL.

With this commit, we duplicate the optins that should be passed through
to `newapkbuild` and use argparse to fully sanitize the options and
display a help page (`pmbootstrap newapkbuild -h`) that is consistent
with the other help pages.

Details:
* The `-f` (force) flag does not get passed through anymore. Instead we
  use it in Python to skip asking if an existing aport should be
  overwritten (the aports are outside of the chroot, so `newapkbuild`
  can't handle it in a way that makes sense for pmbootstrap).
* Output of `newapkbuild` gets redirected to the log file now, as we
  don't need it to display a help page.
* Don't verify the pkgver while creating the new APKBUILD. When passing
  a SRCURL, the pkgver gets extracted from the end of the URL and may
  not have a valid format yet (but we want the APKBUILD anyway).
* Stored options passed through in `pmb/config/__init__.py` and use it
  in both `pmb/parse/arguments.py` and `pmb/helpers/frontend.py`.
* Only allow `-n` with SRCURL
* The postmarketOS aports folder gets specified with `--folder` now.
  That way the generated help page is much closer to the original one
  from `newapkbuild`. The default is `main`.
* Made the package type flags (CMake, autotools, ...) exclusive so only
  one of them can be specified
2018-03-15 21:42:34 +00:00
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00
Oliver Smith e72afc53fd
pmbootstrap zap -m: delete outdated packages too (#1306)
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.

That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).

Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
  list of blocks, instead of the dict with removed duplicates with
  lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
  what they are doing now.
  ```
  short  long (old)         long (new)
  -p     --packages         --pkgs-local
  -m     --mismatch-bins    --pkgs-local-mismatch
  -o,    --old-bins         --pkgs-online-mismatch
  ```
2018-03-10 12:08:02 +00:00
Oliver Smith 99127111a1
Only download APKINDEX for relevant architectures (#1201)
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.

### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
  * Default is updating all arches where the APKBUILD files exist
  * Add existing_only parameter
  * Return True when files have been downloaded
  * Properly print which arches will be updated
  * Print update reason only in verbose log
  * Add and improve comments
* pmb.parse.arguments(), update action:
  * Add --non-existing parameter
  * Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
  * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
Oliver Smith db5e69630e
Index parser: support multiple package providers (#1202)
* The APKINDEX parser used to return a dictionary with one package for
  a given package name. This works for the installed packages database,
  because there can only be one provider for a package. But when
  parsing packages from binary repositories, we need to support
  multiple providers for one package. It is now possible to get a
  dictionary with either multiple providers, or just a single provider
  for each package.
* Dependency parsing logic has been adjusted, to support multiple
  providers. For multiple providers, the one with the same package
  name as the package we are looking up is prefered. If there is none
  (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
  that will be installed anyway, and after that packages that are
  already installed. When all else fails, it just picks the first one
  and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
  pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
  that pmbootstrap should build if neccessary, and install in the
  chroot. This can be used to quickly test the depencency resolution
  of pmbootstrap without doing a full "pmbootstrap install".

Fixes #1122.
2018-02-20 19:52:28 +00:00
Oliver Smith 0f371e426f
pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
2018-02-19 22:04:01 +00:00
Oliver Smith 5e85d72ca0
Testsuite: Run UIs in Qemu and check running processes and more (#982)
* Testsuite: Run UIs in Qemu and check running processes (and other changes)

* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
  it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
  built before running tests (otherwise tests would hang without any output
  while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
  plasma-mobile currently, easy to extend), runs it via Qemu and checks the
  running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
  our source tree, so we don't need to clone their git repo anymore. Now it
  is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.

* Various fixes
* Build device-packages for the device arch and don't raise an
  exception, but print a note if --ignore-depends is not specified
  and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
  check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
  completion
* Exit with 1 when showing usage in check_checksums.py
2018-02-02 00:16:29 +00:00
Oliver Smith e8c27795a8
Remove rest of 'pmbootstrap challenge' left overs (#1173)
Follow up to #1162.

* `pmb.build.buildinfo()`: Used to record the build environment. It is
  flawed because it scans the repo APKINDEX files instead of using the
  actually installed packages list. When it was implemented we were not
  able to do the latter. After this is removed, `pmb.parse.depends` can
  be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
  used exclusively by `pmb.build.buildinfo()`, to learn about which
  files have been changed in the local repository folder after a
  package was built. The idea was, that we could find subpackages that
  way. But this information is present in the installed package list as
  well, which is a much cleaner approach.
2018-02-01 22:03:21 +00:00
Oliver Smith 0f5056f6b9
Fix pmbootstrap zap -m / various zap improvements (#1166)
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
  where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
  have an aport or where the aport has another version. This also
  works with subpackages now,
  we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"

zap in general:
* Show the amount of cleared up space after the deletion instead
  of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
  (we need to call it twice during zap now to get the space
  calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
  for debugging) to list the packages/chroots that would get
  deleted
* Roughly output the command that would get executed to delete
  files, so it's obvious what's going on in --dry mode. (% rm ...)
2018-01-31 19:34:02 +00:00
Oliver Smith 3c59126bc1
Remove timestamp based rebuilds (#1174)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:

    pmbootstrap build --force hello-world

Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close #1167, close #1156, close #1023 and
close #985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
2018-01-28 23:27:33 +00:00
Pavel Machek bdeec7a255 Optional --rsync parameter for pmbootstrap install only copies the diff (#1151)
Should reduce wear of sd card. Example usage:
pmbootstrap install --sdcard=/dev/mmcblk0 --no-fde --rsync
More rsync output with pmbootstrap -v.
2018-01-28 23:25:21 +00:00
Oliver Smith 0479031f7e
Remove lots of legacy 'challenge' code (#1162)
Preparation for #1122.

* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
  an exception when two entries in the apkindex provided the same package.
  Turns out this is *not* invalid after all, two packages can provide the same
  soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
  listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
  which was a dead end. This code uses the broken strict feature.
2018-01-25 18:08:39 +00:00
Oliver Smith 0ae23afa60
Fix #839: Check pkgver after parsing APKBUILD / various small improvements (#854)
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
  (also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
2018-01-18 22:05:27 +00:00
Oliver Smith 9a3ce3ee70
Wrap Alpine's newapkbuild (#894)
Closes #836.
2018-01-15 22:00:11 +00:00
Oliver Smith 1992f37036
Gracefully handle packages breaking because of soname bumps (#1116)
Fixes #893. Changes:
* New action: "pmbootstrap pkgrel_bump"
* pmbootstrap detects missing soname depends when trying to install
  anyting, and suggests "pkgrel_bump --auto" to fix it
* Testcase test_soname_bump.py checks the pmOS binary package repo
  for soname breakage, so we see it when CI runs for new PRs
* libsamsung-ipc: bump pkgrel because of soname bump
2018-01-14 01:26:42 +00:00
Oliver Smith 7750c1dd40
Happy new year! (update copyright to 2018) 2018-01-04 04:53:35 +01:00
Oliver Smith 567ac64e26
ccache: Fix for distcc cross-compiling / various improvements (#1026)
* ccache: Fix for distcc cross-compiling / various improvements

* Make ccache work when cross-compiling with distcc (fix #716)
* Allow to configure the ccache size in "pmbootstrap init"
* Moved ccache stats code from pmb/build/other.py to
  pmb/helpers/frontend.py
* Grouped job count, ccache size and timestamp based rebuilds
  together to "build options" and allow to skip them
* Sorted config options that had to be modified anyway
  alphabetically

* Improve comment in arch-bin-masquerade APKBUILD
2017-12-21 16:42:29 +00:00
Oliver Smith e9ca68dc05
Fix #941: Use the right arch for foreign-arch-only packages (#943)
This is a follow-up to #935.

* fix regression #941: pmbootstrap doesn't automatically pick the
  right architecture for building when none is specified
* remove obsolete --noarch-arch parameter
2017-11-28 19:12:16 +00:00
Oliver Smith d3c77c39ac
Fix #824: Refactor pmb/build/package.py (make depends work like in abuild) (#935)
* Rename pmb/build/package.py to pmb/build/_package.py, so we can
  access the functions it contains in testcases, and still use
  pmb.build.package()
* Refactor the entire file. Instead of one big function that does
  too many things, we have many small ones now, that are tested
  in the testsuite and easier to modify
* Whenever building a package, pmbootstrap does not only build and
  install the "makedepends" (like we did before), now it does the
  same for the "depends". That's required to be compatible with
  abuild. The old behavior can still be used with 'pmbootstrap
  build --ignore-depends'.
* Because of that change, noarch packages can no longer be built in
  the native chroot if we need them for a foreign chroot. A device-
  package depending on a kernel would pull in the same kernel for
  the native architecture otherwise.
* Running 'pmbootstrap build device-...' without '--ignore-depends'
  and without a matching '--arch' displays a note that explains
  this change to the user and tells how to use it instead.
* Noarch packages no longer get symlinked. That was only
  implemented for packages built in the native chroot, and now that
  is not always the case anymore. Symlinking these packages creates
  packages with broken dependencies anyway (e.g.
  device-samsung-i9100 can't be installed in x86_64, because
  linux-samsung-i9100 is armhf only).
* Rename "carch" to "arch" wherever used. Naming it "carch"
  sometimes is confusing with no benefit.
* Add a testcase for the aarch64 qemu workaround (because it failed
  first and I needed to know for sure if it is working again).
* Improved some verbose logging, which helped with development of
  this feature.
* Removed the old "build" test case (which was disabled in
  testcases_fast.sh) as the new "build_package" test case covers its
  functionallity.
* Only build indexes if the packages folder exists for that arch (Travis
  couldn't run a test case otherwise)
2017-11-26 14:32:02 +00:00
Oliver Smith a7b881e4cc
Close #871: Enable binary repository (#887)
* add my own build key
* enable the repo in the config
* update the README file
* Adjust testcase, that validates the keys and enable it in testcases_fast.sh
* Only save/load keys to/from the config file, which we ask for during
  'pmbootstrap init', so the binary repo gets used even if a config file
  already exists (this also removes a workaround, that deletes the work
  folder path from the config dictionary before writing it)
* Download missing APKINDEX.tar.gz files with Python code, before
  attempting to build packages (so we know which ones aleady exist in
  the binary packages repository)
* Consider APKINDEX files older than 4 hours as outdated and download
  them again (also in Python code)
* Provide 'pmbootstrap update' to force-update the APKINDEX files
* Travis: more logging output on failure
* Only allow keys from config_keys to be used by "pmbootstrap config"
2017-11-19 15:04:08 +00:00
drebrez 94e2387af5 Add `pmbootstrap bootimg_analyze` / prompt during new device wizard (#905) 2017-11-19 14:35:23 +00:00
drebrez ba3e2f6d26 Add argument to zap (-o) older versions of packages from all chroots (#841)
- also moved the mismatch clean (-m) before zapping the chroots
- added confirmation questions to -o and -m
2017-11-02 17:33:14 +00:00
Oliver Smith 6627599cf0
pmbootstrap init: Wizard for new port device- and linux-packages (#821)
* pmbootstrap init: Generate new port device- and linux-package
* adds `pmbootstrap aportgen device-*` and
  `pmbootstrap aportgen linux-*`
* ask for confirmation when selecting a non-existing device
* generate the packages directly from init
* refactor aportgen code
* fixed some easy things in the linux- APKBUILD (more to come in
  follow-up PRs!)

Testing:
* Test all questions to the user from pmb.config.init and pmb.aportgen.device
  (except for the timezone question, because we would need to monkeypatch the
  os.path.exists() function, which messes up pytest, so we'd need to refactor
  the timezone function to be more testsuite friendly first)
* Run the device wizard in a testcase a few times and check the output, that
  pmbootstrap.aportgen.device and pmbootstrap.aportgen.linux create by parsing
  the resulting APKBUILDs and deviceinfo and checking its contents.
* Build the generated device package once in the same testcase

Thanks a lot to @drebrez for all the help with this one:
<https://github.com/postmarketOS/pmbootstrap/pull/821>

See also the updated porting guide:
<https://wiki.postmarketos.org/wiki/Porting_to_a_new_device>
2017-10-30 19:56:38 +00:00
drebrez 9c1ef821b3 Add architecture argument for the buildroot chroot (#832)
* Add architecture argument for the buildroot chroot, defaults to device architecture
* Output pmbootstrap log file after failure to debug Travis failure
* Travis: disable timestamp based rebuilds
2017-10-28 00:45:15 +00:00
drebrez 732a94fd45 Unregister all binfmt architectures during `pmbootstrap shutdown` (#822)
* Unregister all binfmt architectures during `pmbootstrap shutdown`
* Remove `-r, --rootfs` argument from `pmbootstrap build_init` command
2017-10-25 22:54:17 +00:00
drebrez 459f97ef1d Support multiple architectures in `pmbootstrap menuconfig` command (#814) 2017-10-24 16:18:42 +00:00
Pablo Castellano b4dd7a89d2 Close #709: Improve user creation (#725)
* Allow to specify a custom username in "pmbootstrap init"
* Build chroots have "pmos" instead of "user" as username now
* Installation user UID is 1000 now (as in all other Linux distributions)
* Adjust autologins
* postmarketos-base: enable wheel group for sudo, removed previous sudoers file
* Implement safe upgrade path:
We save the version of the work folder format now, in $WORK/version.
When this file does not exist, it defaults to 0.
In case it does not match the currently required version
(pmb.config.work_version), then ask the user if it should
automatically be upgraded.
2017-10-12 20:08:10 +00:00
Oliver Smith 1285f74c5f Fix #731: Create symlinks for noarch-subpackages (#740)
* apkindex:
  * Also parse the architecture field
* symlink_noarch_package:
  * Renamed to symlink_noarch_packages
  * Always work on all packages (so we don't need to guess which
    subpackages have been generated after a certain build)
  * Get invoked when running 'pmbootstrap index'
  * Use 'apk index' to generate one index, where the architecture
    does not get rewritten (abuild does that by default, due to
    Alpine's repos not having a 'noarch' folder and diverging from
    that doesn't make things easier for us). That goes super fast,
    and then we know which packages are noarch packages and can
    create the symlinks.
* Made output less verbose:
  * Use -q for 'apk index' when calling it directly (when it gets
    called by abuild we can't control that)
  * Output that the APKINDEXes get reindexed only to the 'pmbootstrap
    log'.
2017-10-11 15:11:25 +00:00
Attila Szöllősi ae6a58b6ed Close #554: kernel config checking (#589)
* Check kernel config
* Allow specifying multiple kernel packages, and also no packages
  which defaults to scanning all kernel configs (it is super fast
  anyway)
* Add the check to Travis CI
* Adjust existing kernel configs, so they pass the kconfig_check.
(We've had to put in a lot of defaults in the aarch64
linux-postmarketos configs, that's why the diff is a bit unclean.)
* Increase modified kernel pkgrels
2017-09-18 21:36:54 +00:00
drebrez 207c200229 zap: add `-d`/`--distfiles` argument to clear downloaded files cache (#576) 2017-09-14 18:28:10 +00:00
Oliver Smith bed1eacbb5 Close #256: Implement strict package building mode (#532)
Contrary to abuild, pmbootstrap only installs makedepends, and
keeps the installed packages around - both hacks save lots of time.

However, they may introduce missing makedepends in the APKBUILDs,
that the authors of the APKBUILDs do not notice because it works
for them.

This PR adss a strict mode, which will always clean the chroots
before building a package, and also remove all installed dependencies
after the package was built. You can use the following syntax to
only zap once, but build many packages at once:
`pmbootstrap build --strict hello-world 0xffff heimdall`

It also builds dependencies properly without leaving makedepends
behind.
2017-09-08 23:50:59 +00:00
Oliver Smith b29cc877a7 Alias: "pmbootstrap export" for "pmbootstrap flasher export" (#417)
* moved export code to pmb/export and split it up
* added deprecation notice to "pmbootstrap flasher export"
* made "pmbootstrap export" work
* adjusted the "pmbootstrap flasher export" hints in the code
2017-09-06 20:14:03 +00:00
Attila Szöllősi c6fe6e134a Flavor instead of device name when creating recovery zip (#491)
* Flavor instead of device name when creating recovery zip.
* Move parse_flavor_arg function
2017-09-02 03:53:58 +00:00
clayton craft a9e5b362dc Implement zapping of apk packages with ver different than aports (466) (#474)
This adds a new option to `zap`: `-m / --mismatch-bins`

When set, any binary apks in the work directory packages folder will be
removed if their version differs from the version in the relevant
APKBUILD in aports.
2017-08-28 20:34:03 +00:00
clayton craft d0f09ca0d0 Resolve #361 by zapping existing chroots after init (#385)
This extends zap() to add a 'no_confirm' option (False by default), and
zap() is now called by init with no_confirm=True to automatically zap
any existing chroots after the user runs init. This helps insure that
what is installed in the chroots is exactly what the user expects after
setting options in init.

Additionally, we create `cache_http` to verify write access to the work
folder instead of `chroot_native`. So we can ask for zapping only if
no chroot folder exists.
2017-08-18 16:25:58 +00:00
Pablo Castellano 401c29af76 Close #326: Implement command to retrieve and set configuration values (#359)
* Implement command to retrieve and set configuration values
* qemu: show advice to use "pmbootstrap config"
* Allow "pmbootstrap config" without positional arguments (prints the full config)
* Check valid variable names
2017-08-14 14:25:28 +00:00