### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.
### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
* Default is updating all arches where the APKBUILD files exist
* Add existing_only parameter
* Return True when files have been downloaded
* Properly print which arches will be updated
* Print update reason only in verbose log
* Add and improve comments
* pmb.parse.arguments(), update action:
* Add --non-existing parameter
* Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
* Inform about --non-existing if no APKBUILDs have been updated
Right now, they appear on screen when using --details-to-stdout. This
does not work well with Travis CI and screws up the log.
Disabling the progress bars in abuild works just like Alpine does it in
their Travis CI script: Exporting SUDO_APK as
"abuild-apk --no-progress" instead of "abuild-apk".
test_check_checksums.py: Run "pmbootstrap build_init" before building
any packages, so it is a bit less verbose (downloading the APKINDEX
files etc.). Later we run the build init code again (because we use
--strict while building the packages), but then the APKINDEX files
are already present. So overall the log is a bit shorter before the
building starts. (It is still logged to the logfile, which gets
printed on error anyway.)
This PR makes the workflow faster and pmbootstrap will
produce less traffic. Details:
* Check if it's possible to create and read from a device
node directly when initializing a chroot (closes#472)
* Copy the Qemu binary into the forign-arch chroots
before initializing them, so the post-install script
directly work during the chroot setup and we don't need
to call apk fix afterwards
* Use pmb.helpers.repo.update(), which only updates the
APKINDEX files if they are older than 4 hours, instead
of using apk's repo update function which always
downloads the APKINDEX files
* Chroot initialization
* Getting the initial APKINDEX to download apk-tools-static
* Updating the APKINDEX at the start of pmbootstrap install
* Fixed a bug in from_chroot_suffix: the buildroot_x86_64 has
architecture x86_64, not x86.
This removes any existing symlinks (which always seem to be broken when
this is encountered) to <workdir>/chroot_native/etc/apk/cache before
creating the symlink.
* Allow to specify a custom username in "pmbootstrap init"
* Build chroots have "pmos" instead of "user" as username now
* Installation user UID is 1000 now (as in all other Linux distributions)
* Adjust autologins
* postmarketos-base: enable wheel group for sudo, removed previous sudoers file
* Implement safe upgrade path:
We save the version of the work folder format now, in $WORK/version.
When this file does not exist, it defaults to 0.
In case it does not match the currently required version
(pmb.config.work_version), then ask the user if it should
automatically be upgraded.
This is important for the binary repository scripts, so it's feasible
to test the binary package build and challenge process locally without
setting up a new chroot whenever changing the repo URLs.
Also it behaves a bit more intuitively, because it really uses the
repo URL specified on the commandline, even when the chroot is already
set up.
TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution.
---
pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has.
I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository.
To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on.
In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option.
In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds".
This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before.
Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases).
The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
* New commandline parameter --mirror-pmOS, where the binary repository
URL for postmarketOS can be specified (empty by default as of now,
this will be filled with the real URL once the repo works)
* Do not build packages, when they are in the binary repository and
the version of the package in the binary repository is up-to-date.
* Add a testcase for pmb.build.is_necessary().
As reported in #53, it appears that older versions of eCryptfs don't
really create the device nodes, although `mknod` does not fail on
the commandline. That's fixed now with this extra check.