In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.
After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).
The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:
pmbootstrap chroot --add=git --user -- \
git -C /mnt/pmbootstrap-git/aports_upstream pull
Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
the absolute path to the upstream aport on disk, after checking the
version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
APKINDEX if necessary and returns the absolute path to the APKINDEX.
This code was basically present already, but not as function, so now
we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
the official gcc APKBUILD before we modify it (the current APKBUILD
parser is not meant to be perfect, as this would require a full shell
parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
`testcases_fast.sh`. Previously it was disabled due to traffic
concerns (cloning the aports repo, but then again we do a full KDE
plasma mobile installation in Travis now, so that shouldn't matter
too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
from the disabled-by-default list (left over from timestamp based
rebuilds), there were no more test cases disabled by default. I've
changed it, so now the qemu_running_processes test case is disabled,
and added an `--all` parameter to the script to disable no test
cases. Travis runs with the `--all` parameter while it's useful to
do a quick local test without `--all` in roughly 2 minutes instead of
10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
default Alpine mirror (so the aportgen testcase runs through).
Example: Building gcc-armhf for armhf does not make sense, so this
commit changes arch="all" to arch="aarch64 x86_64". This helps to
simplify the repository scripts (#970).
* pmbootstrap init: Generate new port device- and linux-package
* adds `pmbootstrap aportgen device-*` and
`pmbootstrap aportgen linux-*`
* ask for confirmation when selecting a non-existing device
* generate the packages directly from init
* refactor aportgen code
* fixed some easy things in the linux- APKBUILD (more to come in
follow-up PRs!)
Testing:
* Test all questions to the user from pmb.config.init and pmb.aportgen.device
(except for the timezone question, because we would need to monkeypatch the
os.path.exists() function, which messes up pytest, so we'd need to refactor
the timezone function to be more testsuite friendly first)
* Run the device wizard in a testcase a few times and check the output, that
pmbootstrap.aportgen.device and pmbootstrap.aportgen.linux create by parsing
the resulting APKBUILDs and deviceinfo and checking its contents.
* Build the generated device package once in the same testcase
Thanks a lot to @drebrez for all the help with this one:
<https://github.com/postmarketOS/pmbootstrap/pull/821>
See also the updated porting guide:
<https://wiki.postmarketos.org/wiki/Porting_to_a_new_device>
This fixes https://github.com/postmarketOS/binary-package-repo/issues/1
GCC generates hardlinks between files `A` and `B` in its `make install` step. The problem is, that `tar` randomly packages `A` as full binary, and links `B` to `A`, or the other way around! I was able to reproduce this issue consistently when re-building `gcc-aarch64` on Travis CI (interestingly, this did not appear for `gcc-armhf`).
The fix is, to delete `B` and create a symlink `B` that points to `A` instead.
libstdc++.a from gcc-armhf was not reproducible on Travis (it was, when built locally!). These .a files are just archives of object files .o, and in this case it was caused by a random order of the .o files in the archive.
This PR patches the package() function of the APKBUILD when running pmbootstrap aportgen gcc-armhf (same for aarch64 of course), to extract all .a files, and repack them to be reproducible (by sorting the files before packing them).
As usually, we can still inherit everything from the upstream gcc aport from Alpine, and apply our changes on top of that.
Travis without the patch:
https://api.travis-ci.org/jobs/260402679/log.txt?deansi=true
> CHALLENGE FAILED for usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a:File 'usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a' is different!
Travis with the patch (I've instructed Travis to run off this branch to test it):
https://api.travis-ci.org/jobs/260806203/log.txt?deansi=true
> Done. Your build exited with 0.
100% tested. Assuming that you don't need to test an obvious change,
because it only changes one line is dangerous. I will learn from this,
sorry for the inconvenience.
`gcc-armhf` compiles fine again with that change, and it is able to
cross-compile packages as it used to.