Add the "args" fixture to test_skip_already_built, so it correctly
initializes logging (and indirectly pmb.helpers.other.cache). With this
it becomes possible to run the test file on its own. See "def args" on
top of the file for implementation of the "args" fixture.
Fix for:
$ pytest -xvv test/test_build_package.py
…
> assert pmb.helpers.other.cache["built"] == {} E TypeError: 'NoneType' object is not subscriptable
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230724070509.1885-2-ollieparanoid@postmarketos.org%3E
Fix that the list "cmd" was turned into a string if one of the proxy
vars was set in the environment. Add a test for this code path. Before
this patch:
$ FTP_PROXY=test pmbootstrap -v --details-to-stdout status
…
% cd /home/user/.local/var/pmbootstrap/cache_git/pmaports; git remote -v
run: FTP_PROXY=test ['git', 'remote', '-v']
ERROR: [Errno 2] No such file or directory: "FTP_PROXY=test ['git', 'remote', '-v']"
Fixes: 1a00c04f ("pmb.helpers.run_core: always configure proxy vars if set in environment")
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230713182337.6185-3-ollieparanoid@postmarketos.org%3E
With this code path, pmbootstrap would start a distccd + sshd in the
native chroot, and configure it so it runs the cross compiler. The
foreign arch chroots would then call this cross compiler from localhost
by calling the distcc client instead of gcc.
This code has been obsoleted by the much simpler crossdirect in 2019.
Let's finally remove it.
Fixes: issue 2179
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230613161437.570196-4-ollieparanoid@postmarketos.org%3E
Rename the file so it runs last, as it's currently failing a lot of
times on sourcehut. When I looked into it last time, it seemed to be
related to CDN caching and then it worked again so I didn't look further
into it at the time. Move it to the end until this is figured out
properly, so at least we see if all other tests are passing.
Replace the old kconfig check tests with new ones, that do not rely on:
* any kernel config in pmaports
* the currently required kconfig options set in pmb/config/__init__.py
This will make it much easier to change the required options, and moves
the testsuite a bit more towards making it completely independent from
current pmaports state.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230312151325.1968-13-ollieparanoid@postmarketos.org%3E
pmOS cannot boot without these two. There was recently someone in the
unofficial postmarketOS Telegram group who used some minimal config and
couldn't figure out why the initramfs couldn't start up, and these
options being disabled turned out to be the culprit. As such, let's
ensure these are enabled to make life easier for people setting up new
kernel configurations.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230128203716.152281-1-newbyte@postmarketos.org%3E
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.
Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.
This speeds up multiple iterations of building the same package
significantly. I'm using it for:
$ pmbootstrap build postmarketos-mkinitfs --src=...
I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.
Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
Running abuild on the host directly creates directories in the
aport where it gets built. Interrupting abuild results in those
working-dirs not getting deleted.
We don't want to copy those entries to our builder.
It's only really noticeable if pmbootstrap tries to copy a broken
symlink in src/ and thus fails.
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221207205201.22139-1-jane400@bingo-ev.de%3E
The typical workflow for upgrading cross/gcc-* is:
$ cd pmaports/cross
$ pmbootstrap aportgen gcc-*
Currently this is failing because the APKBUILD has been updated in
Alpine for gcc, but gcc has not been built for all arches yet. This
shouldn't prevent us from generating the proper updated APKBUILDs in
pmaports so just print a note and don't fail here.
I'm committing this directly to master as this currently breaks
test/test_aportgen.py::test_aportgen.
Replace .gitlab-ci.yml with a minimal .build.yml that runs a script in
the .ci dir for each task. Each of these scripts runs as root first,
installs dependencies, and then drops rights to the testuser and runs
the actual test. I went with this design, so we can add a
'pmbootstrap ci' command in following patches that run these in a
pmbootstrap chroot (see pmaports#2169).
Looking at flake8, we currently ignore W504 and W604. Would be nice to
fix these instead, but let's do that in another patch.
I've added a minver package to Alpine, so we don't need to install it
form pip anymore. Use minver's new --lint argument to simplify the
script, and as recommended by minver itself, use --no-parse-comments as
speed optimization since we don't have minver specific comments.
Using doas instead of sudo in build.yml would be nice, but this doesn't
work out of the box. Can be changed in the future.
Add a note to the pytest script, that 'pmbootstrap log' can be used to
follow the log while tests are running.
Device packages shouldn't depend on it anymore, in order to end up with
a smaller installation for ui=none this package only gets pulled in by
postmarketos-base if mesa is installed.
Related: pmaports MR 3478
abuild traces shared library dependencies for binaries in built
packages, and adds all of them to depends. This is needed for the cross
gcc builds, as otherwise it will not add the isl dependency (currently
isl25).
With the crossdirect compilation method we mount the cross-gcc from the
native chroot in the foreign arch chroot. Due to the missing dependency,
isl was not installed in the native chroot, and so when attempting to
use that native cross gcc from the foreign arch chroot, it would try to
load the foreign arch isl instead of the native one. Resulting in lots
of the following errors (and of course not a working gcc):
Error relocating /usr/lib/libisl.so.23: unsupported relocation type 1026
I had probably added '!tracedeps' because abuild complains about it
otherwise... but as noted in [1] these complaints are safe to ignore.
Keep !strip for now even though it's not in Alpine's APKBUILD. I've
tried to build without and the build fails with:
strip: Unable to recognise the format of the input file `./usr/aarch64-alpine-linux-musl/lib/libgcc_s.so.1'
[1]: 0415ebf6f9
Related: pmaports issue 1732
Do not build the foreign arch libgcc. We are using the one provided by
Alpine's binary repository instead, and when building it here and trying
to use it, we actually run into a conflict.
I tried setting BOOTSTRAP="nolibc" first since there is already logic to
then disable libgcc in the APKBUILD. But that also disables g++ which we
do want to build, so we can't use this.
Fixes: issue 2168
Remove it from depends, as the package has been removed in Alpine and
since quite some time we shouldn't have depended on it anyway.
Related: issue 2167
We'll be enabling the kernel options for waydroid which would break this
test. Change it to UEFI check instead which will probably really never
be used on N900.
Set CTARGET and CTARGET_ARCH on top of the generated APKBUILD and use
the existing code in the APKBUILD for build(), package(), makedepends,
subpackages. This is now possible, because the pmbootstrap parser for
APKBUILDs has been improved a lot since this was originally written and
because the Alpine APKBUILD does everything that was hacked in here,
like removing man, info directories and bfd-plugins if doing a cross
build.
This is now very similar to bootstrapping binutils for another
architecture via scripts/bootstrap.sh in aports.git.
Most importantly this sets --disable-gprofng, which fails to build
against musl and is the reason why our cross/binutils-* packages
generated from binutils 2.39.x didn't build anymore.
Related: https://git.alpinelinux.org/aports/tree/main/binutils/APKBUILD
Related: https://sourceware.org/bugzilla/show_bug.cgi?id=29477
This includes setting header_version="2" as well as dtb_offset
according to the input boot.img when header v2 is detected.
Also adds the following previously missed deviceinfo_attributes:
* "header_version"
* "bootimg_custom_args"
And fixes failing tests now that header_version is always parsed from
boot.img files.
When using a kernel from Alpine the kernel configuration cannot be
found in pmaports. We cannot check the kernel config for missing
options in that case, but that's no reason to break the flasher
entirely.
Check if the pkgnames are sane in install_run_apk, right before running
apk. This makes sure that we really run it on all arguments that are
supposed to be packages / files and not options to apk.
Previously to this patch, pmbootstrap would pass a full dependency tree
to "apk add". It would use a virtual package to ensure only the right
packages get added to /etc/apk/world. For example:
apk add -u --virtual .pmbootstrap postmarketos-base device-asus-me176c \
postmarketos-ui-sxmo-de-sway device-asus-me176c-nonfree-firmware w3m \
sfeed clickclack firefox-esr font-noto font-noto-emoji gnome-icon-theme \
imv megapixels mobile-config-firefox ttyescape postmarketos-base-nofde \
eudev openssh postmarketos-mkinitfs postmarketos-mvcfg postmarketos-keys \
...
Instead of doing that, only pass the packages we want to install and let
apk figure out the dependencies. Most of the time we can even avoid
using the virtual package now.
== Remaining edge case: locally built packages
apk will only upgrade a package with the same pkgver + pkgrel but a
different build date if the full path to an apk file gets passed as
argument. So if the user built a package locally that will be installed,
or one of its dependencies then we still need to use a virtual package
and possibly pass a dependency to apk. Replace
replace_aports_packages_with_path() with packages_get_locally_built_apks()
to get a list of such packages and adjust install() and
install_run_apk() to use it.
Make the code easier to read by moving split_to_add_del() to a separate
function and do some related refactoring. A future patch will use it
twice in install().
Move "arch = ..." to the top of the function while at it, since it's
needed later in the function in 2 places and is not needed for figuring
out packages_with_depends, to_add and to_del.
Remove "# Add depends to packages" because it's obvious from the
packages_with_depends variable name, and getting to_add/to_del is a
different action that stood under the same comment.
install_is_necessary used to do the following things:
1. Error out if there's no binary package but pmb was invoked as
"pmbootstrap install" and build_pkgs_on_install is disabled.
2. Build the package if necessary.
3. Return if a package "needs to be installed" (Boolean or Float).
The only caller of the function is pmb.chroot.apk.install. It would not
add the package to the long "apk add" command if according to 3. it does
not need to be installed.
When I implemented this a few years ago, I probably thought it would be
useful to not unnecessarily pass packages to apk. But this actually
makes it more complicated and doesn't have a benefit, apk is perfectly
capable of recognizing which packages it had already installed.
Replace the function with a much simpler pmb.chroot.apk.install_build,
which only does 1. and 2. Change the order of the package, arch
arguments to match called functions pmb.parse.apkindex.package and
pmb.build.package.
So it looks like some packages might be mentioned in the "depends=" of
other packages and marked as conflict, for example:
depends="!foo"
..and the conflicting package can be dropped from the repo entirely if
it is no longer needed, however the reference to it in `depends=` still
exists. This handles that situation by just ignoring the missing package
if it is only a conflicting dependency.
fixes https://gitlab.com/postmarketOS/pmaports/-/issues/1525
Oliver Smith
Pablo Correa Gómez
Caleb Connolly
Hugo Osvaldo Barrera
Newbyte
Jane Rachinger
David Wronek
Luca Weiss
Jami Kettunen
Minecrell
Clayton Craft
Bart Ribbers
Laszlo Molnar
Johannes Marbach
Anjandev Momi
Alexey Min