Unmaintained devices are device packages that:
- Are known to be broken in some way without an active maintainer
who can investigate how to fix it, or
- Have not received any updates for a very long time, or
- Are discouraged from using because they are just intended for testing.
An example for this are ports using the downstream kernel for devices
which have a mainline port that is working quite well.
Unmaintained devices are still built by bpo (otherwise it would not make
sense to keep them), but they do not show up in "pmbootstrap init".
However, it is possible to manually select them by entering the name.
pmbootstrap will warn in that case.
Unmaintained packages should have a # Unmaintained: <reason> comment
in the APKBUILD, this comment is displayed in "pmbootstrap init"
so that the user knows why the device should not be used unless they
know what they are doing.
Made changes to limit the line length in following files,
- pmb/parse/_apkbuild.py
- pmb/parse/apkindex.py
- pmb/parse/binfmt_info.py
- pmb/parse/deviceinfo.py
- test/test_parse_apkbuild.py
Added the above files in E501 flake8 command list.
Substitute f-string for string concatenation.
Alpine indicates with arch="", that a package should temporarily not be
built for any architecture. Support this in postmarketOS too by not
complaining in the APKBUILD parser if arch is empty.
Adjust pmb.build.autodetect.arch and pmb.build.menuconfig.get_arch, so
both don't fail with an IndexError when encountering a disabled package.
Co-Authored-By: Luca Weiss <luca@z3ntu.xyz>
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.
Replacing variables immediately after parsing (before splitting)
has several advantages:
- No need to handle different value types
(e.g. lists by joining them every time they are accessed)
- Variables like depends="$depends ..." are handled directly
by the variable parser
- APKBUILDs are shell scripts, so we match abuild more closely
if variables defined later do not affect previous attributes
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.
We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.
This will also avoid a (somewhat unrelated) error in the bootimg test:
File "pmb/parse/_apkbuild.py", line 46, in replace_variable
apkbuild["pkgname"], match.group(0),
KeyError: 'pkgname'
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:
ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'
This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.
We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().
For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
Properly ignore comments at the end of lines, instead of assuming that
all lines below belong to the attribute:
subpackages="$pkgname-dev" # $pkgname-lang
Fixes build.postmarketos.org#61, where pmbootstrap would assume that a
random package provides "make", just because the word "make" is written
somewhere below subpackages=" in the APKBUILD and it is parsed
incorrectly.
While at it, also support the ' character for quotations and detect if
a quotation for a value was started, but there is no end quotation sign
in the rest of the file.
I've added tests, and manually checked that this did not introduce any
parsing bugs for all the APKBUILDs in pmaports.git, by running
'pmbootstrap apkbuild_parse' with the old and new code, and diffing the
result.
Properly handle the following two cases in APKBUILDs:
* depends="$depends ..."
* depends="${depends} ..."
First I've attempted to refactor the parsing code to do this in a more
generic way. But I've realized that it would make more sense to retire
the python based APKBUILD parsing approach altogether and finally use a
shell script parser. Let's discuss this in #1801.
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
for now. The kernel isn't used right now, and we save lots of
maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
`"downstream", "mainline", "stable"` (downstream is always
`linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
the `deviceinfo_dtb` file is missing (because we expect it to be
missing for downstream kernels)
* device-sony-amami:
* add kernel subpackages for downstream, mainline
* set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
* don't mix specifying kernels in depends *and* subpackages
* 1 kernel in depends is maximum
* kernel subpackages must have a valid name
* Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
`--ignore-depends` can be used with device packages to avoid building
the kernel. The idea was to make the transition easier after a change
we did months ago, and now the kernel doesn't always get built before
building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
* Add autoinstall=True to kernel_flavors_installed(). When the flag
is set, the function makes sure that at least one kernel for the
device is installed.
* Remove kernel_flavor_autodetect() function, wherever it was used,
it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
kernel_flavors_installed() takes care of that now.
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.
After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).
The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:
pmbootstrap chroot --add=git --user -- \
git -C /mnt/pmbootstrap-git/aports_upstream pull
Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
the absolute path to the upstream aport on disk, after checking the
version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
APKINDEX if necessary and returns the absolute path to the APKINDEX.
This code was basically present already, but not as function, so now
we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
the official gcc APKBUILD before we modify it (the current APKBUILD
parser is not meant to be perfect, as this would require a full shell
parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
`testcases_fast.sh`. Previously it was disabled due to traffic
concerns (cloning the aports repo, but then again we do a full KDE
plasma mobile installation in Travis now, so that shouldn't matter
too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
from the disabled-by-default list (left over from timestamp based
rebuilds), there were no more test cases disabled by default. I've
changed it, so now the qemu_running_processes test case is disabled,
and added an `--all` parameter to the script to disable no test
cases. Travis runs with the `--all` parameter while it's useful to
do a quick local test without `--all` in roughly 2 minutes instead of
10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
default Alpine mirror (so the aportgen testcase runs through).
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.
Changes:
* New config file options nonfree_firmware and nonfree_userland, which
we ask for during "pmbootstrap init" if there are non-free components
for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
packages are called $pkgname-nonfree-firmware and
$pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
subpackages. Parsing that is implemented in
pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
the regular APKBUILD parsing, as this would need a change in the
output format, and it is a lot *less* code if done like in this
commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
pmb/install/install.py to _install.py: needed to call the function in
the usual way (e.g. pmb.parse.apkbuild()) but still being able to
test the individual functions from these files in the test suite.
We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
packages that will be installed, based on the user's choice in
"pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
refactored test/test_questions.py to have multiple functions for
testing the various questions / question types from
"pmbootstrap init" instead of having it all in one big function.
This allows to use another aport folder for testing the new
non-free related questions in init.
Minecrell
Newbyte
Shubham Naik
Oliver Smith
Luca Weiss
Oliver Smith
Robert Yang