In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.
Replacing variables immediately after parsing (before splitting)
has several advantages:
- No need to handle different value types
(e.g. lists by joining them every time they are accessed)
- Variables like depends="$depends ..." are handled directly
by the variable parser
- APKBUILDs are shell scripts, so we match abuild more closely
if variables defined later do not affect previous attributes
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:
ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'
This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.
We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().
For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
postmarketOS/pmaports!700 adds a new "deviceinfo_bootimg_dtb_second"
option that places the DTB in the "second" area of the Android boot
image.
Attempt to detect this automatically by checking the extracted
second binary for the FDT magic (0xd00dfeed).
This MR add the --fork-alpine argument to the pmboostrap aportgen
command, which downloads the APKBUILD and related files and copies them
into the pmaports/temp folder.
Properly handle the following two cases in APKBUILDs:
* depends="$depends ..."
* depends="${depends} ..."
First I've attempted to refactor the parsing code to do this in a more
generic way. But I've realized that it would make more sense to retire
the python based APKBUILD parsing approach altogether and finally use a
shell script parser. Let's discuss this in #1801.
Explicitly depend on mpc1. Our generated gcc aports use the !tracedeps
option, so we need to explicitly set the libraries it depends on.
This has mostly not been an issue, as we are installing our gcc
packages together with Alpine's gcc package, which causes the libraries
to get installed anyway.
Related: pmaports#236
Provides a quick way to incrementally compile a kernel and push it to
device.
Example usage.
Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX
Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
APKBUILD should be indented with tabs not spaces.
This also resolves a bug which the replace_functions feature was always
trimming off the beginning of a line.
For example:
replace_functions = {"build": "return 0"}
would be formatted as "rn 0".
test_aportgen.py fails because some test data is missing.
Make the test pass by fixing the test data with the following changes:
- Update testdata pmaports/cross/*
- Don't git ignore test/testdata/aportgen directory
- Add testdata aports/main/* from upstream Alpine Linux aports
Patch files in pmaports/cross/binutils-armhf were removed, as they are
not needed to test the pmbootstrap code.
* Travis and Coveralls badges
* aports: instead of <https://github.com/postmarketOS>, use
<https://postmarketos.org>
* References to full URLs to issues and pull requests replaced with
a hash and the number
* grsec check: simplify error message, remove link to github issue
(nobody is using that anymore anyway)
`pmbootstrap pkgrel_bump --auto` automatically increases the pkgrel for
packages linking against libraries, which don't exist anymore (because
the soname has been changed). The feature is explained in detail in
The previous implementation did not detect soname breakage, when a
subpackage linked against a certain library, but the main package
did not (e.g. `qt5-qtbase-mysql` and `qt5-qtbase`). This was, because
we iterated over the aports/* to find the packages to be checked.
To fix this, we are iterating over the packages found in the APKINDEX
files instead (of both the locally compiled packages and the downloaded
index from the pmOS mirror).
Details:
* `pmb/helpers/pkgrel_bump.py`:
* Rewrite `auto_apkindex_package()` to act upon a given parsed
`aport` and `apk` (from the index) instead of finding the `apk`
dict by itself (we need it earlier anyway).
* Rewrite `auto()` to iterate over APKINDEX files instead of aports
* Skip packages already found, so the `pkgrel` does not get
increased multiple times when the same package was found in
multipe index files.
* Put the package name at the beginning of the log messages to make
them more readable
* testdata: Create a new `testsubpkg` aport, where only the subpackage
links against `testlib`
* Adjust testing code to test everything with `testsubpkg` as well.
NOTE: This makes the command a bit slower, but we could improve
performance again by smart caching of `pmb.parse.apkindex.depends()`.
This could come in a future PR, the important part here is that the
command is bug-free again with this fix.
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
for now. The kernel isn't used right now, and we save lots of
maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
`"downstream", "mainline", "stable"` (downstream is always
`linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
the `deviceinfo_dtb` file is missing (because we expect it to be
missing for downstream kernels)
* device-sony-amami:
* add kernel subpackages for downstream, mainline
* set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
* don't mix specifying kernels in depends *and* subpackages
* 1 kernel in depends is maximum
* kernel subpackages must have a valid name
* Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
`--ignore-depends` can be used with device packages to avoid building
the kernel. The idea was to make the transition easier after a change
we did months ago, and now the kernel doesn't always get built before
building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
* Add autoinstall=True to kernel_flavors_installed(). When the flag
is set, the function makes sure that at least one kernel for the
device is installed.
* Remove kernel_flavor_autodetect() function, wherever it was used,
it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
kernel_flavors_installed() takes care of that now.
Since PR #1247 we are using the virtual package `.pmbootstrap` to mark
packages as not explcitly installed. In case `pmbootstrap` doesn't
finish the installation because of a bug, the `.pmbootstrap` virtual
package may not get uninstalled.
As virtual packages don't have the "timestamp" attribute set in the
package database (which indicates when the package was built), the
APKINDEX parser fails to parse them.
Changes:
* pmb.parse.apkindex.parse_next_block(): don't require the "timestamp"
attribute to be set (but the arch attribute instead, which is always
present)
* pmb.parse.apkindex.parse(): when a block does not have a `timestamp`
attribute, skip it, because it must be a virtual package.
* add test cases for both functions with a package database that
contains a virtual package.
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.
Changes:
* New config file options nonfree_firmware and nonfree_userland, which
we ask for during "pmbootstrap init" if there are non-free components
for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
packages are called $pkgname-nonfree-firmware and
$pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
subpackages. Parsing that is implemented in
pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
the regular APKBUILD parsing, as this would need a change in the
output format, and it is a lot *less* code if done like in this
commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
pmb/install/install.py to _install.py: needed to call the function in
the usual way (e.g. pmb.parse.apkbuild()) but still being able to
test the individual functions from these files in the test suite.
We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
packages that will be installed, based on the user's choice in
"pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
refactored test/test_questions.py to have multiple functions for
testing the various questions / question types from
"pmbootstrap init" instead of having it all in one big function.
This allows to use another aport folder for testing the new
non-free related questions in init.
* The APKINDEX parser used to return a dictionary with one package for
a given package name. This works for the installed packages database,
because there can only be one provider for a package. But when
parsing packages from binary repositories, we need to support
multiple providers for one package. It is now possible to get a
dictionary with either multiple providers, or just a single provider
for each package.
* Dependency parsing logic has been adjusted, to support multiple
providers. For multiple providers, the one with the same package
name as the package we are looking up is prefered. If there is none
(eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
that will be installed anyway, and after that packages that are
already installed. When all else fails, it just picks the first one
and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
that pmbootstrap should build if neccessary, and install in the
chroot. This can be used to quickly test the depencency resolution
of pmbootstrap without doing a full "pmbootstrap install".
Fixes#1122.
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
we can write into the source folder if necessary, for better
compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
usually), we extend the APKBUILD with overrides to make it use
mountpoint's source instead of downloading the package's source
from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
* Testsuite: Run UIs in Qemu and check running processes (and other changes)
* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
built before running tests (otherwise tests would hang without any output
while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
plasma-mobile currently, easy to extend), runs it via Qemu and checks the
running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
our source tree, so we don't need to clone their git repo anymore. Now it
is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.
* Various fixes
* Build device-packages for the device arch and don't raise an
exception, but print a note if --ignore-depends is not specified
and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
completion
* Exit with 1 when showing usage in check_checksums.py
Fixes#893. Changes:
* New action: "pmbootstrap pkgrel_bump"
* pmbootstrap detects missing soname depends when trying to install
anyting, and suggests "pkgrel_bump --auto" to fix it
* Testcase test_soname_bump.py checks the pmOS binary package repo
for soname breakage, so we see it when CI runs for new PRs
* libsamsung-ipc: bump pkgrel because of soname bump