Use a deterministic mirror URL instead of CDN for aportgen. Otherwise we
may generate a pmaport that wraps an apk from Alpine (e.g. musl-armv7)
locally with one up-to-date mirror given by the CDN. But then the build
will fail if CDN picks an outdated mirror for CI or BPO.
This adds support for specifying an arbitrary git ref (e.g. commits,
tags, branches) to upgrade to. This can be useful if a specific commit
needs to be packaged instead of the latest available. Alternatively you
can also specify a branch to be used if the default branch is 'stable'
but 'develop' should be packaged.
This also removes old code to use the 'bionic' branch for UBports Lomiri
(formerly Unity 8) packages.
The Nokia n900 XkbLayout is a bit peculiar and sometimes
join two keymaps into one, for example:
Option "XkbLayout" "fise"
For the combined finnish/swedish layout. Add the common
joined keymaps, even if not all of these countries are
yet supported.
For details see:
https://gitlab.freedesktop.org/xkeyboard-config/xkeyboard-config/-/blob/master/symbols/nokia_vndr/rx-51
I also include this link in the code so no-one gets confused.
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Some Mediatek devices have a special 512-byte header around the zImage
which must be generated so the device boots.
Support for that exists for a while in postmarketOS but detection was
missing. Add that.
Do not fail in "pmbootstrap setup" if a keymap was selected, but no
/etc/X11/xorg.conf.d path exists in the rootfs chroot. The grep output
is not empty in that case (it would be empty if the directory exists and
there are no matches), so we need to add this extra check:
(rootfs_nokia-n900) % grep -rl XkbLayout /etc/X11/xorg.conf.d/
grep: /etc/X11/xorg.conf.d/: No such file or directory
Replace dl-cdn mirror with dl-2 temporarily to avoid "BAD signature"
errors.
This is related to the rebuild of Alpine edge x86, armhf, armv7 packages
against musl-1.2, that is currently going on. The packages are rebuilt
without a version change and therefore have the same resulting file
name, but a different checksum. Due to caching of the CDN and due to the
same file name, users may get an old package from before the rebuild.
The APKINDEX has a checksum of the new package, and so the download will
fail with a "BAD signature" error.
Alpine developers clear the cache every now and then, but this issue
will happen over and over again, until all packages are rebuilt (some
packages are still not rebuilt and have been disabled).
Let's switch back in two months or so.
Before this fix tab completion of names with hyphens were not done:
try for example to complete UI package name in 'pmbootstrap init' with
hyphen, like "plasma-mobile" / "plasma-desktop". Now this is handled
correctly (hyphen is not considered as delimeter).
Let UI meta-packages specify apps in "pmb_recommends" to be explicitly
installed by default, and not implicitly as dependency of the UI
meta-package ("depends"). Therefore make these apps uninstallable,
without removing the meta-package.
Add pmbootstrap install --no-recommends to disable this feature.
Add a question at the end of "pmbootstrap init", to ask if the user
wants to build outdated packages during "pmbootstrap install". Store the
result in the new pmbootstrap.cfg key "build_pkgs_on_install". I've put it at
the end, because it is a rather complicated question compared to the rest.
This is useful to speed up the installation for casual users who can now
avoid compiling packages. But also for the official images where we only
want to ship the official binary packages and not build anything
on-the-fly.
This is likely to fail with the new default cryptsetup cipher of
aes-xts-plain64, as many downstream kernels used in recovery OS (like
TWRP) do not have CRYPTO_XTS set.
Put a minimum version check for postmarketos-ondev in the pmbootstrap
install code and verify it before starting the installation. This avoids
using incompatible versions, similar to the pmaports.cfg version check
we already have. Set the minimum required version to 0.2.0.
Do not pass the arguments to ondev-prepare as command-line arguments in
a specific order, but instead as environment variables. New arguments
will be added in a follow-up patch.
Checkout the aports.git branch for the current channel (e.g.
3.12-stable), before trying to find the APKBUILD.
I had tried to auto-unshallow the git repository earlier, but then the
tags were missing. I decided that it's not worth to provide a migration
path: "pmbootstrap aportgen" is only used by few advanced users (to
maintain the pmaports repo).
This made sense for Alpine's aports.git repository as we were only using
the master branch. But now that we are using more branches, we need the
entire git repository with all its branches cloned.
Add the Alpine mirrordir (e.g. "edge", "v3.12") to the distfile, so
musl-*.apk files of the same version and arch but from different
mirrordirs do not collide.
Let "abuild checksum" download these apks and generate the checksums,
instead of letting apk download them as side-effect of initializing
foreign arch chroots. The latter did not work anymore, because we would
copy the apk file with a glob that may matches the hash of either mirrordir.
Essentially:
glob.glob(f"{args.work}/cache_apk_{arch}/{subpkgname}-{version}.*.apk")[0]
In the context of the on-device installer, I found that calculating this
hash is not trivial, so let's just avoid it here as well.
While at it, order the imports of musl.py alphabetically.
Add initial support for the on-device installer in pmbootstrap. Let
pmbootstrap create a regular split image, then prepare a new installer
rootfs and copy the previously generated rootfs image into the installer
rootfs. Put the installer rootfs into a new image, with reserved space.
There is more to do from here, such as disabling the generation of the
user account when using --ondev. But this requires support in
postmarketos-ondev first, so let's build that iteratively.
Related: https://wiki.postmarketos.org/wiki/On-device_installer
Related: https://gitlab.com/postmarketOS/postmarketos-ondev/-/issues
Move code that prints flashing information from install_system_image()
to its own function. For the on-device installer, we'll need to call
install_system_image() twice, without printing the flashing information
each time. While at it, add "step" and "steps" parameters.
Prepare for a future patch, that adds reserved space in MiB, by changing
size_boot and size_root from bytes to MB everywhere. This is what we need
most of the time and allows to drop some /1024**2 statements.
Add a "split" argument to the function, instead of using "args.split"
directly. "args.split" is only defined when calling "pmbootstrap install",
but the next patch will add a code path that calls the function from
"pmbootstrap chroot".
Alpine APKBUILDs have the concept of "provider priorities" that affect
the choice of the provider to install when multiple packages provide
a virtual package.
One use case for this is to allow installation of different firmware
versions. bq-paella can run unsigned firmware, therefore you have the
choice between using the original firmware from the manufacturer, or
a slightly newer version from Qualcomm for the Dragonboard 410c.
We add provides="firmware-qcom-msm8916-wcnss" (the "virtual package")
to both firmware-qcom-db410c-wcnss and firmware-bq-picmt-wcnss.
At this point, attempting to install "firmware-qcom-msm8916-wcnss"
would still fail with apk. (Because it does not know which provider
to install.)
To pick a default we can set e.g. provider_priority=100 for
firmware-qcom-db410c-wcnss (the slightly newer version).
In that case, firmware-qcom-db410c-wcnss should be installed by default.
However, the user can choose to do "apk add firmware-bq-picmt-wcnss"
to override the default choice in case of problems. In that case,
the conflicting firmware-qcom-db410c-wcnss will be automatically removed.
At the moment, pmbootstrap does not respect the "provider_priority" at all.
In the above case, it would always install "firmware-bq-picmt-wcnss"
during "pmbootstrap install" since that has the shortest name.
Extend the pmbootstrap code to pick a provider with the highest priority
(if any of the providers has a priority set).
Do not substract the estimated size of the home and boot directories
from the root directory size. While that would be the correct way if we
were able to get exact sizes, it isn't helpful with the very rough
estimates we are getting from pmb.helpers.other.folder_size. Replace
"calculate" wording with "estimate".
"Chassis" defines the what kind of device a device is. Currently the
following standardized types exist:
"desktop", "laptop", "convertible", "server", "tablet", "handset",
"watch", "embedded", "vm", "container"
This property is exposed by org.freedesktop.hostname1 (e.g.
openrc-settingsd) and can be used by applications for example to display
appropriate strings instead of "About this phone" for non-phone devices.
Migrate to workdir version 5 and move already built packages into the edge
channel subdir, for example:
$WORK/packages/x86_64/hello-world-1-r5.apk
to:
$WORK/packages/edge/x86_64/hello-world-1-r5.apk
The build.postmarketos.org code has already been adjusted to find built
packages in either directory structure.
Do not go through the pmb.chroot.init() code path when running
pmb.install.losetup.umount() inside pmb.chroot.shutdown(). This is not
necessary, as pmb.install.losetup.umount() only gets called if the
chroot is already initialized and /dev/loop-control is mounted inside
the chroot.
Not going through this code path is important for the upcoming workdir
migration patch. Without this fix, it will fail with the following if
running "pmbootstrap install" before the work migration:
ERROR: Could not figure out on which release channel the 'native' chroot is
Replace the call to this function with the almost identical
pmb.helpers.repo.apkindex_files(), so release channel related changes
only need to be done in one place.
Use mirrordir_pmos and mirrordir_alpine from channels.cfg to generate
the mirror URLs for postmarketOS and Alpine, which get written to
/etc/apk/repositories and which postmarketOS uses to download the
APKINDEX files.
Remove hardcoded "master" at the end of the postmarketOS mirror and use
mirrordir_pmos instead (which is "master" for the edge channel). Let the
postmarketOS mirror end in a '/' for consistency with the Alpine mirror
in pmb/config/__init__.py.
Remove obsolete --alpine-version. To experiment with a different Alpine
version, one should pass a custom --config-channels from now on.
Don't generate the postmarketOS mirror URLs here, let the urls()
function do it. A follow-up commit will touch this code, hence it's
important to have it de-duplicated.
Ask for release channel and switch pmaports branch to the related branch
defined in channels.cfg.
Store in pmbootstrap.cfg whether the user chose a channel (boolean). If
the user did not choose a channel yet, suggest the recommended channel
from channels.cfg (currently "edge").
Do not make the parsed pmaports.cfg from pmaports.git available as
args.pmaports anymore. This de-bloats the args variable a bit.
First I thought that we didn't even need to cache it, but it was pointed
out that later patches do access it frequently to read the current channel
from pmaports.cfg. Therefore it is using a cache now.
Related: #1879, #1855
New pmaports.cfg has the "channel" key described in [1]. This will be
used in "pmbootstrap init" to determine on which release channel a
forked branch is.
[1] https://postmarketos.org/pmaports.cfg
Prepare to base postmarketOS on Alpine stable by parsing the new
channels.cfg file in pmaports.git, that describes which channel
needs which branches and mirror dirs from postmarketOS and Alpine.
Use the information in pmb.helpers.git.get_branches_official() first,
more is coming in follow-up commits.
Read the file from origin/master, so we get the latest fetched version
even if the last checked out master branch is not up-to-date (think of
currently checked out release branch instead of master, master will
never be updated to point to latest origin/master). Allow to override
the file with a new --config-channels parameter.
Related: https://postmarketos.org/channels.cfg
Most device ports that are added to pmaports (particularly the testing
category) are not actually "maintained". Many of them are never updated
after the initial contribution. There is little reason to list a
"Maintainer:" if the device package is actually not actively maintained
by that person.
Let's stop generating the Maintainer: line by default.
Instead, contributors should add the lines themselves if they are willing
to (actively) maintain the device package for a longer period of time.
Once they no longer want to maintain the package, the Maintainer:
line should be removed again.
Let apkbuild-lint check options again, and pass the pmbootstrap specific
options with the new CUSTOM_VALID_OPTIONS variable. Add a test case and
adjust pmb.helpers.lint.check to return the output of apkbuild-lint, so
we can properly test it.
Related: https://gitlab.alpinelinux.org/Leo/atools/-/merge_requests/28
Fixes: pmaports#553
The `mesa-dri-swrast` package was merged into `mesa-dri-gallium` in Alpine.
(See 298e20d04f)
The old alias might be removed in the future, so change the generated APKBUILDs
to depend on `mesa-dri-gallium` instead.
Many of the multi-line strings already contain a new line at the
end of the file. When using .split("\n") this will result in an empty
line at the end of the file that is again followed by a new line.
In other words: we have two new lines at the end, which looks weird.
.rstrip() the whitespace at the end of the string to avoid this.
In case a package can't be built using crossdirect, add an APKBUILD
option to unconditionally disable crossdirect and use the slower
distcc approach instead. This is needed e.g. when using LD_PRELOAD during
the build as crossdirect cannot work with that.
Have explicit selection of the "native" cross compilation method with a
new "pmb:cross-native" option. Deprecate the implicit pkgname pattern
matching.
Related: #1910
The ui-extras questions will attempt to find a postmarketos-ui-<ui>
package in pmaports. If the package does not exist as "root" APKBUILD
it currently attempts to parse all APKBUILDs in case it is somewhere
defined as a subpackage. This is really slow (up to 2-3 seconds),
which feels weird during "pmbootstrap init".
For the UI packages we specifically look for the root UI package,
not the subpackage, so let's skip searching for subpackages in this
case. This makes selecting the "none" UI nice and fast again.
Fix the function, so it does not crash anymore when the
replace_subpkgnames argument is set and a dependency cannot be resolved.
Instead, print a useful warning that shows which pmaport caused the
error (that has always been a pain to figure out), and simply don't
replace the potential subpkgname with the real pkgname, just use the
dependency name as-is.
Resolve annoying crashes in bpo dependency resolving, like this one
(caused by a few linux-* pmaports for bad downstream kernels that depend
on python, not apparent at all from the message):
[09:08:15] Calculate packages that need to be built (all packages, aarch64)
[09:08:26] ERROR: Package 'python': Could not find aport, and could not find this package in any APKINDEX!
Related: https://builds.sr.ht/~postmarketos/job/184022
Previously these two commands would both print the current value:
pmbootstrap config extra_packages
pmbootstrap config extra_packages ''
With this change, the second command will instead set the given config
value to the empty string.
Prevent "pmbootstrap pull" from failing with:
NOTE: your pmaports folder has version 4, but version 6 is required.
ERROR: Run 'pmbootstrap pull' to update your pmaports.
Fixes: #1900
Use pmb.helpers.args.add_cache() with the args used during
autocompletion, so pmb.helpers.pmaports._find_apkbuilds() does not fail
when trying to access args.cache.
No build is necessary if pmaport can't be built for given arch.
pmbootstrap must use Alpine's binary package in that case, even if the
pmaport version is higher than Alpine's binary package version.
Fixes: #1897
The --no-depends option is supposed to stop pmbootstrap if it was
instructed to build a package, but a dependency must be built first. So
far, this only covers the case if there is no binary package for a dependency.
Make it stop if the binary package exists, but is outdated, too.
Fixes: #1895
At the moment we set samsung-i9100 as default device.
This is probably only for historical reasons.
For someone/something using pmbootstrap without a specific device in mind
(e.g. CI) it's really better to use a generic device. QEMU runs natively
in a virtual machine so everyone can use it for testing.
Flashes device vbmeta partition (can be overriden with
"flash_fastboot_partition_vbmeta" setting in deviceinfo)
with custom vbmeta.img which has verity flag disabled,
so device can boot postmarketOS with no problems.
The kconfig check searches the aport with the "linux-" prefix to the
package name passed as argument. This is not working with the full
package name like linux-device-name because it searches a
linux-linux-device-name and fails.
Use the timestamp of .git/FETCH_HEAD in each git repository, to
determine if too much time has passed since the last fetch/pull.
Modify pmb.helpers.git.clone, so FETCH_HEAD is always created if it does
not exist (because "git clone" would not create it).
Related: #1829
Extend "pmbootstrap status" with checks for all git repositories,
derived from relevant checks in pmb.helpers.git.pull (using shared
code):
* on official branch
* workdir is clean
* tracking proper remote
* up to date
Related: #1829
Add dummy function that only returns ["master"] for now, so we can use it
in the upcoming git checks for "pmbootstrap status". More sophisticated
logic to figure out the branches will be added soon, see project
direction 2020 issue.
Related: #1829
Device ports should be under the device/testing/* directory for now.
In the future, devices may be moved to device/{main,community} after
certain preconditions are met.
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.