Use mirrordir_pmos and mirrordir_alpine from channels.cfg to generate
the mirror URLs for postmarketOS and Alpine, which get written to
/etc/apk/repositories and which postmarketOS uses to download the
APKINDEX files.
Remove hardcoded "master" at the end of the postmarketOS mirror and use
mirrordir_pmos instead (which is "master" for the edge channel). Let the
postmarketOS mirror end in a '/' for consistency with the Alpine mirror
in pmb/config/__init__.py.
Remove obsolete --alpine-version. To experiment with a different Alpine
version, one should pass a custom --config-channels from now on.
Prepare to base postmarketOS on Alpine stable by parsing the new
channels.cfg file in pmaports.git, that describes which channel
needs which branches and mirror dirs from postmarketOS and Alpine.
Use the information in pmb.helpers.git.get_branches_official() first,
more is coming in follow-up commits.
Read the file from origin/master, so we get the latest fetched version
even if the last checked out master branch is not up-to-date (think of
currently checked out release branch instead of master, master will
never be updated to point to latest origin/master). Allow to override
the file with a new --config-channels parameter.
Related: https://postmarketos.org/channels.cfg
Use pmb.helpers.args.add_cache() with the args used during
autocompletion, so pmb.helpers.pmaports._find_apkbuilds() does not fail
when trying to access args.cache.
Flashes device vbmeta partition (can be overriden with
"flash_fastboot_partition_vbmeta" setting in deviceinfo)
with custom vbmeta.img which has verity flag disabled,
so device can boot postmarketOS with no problems.
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
For KVM the code is run pretty much natively on the host CPU, so all
CPU extensions available on the host CPU can be also used inside the VM.
To expose that information to the VM we should pass "-cpu host", so the
VM is aware of which CPU is in use.
For CPU emulation, QEMU uses a rather minimal CPU on x86_64 by default.
It does not have support for SSE3/4 etc, which may be required for some
applications to work properly (e.g. Android in Anbox). Add a --cpu flag
to make the emulated CPU configurable. Useful values are for example
--cpu max to emulate all implemented CPU features.
To test QEMU's CPU emulation it is useful to have a switch to disable
KVM, even when it is available (and potentially working fine).
Add --no-kvm for that purpose.
For some reason, the SDL display backend changes the video resolution
to 1024x768, while the GTK display keeps it at 640x480.
This is annoying, because at the moment we can only set one display
resolution for a device in postmarketOS (e.g. for the splash screen).
At the moment, the resolution for the splash screen is set to 640x480,
which therefore shows up too small with the default SDL display.
It seems like the display resolution can be only changed in the guest
directly. Linux has a video= kernel parameter that can be used to
implement this. (See: https://www.kernel.org/doc/html/latest/fb/modedb.html)
Let's set 1024x768 by default, but make it configurable through --video.
The QEMU 'tablet' input device reports absolute positions instead
of relative mouse pointer movements. This can be used to automatically
grab/release the mouse pointer when entering/leaving the QEMU window,
instead of having to release it with CTRL + ALT + G manually.
This is quite convenient and should be the default option normally,
but at least on my PC the mouse pointer is reported with some vertical
offset for some reason (you can't reach the top and it extends below
the QEMU window...). Let's add it as an optional --tablet option for now.
When using pmbootstrap, you usually select the device you want to work
on using 'pmbootstrap init', generate the rootfs and can then run more
commands in the context of the device.
The same needs to be done before using QEMU (to generate the rootfs).
But for some reason 'pmbootstrap qemu' requires setting the --arch
parameter when running QEMU for a foreign architecture, even when the
device is still selected in pmbootstrap.
Even more confusing is that setting "--arch arm" always selects
device-qemu-vexpress, but this is not immediately clear from the name.
Let's make this a lot more intuitive by making sure there is a QEMU
device selected when running 'pmbootstrap qemu'. We can then use the
device information to infer the architecture automatically.
At the moment, the --display argument is a bit complicated to use.
A common use would be to switch between the UIs (sdl, gtk, none)
or to enable the software rasterizer. Split the two use cases
to separate arguments to make it more intuitive.
mesa-dri-swrast and mesa-dri-virtio are both provided by mesa-dri-gallium
now, so this option does not have much use anymore. With both selections,
exactly the same packages are installed.
The SPICE UI option tends to be broken (see #1836), and even when it is
working, it is not working particularly well. QXL requires special handling
in our QEMU packages, when now virtio-gpu (virgl) is working quite well overall.
Apparently it is possible to use virgl with SPICE; but only when using
a Unix socket instead of a TCP port. That again is a bit complicated
because we run QEMU outside the chroot and the SPICE client within.
Overall it does no longer seem to be worth the effort.
The default QEMU UI is working just fine (for the purposes of testing
postmarketOS at least).
Add new "pmbootstrap status" command, which does a quick health check
for the work dir. As first health check, verify that the chroots are not
too old. Replace the reminder text at the end of "pmbootstrap init" to
tell users to run "pmbootstrap status" instead of "pmbootstrap zap" once
a day before working with pmbootstrap.
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.
Replacing variables immediately after parsing (before splitting)
has several advantages:
- No need to handle different value types
(e.g. lists by joining them every time they are accessed)
- Variables like depends="$depends ..." are handled directly
by the variable parser
- APKBUILDs are shell scripts, so we match abuild more closely
if variables defined later do not affect previous attributes
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.
We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.
This will also avoid a (somewhat unrelated) error in the bootimg test:
File "pmb/parse/_apkbuild.py", line 46, in replace_variable
apkbuild["pkgname"], match.group(0),
KeyError: 'pkgname'
The gist of this action is upgrading the specified aport to the latest
version. There are implementations for both stable packages (which check
via the release-monitoring.org API for new versions) and git packages
(which check the GitLab/GitHub API for new commits on the main branch).
There's also the possibility to pass --all, --all-stable & --all-git to
the action which either loops through all packages, or just stable or
git packages and upgrades them.
The --dry argument is also respected.
Note, that the implementation does update the variables pkgver, pkgrel
and _commit but it doesn't update the checksums because that would slow
down the process a lot, and is potentially undesirable.
asus-me176c has a Fastboot interface that can be used for flashing,
but in postmarketOS we do not use Android boot images for it.
This is because is it not very practical - the boot partition is
quite small and there is a (custom) EFI bootloader that can boot
directly from any other FAT32 partition.
At the moment the installation process is manual:
1. pmbootstrap install --split to have separated boot (FAT32)
and rootfs images
2. pmbootstrap export
3. Flash boot and rootfs images manually using Fastboot
The "fastboot-bootpart" flasher implements that process in a more
convenient way. When a device uses the "fastboot-bootpart" flasher:
- We generate --split images on "pmbootstrap install" by default.
(This can be disabled using --no-split instead.)
- pmbootstrap flasher flash_kernel flashes the raw boot partition
(not an Android boot image) using Fastboot, just like the rootfs.
There are some limitations that could be improved in the future:
- "fastboot-bootpart" is not offered in the device wizard.
I think it is special enough that no-one will be starting with it,
and the difference to normal "fastboot" might be confusing.
- Support "pmbootstrap flasher boot". asus-me176c does not support
"fastboot boot" properly, but theoretically we could still generate
Android boot images to use when booting an image directly.
- At the moment the boot partition image is not regenerated when
using "pmbootstrap flasher flash_kernel" (unlike when using Android
boot images). "pmbootstrap install" needs to be run manually first.
Create add_packages_add() function, so we don't need to write the three
lines for adding the "packages" argument to an action with proper
autocompletion.
Remove in favor of similar, but more visible "--no-install" option for
"pmbootstrap flasher" and "pmbootstrap export".
It seems that the option wasn't really used by anybody, so let's just
remove it without a deprication period. If dear reader thinks otherwise
after this is merged, and is not happy with "--no-install", please state
your reasoning in a new issue, so we can discuss it.
Related: !1863
At the moment, pmbootstrap updates the kernel and the initfs whenever
using the flasher or export. This is useful, but sometimes you just want
to boot exactly the same kernel several times. In that case, having to wait
several seconds for the (redundant) update to complete is quite annoying.
Add a --no-install option that allows skipping the kernel/initfs update.
At the moment, sparse images are generated if the device sets
deviceinfo_flash_sparse="true". But for testing purposes it can be
useful to specifically enable or disable the default behavior.
Add a --sparse and --no-sparse option that enables or disables
sparse image generation.
At the moment we assume that everyone running QEMU is either using
ALSA, or has the ALSA PulseAudio plugin configured on the host system.
(Since we run QEMU outside of the chroot at the moment, configuration
files are read from the host system instead of the chroot...)
Other distributions have much wider support for PulseAudio,
so not everyone will actually have the ALSA PulseAudio plugin configured.
In that case, it is better to use QEMU's PulseAudio backend since that
does not require any configuration. It also lets us drop the alsa-lib
fork, since that was only needed for loading the ALSA PulseAudio plugin.
In general, it seems difficult to detect which audio backend the user
wants to run (if any). Selecting the wrong one results in ugly warnings
when running QEMU. So let's not assume any by default, and add a
--audio option instead which accepts one of QEMU's audio backends
(alsa, pa or sdl).
Add a shortcut for "git pull --ff-only" in all repositories cloned by
pmbootstrap (currently pmaports and aports_upstream, new pmdevices
repository coming soon).
'pmbootstrap pull' will only update the repositories, if:
* they are on an officially supported branch (e.g. master)
* the history is not conflicting (fast-forward is possible)
* the git workdirs are clean
Otherwise it shows the user a descriptive message about what to do. The
list of supported branches is only "master" right now, and will be
extended in later commits, so we can have a stable branch for pmaports
based on Alpine's releases. More about that in the project direction
2020 issue.
Closes: #1858
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:
ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'
This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.
We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().
For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
Rust packaging is new and still a bit weird in Alpine and postmarketOS.
As of writing, we only have one package (squeekboard), and use cargo to
download the source of all dependencies at build time (several git
repositories!) and compile it. Usually, this is a no-go, but at least
until this is resolved properly, let's cache the downloads as suggested
in: https://doc.rust-lang.org/cargo/guide/cargo-home.html
Related: #1861
Adjust to new text that "file" outputs for ARM linux kernels:
"ARM OpenFirmware FORTH Dictionary..."
Make test_bootimg_kernel pass again, which was failing with:
AssertionError: assert 'heimdall-isorec' in 'File is not an Android ...
Oliver Smith
Anjandev Momi
Antoine Fontaine
Martijn Braam
Daniele Debernardi
Niklas Cathor
Luca Weiss
Alexey Min
Minecrell
Danct12
Drew DeVault