At the moment, the --display argument is a bit complicated to use.
A common use would be to switch between the UIs (sdl, gtk, none)
or to enable the software rasterizer. Split the two use cases
to separate arguments to make it more intuitive.
So far we tried to configure virtio-gpu using "-vga virtio" only
when the target architecture matches the host architecture.
But that's actually not what it depends on.
virtio-gpu and virgl can be also used when emulating a foreign
architecture. In fact, we already force usage of virtio-gpu for
aarch64 through "-device virtio-gpu-pci".
However, the "-vga virtio" parameter does not exist on aarch64,
no matter if we run QEMU natively on aarch64 or emulate it on x86_64.
(Apparently, -vga is mainly about legacy VGA framebuffer stuff that
we don't necessarily need. This is quite visible since the display
stays uninitialized on aarch64 until the kernel driver loads,
whereas on x86_64 it is initialized by the BIOS...)
In other words, "-vga virtio" belongs to the parameters specific to x86_64.
Now that we have removed the setup question for the Mesa driver to use
(since it was ineffective), it would still be nice to have some way to
choose if you want to use virtio-gpu/virgl or not.
But actually virtio-gpu can be also used with swrast, without virgl.
This happens automatically when QEMU is started without GL support.
We already have a --display parameter for this, so it is possible to
force swrast by using "--display sdl" (instead of the default sdl,gl=on).
Overall this allows simplifying the QEMU package setup because there is
only a single GPU driver in use (virtio-gpu) instead of the 3 we had before
(virtio, qxl, bochs).
mesa-dri-swrast and mesa-dri-virtio are both provided by mesa-dri-gallium
now, so this option does not have much use anymore. With both selections,
exactly the same packages are installed.
The SPICE UI option tends to be broken (see #1836), and even when it is
working, it is not working particularly well. QXL requires special handling
in our QEMU packages, when now virtio-gpu (virgl) is working quite well overall.
Apparently it is possible to use virgl with SPICE; but only when using
a Unix socket instead of a TCP port. That again is a bit complicated
because we run QEMU outside the chroot and the SPICE client within.
Overall it does no longer seem to be worth the effort.
The default QEMU UI is working just fine (for the purposes of testing
postmarketOS at least).
Work around old rust version in Alpine's armhf repository (builder
currently stuck?) compared to other arches. Armv7 is fine, so switch to
that and avoid this error:
error[E0514]: found crate `std` compiled by an incompatible version of rustc
Fix finding subpackages in pmaports, if no binary repo exists (-mp "" makes
pmbootstrap not use a binary packages mirror for pmOS packages). Discovered
with failing "test_depends_SLOW_40s" in bpo.
$ pmbootstrap -v -mp "" repo_missing --built --arch x86_64
[18:40:51] Calculate packages that need to be built (all packages, x86_64)
[18:40:56] ERROR: Package 'musl-dev-aarch64': Could not find aport, and could not find this package in any APKINDEX!
Newer kernels require "bison" and "flex", so to easy porting we should
add them to generated APKBUILDs by default. While we are it, clean up
the default set of dependencies a bit:
- installkernel is only needed when using "make zinstall", which
we are not using in downstream_package (it requires special kernel
configuration, so it's not really suitable for downstream...)
- most kernels seem to compile fine without gmp-dev and elfutils-dev,
so they should only be added when needed
- the kernel has its own headers, why do we need linux-headers?
(probably copied from Alpine, but anyway...)
- add openssl-dev since it is needed for building/signing kernel modules
Useful reference: https://www.kernel.org/doc/html/latest/process/changes.html
Although it mentions much more than what is really needed...
Building the QCDT image in build() complicates use of envkernel.sh
- in that case, make is done independently of the APKBUILD, and only
the package() function is run to assemble the kernel package.
Building it entirely in package() allows using envkernel.sh without
having to set up some "post make script" (see
https://wiki.postmarketos.org/wiki/Compiling_kernels_with_envkernel.sh#Post_make_scripts).
package() might actually fit better in general, since QCDT is only
a special "package" format for (multiple) DTB files at the end :)
Add a helper class that provides readline-based tab completion and an
extra optional argument "completion_choices" with possible completion
targets.
While at this, also improve paramters documentation for ask() func.
Add new "pmbootstrap status" command, which does a quick health check
for the work dir. As first health check, verify that the chroots are not
too old. Replace the reminder text at the end of "pmbootstrap init" to
tell users to run "pmbootstrap status" instead of "pmbootstrap zap" once
a day before working with pmbootstrap.
Related: #1829
Whenever initializing new chroots, save the date in $WORK/workdir.cfg.
Add pmb.config.workdir.chroots_outdated() to check if it's time to zap
the chroots or not (since we don't update them automatically). Mark them
as outdated after two days.
This will be the first check in "pmbootstrap status" (future patches).
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
Replace the "Add topdir to import path" boilerplate lines in each test
file with a simple "import pmb_test". Use the "# noqa" comment if
"pmb_test" is not used further in the test file, so flake8 does not
complain about an unused module.
Make the path to the testdata available as pmb_test.const.testdata, and
use pmb.config.pmb_src to access the topdir in all tests.
This is in preparation for new "pmbootstrap status" related tests, which
will have shared test code in test/pmb_test/. Also, this makes the
pmbootstrap codebase more consistent with the bpo codebase, which has a
similar "import bpo_test" mechanism.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.
Replacing variables immediately after parsing (before splitting)
has several advantages:
- No need to handle different value types
(e.g. lists by joining them every time they are accessed)
- Variables like depends="$depends ..." are handled directly
by the variable parser
- APKBUILDs are shell scripts, so we match abuild more closely
if variables defined later do not affect previous attributes
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.
We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.
This will also avoid a (somewhat unrelated) error in the bootimg test:
File "pmb/parse/_apkbuild.py", line 46, in replace_variable
apkbuild["pkgname"], match.group(0),
KeyError: 'pkgname'
The gist of this action is upgrading the specified aport to the latest
version. There are implementations for both stable packages (which check
via the release-monitoring.org API for new versions) and git packages
(which check the GitLab/GitHub API for new commits on the main branch).
There's also the possibility to pass --all, --all-stable & --all-git to
the action which either loops through all packages, or just stable or
git packages and upgrades them.
The --dry argument is also respected.
Note, that the implementation does update the variables pkgver, pkgrel
and _commit but it doesn't update the checksums because that would slow
down the process a lot, and is potentially undesirable.
asus-me176c has a Fastboot interface that can be used for flashing,
but in postmarketOS we do not use Android boot images for it.
This is because is it not very practical - the boot partition is
quite small and there is a (custom) EFI bootloader that can boot
directly from any other FAT32 partition.
At the moment the installation process is manual:
1. pmbootstrap install --split to have separated boot (FAT32)
and rootfs images
2. pmbootstrap export
3. Flash boot and rootfs images manually using Fastboot
The "fastboot-bootpart" flasher implements that process in a more
convenient way. When a device uses the "fastboot-bootpart" flasher:
- We generate --split images on "pmbootstrap install" by default.
(This can be disabled using --no-split instead.)
- pmbootstrap flasher flash_kernel flashes the raw boot partition
(not an Android boot image) using Fastboot, just like the rootfs.
There are some limitations that could be improved in the future:
- "fastboot-bootpart" is not offered in the device wizard.
I think it is special enough that no-one will be starting with it,
and the difference to normal "fastboot" might be confusing.
- Support "pmbootstrap flasher boot". asus-me176c does not support
"fastboot boot" properly, but theoretically we could still generate
Android boot images to use when booting an image directly.
- At the moment the boot partition image is not regenerated when
using "pmbootstrap flasher flash_kernel" (unlike when using Android
boot images). "pmbootstrap install" needs to be run manually first.
x86_64 kernels require some additional dependencies to build properly.
At this point, the list is getting quite long. Ideally we should only
install the build dependencies that are necessary for a particular
kernel package. Add a FIXME to document this for the future.
Attempting to use envkernel.sh on x86_64 fails at the moment:
$ . ~/pmos/pmbootstrap/helpers/envkernel.sh
Initializing Alpine chroot (details: 'pmbootstrap log')
ERROR: unsatisfiable constraints:
binutils-x86_64 (missing):
required by: world[binutils-x86_64]
gcc-x86_64 (missing):
required by: world[gcc-x86_64]
We do not need to install a cross compiler if the device arch is
equal to the host architecture. Check if we need a cross compiler
earlier, and avoid installing it if not necessary.
Also rename "is_cc" to "need_cross_compiler". "is_cc" is confusingly
named since "cc" can refer to the compiler symlink ("C compiler")
or "cross compiler".
Create add_packages_add() function, so we don't need to write the three
lines for adding the "packages" argument to an action with proper
autocompletion.
Remove in favor of similar, but more visible "--no-install" option for
"pmbootstrap flasher" and "pmbootstrap export".
It seems that the option wasn't really used by anybody, so let's just
remove it without a deprication period. If dear reader thinks otherwise
after this is merged, and is not happy with "--no-install", please state
your reasoning in a new issue, so we can discuss it.
Related: !1863
Avoid generating an APKBUILD with the following linting issues:
- unnecesary usage of braces: ${_flavor}
- unnecesary usage of braces: ${arch}
- unnecesary usage of braces: ${_commit}
- unnecesary usage of braces: ${_repository}
Commit 43520967 ("pmb.qemu.run.install_depends: adjust to mesa pkgs (!1865)")
replaced mesa-dri-swrast, mesa-dri-virtio and mesa-dri-vmwgfx with
mesa-va-gallium. But the Gallium DRI drivers are actually provided by
mesa-dri-gallium. mesa-va-gallium contains libva drivers for hardware
video acceleration.
Fortunately this did not break anything because mesa-dri-gallium
was still installed as dependency of mesa-dri-ati/intel/nouveau.
QEMU does not make use of libva, so lets just replace mesa-va-gallium
with mesa-dri-gallium. While we are at it, stop depending on the
deprecated mesa-dri-ati/intel/nouveau packages. Those are only kept
for compatibility reasons and install more dependencies than necessary.
At the moment, pmbootstrap updates the kernel and the initfs whenever
using the flasher or export. This is useful, but sometimes you just want
to boot exactly the same kernel several times. In that case, having to wait
several seconds for the (redundant) update to complete is quite annoying.
Add a --no-install option that allows skipping the kernel/initfs update.
At the moment, sparse images are generated if the device sets
deviceinfo_flash_sparse="true". But for testing purposes it can be
useful to specifically enable or disable the default behavior.
Add a --sparse and --no-sparse option that enables or disables
sparse image generation.
Adjust the mesa related packages installed for running in qemu, after
the mesa APKBUILD in Alpine was changed. Install mesa-va-gallium instead
of:
* mesa-dri-freedreno
* mesa-dri-swrast
* mesa-dri-virtio
* mesa-dri-vmwgfx
Fix error in pmbootstrap-qemu CI test:
ERROR: Could not find dependency 'mesa-dri-freedreno' in any aports folder or APKINDEX.
Related: 298e20d04f
Now that the qemu-user binary is bind-mounted, we no longer copy
the binary to the device rootfs. However, there is still the empty
stub file that we used as a destination mount point.
Let's remove it before copying it to the device rootfs.
It is automatically re-created the next time the qemu-user binary
is needed.