Do not pass the arguments to ondev-prepare as command-line arguments in
a specific order, but instead as environment variables. New arguments
will be added in a follow-up patch.
Checkout the aports.git branch for the current channel (e.g.
3.12-stable), before trying to find the APKBUILD.
I had tried to auto-unshallow the git repository earlier, but then the
tags were missing. I decided that it's not worth to provide a migration
path: "pmbootstrap aportgen" is only used by few advanced users (to
maintain the pmaports repo).
This made sense for Alpine's aports.git repository as we were only using
the master branch. But now that we are using more branches, we need the
entire git repository with all its branches cloned.
Add the Alpine mirrordir (e.g. "edge", "v3.12") to the distfile, so
musl-*.apk files of the same version and arch but from different
mirrordirs do not collide.
Let "abuild checksum" download these apks and generate the checksums,
instead of letting apk download them as side-effect of initializing
foreign arch chroots. The latter did not work anymore, because we would
copy the apk file with a glob that may matches the hash of either mirrordir.
Essentially:
glob.glob(f"{args.work}/cache_apk_{arch}/{subpkgname}-{version}.*.apk")[0]
In the context of the on-device installer, I found that calculating this
hash is not trivial, so let's just avoid it here as well.
While at it, order the imports of musl.py alphabetically.
Add initial support for the on-device installer in pmbootstrap. Let
pmbootstrap create a regular split image, then prepare a new installer
rootfs and copy the previously generated rootfs image into the installer
rootfs. Put the installer rootfs into a new image, with reserved space.
There is more to do from here, such as disabling the generation of the
user account when using --ondev. But this requires support in
postmarketos-ondev first, so let's build that iteratively.
Related: https://wiki.postmarketos.org/wiki/On-device_installer
Related: https://gitlab.com/postmarketOS/postmarketos-ondev/-/issues
Move code that prints flashing information from install_system_image()
to its own function. For the on-device installer, we'll need to call
install_system_image() twice, without printing the flashing information
each time. While at it, add "step" and "steps" parameters.
Prepare for a future patch, that adds reserved space in MiB, by changing
size_boot and size_root from bytes to MB everywhere. This is what we need
most of the time and allows to drop some /1024**2 statements.
Add a "split" argument to the function, instead of using "args.split"
directly. "args.split" is only defined when calling "pmbootstrap install",
but the next patch will add a code path that calls the function from
"pmbootstrap chroot".
Alpine APKBUILDs have the concept of "provider priorities" that affect
the choice of the provider to install when multiple packages provide
a virtual package.
One use case for this is to allow installation of different firmware
versions. bq-paella can run unsigned firmware, therefore you have the
choice between using the original firmware from the manufacturer, or
a slightly newer version from Qualcomm for the Dragonboard 410c.
We add provides="firmware-qcom-msm8916-wcnss" (the "virtual package")
to both firmware-qcom-db410c-wcnss and firmware-bq-picmt-wcnss.
At this point, attempting to install "firmware-qcom-msm8916-wcnss"
would still fail with apk. (Because it does not know which provider
to install.)
To pick a default we can set e.g. provider_priority=100 for
firmware-qcom-db410c-wcnss (the slightly newer version).
In that case, firmware-qcom-db410c-wcnss should be installed by default.
However, the user can choose to do "apk add firmware-bq-picmt-wcnss"
to override the default choice in case of problems. In that case,
the conflicting firmware-qcom-db410c-wcnss will be automatically removed.
At the moment, pmbootstrap does not respect the "provider_priority" at all.
In the above case, it would always install "firmware-bq-picmt-wcnss"
during "pmbootstrap install" since that has the shortest name.
Extend the pmbootstrap code to pick a provider with the highest priority
(if any of the providers has a priority set).
Do not substract the estimated size of the home and boot directories
from the root directory size. While that would be the correct way if we
were able to get exact sizes, it isn't helpful with the very rough
estimates we are getting from pmb.helpers.other.folder_size. Replace
"calculate" wording with "estimate".
"Chassis" defines the what kind of device a device is. Currently the
following standardized types exist:
"desktop", "laptop", "convertible", "server", "tablet", "handset",
"watch", "embedded", "vm", "container"
This property is exposed by org.freedesktop.hostname1 (e.g.
openrc-settingsd) and can be used by applications for example to display
appropriate strings instead of "About this phone" for non-phone devices.
Migrate to workdir version 5 and move already built packages into the edge
channel subdir, for example:
$WORK/packages/x86_64/hello-world-1-r5.apk
to:
$WORK/packages/edge/x86_64/hello-world-1-r5.apk
The build.postmarketos.org code has already been adjusted to find built
packages in either directory structure.
Do not go through the pmb.chroot.init() code path when running
pmb.install.losetup.umount() inside pmb.chroot.shutdown(). This is not
necessary, as pmb.install.losetup.umount() only gets called if the
chroot is already initialized and /dev/loop-control is mounted inside
the chroot.
Not going through this code path is important for the upcoming workdir
migration patch. Without this fix, it will fail with the following if
running "pmbootstrap install" before the work migration:
ERROR: Could not figure out on which release channel the 'native' chroot is
Replace the call to this function with the almost identical
pmb.helpers.repo.apkindex_files(), so release channel related changes
only need to be done in one place.
Use mirrordir_pmos and mirrordir_alpine from channels.cfg to generate
the mirror URLs for postmarketOS and Alpine, which get written to
/etc/apk/repositories and which postmarketOS uses to download the
APKINDEX files.
Remove hardcoded "master" at the end of the postmarketOS mirror and use
mirrordir_pmos instead (which is "master" for the edge channel). Let the
postmarketOS mirror end in a '/' for consistency with the Alpine mirror
in pmb/config/__init__.py.
Remove obsolete --alpine-version. To experiment with a different Alpine
version, one should pass a custom --config-channels from now on.
Don't generate the postmarketOS mirror URLs here, let the urls()
function do it. A follow-up commit will touch this code, hence it's
important to have it de-duplicated.
Ask for release channel and switch pmaports branch to the related branch
defined in channels.cfg.
Store in pmbootstrap.cfg whether the user chose a channel (boolean). If
the user did not choose a channel yet, suggest the recommended channel
from channels.cfg (currently "edge").
Do not make the parsed pmaports.cfg from pmaports.git available as
args.pmaports anymore. This de-bloats the args variable a bit.
First I thought that we didn't even need to cache it, but it was pointed
out that later patches do access it frequently to read the current channel
from pmaports.cfg. Therefore it is using a cache now.
Related: #1879, #1855
New pmaports.cfg has the "channel" key described in [1]. This will be
used in "pmbootstrap init" to determine on which release channel a
forked branch is.
[1] https://postmarketos.org/pmaports.cfg
Prepare to base postmarketOS on Alpine stable by parsing the new
channels.cfg file in pmaports.git, that describes which channel
needs which branches and mirror dirs from postmarketOS and Alpine.
Use the information in pmb.helpers.git.get_branches_official() first,
more is coming in follow-up commits.
Read the file from origin/master, so we get the latest fetched version
even if the last checked out master branch is not up-to-date (think of
currently checked out release branch instead of master, master will
never be updated to point to latest origin/master). Allow to override
the file with a new --config-channels parameter.
Related: https://postmarketos.org/channels.cfg
Most device ports that are added to pmaports (particularly the testing
category) are not actually "maintained". Many of them are never updated
after the initial contribution. There is little reason to list a
"Maintainer:" if the device package is actually not actively maintained
by that person.
Let's stop generating the Maintainer: line by default.
Instead, contributors should add the lines themselves if they are willing
to (actively) maintain the device package for a longer period of time.
Once they no longer want to maintain the package, the Maintainer:
line should be removed again.
Let apkbuild-lint check options again, and pass the pmbootstrap specific
options with the new CUSTOM_VALID_OPTIONS variable. Add a test case and
adjust pmb.helpers.lint.check to return the output of apkbuild-lint, so
we can properly test it.
Related: https://gitlab.alpinelinux.org/Leo/atools/-/merge_requests/28
Fixes: pmaports#553
The `mesa-dri-swrast` package was merged into `mesa-dri-gallium` in Alpine.
(See 298e20d04f)
The old alias might be removed in the future, so change the generated APKBUILDs
to depend on `mesa-dri-gallium` instead.
Many of the multi-line strings already contain a new line at the
end of the file. When using .split("\n") this will result in an empty
line at the end of the file that is again followed by a new line.
In other words: we have two new lines at the end, which looks weird.
.rstrip() the whitespace at the end of the string to avoid this.
In case a package can't be built using crossdirect, add an APKBUILD
option to unconditionally disable crossdirect and use the slower
distcc approach instead. This is needed e.g. when using LD_PRELOAD during
the build as crossdirect cannot work with that.
Have explicit selection of the "native" cross compilation method with a
new "pmb:cross-native" option. Deprecate the implicit pkgname pattern
matching.
Related: #1910
The ui-extras questions will attempt to find a postmarketos-ui-<ui>
package in pmaports. If the package does not exist as "root" APKBUILD
it currently attempts to parse all APKBUILDs in case it is somewhere
defined as a subpackage. This is really slow (up to 2-3 seconds),
which feels weird during "pmbootstrap init".
For the UI packages we specifically look for the root UI package,
not the subpackage, so let's skip searching for subpackages in this
case. This makes selecting the "none" UI nice and fast again.
Fix the function, so it does not crash anymore when the
replace_subpkgnames argument is set and a dependency cannot be resolved.
Instead, print a useful warning that shows which pmaport caused the
error (that has always been a pain to figure out), and simply don't
replace the potential subpkgname with the real pkgname, just use the
dependency name as-is.
Resolve annoying crashes in bpo dependency resolving, like this one
(caused by a few linux-* pmaports for bad downstream kernels that depend
on python, not apparent at all from the message):
[09:08:15] Calculate packages that need to be built (all packages, aarch64)
[09:08:26] ERROR: Package 'python': Could not find aport, and could not find this package in any APKINDEX!
Related: https://builds.sr.ht/~postmarketos/job/184022
Previously these two commands would both print the current value:
pmbootstrap config extra_packages
pmbootstrap config extra_packages ''
With this change, the second command will instead set the given config
value to the empty string.
Prevent "pmbootstrap pull" from failing with:
NOTE: your pmaports folder has version 4, but version 6 is required.
ERROR: Run 'pmbootstrap pull' to update your pmaports.
Fixes: #1900
Use pmb.helpers.args.add_cache() with the args used during
autocompletion, so pmb.helpers.pmaports._find_apkbuilds() does not fail
when trying to access args.cache.
No build is necessary if pmaport can't be built for given arch.
pmbootstrap must use Alpine's binary package in that case, even if the
pmaport version is higher than Alpine's binary package version.
Fixes: #1897
The --no-depends option is supposed to stop pmbootstrap if it was
instructed to build a package, but a dependency must be built first. So
far, this only covers the case if there is no binary package for a dependency.
Make it stop if the binary package exists, but is outdated, too.
Fixes: #1895
At the moment we set samsung-i9100 as default device.
This is probably only for historical reasons.
For someone/something using pmbootstrap without a specific device in mind
(e.g. CI) it's really better to use a generic device. QEMU runs natively
in a virtual machine so everyone can use it for testing.
Flashes device vbmeta partition (can be overriden with
"flash_fastboot_partition_vbmeta" setting in deviceinfo)
with custom vbmeta.img which has verity flag disabled,
so device can boot postmarketOS with no problems.
The kconfig check searches the aport with the "linux-" prefix to the
package name passed as argument. This is not working with the full
package name like linux-device-name because it searches a
linux-linux-device-name and fails.
Use the timestamp of .git/FETCH_HEAD in each git repository, to
determine if too much time has passed since the last fetch/pull.
Modify pmb.helpers.git.clone, so FETCH_HEAD is always created if it does
not exist (because "git clone" would not create it).
Related: #1829
Extend "pmbootstrap status" with checks for all git repositories,
derived from relevant checks in pmb.helpers.git.pull (using shared
code):
* on official branch
* workdir is clean
* tracking proper remote
* up to date
Related: #1829
Add dummy function that only returns ["master"] for now, so we can use it
in the upcoming git checks for "pmbootstrap status". More sophisticated
logic to figure out the branches will be added soon, see project
direction 2020 issue.
Related: #1829
Device ports should be under the device/testing/* directory for now.
In the future, devices may be moved to device/{main,community} after
certain preconditions are met.
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
For KVM the code is run pretty much natively on the host CPU, so all
CPU extensions available on the host CPU can be also used inside the VM.
To expose that information to the VM we should pass "-cpu host", so the
VM is aware of which CPU is in use.
For CPU emulation, QEMU uses a rather minimal CPU on x86_64 by default.
It does not have support for SSE3/4 etc, which may be required for some
applications to work properly (e.g. Android in Anbox). Add a --cpu flag
to make the emulated CPU configurable. Useful values are for example
--cpu max to emulate all implemented CPU features.
To test QEMU's CPU emulation it is useful to have a switch to disable
KVM, even when it is available (and potentially working fine).
Add --no-kvm for that purpose.
For some reason, the SDL display backend changes the video resolution
to 1024x768, while the GTK display keeps it at 640x480.
This is annoying, because at the moment we can only set one display
resolution for a device in postmarketOS (e.g. for the splash screen).
At the moment, the resolution for the splash screen is set to 640x480,
which therefore shows up too small with the default SDL display.
It seems like the display resolution can be only changed in the guest
directly. Linux has a video= kernel parameter that can be used to
implement this. (See: https://www.kernel.org/doc/html/latest/fb/modedb.html)
Let's set 1024x768 by default, but make it configurable through --video.
The QEMU 'tablet' input device reports absolute positions instead
of relative mouse pointer movements. This can be used to automatically
grab/release the mouse pointer when entering/leaving the QEMU window,
instead of having to release it with CTRL + ALT + G manually.
This is quite convenient and should be the default option normally,
but at least on my PC the mouse pointer is reported with some vertical
offset for some reason (you can't reach the top and it extends below
the QEMU window...). Let's add it as an optional --tablet option for now.
To ensure consistent behavior for QEMU on all architectures,
it is helpful to try to use the same hardware elements where possible.
A few examples of current inconsistent behavior:
- x86_64 uses a SCSI disk, while aarch64 uses virtio-blk
- x86_64 uses e1000 network, while aarch64 uses virtio-net-device
- x86_64 uses PS/2 mouse, while aarch64 uses usb-mouse
- only x86_64 prints serial output to console
At least the virtio components are usually independent of the selected
architecture, so we can use them for both architectures.
This commit makes most of the hardware configuration shared:
- Redirect serial output to stdio
- virtio-blk for the disk image
- virtio-gpu-pci (this was already implicit for both architectures)
- virtio-net-pci for the network interface
- virtio-mouse-pci/virtio-keyboard-pci as input devices
- intel-hda for audio
We also set -nodefaults to avoid setting up unneeded devices
especially for x86_64.
Now that we try to use the IP assigned by QEMU via DHCP,
the debug-shell is no longer working via telnet.
This is because the VM does not have any IP assigned when it is running.
We would need to start a DHCP client from the initfs to make it work.
busybox provides both udhcpc (client) and udhcpd (server) so this is
not a big problem. But the question is: Is it worth it?
The debug-shell will be only used for debugging, and NetworkManager
handles starting a proper DHCP client once the rootfs is mounted.
Meanwhile the debug-shell can be also accessed via serial output/input,
(available in the pmbootstrap stdout/stdin). So overall it does not
seem worth the effort. Let's just recommend using serial instead.
The current network setup has weird side effects.
Normally, QEMU would automatically make the guest set up necessary
IP routes through its integrated DHCP server.
When running QEMU through pmbootstrap they are missing.
First, we change the DHCP range in a way that could potentially
conflict with default IPs used for QEMU's own services:
QEMU has the default gateway at <network>.2, and DNS at <network>.3.
We set the DHCP range to start at <network>.1, and will therefore
potentially give out one of these addresses (QEMU's default starts at
<network>.15).
See: https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29
In practice this does not cause immediate problems because there is
just one guest in the network, and it will get <network>.1, which is
not used by QEMU.
More problematic is that we start a DHCP server from postmarketOS
at the same time (normally used for the USB network) and there are
actually two DHCP servers running at the same time.
QEMU's user networking is local to the process, therefore it is not
possible to access the QEMU guest through its IP from the host.
That's why we have the port forwardings so you can access SSH at
localhost:2222 for example.
In practice the network interface in the QEMU guest is only used to
access the Internet. For that, we don't care which IP address we get,
we just want to get a working setup (IP + routes + DNS) automatically
through DHCP.
To make this work nicely we just need to stop trying to fit QEMU's
network setup into our usual setup for USB networking. When we remove
the custom DHCP option, and avoid starting a DHCP server from postmarketOS
(deviceinfo_disable_dhcpd) everything is suddenly working fine. :)
device-qemu-vexpress appears to be completely broken at the moment.
I was not able to make it show anything, get a log or even just get
any indication that it is actually doing something.
In general, it does also not fit well to the other QEMU ports.
The vexpress machine type in QEMU seems to be quite limited,
e.g. it has no PCIe bus and therefore it is impossible to configure it
with virtio-gpu. (Unlike already configured for x86_64 and aarch64).
I had some success with a setup similar to aarch64 with -M virt,highmem=off
but was unable to make it work in the end:
- Alpine's virt kernel was missing some options preventing boot completely
- Alpine's lts kernel was missing PCI support for virtio-gpu
- Even after adding the options the VM usually freezed after boot
Overall it does not quite seem worth the effort at the moment,
compared to how well x86_64 and aarch64 are working.
In any case, vexpress is too different (and broken) to continue maintaining it.
When using pmbootstrap, you usually select the device you want to work
on using 'pmbootstrap init', generate the rootfs and can then run more
commands in the context of the device.
The same needs to be done before using QEMU (to generate the rootfs).
But for some reason 'pmbootstrap qemu' requires setting the --arch
parameter when running QEMU for a foreign architecture, even when the
device is still selected in pmbootstrap.
Even more confusing is that setting "--arch arm" always selects
device-qemu-vexpress, but this is not immediately clear from the name.
Let's make this a lot more intuitive by making sure there is a QEMU
device selected when running 'pmbootstrap qemu'. We can then use the
device information to infer the architecture automatically.
At the moment, the --display argument is a bit complicated to use.
A common use would be to switch between the UIs (sdl, gtk, none)
or to enable the software rasterizer. Split the two use cases
to separate arguments to make it more intuitive.
So far we tried to configure virtio-gpu using "-vga virtio" only
when the target architecture matches the host architecture.
But that's actually not what it depends on.
virtio-gpu and virgl can be also used when emulating a foreign
architecture. In fact, we already force usage of virtio-gpu for
aarch64 through "-device virtio-gpu-pci".
However, the "-vga virtio" parameter does not exist on aarch64,
no matter if we run QEMU natively on aarch64 or emulate it on x86_64.
(Apparently, -vga is mainly about legacy VGA framebuffer stuff that
we don't necessarily need. This is quite visible since the display
stays uninitialized on aarch64 until the kernel driver loads,
whereas on x86_64 it is initialized by the BIOS...)
In other words, "-vga virtio" belongs to the parameters specific to x86_64.
Now that we have removed the setup question for the Mesa driver to use
(since it was ineffective), it would still be nice to have some way to
choose if you want to use virtio-gpu/virgl or not.
But actually virtio-gpu can be also used with swrast, without virgl.
This happens automatically when QEMU is started without GL support.
We already have a --display parameter for this, so it is possible to
force swrast by using "--display sdl" (instead of the default sdl,gl=on).
Overall this allows simplifying the QEMU package setup because there is
only a single GPU driver in use (virtio-gpu) instead of the 3 we had before
(virtio, qxl, bochs).
mesa-dri-swrast and mesa-dri-virtio are both provided by mesa-dri-gallium
now, so this option does not have much use anymore. With both selections,
exactly the same packages are installed.
The SPICE UI option tends to be broken (see #1836), and even when it is
working, it is not working particularly well. QXL requires special handling
in our QEMU packages, when now virtio-gpu (virgl) is working quite well overall.
Apparently it is possible to use virgl with SPICE; but only when using
a Unix socket instead of a TCP port. That again is a bit complicated
because we run QEMU outside the chroot and the SPICE client within.
Overall it does no longer seem to be worth the effort.
The default QEMU UI is working just fine (for the purposes of testing
postmarketOS at least).
Fix finding subpackages in pmaports, if no binary repo exists (-mp "" makes
pmbootstrap not use a binary packages mirror for pmOS packages). Discovered
with failing "test_depends_SLOW_40s" in bpo.
$ pmbootstrap -v -mp "" repo_missing --built --arch x86_64
[18:40:51] Calculate packages that need to be built (all packages, x86_64)
[18:40:56] ERROR: Package 'musl-dev-aarch64': Could not find aport, and could not find this package in any APKINDEX!
Newer kernels require "bison" and "flex", so to easy porting we should
add them to generated APKBUILDs by default. While we are it, clean up
the default set of dependencies a bit:
- installkernel is only needed when using "make zinstall", which
we are not using in downstream_package (it requires special kernel
configuration, so it's not really suitable for downstream...)
- most kernels seem to compile fine without gmp-dev and elfutils-dev,
so they should only be added when needed
- the kernel has its own headers, why do we need linux-headers?
(probably copied from Alpine, but anyway...)
- add openssl-dev since it is needed for building/signing kernel modules
Useful reference: https://www.kernel.org/doc/html/latest/process/changes.html
Although it mentions much more than what is really needed...
Building the QCDT image in build() complicates use of envkernel.sh
- in that case, make is done independently of the APKBUILD, and only
the package() function is run to assemble the kernel package.
Building it entirely in package() allows using envkernel.sh without
having to set up some "post make script" (see
https://wiki.postmarketos.org/wiki/Compiling_kernels_with_envkernel.sh#Post_make_scripts).
package() might actually fit better in general, since QCDT is only
a special "package" format for (multiple) DTB files at the end :)
Add a helper class that provides readline-based tab completion and an
extra optional argument "completion_choices" with possible completion
targets.
While at this, also improve paramters documentation for ask() func.
Add new "pmbootstrap status" command, which does a quick health check
for the work dir. As first health check, verify that the chroots are not
too old. Replace the reminder text at the end of "pmbootstrap init" to
tell users to run "pmbootstrap status" instead of "pmbootstrap zap" once
a day before working with pmbootstrap.
Related: #1829
Whenever initializing new chroots, save the date in $WORK/workdir.cfg.
Add pmb.config.workdir.chroots_outdated() to check if it's time to zap
the chroots or not (since we don't update them automatically). Mark them
as outdated after two days.
This will be the first check in "pmbootstrap status" (future patches).
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.
Replacing variables immediately after parsing (before splitting)
has several advantages:
- No need to handle different value types
(e.g. lists by joining them every time they are accessed)
- Variables like depends="$depends ..." are handled directly
by the variable parser
- APKBUILDs are shell scripts, so we match abuild more closely
if variables defined later do not affect previous attributes
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.
We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.
This will also avoid a (somewhat unrelated) error in the bootimg test:
File "pmb/parse/_apkbuild.py", line 46, in replace_variable
apkbuild["pkgname"], match.group(0),
KeyError: 'pkgname'
The gist of this action is upgrading the specified aport to the latest
version. There are implementations for both stable packages (which check
via the release-monitoring.org API for new versions) and git packages
(which check the GitLab/GitHub API for new commits on the main branch).
There's also the possibility to pass --all, --all-stable & --all-git to
the action which either loops through all packages, or just stable or
git packages and upgrades them.
The --dry argument is also respected.
Note, that the implementation does update the variables pkgver, pkgrel
and _commit but it doesn't update the checksums because that would slow
down the process a lot, and is potentially undesirable.
asus-me176c has a Fastboot interface that can be used for flashing,
but in postmarketOS we do not use Android boot images for it.
This is because is it not very practical - the boot partition is
quite small and there is a (custom) EFI bootloader that can boot
directly from any other FAT32 partition.
At the moment the installation process is manual:
1. pmbootstrap install --split to have separated boot (FAT32)
and rootfs images
2. pmbootstrap export
3. Flash boot and rootfs images manually using Fastboot
The "fastboot-bootpart" flasher implements that process in a more
convenient way. When a device uses the "fastboot-bootpart" flasher:
- We generate --split images on "pmbootstrap install" by default.
(This can be disabled using --no-split instead.)
- pmbootstrap flasher flash_kernel flashes the raw boot partition
(not an Android boot image) using Fastboot, just like the rootfs.
There are some limitations that could be improved in the future:
- "fastboot-bootpart" is not offered in the device wizard.
I think it is special enough that no-one will be starting with it,
and the difference to normal "fastboot" might be confusing.
- Support "pmbootstrap flasher boot". asus-me176c does not support
"fastboot boot" properly, but theoretically we could still generate
Android boot images to use when booting an image directly.
- At the moment the boot partition image is not regenerated when
using "pmbootstrap flasher flash_kernel" (unlike when using Android
boot images). "pmbootstrap install" needs to be run manually first.
Create add_packages_add() function, so we don't need to write the three
lines for adding the "packages" argument to an action with proper
autocompletion.
Remove in favor of similar, but more visible "--no-install" option for
"pmbootstrap flasher" and "pmbootstrap export".
It seems that the option wasn't really used by anybody, so let's just
remove it without a deprication period. If dear reader thinks otherwise
after this is merged, and is not happy with "--no-install", please state
your reasoning in a new issue, so we can discuss it.
Related: !1863
Avoid generating an APKBUILD with the following linting issues:
- unnecesary usage of braces: ${_flavor}
- unnecesary usage of braces: ${arch}
- unnecesary usage of braces: ${_commit}
- unnecesary usage of braces: ${_repository}
Commit 43520967 ("pmb.qemu.run.install_depends: adjust to mesa pkgs (!1865)")
replaced mesa-dri-swrast, mesa-dri-virtio and mesa-dri-vmwgfx with
mesa-va-gallium. But the Gallium DRI drivers are actually provided by
mesa-dri-gallium. mesa-va-gallium contains libva drivers for hardware
video acceleration.
Fortunately this did not break anything because mesa-dri-gallium
was still installed as dependency of mesa-dri-ati/intel/nouveau.
QEMU does not make use of libva, so lets just replace mesa-va-gallium
with mesa-dri-gallium. While we are at it, stop depending on the
deprecated mesa-dri-ati/intel/nouveau packages. Those are only kept
for compatibility reasons and install more dependencies than necessary.
At the moment, pmbootstrap updates the kernel and the initfs whenever
using the flasher or export. This is useful, but sometimes you just want
to boot exactly the same kernel several times. In that case, having to wait
several seconds for the (redundant) update to complete is quite annoying.
Add a --no-install option that allows skipping the kernel/initfs update.
At the moment, sparse images are generated if the device sets
deviceinfo_flash_sparse="true". But for testing purposes it can be
useful to specifically enable or disable the default behavior.
Add a --sparse and --no-sparse option that enables or disables
sparse image generation.
Adjust the mesa related packages installed for running in qemu, after
the mesa APKBUILD in Alpine was changed. Install mesa-va-gallium instead
of:
* mesa-dri-freedreno
* mesa-dri-swrast
* mesa-dri-virtio
* mesa-dri-vmwgfx
Fix error in pmbootstrap-qemu CI test:
ERROR: Could not find dependency 'mesa-dri-freedreno' in any aports folder or APKINDEX.
Related: 298e20d04f
Now that the qemu-user binary is bind-mounted, we no longer copy
the binary to the device rootfs. However, there is still the empty
stub file that we used as a destination mount point.
Let's remove it before copying it to the device rootfs.
It is automatically re-created the next time the qemu-user binary
is needed.
Right now we explicitly add ttf-droid to /etc/apk/world, and force
installation of it on every postmarketOS installation. But there really
is no reason to force this from pmbootstrap:
- If ttf-droid is really required for every postmarketOS installation,
it should be a depends of postmarketos-base.
- Even with this change it is still installed on every device,
because postmarketos-splash depends on it.
- There is no need for fonts when using the "none" UI.
For now, the only difference of this change is that we no longer add
ttf-droid explicitly to the APK world. Instead, it is installed as
dependency of postmarketos-splash.
At the moment we assume that everyone running QEMU is either using
ALSA, or has the ALSA PulseAudio plugin configured on the host system.
(Since we run QEMU outside of the chroot at the moment, configuration
files are read from the host system instead of the chroot...)
Other distributions have much wider support for PulseAudio,
so not everyone will actually have the ALSA PulseAudio plugin configured.
In that case, it is better to use QEMU's PulseAudio backend since that
does not require any configuration. It also lets us drop the alsa-lib
fork, since that was only needed for loading the ALSA PulseAudio plugin.
In general, it seems difficult to detect which audio backend the user
wants to run (if any). Selecting the wrong one results in ugly warnings
when running QEMU. So let's not assume any by default, and add a
--audio option instead which accepts one of QEMU's audio backends
(alsa, pa or sdl).
Add a shortcut for "git pull --ff-only" in all repositories cloned by
pmbootstrap (currently pmaports and aports_upstream, new pmdevices
repository coming soon).
'pmbootstrap pull' will only update the repositories, if:
* they are on an officially supported branch (e.g. master)
* the history is not conflicting (fast-forward is possible)
* the git workdirs are clean
Otherwise it shows the user a descriptive message about what to do. The
list of supported branches is only "master" right now, and will be
extended in later commits, so we can have a stable branch for pmaports
based on Alpine's releases. More about that in the project direction
2020 issue.
Closes: #1858
Remove a legacy code path from back in the day, where pmaports were in
the same repository as pmbootstrap ([1]). Nowadays, pmaports are always in a
git repository, so this is not necessary anymore. If git fails at this
point, crash hard and don't carry on with an empty string.
This is in preparation for using rev_parse in new code paths, so we
don't need to check the return value of rev_parse.
[1] 39548835 "Write custom os-release (closes#324) (#439)"
https://github.com/postmarketOS/pmbootstrap/pull/439
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:
ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'
This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.
We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().
For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
This adds a new deviceinfo parameter, 'boot_part_start' which accepts an
int and indicates the number of sectors from the start of the drive to
place the boot partition.
The librem5 devkit (and actual phone) u-boot has grown beyond the 2048
sector space previously before the boot partition, so this is necessary in
order to boot pmos on this device.
Install rust (rustc) in native chroot, if it is in the APKBUILD's
build dependencies. Add a test to verify that crossdirect + rust works
as expected.
Closes: #1861
Rust packaging is new and still a bit weird in Alpine and postmarketOS.
As of writing, we only have one package (squeekboard), and use cargo to
download the source of all dependencies at build time (several git
repositories!) and compile it. Usually, this is a no-go, but at least
until this is resolved properly, let's cache the downloads as suggested
in: https://doc.rust-lang.org/cargo/guide/cargo-home.html
Related: #1861
When initializing the building chroots, some symlinks are created to
/mnt/pmbootstrap* dirs. With upcoming rust-related symlinks, they will
point to subdirs in /mnt/pmbootstrap-rust, such as
/mnt/pmbootstrap-rust/registry/cache. Create this directory structure if
it does not exist.
Do not install git in the native chroot and use it from there. Remove the
chown_to_user argument from pmb.helpers.git.clone(), the resulting dir
is now always owned by the user. While at it, refactor the function and
display the clone URL.
Previously we had cloned aports_upstream (from Alpine) with
chown_to_user=False (legacy) and pmaports with chown_to_user=True.
pmb.helpers.git.rev_parse() would only work after chown_to_user=True.
Check if git is installed in "pmbootstrap init", and remove the same
check from rev_parse(). Add a new work dir version, that checks for git
and changes ownership of already checked out aports_upstream to the
host system's user.
When creating a new work dir, create cache_git instead of cache_http.
cache_http is created on demand already, with proper permissions. But
cache_git must be created, otherwise pmb.helpers.mount.bind will create it
as root.
This is in preparation for the "pmbootstrap pull" feature, as it allows
using the host system's git in all new code paths. We will be able to
handle repositories even if they were cloned outside of the work dir
(which we do in a few CI scripts for example).
Related: #1858
Remove /home/pmos/git from pmb.config.chroot_home_symlinks, so
pmb.chroot.init.init() will not change the ownership of the
$WORK/cache_git dir.
The symlink was added two years ago in [1] as compatibility to previous
behavior, because the mountpoint of cache_git was changed from
/home/user/git to /mnt/pmbootstrap-git. Forcing the ownership of that
dir to be the pmos user inside the chroot made sense, because the only
cloned repository (aports_upstream) would also be owned by the pmos
user. With the upcoming change to have all cloned repos owned by the
host-system's user, this is just confusing.
The /home/pmos/git symlink does not seem to be used by anybody, it is
not mentioned in the pmbootstrap code and in the wiki.
[1] b4dd7a89 "Close #709: Improve user creation (#725)"
https://github.com/postmarketOS/pmbootstrap/pull/725
The APKBUILD reference for "provides" [1] is not worded clearly; but
after reading it over and over again, my understanding is:
* package with provides='foo=1.2' will be automatically installed if
user requests installing "foo"
* package with provides='foo' (without version) will NOT get
automatically installed if user requests installing "foo"
For pmbootstrap, this means, that we must not attempt to build a package
where the pkgname mentioned in provides matches what we are currently
resolving, unless there is an equals sign in the provides entry.
Fixes: #1862, pmaports#404
[1] https://wiki.alpinelinux.org/wiki/APKBUILD_Reference#provides says:
"List of package names (and optionally version info) this package
provides.
If package with a version is provided (provides='foo=1.2') apk will
consider it as an alternate name and it will automatically consider the
package for installation by the alternate name, and conflict with other
packages having the same name, or provides.
If version is not provided (provides='foo'), apk will consider it as
virtual package name. Several package with same non-versioned provides
can be installed simultaneously. However, none of them will be installed
by default when requested by the virtual name - instead, error message
is given and user is asked to choose which package providing the virtual
name should be installed."
Changes:
* Added the ALSA_PLUGIN_DIRS environment variable otherwise it tries
to load the plugins of the host pc and they fails.
* Added the /usr/lib/pulseaudio path to the library-path argument
* Added the audiodev and soundhw parameters to the qemu command
* Added the required packages necessary to be installed in the chroot
Adjust to new text that "file" outputs for ARM linux kernels:
"ARM OpenFirmware FORTH Dictionary..."
Make test_bootimg_kernel pass again, which was failing with:
AssertionError: assert 'heimdall-isorec' in 'File is not an Android ...
This fixes#1845 without breaking #1717 (closed). The amount of inodes
only needs to be explicitly set when we don't know the size of the
filesystem before hand, for example when using fastboot or Heimdall. In
all other cases it's actually better to let it figure out automatically
how many inodes is set.
On my 16GB sdcard the amount of available inodes got increased by 10
times (from ~100K to ~1M).
Add default user to netdev group. Together with MR [1] in pmaports.git,
this fixes the permissions to use nmtui and nm-applet from the user.
[1] https://gitlab.com/postmarketOS/pmaports/merge_requests/777
Signed-off-by: Danct12 <danct12@disroot.org>
Properly ignore comments at the end of lines, instead of assuming that
all lines below belong to the attribute:
subpackages="$pkgname-dev" # $pkgname-lang
Fixes build.postmarketos.org#61, where pmbootstrap would assume that a
random package provides "make", just because the word "make" is written
somewhere below subpackages=" in the APKBUILD and it is parsed
incorrectly.
While at it, also support the ' character for quotations and detect if
a quotation for a value was started, but there is no end quotation sign
in the rest of the file.
I've added tests, and manually checked that this did not introduce any
parsing bugs for all the APKBUILDs in pmaports.git, by running
'pmbootstrap apkbuild_parse' with the old and new code, and diffing the
result.
Because of an APKINDEX subpkg parsing bug, pmbootstrap currently assumes
that "ktp-common-internals" provides "make". It fails with a cryptic
error when trying to build a package without an existing postmarketOS
binary packages mirror (like done in the build.postmarketos.org
testsuite):
File "/home/user/code/pmbootstrap/pmb/chroot/apk.py", line 174, in replace_aports_packages_with_path
package + "-" + data_repo["version"] + ".apk")
TypeError: 'NoneType' object is not subscriptable
Replace this error with a more meaningful message, that points to the
issue analyzing this bug.
Related: https://gitlab.com/postmarketOS/build.postmarketos.org/issues/61
Always install git when building packages, so abuild won't behave
slightly different if it is (not) installed. As nice side-effect, we
will always get the pmaports.git commit saved in the resulting .apk file
from now on (in .PKGINFO).
With the nokia N900 the user can choose between different keyboard
layouts and during the install the setup-keymap installs the choosen
layout.
This only changes the keyboard layout of the console, while installing
a UI using xorg, the layout was hardcoded to "us".
With a simple check for the existence of an xorg config containing
XkbLayout, this will run sed to replace the default with the layout
chosen by the user.
Tested with a clean install of the nokia N900 with the xfce4 UI.
elogind requires cgroup support in the kernel.
It does not start without it, resulting into errors like:
elogind-daemon[1654]: Failed to mount tmpfs at /sys/fs/cgroup: No such file or directory
elogind-daemon[1654]: Failed to mount cgroup at /sys/fs/cgroup/elogind: No such file or directory
elogind-daemon[1654]: Failed to allocate manager object: No such file or directory
dbus-daemon[1437]: [system] Activating service name='org.freedesktop.login1' requested by ':1.1' (uid=0 pid=1642 comm="/usr/bin/lightdm ") (using servicehelper)
dbus-daemon[1437]: [system] Failed to activate service 'org.freedesktop.login1': timed out (service_start_timeout=25000ms)
Most of the supported desktop environments (e.g. Weston) do not
start without elogind, so we should require CONFIG_CGROUPS=y in
all kernels to avoid this problem.
Make /home/pmos/build/.git point to the .git dir from pmaports.git, with
a symlink so abuild does not fail.
abuild expects the current working directory to be a subdirectory of a
cloned git repository (e.g. main/openrc from aports.git). If git is
installed, it will try to get the last git commit from that repository,
and place it in the resulting apk (.PKGINFO) as well as use the date
from that commit as SOURCE_DATE_EPOCH (for reproducible builds).
With that symlink, we actually make it use the last git commit from
pmaports.git for SOURCE_DATE_EPOCH and have that in the resulting apk's
.PKGINFO.
Fixes: #1841
When 'pmbootstrap build' is started with '--no-depends', we make sure
that binary packages of all dependencies exist before proceeding with a
build. Make sure that we have downloaded the APKINDEX for the given arch
first, otherwise this may complain that a binary package is missing
although it does exist.
This happened when using the "native" cross compile method, e.g. when
building kernels for different architectures. It would complain that
there is no binary package for "bash" (which is in Alpine obviously):
https://builds.sr.ht/~postmarketos/job/103882
When running pmbootstrap init, first select device vendor, then device
codename. Also fixed tests for new behavior and added some new ones for
new scenarios.
postmarketOS/pmaports!700 adds a new "deviceinfo_bootimg_dtb_second"
option that places the DTB in the "second" area of the Android boot
image.
Attempt to detect this automatically by checking the extracted
second binary for the FDT magic (0xd00dfeed).
Packages ending in -dev: just assume that the originating aport has the
same pkgname, except for the -dev at the end. Otherwise we may end up
with the wrong package.
For example, if something depends on plasma-framework-dev, and
plasma-framework is in Alpine, but plasma is in pmaports, then the
regular guess_main() algorithm below would pick plasma instead of
plasma-framework.
Fixes: build.postmarketos.org#52
mkbootimg from osm0sis changed the filename of the dtb section in [1]
which breaks the detection of qcdt in pmbootstrap.
Fix this by adjusting the filename pmbootstrap checks to match the
unpackbootimg behavior.
[1] https://github.com/osm0sis/mkbootimg/commit/d8222a4d
Currently, building with envkernel.sh entirely ignores the
makedepends listed in the kernel APKBUILD. Common dependencies
needed by most kernels are hardcoded in envkernel.sh.
However, some kernels may need extra dependencies either during
the build process or when the kernel is packaged.
Installing the makedepends when the build environment is initialized
is difficult because "source helpers/envkernel.sh" is not aware
of the exact kernel package that is going to be built later.
However, we can easily modify the packaging step
(i.e. pmbootstrap build --envkernel linux-...)
to install the required makedepends.
This fixes building "linux-postmarketos-qcom-msm8916" using envkernel,
since it requires "dtbTool" and "installkernel" to be installed when
the APKBUILD package() function is executed.
The sparse image tools (simg2img, img2simg, append2simg) are now
part of Alpine's android-tools where they are continously updated.
(See https://github.com/nmeum/android-tools/pull/8)
Therefore, "libsparse" now conflicts with "android-tools", which
causes the fastboot flasher to fail if "libsparse" is already
installed.
Install "android-tools" instead of "libsparse" before generating
a sparse image to avoid this problem.
Make it more obvious that we are generating packages during
'pmbootstrap init', when using the new device wizard:
[22:32:32] *** pmaport generated: /home/user/.local/var/pmbootstrap/cache_git/pmaports/device/device-openphoenux-neo900
[22:32:32] *** pmaport generated: /home/user/.local/var/pmbootstrap/cache_git/pmaports/device/linux-openphoenux-neo900
This file is used by the package browser frontend build by the alpine
developers. It uses the contents of the DESCRIPTION file to validate the
cache in the database.
Remove the recursive check, as it caused an infinite loop. It took a
very long time to complete anyway, even when it worked. The reasoning
for having the recursive check in the first place was, that we would
have device packages with arch=noarch set in the APKBUILD, but which
could only be built for a certain architecture (the device
architecture). Nowadays using arch=noarch in device packages is
forbidden, and we enforce that rule [1]. So the recursive arch check
isn't necessary anymore.
[1] ac6c0a2997
This reverts commit 6fb5b28e2f.
The -i option was removed from fastboot, so we can't use it anymore
unless we fork the package. There was only one device using it,
amazon-thor. I will add a note to the wiki page.
Fixes#1830.
Linking big libraries may take more than five minutes, so change the
default timeout to fifteen minutes to save users from frustration.
This is the same value that I'm using to build binary packages for the
repository.
Pushing directly to master without making a merge request, because this
is a trivial change.
Make sure, that "args.fork_alpine" is always present. Otherwise,
pmbootstrap will fail if the aportgen code is called by anything but
"pmbootstrap aportgen". For example, when the user is adding a new
device during "pmbootstrap init".
Fixes: 54e51759ad ("aportgen: add feature to fork upstream packages")
The important part about this patch is the change for armhf, which
adjusts the hostspec to the one used by Alpine.
Fixes a part of postmarketOS/pmaports#363
This MR add the --fork-alpine argument to the pmboostrap aportgen
command, which downloads the APKBUILD and related files and copies them
into the pmaports/temp folder.
At the moment, attempting to install to SD card when
flash_sparse is set to "true" will always fail because
/home/pmos/rootfs does not exist in that case.
Sparse images are only useful to speed up Fastboot flashing.
Nothing will be able to read it from the SD card.
The problem can therefore be avoided by simply skipping the
generation of the sparse image when SD card installation is used.
Remove u-boot from the native_cross_compile list as we now have
crossdirect. Compiling is not terribly slow anymore, and when not using
the "native" method, it is possible to depend on firmware packages like
arm-trusted-firmware-sun50i.
This reverts commit d7d7ccb672.
This allows for example for me to call the kconfig check function on the
.config file in my Linux tree: $ pmbootstrap kconfig check --file
.config and it reports me which kconfig options I need to enable.
From https://docs.python.org/3/library/re.html
"Backslashes are not handled in any special way
in a string literal prefixed with 'r'."
Signed-off-by: Steffen Pankratz <kratz00@gmx.de>
Properly handle the following two cases in APKBUILDs:
* depends="$depends ..."
* depends="${depends} ..."
First I've attempted to refactor the parsing code to do this in a more
generic way. But I've realized that it would make more sense to retire
the python based APKBUILD parsing approach altogether and finally use a
shell script parser. Let's discuss this in #1801.
Add check for UEVENT_HELPER for kernels >= 4.0.0. Change LBDAF check to
be only required for kernels <5.2.0. Adjust the config format and
checking code to support such range specific checks.
Compare a version against a check string. This will be used in
"pmbootstrap kconfig check", to only require certain options if the
pkgver is in a specified range.
This fixes regression from commits 0431a519 and 4daf9916.
pmbootstrap kconfig_edit raised an error while trying to checksum a package.
Error was:
(028793) [16:13:22] ERROR: 'module' object is not callable
(028793) [16:13:22] See also: <https://postmarketos.org/troubleshooting>
(028793) [16:13:22] Traceback (most recent call last):
File "/home/lexx/dev/pmos/pmbootstrap/pmb/__init__.py", line 63, in main
getattr(frontend, args.action)(args)
File "/home/lexx/dev/pmos/pmbootstrap/pmb/helpers/frontend.py", line 279, in kconfig
pmb.build.menuconfig(args, args.package)
File "/home/lexx/dev/pmos/pmbootstrap/pmb/build/menuconfig.py", line 158, in menuconfig
pmb.build.checksum(args, pkgname)
TypeError: 'module' object is not callable
The function was renamed, call new function.
(Some?) Mediatek kernels don't have the kernel in the top-level
directory but in a subdirectory called 'kernel' next to a folder
'mediatek' containing most device-specific code.
Adjust the menuconfig code to be able to work with that.
Download all sources and verify their checksums.
This will be used in pmaports.git CI, if ci:skip-build is set in the
commit message (currently it just skips the build, and we don't test if
the source checksums are valid or not).
Fix configure crash when building mozjs60 by mounting tmpfs as /dev/shm
in chroots. This is an important fix, because of this dependency chain,
that is currently broken in armhf and armv7 (see pmaports#244):
postmarketos-base -> networkmanager -> polkit -> mozjs60
Explicitly depend on mpc1. Our generated gcc aports use the !tracedeps
option, so we need to explicitly set the libraries it depends on.
This has mostly not been an issue, as we are installing our gcc
packages together with Alpine's gcc package, which causes the libraries
to get installed anyway.
Related: pmaports#236
abuild depends on the gcc binary in order to define a default CBUILD value.
When using an alternative gcc version (e.g. envkernel.sh with gcc6), the
gcc binary is not installed by the envkernel.sh script.
This change sets the CBUILD env variable so that abuild doesn't need to
depend on the gcc package.
Do not crash during "pmbootstrap zap" and other actions when the user
needs to type in the sudo password before running "du" to calculate the
work folder size. We're trying to parse the "du" output as integer, but
of course the 'Sorry, try again' message from sudo is not a valid
integer.
$ pmbootstrap --details-to-stdout zap
(008707) [14:00:57] Shutdown complete
(008707) [14:00:57] Calculate work folder size
(008707) [14:00:57] % sudo du --summarize --block-size=1 /home/luca/.local/var/pmbootstrap
[sudo] password for luca:
[sudo] password for luca: Sorry, try again.
12966293504 /home/luca/.local/var/pmbootstrap
(008707) [14:01:03] ERROR: invalid literal for int() with base 10: 'Sorry, try again.\n12966293504'
It would be a good idea to separate the sudo output from the process
output in general, see #1677 for a related proposal.
When the pmaport arch does not match, but there's a matching binary
package, return the binary package instead of the pmaport.
This is needed to fix "pmbootstrap repo_missing" for x86_64
networkmanager-qt, which indirectly depends on mesa (and mesa needs to
be taken from the binary packages for x86_64, not from the pmaports,
see #1741).
Split the part that actually checks the arch against the arches list
into pmb.helpers.pmaports.check_arches(arches, arch), and call it
from pmb.helpers.package.check_arch(args, pkgname, arch, binary).
This will be used in a follow up commit, where we have already resolved
the package data and only need to check the architecture.
Packages can add pmb:strict to their options to enable the --strict
mode. This can be used if packages need to get build in a clean chroot
or their make dependencies need to get removed.
Depend on new pmaports version, where crossdirect uses the native ccache
binary instead of going through the foreign arch ccache first and then
going through crossdirect.
Old:
ccache (foreign) -> crossdirect (native) -> gcc (native)
New:
crossdirect (native) -> ccache (native) -> gcc (native)
Set the PATH to the crossdirect binaries, and don't set CCACHE_PATH or
CCACHE_COMPILERCHECK from pmbootstrap anymore. crossdirect sets the
CCACHE_PATH to /native/usr/bin now, along with all other required
environment variables. CCACHE_COMPILERCHECK isn't necessary anymore,
because ccache will call gcc directly and therefore be able to use the
file's timestamp and size directly. Also passing that would not work
with the current crossdirect package.
Launch native cross compilers inside foreign chroot. Enable by default,
but allow disabling with --no-crossdirect for now. This option and the
distcc-sshd related code will be removed in the future.
Aborts the build if any dependencies would have to be build first. This
is useful for build.postmarketos.org, because we want to build exactly
one package in one build job. If dependencies would need to be built, we
made a mistake earlier, and not aborting the build makes it harder to
find that orginal mistake.
When a pmaport can not be built for the desired architecture, fall back
to the binary package (from postmarketOS or Alpine) if it exists. This
allows us to provide an updated version of mesa for arm arches, but
using Alpine's mesa package on x86* arches.
Follow up to last commit, turns out we need to use it in both functions
or otherwise at least gitlab-ci can fail, as seen in this MR:
https://gitlab.com/postmarketOS/pmaports/merge_requests/270
After using 'busybox tar' instead of 'tar' in both places in the
APKBUILDs, it went through.
Use 'busybox tar' to avoid 'tar: Child returned status 141' on some
machines (builds.sr.ht, gitlab-ci).
See pmaports#26.
[skip ci]: already ran through successfully
This adds flasher support for uuu, a utility used by NXP devices for
flashing images.
The flasher expects a uuu command list script to be installed in the
device rootfs at /usr/share/uuu/flash_script.lst.
This allows building u-boot much faster, previously it used to build
whole u-boot with distcc+qemu cross-compile, now it builds u-boot on
native chroot using cross-compilers.
In pmb.helpers.repo.alpine_apkindex_path(), default to
arch = args.arch_native *before* calling pmb.helpers.repo.update().
Because otherwise update() defaults to all arches instead of
args.arch_native.
This caused the APKINDEX files for all arches to get downloaded, as
pmb.chroot.apk_static.init() calls alpine_apkindex_path() without
setting arch = args.arch_native explicitly.
Related to a proprietary "T-Mobile's WiFi-Calling" feature. It should
not affect normal calls, so it should be safe to disable it. Having it
enabled causes sudo, ip and other commands to hang.
It is dangerous to flash the boot partition on the Ouya. Doing so can
result in a bricked device. Allow deviceinfo to blacklist partitions from
ever being flashed.
Ignore APKBUILDs that have "!pmb:kconfigcheck" in their options by
default in "pmbootstrap kconfig check", but print a note that they have
been skipped. Check all kernels with "pmbootstrap kconfig check -f".
This is necessary, because the Librem 5 devboard kernel's config does
not have CONFIG_DM_CRYPT enabled in their config, and we check for
that. As the device is still under heavy development, we will make our
lives easier by just using the upstream kernel config without any
changes and ignoring it in our check by default.
"breeze-icons" depends on "qt5-qtbase-dev", but
"pmbootstrap repo_missing" should return "qt5-qtbase" instead.
This patch fixes it, as one can see with:
$ pmbootstrap repo_missing --built breeze-icons --overview
Provides a quick way to incrementally compile a kernel and push it to
device.
Example usage.
Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX
Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Set the log level to DEBUG if args (and therefore logging) failed to
initialize. This causes the default logging handler to be used, and
everything that gets written to the log (error message and trace!) will
get printed to stdout instead of not being printed anywhere at all.
This can be tested by removing x86_64 from the mapping in
pmb.parse.arch.alpine_native() (assuming the host is x86_64). Without
this patch, no meaningful error appears. With the patch, the real error
message, and a stack trace gets printed.
Test with flake8: *.py
./test/check_checksums.py:13:13: E117 over-indented
./pmb/config/init.py:97:8: F632 use ==/!= to compare str, bytes, and int literals
./pmb/parse/arguments.py:229:13: E117 over-indented
Increase pakage version and minimum required apk version. The latter
should be done with every new release from now on, so we always have a
recent version there. I've updated the pmbootstrep release checklist
accordingly: https://wiki.postmarketos.org/wiki/Pmbootstrap_release
This drops the --apparent-size parameter when calculating the size
required for rootfs, which seemed to return a size that was too small
for some devices. This also includes specifying the number of inodes to
support when formatting rootfs.
Fixes#1717
Allow changing the kernel partition for fastboot and heimdall in
deviceinfo and on the fly while doing "pmbootstrap flasher
flash_kernel". Also allow changing the partition for
"... flash_rootfs" with fastboot (this was only possible with
heimdall so far).
Introduce two new deviceinfo variables:
* flash_fastboot_partition_kernel
* flash_fastboot_partition_system
This is useful for devices with dual partitioning that have boot_a
and boot_b.
This allows embedding multiple firmware binaries into SD images.
Firmware images for most devices that require this functionality consist
mainly of u-boot, but some devices (e.g. librem5) have multiple firmware
images that need to be embedded in the SD image created by pmbootstrap.
This functionality uses two new deviceinfo parameters:
- deviceinfo_sd_embed_firmware: a comma-separated list of
binary:offset (where binary is under /usr/share/firmware)
- deviceinfo_sd_embed_firmware_step_size: The number of bytes for
each increment of the offset specified in the
deviceinfo_sd_embed_firmware (typically 1024 or 2048)
deviceinfo_write_uboot_spl has been obsoleted by these new parameters.
Rename pmb.helpers.devices.list() to
pmb.helpers.devices.list_codenames(). Python already has a list()
function, so we name our function to make the calls to the codenames
listing function inside pmb/helpers/devices.py less confusing.
This fixes list_apkbuilds which is dead code currently but is useful
for external software using pmbootstrap. It also adds list_deviceinfos
which does the same thing but for the deviceinfo data.
The plan is to add an API later, where both functions can be used.
APKBUILD should be indented with tabs not spaces.
This also resolves a bug which the replace_functions feature was always
trimming off the beginning of a line.
For example:
replace_functions = {"build": "return 0"}
would be formatted as "rn 0".
Python < 3.6 randomized the order of keys in dictionaries, unless
OrderedDict was used. Use OrderedDict to store the version suffixes.
When the order was randomized, the valid version string 3.0.0_pre1 did
not always pass the validation check. The suffix "pre" should always be
detected as such, but with the random order, it was sometimes detected
as "p" suffix (see below). The following letters "re" are not a valid
suffix_no (the number expected to follow the suffix) and so it failed.
suffixes = {
"pre": ["pre", ...],
"post": ["p". ...]
}
Display the resolved deviceinfo file path in exceptions from the
deviceinfo parser. Instead of messages like these:
RuntimeError: Please add 'deviceinfo_codename="lg-mako"' to: /home/user/.local/var/pmbootstrap/cache_git/pmaports/.gitlab-ci/testcases/../../device/device-lg-mako/deviceinfo
We get the more readable version:
RuntimeError: Please add 'deviceinfo_codename="lg-mako"' to: /home/user/.local/var/pmbootstrap/cache_git/pmaports/device/device-lg-mako/deviceinfo
deviceinfo_codneame holds the device's code name, so we can easily look
it up in the finished postmarketOS installation by reading
/etc/deviceinfo.
Related: postmarketOS/pmaports#157
Properly replace ~ with $HOME in all paths. Fix tab completion when
passing a path starting with ~ to pmaports:
$ pmbootstrap --aports ~/src/pmaports/ build linux-<TAB>
Make tab completion work again for packages, if the default pmaports
dir is used. In the last "args" code refactor, the variable replacing
code was moved into its own function. We did not call it in the
packagecompleter() yet, so it could not replace the $WORK variable in
the default pmaports path.
Show all config keys that can be queried and set in the
'pmbootstrap config -h' output and make tab completion work for the
key names.
I've set "metavar" and placed the variables in the helpstring. That
way, argparse will not generate a huge "positional arguments" string
that blows up the layout of the help output:
[{ccache_size,device,extra_packages,hostname,jobs,kernel,keymap,...
Make it possible to point the pmaports dir to an existing repository
clone, while not having to give the --aports/-p parameter with every
command.
pmbootstrap --aports=/my/pmaports/path init
If the config file exists already, the dir can also be set with:
pmbootstrap config aports /my/pmaports/path
Adds an optional deviceinfo variable, `deviceinfo_rootfs_image_sector_size`,
which specifies the logical sector size of the device's storage.
Some devices made after 2016 with UFS storage uses 4096 byte sectors
instead of the normal 512 bytes. The partition table in our rootfs
must match, otherwise the root filesystem won't mount on the device.
This change passes the sector size to `losetup` when creating the image
if the deviceinfo specifies it, so the image will have the correct
sector size.
If the deviceinfo doesn't specify the new option, the behaviour
is the same as previous versions of pmbootstrap.
Note that the sector size option only works on Linux 4.14 and above,
so pmbootstrap should be run on a >4.14 computer when installing to
devices with non-standard sector size.
To find if a device needs this parameter, run `fdisk -l` on the device.
If the output shows
`Note: sector size is 4096 (not 512)`
then add `deviceinfo_rootfs_image_sector_size="4096"` to the deviceinfo.
This is needed by the Pixel 3 XL (google-crosshatch) port.
See https://gitlab.com/postmarketOS/pmbootstrap/issues/1696.
Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.
This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.
Change the cache format from args.cache["offline_msg_shown"] to
args.cache["pmb.helpers.repo.update"]["offline_msg_shown"]. This is in
preparation for saving more data in the cache of
pmb.helpers.repo.update in the next commit.
Do not fail anymore when attempting to start a new binary repository
build without any existing binary packages:
pmbootstrap -mp="" repo_missing
Find subpackages defined with subpackage functions:
subpackages="dev:mydevfunc"
Find provides defined with specific versions:
provides="mkbootimg=1.0.0"
Weston and Plasma Mobile might choose to draw on this virtual frame
buffer, just like it happened for the samsung-jflte. Disabling the
USE_VFB option fixed this, so let's make sure we don't have that option
enabled for any kernel.
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories
This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
Obscure feature: it was possible to specify a local path as
--mirror-pmOS. It would then get mounted to /mnt/postmarketos-mirror
inside the chroot, and be specified as such in the generated
/etc/apk/repositories file.
I had used this once for some testing scripts, but I am sure nobody is
using this anymore. The same can be achieved with running a local http
server anyway:
<https://wiki.postmarketos.org/wiki/Installing_packages_on_a_running_phone>
Removing this makes it easier to support multiple postmarketOS mirrors
(next commit).
-m is the Alpine mirror, -mp is the postmarketOS mirror. Use "URL" as
metavar and add help text that explains how to disable the postmarketOS
mirror (so all pmaports get built locally).
In pmb.helpers.package.get(), we are differentiating between packages
that do not exist at all, and packages that do not exist for the
specified architecture. Make sure to actuallyy download the foreign arch
APKINDEX files, before trying to find the packages in there. Otherwise
the "could not find aport, and could not find this package in any
APKINDEX" package would appear.
We ran into this when testing on sr.ht, because pmbootstrap runs on a
fresh install every time, where no old APKINDEX files are present
(which would work around the bug).
This was meant to be part of the previous commit already, and was
tested, but it was not submitted correctly. So here it is as separate
commit straight to master. It is trivial anyway.
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.
Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
Target arch is the system the package will run on. Host arch is the
system the package is compiled on.
For example kernel packages can be compiled on host arch x86_64 and
intended to run on target arch armhf. A build is necessary check
against host arch will always return True. The correct way is to check if
the package needs to be built for target arch.
Do not try to build and install dependencies for the package's
architecture when compiling in the "native" mode. That mode is
described here in detail:
https://wiki.postmarketos.org/wiki/Build_internals#Cross-compile_types
This makes it possible to cross compile kernels again, which need to
be built with GCC6. We have switched to Alpine's GCC6 package, but it is
not available for armhf/aarch64 on edge yet, because Alpine's build bots
are stuck (right now armhf is not even listed):
http://build.alpinelinux.org/
Huge thanks to ryang2478/Decatf for posting this patch in pmaports#138!
Oliver Smith
Minecrell
HenriDellal
Anjandev Momi
Luca Weiss
Antoine Fontaine
Daniele Debernardi
Dolphin von Chips
Martijn Braam
Niklas Cathor
Alexey Min
KuroLevin
Danct12
Clayton Craft
Drew DeVault
Bart Ribbers
lambdadroid
Steffen Pankratz
Oliver Smith
Idan Horo
Robert Yang
Nick Reitemeyer
Zhuowei Zhang
Bhushan Shah
Grant Miller
Ion Agorria
Craig Comstock