At the moment, attempting to install to SD card when
flash_sparse is set to "true" will always fail because
/home/pmos/rootfs does not exist in that case.
Sparse images are only useful to speed up Fastboot flashing.
Nothing will be able to read it from the SD card.
The problem can therefore be avoided by simply skipping the
generation of the sparse image when SD card installation is used.
Remove u-boot from the native_cross_compile list as we now have
crossdirect. Compiling is not terribly slow anymore, and when not using
the "native" method, it is possible to depend on firmware packages like
arm-trusted-firmware-sun50i.
This reverts commit d7d7ccb672.
This allows for example for me to call the kconfig check function on the
.config file in my Linux tree: $ pmbootstrap kconfig check --file
.config and it reports me which kconfig options I need to enable.
From https://docs.python.org/3/library/re.html
"Backslashes are not handled in any special way
in a string literal prefixed with 'r'."
Signed-off-by: Steffen Pankratz <kratz00@gmx.de>
Properly handle the following two cases in APKBUILDs:
* depends="$depends ..."
* depends="${depends} ..."
First I've attempted to refactor the parsing code to do this in a more
generic way. But I've realized that it would make more sense to retire
the python based APKBUILD parsing approach altogether and finally use a
shell script parser. Let's discuss this in #1801.
Add check for UEVENT_HELPER for kernels >= 4.0.0. Change LBDAF check to
be only required for kernels <5.2.0. Adjust the config format and
checking code to support such range specific checks.
Compare a version against a check string. This will be used in
"pmbootstrap kconfig check", to only require certain options if the
pkgver is in a specified range.
This fixes regression from commits 0431a519 and 4daf9916.
pmbootstrap kconfig_edit raised an error while trying to checksum a package.
Error was:
(028793) [16:13:22] ERROR: 'module' object is not callable
(028793) [16:13:22] See also: <https://postmarketos.org/troubleshooting>
(028793) [16:13:22] Traceback (most recent call last):
File "/home/lexx/dev/pmos/pmbootstrap/pmb/__init__.py", line 63, in main
getattr(frontend, args.action)(args)
File "/home/lexx/dev/pmos/pmbootstrap/pmb/helpers/frontend.py", line 279, in kconfig
pmb.build.menuconfig(args, args.package)
File "/home/lexx/dev/pmos/pmbootstrap/pmb/build/menuconfig.py", line 158, in menuconfig
pmb.build.checksum(args, pkgname)
TypeError: 'module' object is not callable
The function was renamed, call new function.
(Some?) Mediatek kernels don't have the kernel in the top-level
directory but in a subdirectory called 'kernel' next to a folder
'mediatek' containing most device-specific code.
Adjust the menuconfig code to be able to work with that.
Download all sources and verify their checksums.
This will be used in pmaports.git CI, if ci:skip-build is set in the
commit message (currently it just skips the build, and we don't test if
the source checksums are valid or not).
Fix configure crash when building mozjs60 by mounting tmpfs as /dev/shm
in chroots. This is an important fix, because of this dependency chain,
that is currently broken in armhf and armv7 (see pmaports#244):
postmarketos-base -> networkmanager -> polkit -> mozjs60
Explicitly depend on mpc1. Our generated gcc aports use the !tracedeps
option, so we need to explicitly set the libraries it depends on.
This has mostly not been an issue, as we are installing our gcc
packages together with Alpine's gcc package, which causes the libraries
to get installed anyway.
Related: pmaports#236
abuild depends on the gcc binary in order to define a default CBUILD value.
When using an alternative gcc version (e.g. envkernel.sh with gcc6), the
gcc binary is not installed by the envkernel.sh script.
This change sets the CBUILD env variable so that abuild doesn't need to
depend on the gcc package.
Do not crash during "pmbootstrap zap" and other actions when the user
needs to type in the sudo password before running "du" to calculate the
work folder size. We're trying to parse the "du" output as integer, but
of course the 'Sorry, try again' message from sudo is not a valid
integer.
$ pmbootstrap --details-to-stdout zap
(008707) [14:00:57] Shutdown complete
(008707) [14:00:57] Calculate work folder size
(008707) [14:00:57] % sudo du --summarize --block-size=1 /home/luca/.local/var/pmbootstrap
[sudo] password for luca:
[sudo] password for luca: Sorry, try again.
12966293504 /home/luca/.local/var/pmbootstrap
(008707) [14:01:03] ERROR: invalid literal for int() with base 10: 'Sorry, try again.\n12966293504'
It would be a good idea to separate the sudo output from the process
output in general, see #1677 for a related proposal.
When the pmaport arch does not match, but there's a matching binary
package, return the binary package instead of the pmaport.
This is needed to fix "pmbootstrap repo_missing" for x86_64
networkmanager-qt, which indirectly depends on mesa (and mesa needs to
be taken from the binary packages for x86_64, not from the pmaports,
see #1741).
Split the part that actually checks the arch against the arches list
into pmb.helpers.pmaports.check_arches(arches, arch), and call it
from pmb.helpers.package.check_arch(args, pkgname, arch, binary).
This will be used in a follow up commit, where we have already resolved
the package data and only need to check the architecture.
Packages can add pmb:strict to their options to enable the --strict
mode. This can be used if packages need to get build in a clean chroot
or their make dependencies need to get removed.
Depend on new pmaports version, where crossdirect uses the native ccache
binary instead of going through the foreign arch ccache first and then
going through crossdirect.
Old:
ccache (foreign) -> crossdirect (native) -> gcc (native)
New:
crossdirect (native) -> ccache (native) -> gcc (native)
Set the PATH to the crossdirect binaries, and don't set CCACHE_PATH or
CCACHE_COMPILERCHECK from pmbootstrap anymore. crossdirect sets the
CCACHE_PATH to /native/usr/bin now, along with all other required
environment variables. CCACHE_COMPILERCHECK isn't necessary anymore,
because ccache will call gcc directly and therefore be able to use the
file's timestamp and size directly. Also passing that would not work
with the current crossdirect package.
Launch native cross compilers inside foreign chroot. Enable by default,
but allow disabling with --no-crossdirect for now. This option and the
distcc-sshd related code will be removed in the future.
Aborts the build if any dependencies would have to be build first. This
is useful for build.postmarketos.org, because we want to build exactly
one package in one build job. If dependencies would need to be built, we
made a mistake earlier, and not aborting the build makes it harder to
find that orginal mistake.
When a pmaport can not be built for the desired architecture, fall back
to the binary package (from postmarketOS or Alpine) if it exists. This
allows us to provide an updated version of mesa for arm arches, but
using Alpine's mesa package on x86* arches.
Follow up to last commit, turns out we need to use it in both functions
or otherwise at least gitlab-ci can fail, as seen in this MR:
https://gitlab.com/postmarketOS/pmaports/merge_requests/270
After using 'busybox tar' instead of 'tar' in both places in the
APKBUILDs, it went through.
Use 'busybox tar' to avoid 'tar: Child returned status 141' on some
machines (builds.sr.ht, gitlab-ci).
See pmaports#26.
[skip ci]: already ran through successfully
This adds flasher support for uuu, a utility used by NXP devices for
flashing images.
The flasher expects a uuu command list script to be installed in the
device rootfs at /usr/share/uuu/flash_script.lst.
This allows building u-boot much faster, previously it used to build
whole u-boot with distcc+qemu cross-compile, now it builds u-boot on
native chroot using cross-compilers.
In pmb.helpers.repo.alpine_apkindex_path(), default to
arch = args.arch_native *before* calling pmb.helpers.repo.update().
Because otherwise update() defaults to all arches instead of
args.arch_native.
This caused the APKINDEX files for all arches to get downloaded, as
pmb.chroot.apk_static.init() calls alpine_apkindex_path() without
setting arch = args.arch_native explicitly.
Related to a proprietary "T-Mobile's WiFi-Calling" feature. It should
not affect normal calls, so it should be safe to disable it. Having it
enabled causes sudo, ip and other commands to hang.
It is dangerous to flash the boot partition on the Ouya. Doing so can
result in a bricked device. Allow deviceinfo to blacklist partitions from
ever being flashed.
Ignore APKBUILDs that have "!pmb:kconfigcheck" in their options by
default in "pmbootstrap kconfig check", but print a note that they have
been skipped. Check all kernels with "pmbootstrap kconfig check -f".
This is necessary, because the Librem 5 devboard kernel's config does
not have CONFIG_DM_CRYPT enabled in their config, and we check for
that. As the device is still under heavy development, we will make our
lives easier by just using the upstream kernel config without any
changes and ignoring it in our check by default.
"breeze-icons" depends on "qt5-qtbase-dev", but
"pmbootstrap repo_missing" should return "qt5-qtbase" instead.
This patch fixes it, as one can see with:
$ pmbootstrap repo_missing --built breeze-icons --overview
Provides a quick way to incrementally compile a kernel and push it to
device.
Example usage.
Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX
Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
Set the log level to DEBUG if args (and therefore logging) failed to
initialize. This causes the default logging handler to be used, and
everything that gets written to the log (error message and trace!) will
get printed to stdout instead of not being printed anywhere at all.
This can be tested by removing x86_64 from the mapping in
pmb.parse.arch.alpine_native() (assuming the host is x86_64). Without
this patch, no meaningful error appears. With the patch, the real error
message, and a stack trace gets printed.
Test with flake8: *.py
./test/check_checksums.py:13:13: E117 over-indented
./pmb/config/init.py:97:8: F632 use ==/!= to compare str, bytes, and int literals
./pmb/parse/arguments.py:229:13: E117 over-indented
Increase pakage version and minimum required apk version. The latter
should be done with every new release from now on, so we always have a
recent version there. I've updated the pmbootstrep release checklist
accordingly: https://wiki.postmarketos.org/wiki/Pmbootstrap_release
This drops the --apparent-size parameter when calculating the size
required for rootfs, which seemed to return a size that was too small
for some devices. This also includes specifying the number of inodes to
support when formatting rootfs.
Fixes#1717
Allow changing the kernel partition for fastboot and heimdall in
deviceinfo and on the fly while doing "pmbootstrap flasher
flash_kernel". Also allow changing the partition for
"... flash_rootfs" with fastboot (this was only possible with
heimdall so far).
Introduce two new deviceinfo variables:
* flash_fastboot_partition_kernel
* flash_fastboot_partition_system
This is useful for devices with dual partitioning that have boot_a
and boot_b.
This allows embedding multiple firmware binaries into SD images.
Firmware images for most devices that require this functionality consist
mainly of u-boot, but some devices (e.g. librem5) have multiple firmware
images that need to be embedded in the SD image created by pmbootstrap.
This functionality uses two new deviceinfo parameters:
- deviceinfo_sd_embed_firmware: a comma-separated list of
binary:offset (where binary is under /usr/share/firmware)
- deviceinfo_sd_embed_firmware_step_size: The number of bytes for
each increment of the offset specified in the
deviceinfo_sd_embed_firmware (typically 1024 or 2048)
deviceinfo_write_uboot_spl has been obsoleted by these new parameters.
Rename pmb.helpers.devices.list() to
pmb.helpers.devices.list_codenames(). Python already has a list()
function, so we name our function to make the calls to the codenames
listing function inside pmb/helpers/devices.py less confusing.
This fixes list_apkbuilds which is dead code currently but is useful
for external software using pmbootstrap. It also adds list_deviceinfos
which does the same thing but for the deviceinfo data.
The plan is to add an API later, where both functions can be used.
APKBUILD should be indented with tabs not spaces.
This also resolves a bug which the replace_functions feature was always
trimming off the beginning of a line.
For example:
replace_functions = {"build": "return 0"}
would be formatted as "rn 0".
Python < 3.6 randomized the order of keys in dictionaries, unless
OrderedDict was used. Use OrderedDict to store the version suffixes.
When the order was randomized, the valid version string 3.0.0_pre1 did
not always pass the validation check. The suffix "pre" should always be
detected as such, but with the random order, it was sometimes detected
as "p" suffix (see below). The following letters "re" are not a valid
suffix_no (the number expected to follow the suffix) and so it failed.
suffixes = {
"pre": ["pre", ...],
"post": ["p". ...]
}
Display the resolved deviceinfo file path in exceptions from the
deviceinfo parser. Instead of messages like these:
RuntimeError: Please add 'deviceinfo_codename="lg-mako"' to: /home/user/.local/var/pmbootstrap/cache_git/pmaports/.gitlab-ci/testcases/../../device/device-lg-mako/deviceinfo
We get the more readable version:
RuntimeError: Please add 'deviceinfo_codename="lg-mako"' to: /home/user/.local/var/pmbootstrap/cache_git/pmaports/device/device-lg-mako/deviceinfo
deviceinfo_codneame holds the device's code name, so we can easily look
it up in the finished postmarketOS installation by reading
/etc/deviceinfo.
Related: postmarketOS/pmaports#157
Properly replace ~ with $HOME in all paths. Fix tab completion when
passing a path starting with ~ to pmaports:
$ pmbootstrap --aports ~/src/pmaports/ build linux-<TAB>
Make tab completion work again for packages, if the default pmaports
dir is used. In the last "args" code refactor, the variable replacing
code was moved into its own function. We did not call it in the
packagecompleter() yet, so it could not replace the $WORK variable in
the default pmaports path.
Show all config keys that can be queried and set in the
'pmbootstrap config -h' output and make tab completion work for the
key names.
I've set "metavar" and placed the variables in the helpstring. That
way, argparse will not generate a huge "positional arguments" string
that blows up the layout of the help output:
[{ccache_size,device,extra_packages,hostname,jobs,kernel,keymap,...
Make it possible to point the pmaports dir to an existing repository
clone, while not having to give the --aports/-p parameter with every
command.
pmbootstrap --aports=/my/pmaports/path init
If the config file exists already, the dir can also be set with:
pmbootstrap config aports /my/pmaports/path
Adds an optional deviceinfo variable, `deviceinfo_rootfs_image_sector_size`,
which specifies the logical sector size of the device's storage.
Some devices made after 2016 with UFS storage uses 4096 byte sectors
instead of the normal 512 bytes. The partition table in our rootfs
must match, otherwise the root filesystem won't mount on the device.
This change passes the sector size to `losetup` when creating the image
if the deviceinfo specifies it, so the image will have the correct
sector size.
If the deviceinfo doesn't specify the new option, the behaviour
is the same as previous versions of pmbootstrap.
Note that the sector size option only works on Linux 4.14 and above,
so pmbootstrap should be run on a >4.14 computer when installing to
devices with non-standard sector size.
To find if a device needs this parameter, run `fdisk -l` on the device.
If the output shows
`Note: sector size is 4096 (not 512)`
then add `deviceinfo_rootfs_image_sector_size="4096"` to the deviceinfo.
This is needed by the Pixel 3 XL (google-crosshatch) port.
See https://gitlab.com/postmarketOS/pmbootstrap/issues/1696.
Do not fail when an APKINDEX can not be downloaded, print a WARNING
instead. This matches, what apk does. Add a new allow_404 parameter to
pmb.helpers.http.download(), and use it in pmb.helpers.repo.update()
for downloading APKINDEX files. Cache the APKINDEX URLs that gave a 404
for the session, so we do not attempt to download these again.
This is needed for the new binary repository: the initial build is done
without existing APKINDEX files, so we must not fail in that case.
Change the cache format from args.cache["offline_msg_shown"] to
args.cache["pmb.helpers.repo.update"]["offline_msg_shown"]. This is in
preparation for saving more data in the cache of
pmb.helpers.repo.update in the next commit.
Do not fail anymore when attempting to start a new binary repository
build without any existing binary packages:
pmbootstrap -mp="" repo_missing
Find subpackages defined with subpackage functions:
subpackages="dev:mydevfunc"
Find provides defined with specific versions:
provides="mkbootimg=1.0.0"
Weston and Plasma Mobile might choose to draw on this virtual frame
buffer, just like it happened for the samsung-jflte. Disabling the
USE_VFB option fixed this, so let's make sure we don't have that option
enabled for any kernel.
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories
This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
Obscure feature: it was possible to specify a local path as
--mirror-pmOS. It would then get mounted to /mnt/postmarketos-mirror
inside the chroot, and be specified as such in the generated
/etc/apk/repositories file.
I had used this once for some testing scripts, but I am sure nobody is
using this anymore. The same can be achieved with running a local http
server anyway:
<https://wiki.postmarketos.org/wiki/Installing_packages_on_a_running_phone>
Removing this makes it easier to support multiple postmarketOS mirrors
(next commit).
-m is the Alpine mirror, -mp is the postmarketOS mirror. Use "URL" as
metavar and add help text that explains how to disable the postmarketOS
mirror (so all pmaports get built locally).
In pmb.helpers.package.get(), we are differentiating between packages
that do not exist at all, and packages that do not exist for the
specified architecture. Make sure to actuallyy download the foreign arch
APKINDEX files, before trying to find the packages in there. Otherwise
the "could not find aport, and could not find this package in any
APKINDEX" package would appear.
We ran into this when testing on sr.ht, because pmbootstrap runs on a
fresh install every time, where no old APKINDEX files are present
(which would work around the bug).
This was meant to be part of the previous commit already, and was
tested, but it was not submitted correctly. So here it is as separate
commit straight to master. It is trivial anyway.
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.
This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.
For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.
Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.
Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
distinguish it from the other check_arch function
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.
Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
lambdadroid
Luca Weiss
Steffen Pankratz
Oliver Smith
Daniele Debernardi
Idan Horo
Martijn Braam
Alexey Min
Robert Yang
Nick Reitemeyer
Zhuowei Zhang
Clayton Craft
Bhushan Shah
Grant Miller
Ion Agorria
Craig Comstock