The `suffix` argument was not specified in chroot commands executed in
`pmb.build._package.override_source()`. Because of that, it was not
possible to use "build --src" when compiling in a non-native chroot,
for example:
```
$ pmbootstrap build hello-world --arch=armhf
...
(native) % rm /tmp/APKBUILD.append
rm: can't remove '/tmp/APKBUILD.append': No such file or directory
```
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.
### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
* Default is updating all arches where the APKBUILD files exist
* Add existing_only parameter
* Return True when files have been downloaded
* Properly print which arches will be updated
* Print update reason only in verbose log
* Add and improve comments
* pmb.parse.arguments(), update action:
* Add --non-existing parameter
* Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
* Inform about --non-existing if no APKBUILDs have been updated
In case the user does not specify for which arch packages should be
built with `pmbootstrap build`, we detect it automatically.
Previous logic was, that if the APKBUILD's arch is "all" or "noarch",
then prefer the native arch, and otherwise use the first one in the
list of available arches.
New behavior is, that we also check if the list of possible arches
contains the native arch (and if that fails, the device arch). If that
is the case, we return the native/device arch instead of the first one
in the list.
### Use case
The arch from `gcc-armhf` and similar packages (as generated by
`pmbootstrap aportgen`) used to be "all", but is nowadays a specific
list of arches. This means, that after updating the `gcc-armhf` and
`gcc-aarch64` packages, and calling `pmbootstrap build gcc-armhf`,
it will try to build `gcc-armhf` for `aarch64` instead of the native
architecture, because that is the first one listed.
And since compiling to `aarch64` requires `gcc-aarch64`, it will build
that for the native architecture first.
So you're asking for `gcc-armhf` and it compiles `gcc-aarch64`, which
is very confusing (see #1272).
pmbootstrap does dependency resolving on its own, and passes the list
of resolved packages to apk when we want it to install something. The
reason was outlined in #129:
> fixing #120: packages do not get updated in "pmbootstrap install"
> after they have been rebuilt. For this to work, we specify all
> packages explicitly for abuild, instead of letting abuild do the
> resolving.
This new PR fixes#1212 (which noted that all of these dependencies
were explicitly marked for installation) by doing the following:
1. All packages and dependencies get attached to the virtual package
.pmbootstrap instead of world
2. We install the packages (without depends) explcitly
3. .pmbootstrap gets removed, which means that all packages from 1.
stay installed, but are no longer marked as explicitly installed.
They will get removed automatically, when the depending packages get
removed.
In addition, the mechanism for replacing the package of locally built
packages with their full path, was broken and has been fixed in this
commit. This is necessary to update packages of the same version with
apk.
* device-*: add postmarketos-base to depends
* aportgen: add postmarketos-base to depends
* Add test case
* postmarketos-base: Don't depend on devicepkg
* msm-fb-refresher: Enable service in post-install
We don't use 'arch="all"' in our kernels anymore (that does not make sense,
since each arch needs its own kernel config). This patch fixes the menuconfig
code to work with multiple values in the "arch" field.
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.
Changes:
* New config file options nonfree_firmware and nonfree_userland, which
we ask for during "pmbootstrap init" if there are non-free components
for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
packages are called $pkgname-nonfree-firmware and
$pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
subpackages. Parsing that is implemented in
pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
the regular APKBUILD parsing, as this would need a change in the
output format, and it is a lot *less* code if done like in this
commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
pmb/install/install.py to _install.py: needed to call the function in
the usual way (e.g. pmb.parse.apkbuild()) but still being able to
test the individual functions from these files in the test suite.
We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
packages that will be installed, based on the user's choice in
"pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
refactored test/test_questions.py to have multiple functions for
testing the various questions / question types from
"pmbootstrap init" instead of having it all in one big function.
This allows to use another aport folder for testing the new
non-free related questions in init.
The message appears 20 or so times in a row when setting up a new
chroot. Right now we log it into the debug log (which is what
"pmbootstrap log" and "pmbootstrap --details-to-stdout" show). This
is annoying for normal pmbootstrap usage, and really clutters up the
Travis output.
This commit changes the log level to verbose, so it only shows up
when "-v" is passed to pmbootstrap.
Without this fix, `pmbootstrap zap -m` fails with:
File "/home/user/code/pmbootstrap/pmb/__init__.py", line 61, in main
getattr(frontend, args.action)(args)
File "/home/user/code/pmbootstrap/pmb/helpers/frontend.py", line 322, in zap
distfiles=args.distfiles)
File "/home/user/code/pmbootstrap/pmb/chroot/zap.py", line 54, in zap
zap_mismatch_bins(args, confirm, dry)
File "/home/user/code/pmbootstrap/pmb/chroot/zap.py", line 110, in zap_mismatch_bins
if pkgname != bin_data["pkgname"]:
KeyError: 'pkgname'
* The APKINDEX parser used to return a dictionary with one package for
a given package name. This works for the installed packages database,
because there can only be one provider for a package. But when
parsing packages from binary repositories, we need to support
multiple providers for one package. It is now possible to get a
dictionary with either multiple providers, or just a single provider
for each package.
* Dependency parsing logic has been adjusted, to support multiple
providers. For multiple providers, the one with the same package
name as the package we are looking up is prefered. If there is none
(eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
that will be installed anyway, and after that packages that are
already installed. When all else fails, it just picks the first one
and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
that pmbootstrap should build if neccessary, and install in the
chroot. This can be used to quickly test the depencency resolution
of pmbootstrap without doing a full "pmbootstrap install".
Fixes#1122.
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
we can write into the source folder if necessary, for better
compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
usually), we extend the APKBUILD with overrides to make it use
mountpoint's source instead of downloading the package's source
from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
The 'necessary_kconfig_options' dictionary in pmb/config/__init__.py
now has the different architectures (space separated) as the keys and
the dictionary, which matches kernel config options and their
expected value, as its value.
For that purpose, the 'check' function in pmb/parse/kconfig.py was
modified, so that it takes the architecture from the kconfig filename
and uses it to find the needed kernel config options.
Closes#1218.
Closes#441. Adjust bootimg_analyze code:
* Install mkbootimg (which now provides unpackbootimg) instead of
unpackbootimg. In theory, pmbootstrap should recognize this
automatically, however right now it does not yet handle this case.
* The file names of the extracted files have changed.
Right now, they appear on screen when using --details-to-stdout. This
does not work well with Travis CI and screws up the log.
Disabling the progress bars in abuild works just like Alpine does it in
their Travis CI script: Exporting SUDO_APK as
"abuild-apk --no-progress" instead of "abuild-apk".
test_check_checksums.py: Run "pmbootstrap build_init" before building
any packages, so it is a bit less verbose (downloading the APKINDEX
files etc.). Later we run the build init code again (because we use
--strict while building the packages), but then the APKINDEX files
are already present. So overall the log is a bit shorter before the
building starts. (It is still logged to the logfile, which gets
printed on error anyway.)
* Testsuite: Run UIs in Qemu and check running processes (and other changes)
* When `pmbootstrap qemu` gets killed, it now takes down the Qemu process with it
* `test/check_checksums.py` got a new optional `--build` parameter, which makes
it build all changed packages instead of just checking the checksums
* We run this before running the testsuite now, so all changed packages get
built before running tests (otherwise tests would hang without any output
while a changed package is building)
* New testcase, that zaps all chroots, installs a specific UI (xfce4 and
plasma-mobile currently, easy to extend), runs it via Qemu and checks the
running processes via SSH.
* Version checking testcase: rewritten to include Alpine's testsuite file in
our source tree, so we don't need to clone their git repo anymore. Now it
is enabled for Travis.
* All this gives us a nice 10% code coverage boost
* Increased the `hello-world` pkgrel to verify that the Travis job is working.
* Various fixes
* Build device-packages for the device arch and don't raise an
exception, but print a note if --ignore-depends is not specified
and therefore the kernel gets installed, too.
* Don't use --force when building in Travis (because abuild doesn't
check the checksums then. Bug report on the way.)
* Don't run the building process in the background, but wait for its
completion
* Exit with 1 when showing usage in check_checksums.py
It used to have an entry for /mnt/pmbootstrap-packages, which only
makes sense while working on the chroot with pmbootstrap. After the
installation on the device, there's no repo in that path.
Furthermore, empty lines were added to the recovery installer script
for readability (thanks @ata2001!)
Follow up to #1162.
* `pmb.build.buildinfo()`: Used to record the build environment. It is
flawed because it scans the repo APKINDEX files instead of using the
actually installed packages list. When it was implemented we were not
able to do the latter. After this is removed, `pmb.parse.depends` can
be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
used exclusively by `pmb.build.buildinfo()`, to learn about which
files have been changed in the local repository folder after a
package was built. The idea was, that we could find subpackages that
way. But this information is present in the installed package list as
well, which is a much cleaner approach.
Nowadays pmb.config.build_device_architectures holds the native
architecture as well, so we don't need to explicitly download the
native architecture APKINDEX.
* Fail if mkbootimg/uboot-tools are not installed, but creating a
boot.img file / u-boot legacy image was requested via deviceinfo
(fixes#312)
* Fail if /boot/dt.img is missing, but we have a qcdt device
* Fail if the dtb file specified in deviceinfo does not exist
* Fail if mkbootimg etc. exit with error code
* Don't try to add the ext4 module into the initramfs. We always
compile it into the kernel. Instead, kconfig_check makes sure it
is enabled now. (fixes#1037)
* Add a note that modprobe warnings can be ignored mostly
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
have an aport or where the aport has another version. This also
works with subpackages now,
we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"
zap in general:
* Show the amount of cleared up space after the deletion instead
of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
(we need to call it twice during zap now to get the space
calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
for debugging) to list the packages/chroots that would get
deleted
* Roughly output the command that would get executed to delete
files, so it's obvious what's going on in --dry mode. (% rm ...)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:
pmbootstrap build --force hello-world
Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close#1167, close#1156, close#1023 and
close#985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
Preparation for #1122.
* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
an exception when two entries in the apkindex provided the same package.
Turns out this is *not* invalid after all, two packages can provide the same
soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
which was a dead end. This code uses the broken strict feature.
The message showed up, when you apk-static download could not be
verified. What the user needs to do instead is checking if openssl
is installed, and possibly delete the http cache ("zap -hc").
This PR makes the workflow faster and pmbootstrap will
produce less traffic. Details:
* Check if it's possible to create and read from a device
node directly when initializing a chroot (closes#472)
* Copy the Qemu binary into the forign-arch chroots
before initializing them, so the post-install script
directly work during the chroot setup and we don't need
to call apk fix afterwards
* Use pmb.helpers.repo.update(), which only updates the
APKINDEX files if they are older than 4 hours, instead
of using apk's repo update function which always
downloads the APKINDEX files
* Chroot initialization
* Getting the initial APKINDEX to download apk-tools-static
* Updating the APKINDEX at the start of pmbootstrap install
* Fixed a bug in from_chroot_suffix: the buildroot_x86_64 has
architecture x86_64, not x86.
Example: Building gcc-armhf for armhf does not make sense, so this
commit changes arch="all" to arch="aarch64 x86_64". This helps to
simplify the repository scripts (#970).
Also don't build "-repack" packages in native chroot anymore.
This was a legacy hack, which has no use anymore, and it prevented
the package from being built for different architectures.
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
(also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
* pmbootstrap: __config_.py - update the deviceinfo_attributes table
Add missing attributes:
* "screen_width"
* "screen_height"
* "dev_touchscreen"
* "dev_touchscreen_calibration"
* "dev_keyboard"
* "bootimg_qcdt"
Reorder the list to correspond to pmb/aportgen/device.py
Add a comment in the aforementioned file to avoid forgetting to update
this list.
Signed-off-by: Mayeul Cantan <mayeul.cantan@gmail.com>
* pmbootstrap: add qcdt generation to the linux aportgen APKBUILDs
This checks the next box in #688
When the device has bootimg_qcdt set to true, the following is done to
the linux APKBUILD:
* Add dtbtool to makedepends
* Call dtbTool during build() to generate dt.img
* Add the generated dt.img in the package's boot/dt.img
Signed-off-by: Mayeul Cantan <mayeul.cantan@gmail.com>
Fixes#893. Changes:
* New action: "pmbootstrap pkgrel_bump"
* pmbootstrap detects missing soname depends when trying to install
anyting, and suggests "pkgrel_bump --auto" to fix it
* Testcase test_soname_bump.py checks the pmOS binary package repo
for soname breakage, so we see it when CI runs for new PRs
* libsamsung-ipc: bump pkgrel because of soname bump
Daniele Debernardi
Oliver Smith
Luca Weiss
Attila Szöllősi
Alexander
Lucas Ramage
Pavel Machek
Mayeul Cantan
rrooij