Commit Graph

1187 Commits

Author SHA1 Message Date
Pablo Correa Gómez bf820f442d
install: make cached remote repositories available on first install
This allows the usage of APK (and apk-polkit-rs) on first boot without
the need to establish an internet connection.

Closes https://gitlab.com/postmarketOS/pmbootstrap/-/issues/2206

Signed-off-by: Pablo Correa Gómez <ablocorrea@hotmail.com>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3CPAXP192MB1293A82207F0983ED6C8665EC7AE9@PAXP192MB1293.EURP192.PROD.OUTLOOK.COM%3E
2023-02-27 08:35:20 +01:00
Pablo Correa Gómez 014509b427
install: run alpine-appstream-downloader if available
This makes sure that the AppStream data is available on the built
image. To avoid unnecessary downloads on multiple invocations of
install, create a new cache chroot.

Signed-off-by: Pablo Correa Gómez <ablocorrea@hotmail.com>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3CPAXP192MB12938C6B570CA56AA162E8A6C7AE9@PAXP192MB1293.EURP192.PROD.OUTLOOK.COM%3E
2023-02-27 08:35:20 +01:00
Oliver Smith f208bba4f2
qemu: warn that network won't work with ui=none
The other day I spent way too long trying to find a regression that
caused network inside qemu not to work anymore, before I realized it was
caused by selecting UI=none instead of anything else.

Print a warning and link to a wiki page describing this in more detail.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230226184546.6869-1-ollieparanoid@postmarketos.org%3E
2023-02-27 08:35:14 +01:00
Newbyte 162867a08c
config: Add BINFMT_ELF and BINFMT_SCRIPT
pmOS cannot boot without these two. There was recently someone in the
unofficial postmarketOS Telegram group who used some minimal config and
couldn't figure out why the initramfs couldn't start up, and these
options being disabled turned out to be the culprit. As such, let's
ensure these are enabled to make life easier for people setting up new
kernel configurations.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230128203716.152281-1-newbyte@postmarketos.org%3E
2023-02-26 19:42:05 +01:00
Oliver Smith a0cea6255a
pmb.config: mount go caching directories
Always set GOCACHE during build and let it point to pmbootstrap's work
dir. This has a similar effect as using ccache for C/C++.

Set GOMODCACHE conditionally: this is for caching the go modules (git
repositories) that get cloned during a build if they are not bundled
with the source. Usually APKBUILDs should cache them, but when using
pmbootstrap build --src, they would get downloaded over and over again.
Set GOMODCACHE automatically for --src and allow enabling/disabling this
manually with new --go-mod-cache / --no-go-mod-cache.

This speeds up multiple iterations of building the same package
significantly. I'm using it for:

$ pmbootstrap build postmarketos-mkinitfs --src=...

I've verified that using the same GOCACHE dir for multiple architectures
doesn't result in build artifacts for the wrong arch getting used.

Reviewed-by: Clayton Craft <clayton@craftyguy.net>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230223064743.1773-1-ollieparanoid@postmarketos.org%3E
2023-02-26 19:37:30 +01:00
Oliver Smith 3ad159f98d
Prepare 1.51.0 release 2023-02-19 16:13:52 +01:00
Oliver Smith 98290f9191
pmbootstrap init: don't allow 'root' as username
While going over the wiki pages recently, I've noticed that somebody
added a note to not use "root" as username. I don't think many people
will attempt to do this, but let's catch it here in pmbootstrap with a
friendly message instead of mentioning it in the wiki.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230126071154.1689-1-ollieparanoid@postmarketos.org%3E
2023-02-19 16:13:52 +01:00
Oliver Smith e8a2ab2e98
check_binfmt_misc: print more helpful error
Instead of letting the mount -t binfmt_misc … command fail, and simply
telling the user that the command failed: rely on the following check
that prints a better error message on error. If it fails at this point,
the kernel option isn't enabled. Add two comments explaining why
check=False is used in this function while at it.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20230123064516.1607-1-ollieparanoid@postmarketos.org%3E
2023-02-19 16:13:52 +01:00
Oliver Smith 9975d373b0
Bump copyright to 2023 2023-01-22 19:18:06 +01:00
Oliver Smith faccbdc2eb
qemu: ssh hostfwd on 127.0.0.1 instead of 0.0.0.0
Make it accessible over localhost only. The idea is that users can login
into the VM on their own PC while developing, not that it's exposed over
the network. If somebody needs to have hostfwd on another IP, send a
patch to make this configurable via argparse.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221221220305.1809-1-ollieparanoid@postmarketos.org%3E
2023-01-20 15:28:07 +01:00
Clayton Craft 917ff92f63
pmb.qemu.run: fix 3d accel on amd64
Recent changes to qemu and Alpine packaging now require using the
virtio-vga-gl device and installing -gl packages to get virglrenderer
support.

Without this, wlroots fails to get an EGL context (among other problems
you'd expect by not having a useful GPU around...)

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Tested-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221210185021.3546-1-clayton@craftyguy.net%3E
2023-01-20 15:28:07 +01:00
Jane Rachinger 42feaf0d49
pmb.build.other: do not copy leftover abuild dirs
Running abuild on the host directly creates directories in the
aport where it gets built. Interrupting abuild results in those
working-dirs not getting deleted.
We don't want to copy those entries to our builder.

It's only really noticeable if pmbootstrap tries to copy a broken
symlink in src/ and thus fails.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221207205201.22139-1-jane400@bingo-ev.de%3E
2023-01-20 15:28:02 +01:00
Oliver Smith 9d93d34b65
Prepare 1.50.1 release 2022-12-05 08:19:27 +01:00
Weijia Wang a56838fb45
pmb/parse/arch.py: add arm64 -> aarch64 to mapping
This allows pmbootstrap to work on aarch64-darwin systems.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C167000939722.18650.8735853729695007724-0@git.sr.ht%3E
2022-12-05 08:19:27 +01:00
Oliver Smith bb5399d6d5
pmb.config.apk_tools_min_version: add alpine 3.17 2022-11-27 19:10:05 +01:00
Sicelo 4b339763db
pmb.parse.kconfig: update version for MEMCG_SWAP change
Without the `_rc1` tag, building kernel results in
"linux-nokia-n900/config-nokia-n900.armv7: CONFIG_MEMCG_SWAP should be set. See
<https://wiki.postmarketos.org/wiki/kconfig#CONFIG_MEMCG_SWAP> for details"
message.

See also https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2220#note_1146219177

Signed-off-by: Sicelo A. Mhlongo <absicsz@gmail.com>
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
2022-11-22 07:43:35 +01:00
Oliver Smith 757d2a0bff
Prepare 1.50.0 release 2022-11-20 16:43:03 +01:00
Oliver Smith 147b3ce4b9
pmb ci: fix typo in URL
Fix a typo in the URL https://postmarketos.org/pmb-ci that is being used
to detect if a script is compatible with 'pmbootstrap ci' or not.
2022-11-20 16:38:37 +01:00
Luca Weiss 0f6c6238f9
pmb.qemu.run: stop forcing bios for riscv64
We decided now not to force the bios firmware being used for riscv64 but
instead just use the one built into QEMU.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221118203424.106861-1-luca@z3ntu.xyz%3E
2022-11-20 15:35:11 +01:00
Oliver Smith 315621d5b8
pmb ci: add --fast argument
Make it easy to only run the fast tests.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-3-ollieparanoid@postmarketos.org%3E
2022-11-20 15:35:11 +01:00
Oliver Smith 3fd22104a8
pmb ci: error on using --all with script names
You can either say you want all scripts, or give a list of script names,
not both. Add it this way and not with an add_mutually_exclusive_group,
as I'll add a add_mutually_exclusive_group in the next patch to only
specify --all or --fast, but having --fast with script names is fine.

Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-2-ollieparanoid@postmarketos.org%3E
2022-11-20 15:35:11 +01:00
Oliver Smith 9a84ad20b1
pmb ci: fix arg desc for --all
Reviewed-by: Luca Weiss <luca@z3ntu.xyz>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111072354.3431-1-ollieparanoid@postmarketos.org%3E
2022-11-20 15:35:05 +01:00
vaino ed7b0273f5
flasher: heimdall: take depends from pmaports.cfg
Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Co-developed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221114115705.1107886-1-vaino@vke.fi%3E
2022-11-16 09:01:32 +01:00
Caleb Connolly ab0aa7f956
pmb: sideload: wait for apk database lock
When installing the APK wait for a while for the APK database lock,
services like apk-polkit-server sometimes hog it for a while but there's
no point bailing out immediately.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221111212251.1360612-1-kc@postmarketos.org%3E
2022-11-12 14:28:41 +01:00
Luca Weiss 4a6c5657d5
pmb.parse.kconfig: don't enforce non-core checks for testing devices
Currently when any device does not conform to the options they declare,
we fail the whole kconfig check.

Now that we start requiring more options, especially with
pmb:kconfigcheck-community it makes sense to relax these restrictions so
we're more free to edit kconfig options and don't have to adjust all
testing devices that may or may not be properly maintained.

As a side effect this patch makes it practically impossible to make
kconfig check actually fail for any testing device which might not be
optimal. If these use cases appear in the future we will want to adjust
pmbootstrap to allow for that.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221105074432.13804-2-luca@z3ntu.xyz%3E
2022-11-12 14:28:41 +01:00
Luca Weiss 381a1ca907
pmb.parse.kconfig: add wireguard, filesystems & more to community check
Add wireguard options, supported file systems and some extra options to
the community kconfig check.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221105074432.13804-1-luca@z3ntu.xyz%3E
2022-11-12 14:28:40 +01:00
Oliver Smith 6f45a5d5fb
Prepare 1.49.0 release 2022-11-09 09:19:33 +01:00
Oliver Smith 30055c14d2
flasher: fastboot: take depends from pmaports.cfg
Adjust to avbtool now being part of android-tools in alpine edge.
Instead of trying to install both (which fails on edge), take the
dependencies from a new pmaports.cfg variable
supported_fastboot_depends, which only contains android-tools in
pmaports.git master branch.

Related: https://postmarketos.org/pmaports.cfg
2022-11-09 09:19:33 +01:00
Oliver Smith b64641bb1c
pmb.flasher.init.install_depends: new function
Move logic to install depends into one shared function.
2022-11-09 09:19:32 +01:00
Oliver Smith 04f8f59208
pmb.config.required_programs: add tar
Add tar as it is required for 'pmbootstrap ci' to get the source into
the pmbootstrap chroot.

Reviewed-by: Caleb Connolly <kc@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221031111614.1377-2-ollieparanoid@postmarketos.org%3E
2022-11-09 09:19:32 +01:00
Oliver Smith f8d186e776
pmbootstrap ci: fix error with deleted files
Check if each file listed by the two git ls-files commands still exists,
and only add the existing ones to the tarball. Otherwise it would crash
when a file has been deleted without having the change commited yet.

Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221031111614.1377-1-ollieparanoid@postmarketos.org%3E
2022-11-09 09:19:26 +01:00
Oliver Smith e8b0b4ba78
aportgen: don't fail if binary ver < APKBUILD ver
The typical workflow for upgrading cross/gcc-* is:

$ cd pmaports/cross
$ pmbootstrap aportgen gcc-*

Currently this is failing because the APKBUILD has been updated in
Alpine for gcc, but gcc has not been built for all arches yet. This
shouldn't prevent us from generating the proper updated APKBUILDs in
pmaports so just print a note and don't fail here.

I'm committing this directly to master as this currently breaks
test/test_aportgen.py::test_aportgen.
2022-11-02 21:09:28 +01:00
Luca Weiss 4771fbac65
pmb.qemu.run: support riscv64
We're using the standard virt machine, adding virtio-gpu-pci for
graphics and are using the bios firmware that is installed from the
device package, instead of the one built-in to QEMU.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221029114536.100268-2-luca@z3ntu.xyz%3E
2022-11-02 21:09:27 +01:00
Luca Weiss 0624a1ae33
pmb.qemu.run: replace -nic option with -netdev and -device
On qemu-system-riscv64 the -nic option doesn't seem to work correctly.

  qemu-system-riscv64: warning: requested NIC (anonymous, model virtio-net-pci) was not created (not supported by this machine?)

Using -netdev and -device provides the same functionality and also works
on riscv64.

Reviewed-by: Oliver Smith <ollieparanoid@postmarketos.org>
Link: https://lists.sr.ht/~postmarketos/pmbootstrap-devel/%3C20221029114536.100268-1-luca@z3ntu.xyz%3E
2022-11-02 21:09:27 +01:00
Alexander Martinz 89c2ff0926
pmb.parse.kconfig: explicitly add dependency on SWAP
Commit d8f2f20186 removed
the requirement of selecting MEMCG_SWAP for Kernels >= 6.1.

However, it did not fully account for the behavior change,
as MEMCG_SWAP depends on both MEMCG and SWAP.

Signed-off-by: Alexander Martinz <amartinz@shiftphones.com>
2022-10-25 23:18:47 +02:00
Oliver Smith 0bcd58765c
pmbootstrap ci: new command
Add a new command that makes running CI scripts easy. The user goes to
the git repository of choice, which has CI scripts written in a certain
format, and then runs 'pmbootstrap ci' to get an interactive selection
of which of the available scripts to run (or "all"). Specifying one or
multiple scripts on the command-line is also possible, e.g.

$ pmbootstrap ci flake8
$ pmbootstrap ci shellcheck flake8 pytest
$ pmbootstrap ci --all

pmbootstrap then either runs the selected scripts in a chroot (and
installs dependencies as defined at the beginning of the CI scripts), or
natively (with checks inside the scripts for having dependencies
installed). Running natively is needed for .ci/pytest.sh in this
pmbootstrap.git repository, as pmbootstrap can't run inside pmbootstrap.
Running natively or in chroot is defined in an "# Options: " comment
inside the script file.

Documentation for this command and how script files look like:
https://postmarketos.org/pmb-ci
2022-10-25 20:31:19 +02:00
Oliver Smith 6ac39d17e7
pmb/helpers/frontend: order imports alphabetically 2022-10-25 20:31:19 +02:00
Oliver Smith 4428c7bcdc
pmbootstrap log: show testsuite log too
When running the testsuite, most logging gets written to a separate
log_testsuite.txt file. Check if it exists, and if so, instruct tail to
print its output as well. This allows immediatelly figuring out what the
testsuite is doing without manually attaching to log_testsuite.txt
(which I often did while running the testsuite).
2022-10-25 20:31:14 +02:00
Alexander Martinz 2ca4518799
pmb.parse.kconfig: drop remaining "_rc1" references (MR 2220)
As discussed at: https://gitlab.com/postmarketOS/pmbootstrap/-/merge_requests/2220#note_1146631834

Signed-off-by: Alexander Martinz <amartinz@shiftphones.com>
2022-10-24 17:36:34 +02:00
Alexander Martinz d8f2f20186
pmb.parse.kconfig: fix container check for kernels >= 6.1 (MR 2220)
Kernel 6.1 dropped CONFIG_MEMCG_SWAP[1].

[1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e55b9f96860f6c6026cff97966a740576285e07b

Signed-off-by: Alexander Martinz <amartinz@shiftphones.com>
2022-10-24 11:14:47 +02:00
Minecrell 3b5492d91e
pmb: build: envkernel: Fix bind mount detection (MR 2217)
At the moment the "envkernel.sh hasn't run, assuming the kernel was
cross compiled on host and using current dir as source" code path
triggers even when using envkernel.sh, which works somewhat but
requires sourcing envkernel.sh again after each invocation of
"pmbootstrap build --envkernel ...".

The reason is that os.path.ismount() does not work for bind mounts
(see https://bugs.python.org/issue29707). There is a workaround for
that already in pmbootstrap but it is not used here for some reason.
2022-10-20 11:41:27 +02:00
Luca Weiss dfada2a522
pmb: enable riscv64 architecture (MR 2215)
Enable pmbootstrap to work on riscv64 packages, and at the same time
also expand the alpine_to_hostspec list to match upstream (adding
riscv32 and loongarch*).
2022-10-19 09:11:54 +02:00
Luca Weiss 61ce6069ed
pmb.config: add missing pmb:kconfigcheck-* for lint (MR 2214)
While introducing multiple pmb:kconfigcheck-* APKBUILD options we missed
adding them to this array leading to potential apkbuild-lint failures.

This hasn't been noticed much because the check in apkbuild-lint is
relatively buggy, see https://gitlab.alpinelinux.org/Leo/atools/-/issues/48
2022-10-18 21:10:56 +02:00
Caleb Connolly 66706f896d
pmb.parse.build: mention host compiled kernels in envkenel help (MR 2175) 2022-10-18 20:08:41 +02:00
Caleb Connolly 6502d8aa28
build: envkernel: support packaging kernels that were built on the host (MR 2175)
Add support for packaging a kernel that was compiled outside of
envkernel. The envkernel.sh wrapper is great for someone new to kernel
development, but it makes it difficult to do things like "make
dt_binding_check". This lets you avoid the wrapper and build on your
host machine but still use pmbootstrap to package your kernel for easier
testing.
2022-10-18 20:08:36 +02:00
Oliver Smith d2c39ce5d8
Prepare 1.48.0 release 2022-10-16 18:22:31 +02:00
Oliver Smith f41659d4ae
pmb.build._package: update isl workaround (MR 2213)
The plan was to drop this workaround, once alpine releases a new gcc
version, and we regenerate our cross compilers against that version
_and_ drop options="!tracedeps", as latest pmbootstrap aportgen would
do. However !tracedeps wasn't dropped in the latest upgrade. This was
too easy to miss, I'll add a comment to the gcc-cross APKBUILDs so we
see it next time. Let's just update the workaround for this version.
2022-10-16 18:02:26 +02:00
Oliver Smith 35784a5fd7
pmb.aportgen.device: no depend on mesa-dri-gallium (MR 2210)
Device packages shouldn't depend on it anymore, in order to end up with
a smaller installation for ui=none this package only gets pulled in by
postmarketos-base if mesa is installed.

Related: pmaports MR 3478
2022-10-12 08:24:53 +02:00
Oliver Smith 248fe447bd
pmb.chroot.root: set PYTHONUNBUFFERED=1 (MR 2209)
Set the env var, so python programs running inside pmbootstrap chroot
don't buffer their output and only end up printing everything when they
are done. I've seen this with meson. This is bad for usability because
we don't see output, but also a problem because pmbootstrap kills
commands if they don't print any output for some time (default: 15 min).
2022-10-07 09:02:53 +02:00
Oliver Smith 40679f7126
pmb.chroot.root: order env vars alphabetically (MR 2209) 2022-10-07 09:02:53 +02:00