Commit Graph

725 Commits

Author SHA1 Message Date
Oliver Smith 9d17ed29e3
pmb aportgen binutils: add texinfo to makedepends (!1901)
The current version of binutils does not build anymore without texinfo.

Related: #1894
Related: pmaports!1114
2020-03-31 23:33:41 +02:00
Daniele Debernardi 11c36fcf65
pmb.helpers.pmaports: cache get_list results (!1896) 2020-03-29 20:19:58 +02:00
Daniele Debernardi 20de238ff6
pmb.helpers.pmaports: search apkbuild path only if package exists (!1896) 2020-03-29 20:19:58 +02:00
Daniele Debernardi 0c179a5cf9
pmb.helpers.pmaports: cache _glob_apkbuilds results (!1896) 2020-03-29 20:19:48 +02:00
Luca Weiss f598a7fd41
pmb.parse._apkbuild: decrease log verbosity (!1897)
Closes #1886
2020-03-29 15:15:55 +02:00
Alexey Min 93514ae149
Fix "local variable '_partition_vbmeta' referenced before assignment" (!1895)
For safety, we should define variable in all code paths.

Fixes #1893
2020-03-23 04:37:13 +03:00
Alexey Min e5e48537b1
pmb.install: offer to flash vbmeta if device declares support for it (!1885) 2020-03-21 20:58:08 +01:00
Alexey Min e6da56d9b0
add "flasher flash_vbmeta" command (!1885)
Flashes device vbmeta partition (can be overriden with
"flash_fastboot_partition_vbmeta" setting in deviceinfo)
with custom vbmeta.img which has verity flag disabled,
so device can boot postmarketOS with no problems.
2020-03-21 20:58:08 +01:00
Daniele Debernardi 45f5ace1c2
pmb.helpers.frontend: fix wrong package name in kconfig check (!1891)
The kconfig check searches the aport with the "linux-" prefix to the
package name passed as argument. This is not working with the full
package name like linux-device-name because it searches a
linux-linux-device-name and fails.
2020-03-21 19:41:08 +01:00
Oliver Smith cfd93bc6ea
pmbootstrap status: check if git repo is outdated (!1882)
Use the timestamp of .git/FETCH_HEAD in each git repository, to
determine if too much time has passed since the last fetch/pull.
Modify pmb.helpers.git.clone, so FETCH_HEAD is always created if it does
not exist (because "git clone" would not create it).

Related: #1829
2020-03-14 19:15:32 +03:00
Oliver Smith c8526f2fcb
pmbootstrap status: basic checks for git repos (!1882)
Extend "pmbootstrap status" with checks for all git repositories,
derived from relevant checks in pmb.helpers.git.pull (using shared
code):
* on official branch
* workdir is clean
* tracking proper remote
* up to date

Related: #1829
2020-03-14 19:15:32 +03:00
Oliver Smith 5ae6f86a1f
pmb.helpers.git.get_branches_official(): new func (!1882)
Add dummy function that only returns ["master"] for now, so we can use it
in the upcoming git checks for "pmbootstrap status". More sophisticated
logic to figure out the branches will be added soon, see project
direction 2020 issue.

Related: #1829
2020-03-14 19:15:32 +03:00
Oliver Smith 1358424703
pmbootstrap status: print checked out git repos (!1882)
Related: #1829
2020-03-14 19:15:31 +03:00
Oliver Smith a659ba5695
Prepare 1.17.0 release 2020-03-14 08:56:50 +01:00
Minecrell 72fdfeb107
pmb: Fail on device ports directly in device/* (!1879)
Device ports should be under the device/testing/* directory for now.
In the future, devices may be moved to device/{main,community} after
certain preconditions are met.
2020-03-14 08:44:53 +01:00
Minecrell fb8de5a553
pmb: Look for device/* files in device/*/... instead (!1879)
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.

Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).

Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
2020-03-14 08:44:16 +01:00
Minecrell 527f5a9512
Update required Python version to 3.6+ (!1879)
We need Python 3.5+ for recursive (**) globs.
While we are at it, we might as well require Python 3.6
so we can use f-strings in the future.
2020-03-14 08:39:13 +01:00
Daniele Debernardi 5f6b8eaf0e
pmb.qemu: set output="tui" to avoid logging the stdout (!1886) 2020-03-14 08:05:57 +01:00
Daniele Debernardi 9718320829
pmb.qemu: drop --flavor option (!1886)
Since we ask for the kernel flavor during init, only the chosen kernel is
installed, hence there's no need to specify a different one.
2020-03-14 08:05:57 +01:00
Daniele Debernardi a76897b4a1
pmb.qemu: set default image size of 4G (!1886) 2020-03-14 08:05:32 +01:00
Minecrell 7bbc507416
pmb.qemu: use -cpu host for KVM, make configurable with --cpu (!1886)
For KVM the code is run pretty much natively on the host CPU, so all
CPU extensions available on the host CPU can be also used inside the VM.
To expose that information to the VM we should pass "-cpu host", so the
VM is aware of which CPU is in use.

For CPU emulation, QEMU uses a rather minimal CPU on x86_64 by default.
It does not have support for SSE3/4 etc, which may be required for some
applications to work properly (e.g. Android in Anbox). Add a --cpu flag
to make the emulated CPU configurable. Useful values are for example
--cpu max to emulate all implemented CPU features.
2020-03-14 08:05:32 +01:00
Minecrell 76dcf8aa0b
pmb.qemu: add --no-kvm to disable KVM even when available (!1886)
To test QEMU's CPU emulation it is useful to have a switch to disable
KVM, even when it is available (and potentially working fine).

Add --no-kvm for that purpose.
2020-03-14 08:05:32 +01:00
Minecrell b4678f0882
pmb.qemu: make video resolution configurable + consistent (!1886)
For some reason, the SDL display backend changes the video resolution
to 1024x768, while the GTK display keeps it at 640x480.
This is annoying, because at the moment we can only set one display
resolution for a device in postmarketOS (e.g. for the splash screen).

At the moment, the resolution for the splash screen is set to 640x480,
which therefore shows up too small with the default SDL display.

It seems like the display resolution can be only changed in the guest
directly. Linux has a video= kernel parameter that can be used to
implement this. (See: https://www.kernel.org/doc/html/latest/fb/modedb.html)

Let's set 1024x768 by default, but make it configurable through --video.
2020-03-14 08:05:32 +01:00
Minecrell f602df0140
pmb.qemu: add --tablet option for QEMU tablet input device (!1886)
The QEMU 'tablet' input device reports absolute positions instead
of relative mouse pointer movements. This can be used to automatically
grab/release the mouse pointer when entering/leaving the QEMU window,
instead of having to release it with CTRL + ALT + G manually.

This is quite convenient and should be the default option normally,
but at least on my PC the mouse pointer is reported with some vertical
offset for some reason (you can't reach the top and it extends below
the QEMU window...). Let's add it as an optional --tablet option for now.
2020-03-14 08:05:32 +01:00
Minecrell 240e10816d
pmb.qemu: use consistent hardware for all architectures (!1886)
To ensure consistent behavior for QEMU on all architectures,
it is helpful to try to use the same hardware elements where possible.

A few examples of current inconsistent behavior:
  - x86_64 uses a SCSI disk, while aarch64 uses virtio-blk
  - x86_64 uses e1000 network, while aarch64 uses virtio-net-device
  - x86_64 uses PS/2 mouse, while aarch64 uses usb-mouse
  - only x86_64 prints serial output to console

At least the virtio components are usually independent of the selected
architecture, so we can use them for both architectures.

This commit makes most of the hardware configuration shared:
  - Redirect serial output to stdio
  - virtio-blk for the disk image
  - virtio-gpu-pci (this was already implicit for both architectures)
  - virtio-net-pci for the network interface
  - virtio-mouse-pci/virtio-keyboard-pci as input devices
  - intel-hda for audio

We also set -nodefaults to avoid setting up unneeded devices
especially for x86_64.
2020-03-14 08:05:32 +01:00
Minecrell 81b3aade07
pmb.qemu: drop telnet port forwarding (!1886)
Now that we try to use the IP assigned by QEMU via DHCP,
the debug-shell is no longer working via telnet.
This is because the VM does not have any IP assigned when it is running.

We would need to start a DHCP client from the initfs to make it work.
busybox provides both udhcpc (client) and udhcpd (server) so this is
not a big problem. But the question is: Is it worth it?

The debug-shell will be only used for debugging, and NetworkManager
handles starting a proper DHCP client once the rootfs is mounted.
Meanwhile the debug-shell can be also accessed via serial output/input,
(available in the pmbootstrap stdout/stdin). So overall it does not
seem worth the effort. Let's just recommend using serial instead.
2020-03-14 08:05:32 +01:00
Minecrell 79a8d04835
pmb.qemu: do not try to change default IP range (!1886)
The current network setup has weird side effects.
Normally, QEMU would automatically make the guest set up necessary
IP routes through its integrated DHCP server.
When running QEMU through pmbootstrap they are missing.

First, we change the DHCP range in a way that could potentially
conflict with default IPs used for QEMU's own services:
QEMU has the default gateway at <network>.2, and DNS at <network>.3.
We set the DHCP range to start at <network>.1, and will therefore
potentially give out one of these addresses (QEMU's default starts at
<network>.15).
See: https://wiki.qemu.org/Documentation/Networking#User_Networking_.28SLIRP.29

In practice this does not cause immediate problems because there is
just one guest in the network, and it will get <network>.1, which is
not used by QEMU.

More problematic is that we start a DHCP server from postmarketOS
at the same time (normally used for the USB network) and there are
actually two DHCP servers running at the same time.

QEMU's user networking is local to the process, therefore it is not
possible to access the QEMU guest through its IP from the host.
That's why we have the port forwardings so you can access SSH at
localhost:2222 for example.

In practice the network interface in the QEMU guest is only used to
access the Internet. For that, we don't care which IP address we get,
we just want to get a working setup (IP + routes + DNS) automatically
through DHCP.

To make this work nicely we just need to stop trying to fit QEMU's
network setup into our usual setup for USB networking. When we remove
the custom DHCP option, and avoid starting a DHCP server from postmarketOS
(deviceinfo_disable_dhcpd) everything is suddenly working fine. :)
2020-03-14 08:05:32 +01:00
Minecrell c7455f7a21
pmb.qemu: drop vexpress (armv7) support (!1886)
device-qemu-vexpress appears to be completely broken at the moment.
I was not able to make it show anything, get a log or even just get
any indication that it is actually doing something.

In general, it does also not fit well to the other QEMU ports.
The vexpress machine type in QEMU seems to be quite limited,
e.g. it has no PCIe bus and therefore it is impossible to configure it
with virtio-gpu. (Unlike already configured for x86_64 and aarch64).

I had some success with a setup similar to aarch64 with -M virt,highmem=off
but was unable to make it work in the end:

  - Alpine's virt kernel was missing some options preventing boot completely
  - Alpine's lts kernel was missing PCI support for virtio-gpu
  - Even after adding the options the VM usually freezed after boot

Overall it does not quite seem worth the effort at the moment,
compared to how well x86_64 and aarch64 are working.

In any case, vexpress is too different (and broken) to continue maintaining it.
2020-03-14 08:05:32 +01:00
Minecrell f536fd9cb9
pmb.qemu: use current device instead of requiring --arch (!1886)
When using pmbootstrap, you usually select the device you want to work
on using 'pmbootstrap init', generate the rootfs and can then run more
commands in the context of the device.

The same needs to be done before using QEMU (to generate the rootfs).
But for some reason 'pmbootstrap qemu' requires setting the --arch
parameter when running QEMU for a foreign architecture, even when the
device is still selected in pmbootstrap.

Even more confusing is that setting "--arch arm" always selects
device-qemu-vexpress, but this is not immediately clear from the name.

Let's make this a lot more intuitive by making sure there is a QEMU
device selected when running 'pmbootstrap qemu'. We can then use the
device information to infer the architecture automatically.
2020-03-14 08:05:32 +01:00
Minecrell 32dca18eb6
pmb.qemu: simplify --display by introducing --no-gl (!1886)
At the moment, the --display argument is a bit complicated to use.
A common use would be to switch between the UIs (sdl, gtk, none)
or to enable the software rasterizer. Split the two use cases
to separate arguments to make it more intuitive.
2020-03-14 08:05:32 +01:00
Minecrell d1f5753c89
pmb.qemu: always use virtio-gpu (!1886)
So far we tried to configure virtio-gpu using "-vga virtio" only
when the target architecture matches the host architecture.
But that's actually not what it depends on.

virtio-gpu and virgl can be also used when emulating a foreign
architecture. In fact, we already force usage of virtio-gpu for
aarch64 through "-device virtio-gpu-pci".

However, the "-vga virtio" parameter does not exist on aarch64,
no matter if we run QEMU natively on aarch64 or emulate it on x86_64.

(Apparently, -vga is mainly about legacy VGA framebuffer stuff that
we don't necessarily need. This is quite visible since the display
stays uninitialized on aarch64 until the kernel driver loads,
whereas on x86_64 it is initialized by the BIOS...)

In other words, "-vga virtio" belongs to the parameters specific to x86_64.

Now that we have removed the setup question for the Mesa driver to use
(since it was ineffective), it would still be nice to have some way to
choose if you want to use virtio-gpu/virgl or not.

But actually virtio-gpu can be also used with swrast, without virgl.
This happens automatically when QEMU is started without GL support.
We already have a --display parameter for this, so it is possible to
force swrast by using "--display sdl" (instead of the default sdl,gl=on).

Overall this allows simplifying the QEMU package setup because there is
only a single GPU driver in use (virtio-gpu) instead of the 3 we had before
(virtio, qxl, bochs).
2020-03-14 08:05:32 +01:00
Minecrell 320b2faa4c
pmb.qemu: remove QEMU mesa driver setup question (!1886)
mesa-dri-swrast and mesa-dri-virtio are both provided by mesa-dri-gallium
now, so this option does not have much use anymore. With both selections,
exactly the same packages are installed.
2020-03-14 08:05:32 +01:00
Minecrell cb1f68817f
pmb.qemu: drop spice support (!1886)
The SPICE UI option tends to be broken (see #1836), and even when it is
working, it is not working particularly well. QXL requires special handling
in our QEMU packages, when now virtio-gpu (virgl) is working quite well overall.

Apparently it is possible to use virgl with SPICE; but only when using
a Unix socket instead of a TCP port. That again is a bit complicated
because we run QEMU outside the chroot and the SPICE client within.

Overall it does no longer seem to be worth the effort.
The default QEMU UI is working just fine (for the purposes of testing
postmarketOS at least).
2020-03-14 08:05:32 +01:00
Minecrell 0997a46b3a
pmb: replace deviceinfo_date with deviceinfo_year, add to wizard (!1888)
See: https://gitlab.com/postmarketOS/pmaports/issues/447
2020-03-14 00:33:22 +01:00
Minecrell 2159a083ef
pmb.parse.deviceinfo: deprecate dev_keyboard (!1888)
This property was never actually used. Let's remove it.
2020-03-14 00:33:22 +01:00
Minecrell 7ae04c5698
pmb.aportgen: remove touchscreen properties by default (!1888)
In most cases it is no longer necessary to set them.
directfb will now try to detect the touchscreen automatically.
2020-03-14 00:33:22 +01:00
Minecrell 7915e94275
pmb.config: remove deviceinfo_nonfree (!1888)
See: https://gitlab.com/postmarketOS/pmaports/-/issues/438
2020-03-14 00:33:22 +01:00
Oliver Smith f1cfd029d6
pmb/helpers/pmaports: fix finding subpackages (!1890)
Fix finding subpackages in pmaports, if no binary repo exists (-mp "" makes
pmbootstrap not use a binary packages mirror for pmOS packages). Discovered
with failing "test_depends_SLOW_40s" in bpo.

$ pmbootstrap -v -mp "" repo_missing --built --arch x86_64
[18:40:51] Calculate packages that need to be built (all packages, x86_64)
[18:40:56] ERROR: Package 'musl-dev-aarch64': Could not find aport, and could not find this package in any APKINDEX!
2020-03-09 18:49:59 +01:00
Luca Weiss 8edb136729
pmb.build._package: fix erroneous if condition (!1889)
This caused rust to be always installed into the native chroot even when
it wasn't required.
2020-03-08 19:22:12 +01:00
Oliver Smith a653ba9273
Prepare 1.16.0 release 2020-03-01 11:19:51 +01:00
KuroLevin b05c0e78c7
Increase the size of boot partition. (!1884)
Give enough space to do atomic replacement of boot files, as is now done
with:
ed3eeeeebb
2020-03-01 11:06:42 +01:00
Minecrell f565d6e153
pmb.aportgen.linux: cleanup makedepends (!1880)
Newer kernels require "bison" and "flex", so to easy porting we should
add them to generated APKBUILDs by default. While we are it, clean up
the default set of dependencies a bit:

  - installkernel is only needed when using "make zinstall", which
    we are not using in downstream_package (it requires special kernel
    configuration, so it's not really suitable for downstream...)

  - most kernels seem to compile fine without gmp-dev and elfutils-dev,
    so they should only be added when needed

  - the kernel has its own headers, why do we need linux-headers?
    (probably copied from Alpine, but anyway...)

  - add openssl-dev since it is needed for building/signing kernel modules

Useful reference: https://www.kernel.org/doc/html/latest/process/changes.html
Although it mentions much more than what is really needed...
2020-03-01 10:27:15 +01:00
Minecrell b0a664dc0e
pmb.aportgen.linux: build qcdt in package() (!1880)
Building the QCDT image in build() complicates use of envkernel.sh
- in that case, make is done independently of the APKBUILD, and only
the package() function is run to assemble the kernel package.

Building it entirely in package() allows using envkernel.sh without
having to set up some "post make script" (see
https://wiki.postmarketos.org/wiki/Compiling_kernels_with_envkernel.sh#Post_make_scripts).

package() might actually fit better in general, since QCDT is only
a special "package" format for (multiple) DTB files at the end :)
2020-03-01 10:27:15 +01:00
Minecrell 21b4e04c77
pmb.aportgen.linux: remove HOSTCC for latest GCC (!1880)
Setting HOSTCC is not necessary when using latest GCC
(since it defaults to "gcc"). Setting it only causes annoying
linter warnings. Let's remove it!
2020-03-01 10:27:15 +01:00
Luca Weiss ec6d40dfcc
pmb.aportgen.linux: use downstreakernel_package helper (!1880) 2020-03-01 10:27:00 +01:00
Alexey Min 383a58594d
pmb.config: provide tab completion for almost every question of "init" (!1875)
Add tab completion options for: vendor, codename, kernel, ui package,
keymap, qemu mesa driver
2020-03-01 00:44:19 +03:00
Alexey Min 7e61e62044
pmb.helpers.cli: add tab completion option for ask() helper (!1875)
Add a helper class that provides readline-based tab completion and an
extra optional argument "completion_choices" with possible completion
targets.

While at this, also improve paramters documentation for ask() func.
2020-02-26 01:20:18 +03:00
Oliver Smith 804743e65a
pmbootstrap status: print config overview (!1878)
Add an overview of the most important config options, which the user has
chosen in "pmbootstrap init".

Related: #1829
2020-02-24 18:18:54 +01:00
Oliver Smith 1724ed4665
pmbootstrap status: show if chroots are outdated (!1878)
Add new "pmbootstrap status" command, which does a quick health check
for the work dir. As first health check, verify that the chroots are not
too old. Replace the reminder text at the end of "pmbootstrap init" to
tell users to run "pmbootstrap status" instead of "pmbootstrap zap" once
a day before working with pmbootstrap.

Related: #1829
2020-02-24 18:18:53 +01:00
Oliver Smith 17673c5bf1
pmb.config.workdir: save/check chroot init date (!1878)
Whenever initializing new chroots, save the date in $WORK/workdir.cfg.
Add pmb.config.workdir.chroots_outdated() to check if it's time to zap
the chroots or not (since we don't update them automatically). Mark them
as outdated after two days.

This will be the first check in "pmbootstrap status" (future patches).
Related: #1829
2020-02-24 18:18:38 +01:00
Oliver Smith f21c216a26
Cosmetic: use SPDX license header (!1877)
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.

This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
2020-02-24 03:11:10 +03:00
Oliver Smith 4a08b5e066
Prepare 1.15.0 release 2020-02-23 03:40:30 +01:00
Luca Weiss b7e47367e5
aportgen: suggest 'pmbootstrap pull' in error message (!1874) 2020-02-21 16:26:08 +01:00
Minecrell 0e27713512
pmb.parse._apkbuild: Extend APKBUILD parser to work for subpackages (!1866)
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.

Simply put this works by:
  - Finding the lines that belong to the subpackage function
  - Stripping indentation (tab)
  - Parsing relevant attributes similar to the apkbuild() function

The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
2020-02-19 14:56:57 +03:00
Minecrell 0dad22a112
pmb.parse._apkbuild: Set "array": False by default (!1866)
This avoids having to set all unused options to False.
2020-02-19 14:56:52 +03:00
Minecrell 5f5050f9a3
pmb.parse._apkbuild: Replace variables immediately after parsing (!1866)
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.

Replacing variables immediately after parsing (before splitting)
has several advantages:

  - No need to handle different value types
    (e.g. lists by joining them every time they are accessed)

  - Variables like depends="$depends ..." are handled directly
    by the variable parser

  - APKBUILDs are shell scripts, so we match abuild more closely
    if variables defined later do not affect previous attributes
2020-02-19 14:56:46 +03:00
Minecrell 3aabaf451c
pmb.parse._apkbuild: Set default attribute values before starting to parse (!866)
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.

We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.

This will also avoid a (somewhat unrelated) error in the bootimg test:

  File "pmb/parse/_apkbuild.py", line 46, in replace_variable
    apkbuild["pkgname"], match.group(0),
  KeyError: 'pkgname'
2020-02-19 14:56:35 +03:00
Luca Weiss 33e3553cdd
aportupgrade command for upgrading APKBUILDs (!1752)
The gist of this action is upgrading the specified aport to the latest
version. There are implementations for both stable packages (which check
via the release-monitoring.org API for new versions) and git packages
(which check the GitLab/GitHub API for new commits on the main branch).

There's also the possibility to pass --all, --all-stable & --all-git to
the action which either loops through all packages, or just stable or
git packages and upgrades them.

The --dry argument is also respected.

Note, that the implementation does update the variables pkgver, pkgrel
and _commit but it doesn't update the checksums because that would slow
down the process a lot, and is potentially undesirable.
2020-02-15 20:24:09 +01:00
Luca Weiss dd72fd44db
pmb.helpers.http: add retrieve & retrieve_json helpers (!1752)
Fetch the content of a URL and returns it as string.
2020-02-15 20:24:09 +01:00
Luca Weiss ddb8ff435c
pmb.helpers.file: add replace_apkbuild helper (!1752)
Replace one key=value line in an APKBUILD and verify it afterwards.
2020-02-15 20:24:04 +01:00
Minecrell 87dd071b32
add "fastboot-bootpart" flasher to flash split images with fastboot (!1871)
asus-me176c has a Fastboot interface that can be used for flashing,
but in postmarketOS we do not use Android boot images for it.
This is because is it not very practical - the boot partition is
quite small and there is a (custom) EFI bootloader that can boot
directly from any other FAT32 partition.

At the moment the installation process is manual:
  1. pmbootstrap install --split to have separated boot (FAT32)
     and rootfs images
  2. pmbootstrap export
  3. Flash boot and rootfs images manually using Fastboot

The "fastboot-bootpart" flasher implements that process in a more
convenient way. When a device uses the "fastboot-bootpart" flasher:

  - We generate --split images on "pmbootstrap install" by default.
    (This can be disabled using --no-split instead.)

  - pmbootstrap flasher flash_kernel flashes the raw boot partition
    (not an Android boot image) using Fastboot, just like the rootfs.

There are some limitations that could be improved in the future:

  - "fastboot-bootpart" is not offered in the device wizard.
    I think it is special enough that no-one will be starting with it,
    and the difference to normal "fastboot" might be confusing.

  - Support "pmbootstrap flasher boot". asus-me176c does not support
    "fastboot boot" properly, but theoretically we could still generate
    Android boot images to use when booting an image directly.

  - At the moment the boot partition image is not regenerated when
    using "pmbootstrap flasher flash_kernel" (unlike when using Android
    boot images). "pmbootstrap install" needs to be run manually first.
2020-02-10 00:05:00 +03:00
Minecrell ba1e39f48e
pmb.config: Fix formatting for flash methods (!1871)
For some reason, the formatting for the flash method definitions
is really messed up. Let's fix it!
2020-02-10 00:04:59 +03:00
Oliver Smith fbbcc1c26f
pmb.parse.arguments: common func for packages arg (!1872)
Create add_packages_add() function, so we don't need to write the three
lines for adding the "packages" argument to an action with proper
autocompletion.
2020-02-06 16:29:31 +03:00
Danct12 a021d1924c
pmb.helpers.lint: add simple linting (!1855)
Run linting, just like in pmaports.git CI checks. Start with
apkbuild-lint, eventually this should run all checks.

Related: #1869
2020-02-06 13:45:21 +01:00
Oliver Smith 2f01c91c1a
pmbootstrap -s: remove option (!1868)
Remove in favor of similar, but more visible "--no-install" option for
"pmbootstrap flasher" and "pmbootstrap export".

It seems that the option wasn't really used by anybody, so let's just
remove it without a deprication period. If dear reader thinks otherwise
after this is merged, and is not happy with "--no-install", please state
your reasoning in a new issue, so we can discuss it.

Related: !1863
2020-02-06 12:33:21 +01:00
Minecrell 630310ebfb
pmb.aportgen.linux: Fix linting issues in generated APKBUILDs (!1869)
Avoid generating an APKBUILD with the following linting issues:
  - unnecesary usage of braces: ${_flavor}
  - unnecesary usage of braces: ${arch}
  - unnecesary usage of braces: ${_commit}
  - unnecesary usage of braces: ${_repository}
2020-02-02 06:49:12 +03:00
Minecrell d3fede3262
pmb.qemu.run.install_depends: Fix Mesa Gallium DRI depends (!1867)
Commit 43520967 ("pmb.qemu.run.install_depends: adjust to mesa pkgs (!1865)")
replaced mesa-dri-swrast, mesa-dri-virtio and mesa-dri-vmwgfx with
mesa-va-gallium. But the Gallium DRI drivers are actually provided by
mesa-dri-gallium. mesa-va-gallium contains libva drivers for hardware
video acceleration.

Fortunately this did not break anything because mesa-dri-gallium
was still installed as dependency of mesa-dri-ati/intel/nouveau.

QEMU does not make use of libva, so lets just replace mesa-va-gallium
with mesa-dri-gallium. While we are at it, stop depending on the
deprecated mesa-dri-ati/intel/nouveau packages. Those are only kept
for compatibility reasons and install more dependencies than necessary.
2020-01-31 04:10:50 +01:00
Minecrell 60217d0818
pmb.flasher/export: Add --no-install option to skip kernel/initfs update (!1863)
At the moment, pmbootstrap updates the kernel and the initfs whenever
using the flasher or export. This is useful, but sometimes you just want
to boot exactly the same kernel several times. In that case, having to wait
several seconds for the (redundant) update to complete is quite annoying.

Add a --no-install option that allows skipping the kernel/initfs update.
2020-01-31 01:35:26 +01:00
Oliver Smith 8f4bd7dea1
Prepare 1.14.0 release 2020-01-27 00:35:05 +01:00
Luca Weiss 859159254e
pmb/parse/_apkbuild: wrap more calls in try-catch (!1864)
Otherwise pmb crashes on some APKBUILDs, for example:

source="https://files.pythonhosted.org/packages/source/${_pyname%${_pyname#?}}/$_pyname/$_pyname-$pkgver.tar.gz"
2020-01-26 23:06:39 +01:00
Minecrell c8737740b1
pmb.install: Add pmbootstrap install --(no-)sparse option (!1862)
At the moment, sparse images are generated if the device sets
deviceinfo_flash_sparse="true". But for testing purposes it can be
useful to specifically enable or disable the default behavior.

Add a --sparse and --no-sparse option that enables or disables
sparse image generation.
2020-01-26 22:56:42 +01:00
Oliver Smith 4352096783
pmb.qemu.run.install_depends: adjust to mesa pkgs (!1865)
Adjust the mesa related packages installed for running in qemu, after
the mesa APKBUILD in Alpine was changed. Install mesa-va-gallium instead
of:
* mesa-dri-freedreno
* mesa-dri-swrast
* mesa-dri-virtio
* mesa-dri-vmwgfx

Fix error in pmbootstrap-qemu CI test:
ERROR: Could not find dependency 'mesa-dri-freedreno' in any aports folder or APKINDEX.

Related: 298e20d04f
2020-01-26 21:05:42 +01:00
Minecrell 44e887f11e
pmb.install: remove qemu-user binary stub from device rootfs (!1861)
Now that the qemu-user binary is bind-mounted, we no longer copy
the binary to the device rootfs. However, there is still the empty
stub file that we used as a destination mount point.

Let's remove it before copying it to the device rootfs.
It is automatically re-created the next time the qemu-user binary
is needed.
2020-01-26 19:47:43 +01:00
Minecrell d0376ee0fb
pmb.chroot.init: mount --bind qemu-user binary (!1861)
Avoid copying the qemu-user binary to each chroot by using a bind mount.
This ensures that we remove it when we shutdown the chroot.
2020-01-26 19:47:42 +01:00
Minecrell 81d9bfe3ba
pmb.helpers.mount: bind_file: Add create_folders option (!1861)
Create the necessary directory structure if create_folders=True.
2020-01-26 19:47:42 +01:00
Minecrell f43d552039
pmb.helpers.mount: Rename bind_blockdevice() to bind_file() (!1861)
bind_blockdevice() can be also used to mount --bind files,
rename it to bind_file().
2020-01-26 19:47:28 +01:00
Minecrell 4b2750e4a9
pmb.config: do not force installation of ttf-droid (!1860)
Right now we explicitly add ttf-droid to /etc/apk/world, and force
installation of it on every postmarketOS installation. But there really
is no reason to force this from pmbootstrap:

  - If ttf-droid is really required for every postmarketOS installation,
    it should be a depends of postmarketos-base.

  - Even with this change it is still installed on every device,
    because postmarketos-splash depends on it.

  - There is no need for fonts when using the "none" UI.

For now, the only difference of this change is that we no longer add
ttf-droid explicitly to the APK world. Instead, it is installed as
dependency of postmarketos-splash.
2020-01-20 14:19:41 +03:00
Minecrell 66fdb74b5b
qemu: Make QEMU audio backend configurable (!1859)
At the moment we assume that everyone running QEMU is either using
ALSA, or has the ALSA PulseAudio plugin configured on the host system.
(Since we run QEMU outside of the chroot at the moment, configuration
files are read from the host system instead of the chroot...)

Other distributions have much wider support for PulseAudio,
so not everyone will actually have the ALSA PulseAudio plugin configured.
In that case, it is better to use QEMU's PulseAudio backend since that
does not require any configuration. It also lets us drop the alsa-lib
fork, since that was only needed for loading the ALSA PulseAudio plugin.

In general, it seems difficult to detect which audio backend the user
wants to run (if any). Selecting the wrong one results in ugly warnings
when running QEMU. So let's not assume any by default, and add a
--audio option instead which accepts one of QEMU's audio backends
(alsa, pa or sdl).
2020-01-20 00:53:52 +03:00
Oliver Smith e04712a636
pmbootstrap pull: new action (!1848)
Add a shortcut for "git pull --ff-only" in all repositories cloned by
pmbootstrap (currently pmaports and aports_upstream, new pmdevices
repository coming soon).

'pmbootstrap pull' will only update the repositories, if:
* they are on an officially supported branch (e.g. master)
* the history is not conflicting (fast-forward is possible)
* the git workdirs are clean

Otherwise it shows the user a descriptive message about what to do. The
list of supported branches is only "master" right now, and will be
extended in later commits, so we can have a stable branch for pmaports
based on Alpine's releases. More about that in the project direction
2020 issue.

Closes: #1858
2020-01-19 20:37:46 +03:00
Oliver Smith 107c9f8b06
pmb.helpers.git.get_path: new function (!1848)
Get the path to the repository, which is either the default one in the
work dir, or a user-specified one in args.
2020-01-19 20:37:46 +03:00
Oliver Smith bbe3f5a219
pmb.helpers.git.rev_parse: add extra_args argument (!1848) 2020-01-19 20:37:46 +03:00
Oliver Smith 9ca06bc6c2
pmb.helpers.git.rev_parse: add path argument (!1848) 2020-01-19 20:37:46 +03:00
Oliver Smith 2b1dfe7cfc
pmb.helpers.git.rev_parse: don't return "" on err (!1848)
Remove a legacy code path from back in the day, where pmaports were in
the same repository as pmbootstrap ([1]). Nowadays, pmaports are always in a
git repository, so this is not necessary anymore. If git fails at this
point, crash hard and don't carry on with an empty string.

This is in preparation for using rev_parse in new code paths, so we
don't need to check the return value of rev_parse.

[1] 39548835 "Write custom os-release (closes #324) (#439)"
    https://github.com/postmarketOS/pmbootstrap/pull/439
2020-01-19 20:37:43 +03:00
Minecrell 7907e91879
pmb/parse/_apkbuild: Fix locating kernel subpackages with "-" in name (!1857)
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:

ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'

This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.

We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().

For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
2020-01-19 19:36:11 +03:00
Luca Weiss 2320e0100c
pmb/parse/_apkbuild: implement parsing for more variable styles (!1854)
Add a test for APKBUILD variable parsing as well
2020-01-19 11:30:59 +01:00
Oliver Smith 3f5cc734d9
pmb.config: add deviceinfo_boot_part_start
Fixes: 9f1dd64fa1 ("install/partition: allow specifying a different boot partition start (!1856)")
2020-01-18 23:03:32 +01:00
Clayton Craft 9f1dd64fa1
install/partition: allow specifying a different boot partition start (!1856)
This adds a new deviceinfo parameter, 'boot_part_start' which accepts an
int and indicates the number of sectors from the start of the drive to
place the boot partition.

The librem5 devkit (and actual phone) u-boot has grown beyond the 2048
sector space previously before the boot partition, so this is necessary in
order to boot pmos on this device.
2020-01-18 19:26:31 +00:00
Oliver Smith 74f76f03f6
crossdirect: support rust (!1850)
Install rust (rustc) in native chroot, if it is in the APKBUILD's
build dependencies. Add a test to verify that crossdirect + rust works
as expected.

Closes: #1861
2020-01-14 02:50:53 +01:00
Oliver Smith 728f304dc7
pmb.config.chroot_home_symlinks: cache rust dirs (!1850)
Rust packaging is new and still a bit weird in Alpine and postmarketOS.
As of writing, we only have one package (squeekboard), and use cargo to
download the source of all dependencies at build time (several git
repositories!) and compile it. Usually, this is a no-go, but at least
until this is resolved properly, let's cache the downloads as suggested
in: https://doc.rust-lang.org/cargo/guide/cargo-home.html

Related: #1861
2020-01-14 02:50:53 +01:00
Oliver Smith b93e83d8b3
pmb.chroot.init: home links: create missing dirs (!1850)
When initializing the building chroots, some symlinks are created to
/mnt/pmbootstrap* dirs. With upcoming rust-related symlinks, they will
point to subdirs in /mnt/pmbootstrap-rust, such as
/mnt/pmbootstrap-rust/registry/cache. Create this directory structure if
it does not exist.
2020-01-14 02:50:53 +01:00
Oliver Smith 02e514f4d3
pmb.helpers.git.clone: use git from host system (!1845)
Do not install git in the native chroot and use it from there. Remove the
chown_to_user argument from pmb.helpers.git.clone(), the resulting dir
is now always owned by the user. While at it, refactor the function and
display the clone URL.

Previously we had cloned aports_upstream (from Alpine) with
chown_to_user=False (legacy) and pmaports with chown_to_user=True.
pmb.helpers.git.rev_parse() would only work after chown_to_user=True.

Check if git is installed in "pmbootstrap init", and remove the same
check from rev_parse(). Add a new work dir version, that checks for git
and changes ownership of already checked out aports_upstream to the
host system's user.

When creating a new work dir, create cache_git instead of cache_http.
cache_http is created on demand already, with proper permissions. But
cache_git must be created, otherwise pmb.helpers.mount.bind will create it
as root.

This is in preparation for the "pmbootstrap pull" feature, as it allows
using the host system's git in all new code paths. We will be able to
handle repositories even if they were cloned outside of the work dir
(which we do in a few CI scripts for example).

Related: #1858
2020-01-12 00:39:47 +01:00
Oliver Smith cba37d5d79
chroot: keep ownership of cache_git (!1845)
Remove /home/pmos/git from pmb.config.chroot_home_symlinks, so
pmb.chroot.init.init() will not change the ownership of the
$WORK/cache_git dir.

The symlink was added two years ago in [1] as compatibility to previous
behavior, because the mountpoint of cache_git was changed from
/home/user/git to /mnt/pmbootstrap-git. Forcing the ownership of that
dir to be the pmos user inside the chroot made sense, because the only
cloned repository (aports_upstream) would also be owned by the pmos
user. With the upcoming change to have all cloned repos owned by the
host-system's user, this is just confusing.

The /home/pmos/git symlink does not seem to be used by anybody, it is
not mentioned in the pmbootstrap code and in the wiki.

[1] b4dd7a89 "Close #709: Improve user creation (#725)"
    https://github.com/postmarketOS/pmbootstrap/pull/725
2020-01-12 00:39:47 +01:00
Oliver Smith c53988b978
git aports_upstream: use gitlab.alpinelinux.org (!1845)
Use Alpine's primary mirror instead of the mirrored version at github
for new clones. A follow up commit will display the clone URL.
2020-01-12 00:39:47 +01:00
Oliver Smith a38026f98e
pmbootstrap init: migrate work dir if necessary (!1845)
This should have already happened, but probably broke when splitting
pmaports into a separate git repository.
2020-01-12 00:39:47 +01:00
Oliver Smith b1cb662645
pmbootstrap init: check for required programs (!1845)
Prepare to require "git" in a follow-up patch.
2020-01-12 00:39:41 +01:00
Daniele Debernardi 3f2a064993
pmb.aportgen: add Contributor/Maintainer to APKBUILD templates (!1852) 2020-01-07 01:46:38 +01:00
Oliver Smith 948e3f931f
Change copyright to 2020 2020-01-06 02:43:00 +01:00
Oliver Smith 680ee1bf97
Prepare 1.13.0 release 2020-01-03 05:28:24 +01:00
Oliver Smith ae1b3fc38c
pmb/helpers/pmaports: ignore provides without ver (!1851)
The APKBUILD reference for "provides" [1] is not worded clearly; but
after reading it over and over again, my understanding is:

* package with provides='foo=1.2' will be automatically installed if
  user requests installing "foo"
* package with provides='foo' (without version) will NOT get
  automatically installed if user requests installing "foo"

For pmbootstrap, this means, that we must not attempt to build a package
where the pkgname mentioned in provides matches what we are currently
resolving, unless there is an equals sign in the provides entry.

Fixes: #1862, pmaports#404

[1] https://wiki.alpinelinux.org/wiki/APKBUILD_Reference#provides says:

"List of package names (and optionally version info) this package
provides.

If package with a version is provided (provides='foo=1.2') apk will
consider it as an alternate name and it will automatically consider the
package for installation by the alternate name, and conflict with other
packages having the same name, or provides.

If version is not provided (provides='foo'), apk will consider it as
virtual package name. Several package with same non-versioned provides
can be installed simultaneously. However, none of them will be installed
by default when requested by the virtual name - instead, error message
is given and user is asked to choose which package providing the virtual
name should be installed."
2020-01-02 17:16:05 +01:00
Oliver Smith 5076069209
pmb/aportgen/gcc.py: add LANG_D=false (!1849)
Alpine's APKBUILD for gcc supports a new language that we don't need, so
disable it just like the irrelevant ones.

Related: https://gitlab.com/postmarketOS/pmaports/merge_requests/839
2019-12-30 12:04:12 +01:00