Commit Graph

388 Commits

Author SHA1 Message Date
Oliver Smith f9c74277ca
cpu_emulation_required: not for armv7 -> armhf (MR 1952)
Make it possible to build armhf packages on armv7 devices, e.g. to help
with stuck build-edge-armhf packages that won't build with qemu.
2020-07-14 22:42:55 +02:00
Mark Hargreaves 4312160c96
pmb/parse/bootimg.py: fix file names after mkbootimg-osm0sis update (MR 1959) 2020-07-10 22:53:00 +03:00
Oliver Smith ddb5d9ae2c
pmbootstrap install --ondev: new option (MR 1946)
Add initial support for the on-device installer in pmbootstrap. Let
pmbootstrap create a regular split image, then prepare a new installer
rootfs and copy the previously generated rootfs image into the installer
rootfs. Put the installer rootfs into a new image, with reserved space.

There is more to do from here, such as disabling the generation of the
user account when using --ondev. But this requires support in
postmarketos-ondev first, so let's build that iteratively.

Related: https://wiki.postmarketos.org/wiki/On-device_installer
Related: https://gitlab.com/postmarketOS/postmarketos-ondev/-/issues
2020-06-19 09:36:51 +02:00
Oliver Smith 63fc2b621a
pmbootstrap chroot --install-blockdev: new option (MR 1946)
Create /dev/install inside the chroot from a block device, just like
done during the installation. This is useful for testing the Calamares
installer.
2020-06-19 09:36:50 +02:00
Minecrell 3ebb994206
pmb.parse: Respect provider_priority if multiple providers exist (MR 1945)
Alpine APKBUILDs have the concept of "provider priorities" that affect
the choice of the provider to install when multiple packages provide
a virtual package.

One use case for this is to allow installation of different firmware
versions. bq-paella can run unsigned firmware, therefore you have the
choice between using the original firmware from the manufacturer, or
a slightly newer version from Qualcomm for the Dragonboard 410c.

We add provides="firmware-qcom-msm8916-wcnss" (the "virtual package")
to both firmware-qcom-db410c-wcnss and firmware-bq-picmt-wcnss.
At this point, attempting to install "firmware-qcom-msm8916-wcnss"
would still fail with apk. (Because it does not know which provider
to install.)

To pick a default we can set e.g. provider_priority=100 for
firmware-qcom-db410c-wcnss (the slightly newer version).
In that case, firmware-qcom-db410c-wcnss should be installed by default.

However, the user can choose to do "apk add firmware-bq-picmt-wcnss"
to override the default choice in case of problems. In that case,
the conflicting firmware-qcom-db410c-wcnss will be automatically removed.

At the moment, pmbootstrap does not respect the "provider_priority" at all.
In the above case, it would always install "firmware-bq-picmt-wcnss"
during "pmbootstrap install" since that has the shortest name.

Extend the pmbootstrap code to pick a provider with the highest priority
(if any of the providers has a priority set).
2020-06-16 01:31:28 +03:00
Oliver Smith 6d013b4472
pmbootstrap chroot --xauth: new option (MR 1944)
Make it easy to start any X11 application from the native chroot.
2020-06-09 04:10:51 +02:00
Anjandev Momi 37b4af19fc
make boot partition size user-overridable (MR 1931)
Ask for the boot partition size during "pmbootstrap init" in the
additional options with a default of 128 MB, and allow to override it
with -B.
2020-06-02 10:48:03 +02:00
Oliver Smith 0aa1dd7145
pmb.parse.deviceinfo.sanity_check: verify chassis (MR 1933) 2020-05-24 08:38:43 +02:00
Antoine Fontaine 28da033267
pmb.parse.kconfig: implement anbox kconfig check (MR 1916)
fixes https://gitlab.com/postmarketOS/pmbootstrap/-/issues/1891.
2020-05-24 04:42:01 +02:00
Antoine Fontaine e36e160167
pmb.parse.kconfig: check for end of line in "CONFIG_…=y" (MR 1916)
it's a nitpick, but now `is_set` won't return true
for (e.g.) CONFIG_XYZ=marmelade.
2020-05-22 23:05:28 +02:00
Oliver Smith ce433ae012
pmb.helpers.repo.urls: use channels.cfg mirrordirs (MR 1912)
Use mirrordir_pmos and mirrordir_alpine from channels.cfg to generate
the mirror URLs for postmarketOS and Alpine, which get written to
/etc/apk/repositories and which postmarketOS uses to download the
APKINDEX files.

Remove hardcoded "master" at the end of the postmarketOS mirror and use
mirrordir_pmos instead (which is "master" for the edge channel). Let the
postmarketOS mirror end in a '/' for consistency with the Alpine mirror
in pmb/config/__init__.py.

Remove obsolete --alpine-version. To experiment with a different Alpine
version, one should pass a custom --config-channels from now on.
2020-05-17 08:08:45 +02:00
Oliver Smith c616874443
pmb.helpers.git: parse channels.cfg (MR 1912)
Prepare to base postmarketOS on Alpine stable by parsing the new
channels.cfg file in pmaports.git, that describes which channel
needs which branches and mirror dirs from postmarketOS and Alpine.

Use the information in pmb.helpers.git.get_branches_official() first,
more is coming in follow-up commits.

Read the file from origin/master, so we get the latest fetched version
even if the last checked out master branch is not up-to-date (think of
currently checked out release branch instead of master, master will
never be updated to point to latest origin/master). Allow to override
the file with a new --config-channels parameter.

Related: https://postmarketos.org/channels.cfg
2020-05-17 08:08:44 +02:00
Oliver Smith ed16a9d080
pmb/parse/arguments.py: --config: add help text (MR 1912) 2020-05-17 08:08:39 +02:00
Martijn Braam df08af7984
arguments: --ccache-disable => --no-ccache (MR 1926)
Rename for consistency.
2020-05-10 16:18:10 +02:00
Daniele Debernardi 34c6f88691
pmb.install._install: add option to not generate the image (MR 1914) 2020-04-14 00:29:35 +03:00
Niklas Cathor 29a3cb2b8c
pmb.helpers.frontend.config: add option to reset config to default (!1907) 2020-04-07 01:02:19 +03:00
Oliver Smith 73401f8606
pmb.parse.arguments: fix pkgname autocompletion (!1905)
Use pmb.helpers.args.add_cache() with the args used during
autocompletion, so pmb.helpers.pmaports._find_apkbuilds() does not fail
when trying to access args.cache.
2020-04-05 13:04:53 +02:00
Luca Weiss f598a7fd41
pmb.parse._apkbuild: decrease log verbosity (!1897)
Closes #1886
2020-03-29 15:15:55 +02:00
Alexey Min e6da56d9b0
add "flasher flash_vbmeta" command (!1885)
Flashes device vbmeta partition (can be overriden with
"flash_fastboot_partition_vbmeta" setting in deviceinfo)
with custom vbmeta.img which has verity flag disabled,
so device can boot postmarketOS with no problems.
2020-03-21 20:58:08 +01:00
Minecrell fb8de5a553
pmb: Look for device/* files in device/*/... instead (!1879)
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.

Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).

Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
2020-03-14 08:44:16 +01:00
Daniele Debernardi 9718320829
pmb.qemu: drop --flavor option (!1886)
Since we ask for the kernel flavor during init, only the chosen kernel is
installed, hence there's no need to specify a different one.
2020-03-14 08:05:57 +01:00
Daniele Debernardi a76897b4a1
pmb.qemu: set default image size of 4G (!1886) 2020-03-14 08:05:32 +01:00
Minecrell 7bbc507416
pmb.qemu: use -cpu host for KVM, make configurable with --cpu (!1886)
For KVM the code is run pretty much natively on the host CPU, so all
CPU extensions available on the host CPU can be also used inside the VM.
To expose that information to the VM we should pass "-cpu host", so the
VM is aware of which CPU is in use.

For CPU emulation, QEMU uses a rather minimal CPU on x86_64 by default.
It does not have support for SSE3/4 etc, which may be required for some
applications to work properly (e.g. Android in Anbox). Add a --cpu flag
to make the emulated CPU configurable. Useful values are for example
--cpu max to emulate all implemented CPU features.
2020-03-14 08:05:32 +01:00
Minecrell 76dcf8aa0b
pmb.qemu: add --no-kvm to disable KVM even when available (!1886)
To test QEMU's CPU emulation it is useful to have a switch to disable
KVM, even when it is available (and potentially working fine).

Add --no-kvm for that purpose.
2020-03-14 08:05:32 +01:00
Minecrell b4678f0882
pmb.qemu: make video resolution configurable + consistent (!1886)
For some reason, the SDL display backend changes the video resolution
to 1024x768, while the GTK display keeps it at 640x480.
This is annoying, because at the moment we can only set one display
resolution for a device in postmarketOS (e.g. for the splash screen).

At the moment, the resolution for the splash screen is set to 640x480,
which therefore shows up too small with the default SDL display.

It seems like the display resolution can be only changed in the guest
directly. Linux has a video= kernel parameter that can be used to
implement this. (See: https://www.kernel.org/doc/html/latest/fb/modedb.html)

Let's set 1024x768 by default, but make it configurable through --video.
2020-03-14 08:05:32 +01:00
Minecrell f602df0140
pmb.qemu: add --tablet option for QEMU tablet input device (!1886)
The QEMU 'tablet' input device reports absolute positions instead
of relative mouse pointer movements. This can be used to automatically
grab/release the mouse pointer when entering/leaving the QEMU window,
instead of having to release it with CTRL + ALT + G manually.

This is quite convenient and should be the default option normally,
but at least on my PC the mouse pointer is reported with some vertical
offset for some reason (you can't reach the top and it extends below
the QEMU window...). Let's add it as an optional --tablet option for now.
2020-03-14 08:05:32 +01:00
Minecrell f536fd9cb9
pmb.qemu: use current device instead of requiring --arch (!1886)
When using pmbootstrap, you usually select the device you want to work
on using 'pmbootstrap init', generate the rootfs and can then run more
commands in the context of the device.

The same needs to be done before using QEMU (to generate the rootfs).
But for some reason 'pmbootstrap qemu' requires setting the --arch
parameter when running QEMU for a foreign architecture, even when the
device is still selected in pmbootstrap.

Even more confusing is that setting "--arch arm" always selects
device-qemu-vexpress, but this is not immediately clear from the name.

Let's make this a lot more intuitive by making sure there is a QEMU
device selected when running 'pmbootstrap qemu'. We can then use the
device information to infer the architecture automatically.
2020-03-14 08:05:32 +01:00
Minecrell 32dca18eb6
pmb.qemu: simplify --display by introducing --no-gl (!1886)
At the moment, the --display argument is a bit complicated to use.
A common use would be to switch between the UIs (sdl, gtk, none)
or to enable the software rasterizer. Split the two use cases
to separate arguments to make it more intuitive.
2020-03-14 08:05:32 +01:00
Minecrell 320b2faa4c
pmb.qemu: remove QEMU mesa driver setup question (!1886)
mesa-dri-swrast and mesa-dri-virtio are both provided by mesa-dri-gallium
now, so this option does not have much use anymore. With both selections,
exactly the same packages are installed.
2020-03-14 08:05:32 +01:00
Minecrell cb1f68817f
pmb.qemu: drop spice support (!1886)
The SPICE UI option tends to be broken (see #1836), and even when it is
working, it is not working particularly well. QXL requires special handling
in our QEMU packages, when now virtio-gpu (virgl) is working quite well overall.

Apparently it is possible to use virgl with SPICE; but only when using
a Unix socket instead of a TCP port. That again is a bit complicated
because we run QEMU outside the chroot and the SPICE client within.

Overall it does no longer seem to be worth the effort.
The default QEMU UI is working just fine (for the purposes of testing
postmarketOS at least).
2020-03-14 08:05:32 +01:00
Minecrell 0997a46b3a
pmb: replace deviceinfo_date with deviceinfo_year, add to wizard (!1888)
See: https://gitlab.com/postmarketOS/pmaports/issues/447
2020-03-14 00:33:22 +01:00
Minecrell 2159a083ef
pmb.parse.deviceinfo: deprecate dev_keyboard (!1888)
This property was never actually used. Let's remove it.
2020-03-14 00:33:22 +01:00
Minecrell 7915e94275
pmb.config: remove deviceinfo_nonfree (!1888)
See: https://gitlab.com/postmarketOS/pmaports/-/issues/438
2020-03-14 00:33:22 +01:00
Oliver Smith 1724ed4665
pmbootstrap status: show if chroots are outdated (!1878)
Add new "pmbootstrap status" command, which does a quick health check
for the work dir. As first health check, verify that the chroots are not
too old. Replace the reminder text at the end of "pmbootstrap init" to
tell users to run "pmbootstrap status" instead of "pmbootstrap zap" once
a day before working with pmbootstrap.

Related: #1829
2020-02-24 18:18:53 +01:00
Oliver Smith f21c216a26
Cosmetic: use SPDX license header (!1877)
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.

This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
2020-02-24 03:11:10 +03:00
Minecrell 0e27713512
pmb.parse._apkbuild: Extend APKBUILD parser to work for subpackages (!1866)
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.

Simply put this works by:
  - Finding the lines that belong to the subpackage function
  - Stripping indentation (tab)
  - Parsing relevant attributes similar to the apkbuild() function

The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
2020-02-19 14:56:57 +03:00
Minecrell 0dad22a112
pmb.parse._apkbuild: Set "array": False by default (!1866)
This avoids having to set all unused options to False.
2020-02-19 14:56:52 +03:00
Minecrell 5f5050f9a3
pmb.parse._apkbuild: Replace variables immediately after parsing (!1866)
At the moment we parse all attributes, split them, and eventually
join them back together for variable replacement.

Replacing variables immediately after parsing (before splitting)
has several advantages:

  - No need to handle different value types
    (e.g. lists by joining them every time they are accessed)

  - Variables like depends="$depends ..." are handled directly
    by the variable parser

  - APKBUILDs are shell scripts, so we match abuild more closely
    if variables defined later do not affect previous attributes
2020-02-19 14:56:46 +03:00
Minecrell 3aabaf451c
pmb.parse._apkbuild: Set default attribute values before starting to parse (!866)
Theoretically it is possible to reference variables before they are defined.
In the shell, these will simply be evaluated to an empty string.
In preparation of replacing variables immediately after parsing attributes,
these variables will be no longer replaced correctly.

We can simplify the code further, and avoid this problem by initializing
the dict with the default values, replacing them with the real values
from the APKBUILD.

This will also avoid a (somewhat unrelated) error in the bootimg test:

  File "pmb/parse/_apkbuild.py", line 46, in replace_variable
    apkbuild["pkgname"], match.group(0),
  KeyError: 'pkgname'
2020-02-19 14:56:35 +03:00
Luca Weiss 33e3553cdd
aportupgrade command for upgrading APKBUILDs (!1752)
The gist of this action is upgrading the specified aport to the latest
version. There are implementations for both stable packages (which check
via the release-monitoring.org API for new versions) and git packages
(which check the GitLab/GitHub API for new commits on the main branch).

There's also the possibility to pass --all, --all-stable & --all-git to
the action which either loops through all packages, or just stable or
git packages and upgrades them.

The --dry argument is also respected.

Note, that the implementation does update the variables pkgver, pkgrel
and _commit but it doesn't update the checksums because that would slow
down the process a lot, and is potentially undesirable.
2020-02-15 20:24:09 +01:00
Minecrell 87dd071b32
add "fastboot-bootpart" flasher to flash split images with fastboot (!1871)
asus-me176c has a Fastboot interface that can be used for flashing,
but in postmarketOS we do not use Android boot images for it.
This is because is it not very practical - the boot partition is
quite small and there is a (custom) EFI bootloader that can boot
directly from any other FAT32 partition.

At the moment the installation process is manual:
  1. pmbootstrap install --split to have separated boot (FAT32)
     and rootfs images
  2. pmbootstrap export
  3. Flash boot and rootfs images manually using Fastboot

The "fastboot-bootpart" flasher implements that process in a more
convenient way. When a device uses the "fastboot-bootpart" flasher:

  - We generate --split images on "pmbootstrap install" by default.
    (This can be disabled using --no-split instead.)

  - pmbootstrap flasher flash_kernel flashes the raw boot partition
    (not an Android boot image) using Fastboot, just like the rootfs.

There are some limitations that could be improved in the future:

  - "fastboot-bootpart" is not offered in the device wizard.
    I think it is special enough that no-one will be starting with it,
    and the difference to normal "fastboot" might be confusing.

  - Support "pmbootstrap flasher boot". asus-me176c does not support
    "fastboot boot" properly, but theoretically we could still generate
    Android boot images to use when booting an image directly.

  - At the moment the boot partition image is not regenerated when
    using "pmbootstrap flasher flash_kernel" (unlike when using Android
    boot images). "pmbootstrap install" needs to be run manually first.
2020-02-10 00:05:00 +03:00
Oliver Smith fbbcc1c26f
pmb.parse.arguments: common func for packages arg (!1872)
Create add_packages_add() function, so we don't need to write the three
lines for adding the "packages" argument to an action with proper
autocompletion.
2020-02-06 16:29:31 +03:00
Danct12 a021d1924c
pmb.helpers.lint: add simple linting (!1855)
Run linting, just like in pmaports.git CI checks. Start with
apkbuild-lint, eventually this should run all checks.

Related: #1869
2020-02-06 13:45:21 +01:00
Oliver Smith 2f01c91c1a
pmbootstrap -s: remove option (!1868)
Remove in favor of similar, but more visible "--no-install" option for
"pmbootstrap flasher" and "pmbootstrap export".

It seems that the option wasn't really used by anybody, so let's just
remove it without a deprication period. If dear reader thinks otherwise
after this is merged, and is not happy with "--no-install", please state
your reasoning in a new issue, so we can discuss it.

Related: !1863
2020-02-06 12:33:21 +01:00
Minecrell 60217d0818
pmb.flasher/export: Add --no-install option to skip kernel/initfs update (!1863)
At the moment, pmbootstrap updates the kernel and the initfs whenever
using the flasher or export. This is useful, but sometimes you just want
to boot exactly the same kernel several times. In that case, having to wait
several seconds for the (redundant) update to complete is quite annoying.

Add a --no-install option that allows skipping the kernel/initfs update.
2020-01-31 01:35:26 +01:00
Luca Weiss 859159254e
pmb/parse/_apkbuild: wrap more calls in try-catch (!1864)
Otherwise pmb crashes on some APKBUILDs, for example:

source="https://files.pythonhosted.org/packages/source/${_pyname%${_pyname#?}}/$_pyname/$_pyname-$pkgver.tar.gz"
2020-01-26 23:06:39 +01:00
Minecrell c8737740b1
pmb.install: Add pmbootstrap install --(no-)sparse option (!1862)
At the moment, sparse images are generated if the device sets
deviceinfo_flash_sparse="true". But for testing purposes it can be
useful to specifically enable or disable the default behavior.

Add a --sparse and --no-sparse option that enables or disables
sparse image generation.
2020-01-26 22:56:42 +01:00
Minecrell 66fdb74b5b
qemu: Make QEMU audio backend configurable (!1859)
At the moment we assume that everyone running QEMU is either using
ALSA, or has the ALSA PulseAudio plugin configured on the host system.
(Since we run QEMU outside of the chroot at the moment, configuration
files are read from the host system instead of the chroot...)

Other distributions have much wider support for PulseAudio,
so not everyone will actually have the ALSA PulseAudio plugin configured.
In that case, it is better to use QEMU's PulseAudio backend since that
does not require any configuration. It also lets us drop the alsa-lib
fork, since that was only needed for loading the ALSA PulseAudio plugin.

In general, it seems difficult to detect which audio backend the user
wants to run (if any). Selecting the wrong one results in ugly warnings
when running QEMU. So let's not assume any by default, and add a
--audio option instead which accepts one of QEMU's audio backends
(alsa, pa or sdl).
2020-01-20 00:53:52 +03:00
Oliver Smith e04712a636
pmbootstrap pull: new action (!1848)
Add a shortcut for "git pull --ff-only" in all repositories cloned by
pmbootstrap (currently pmaports and aports_upstream, new pmdevices
repository coming soon).

'pmbootstrap pull' will only update the repositories, if:
* they are on an officially supported branch (e.g. master)
* the history is not conflicting (fast-forward is possible)
* the git workdirs are clean

Otherwise it shows the user a descriptive message about what to do. The
list of supported branches is only "master" right now, and will be
extended in later commits, so we can have a stable branch for pmaports
based on Alpine's releases. More about that in the project direction
2020 issue.

Closes: #1858
2020-01-19 20:37:46 +03:00
Minecrell 7907e91879
pmb/parse/_apkbuild: Fix locating kernel subpackages with "-" in name (!1857)
When defining a new kernel subpackage with a "-" in it
(e.g. $pkgname-kernel-mainline-modem), then pmbootstrap is unable to
find the function that builds the subpackage:

ERROR: Could not find subpackage function, no line starts with 'kernel_mainline-modem() {'

This is because it assumes that a $pkgname-kernel-<name> subpackage
is built by a kernel_<name> function, but this does not have to be the case.

We should really respect the name of the subpackage function that
is specified when defining the subpackage, but unfortunately it is
stripped away in cut_off_function_names().

For now let's fix this by replacing "-" with "_", but ideally the
APKBUILD parser API should be refactored to expose the subpackage
function in the future.
2020-01-19 19:36:11 +03:00
Luca Weiss 2320e0100c
pmb/parse/_apkbuild: implement parsing for more variable styles (!1854)
Add a test for APKBUILD variable parsing as well
2020-01-19 11:30:59 +01:00
Oliver Smith 728f304dc7
pmb.config.chroot_home_symlinks: cache rust dirs (!1850)
Rust packaging is new and still a bit weird in Alpine and postmarketOS.
As of writing, we only have one package (squeekboard), and use cargo to
download the source of all dependencies at build time (several git
repositories!) and compile it. Usually, this is a no-go, but at least
until this is resolved properly, let's cache the downloads as suggested
in: https://doc.rust-lang.org/cargo/guide/cargo-home.html

Related: #1861
2020-01-14 02:50:53 +01:00
Oliver Smith 948e3f931f
Change copyright to 2020 2020-01-06 02:43:00 +01:00
Oliver Smith a790900e31
pmb.parse.bootimg: fix kernel detection (!1846)
Adjust to new text that "file" outputs for ARM linux kernels:
	"ARM OpenFirmware FORTH Dictionary..."

Make test_bootimg_kernel pass again, which was failing with:
	AssertionError: assert 'heimdall-isorec' in 'File is not an Android ...
2019-12-23 20:24:05 +01:00
Drew DeVault 87389fbad3
pmbootstrap install: add --no-base option (!1843) 2019-12-14 02:38:56 +01:00
Oliver Smith 5438085e62
pmb/parse/_apkbuild: fix parsing commented lines (!1837)
Properly ignore comments at the end of lines, instead of assuming that
all lines below belong to the attribute:
	subpackages="$pkgname-dev" # $pkgname-lang

Fixes build.postmarketos.org#61, where pmbootstrap would assume that a
random package provides "make", just because the word "make" is written
somewhere below subpackages=" in the APKBUILD and it is parsed
incorrectly.

While at it, also support the ' character for quotations and detect if
a quotation for a value was started, but there is no end quotation sign
in the rest of the file.

I've added tests, and manually checked that this did not introduce any
parsing bugs for all the APKBUILDs in pmaports.git, by running
'pmbootstrap apkbuild_parse' with the old and new code, and diffing the
result.
2019-11-30 12:44:06 +01:00
Alexey Min 64aa7443e5
pmb.deviceinfo: fix sanity check when "device-" is present in codename (!1834)
Fixes #1842
2019-11-22 00:01:38 +01:00
lambdadroid 67b080b158
Detect if DTB is placed in second area of Android boot image (!1828)
postmarketOS/pmaports!700 adds a new "deviceinfo_bootimg_dtb_second"
option that places the DTB in the "second" area of the Android boot
image.

Attempt to detect this automatically by checking the extracted
second binary for the FDT magic (0xd00dfeed).
2019-10-25 17:36:22 +02:00
Luca Weiss 8a5c8fcc3e
Fix detection of qcdt boot images (!1826)
mkbootimg from osm0sis changed the filename of the dtb section in [1]
which breaks the detection of qcdt in pmbootstrap.

Fix this by adjusting the filename pmbootstrap checks to match the
unpackbootimg behavior.

[1] https://github.com/osm0sis/mkbootimg/commit/d8222a4d
2019-10-18 21:35:44 +02:00
Oliver Smith 5ad84a4c08
Cosmetic: pmb.parse.sanity_check: nicer comments (!1818)
Create one block of legacy errors, instead of having own blocks, each
with their own comment stating the obvious.
2019-09-26 22:55:17 +02:00
Oliver Smith a791bf4ecb
deviceinfo: flash_fastboot_vendor_id is legacy (!1818) 2019-09-26 22:55:13 +02:00
Oliver Smith 69aa7b73f2
timeout: change default from 300 to 900 seconds
Linking big libraries may take more than five minutes, so change the
default timeout to fifteen minutes to save users from frustration.
This is the same value that I'm using to build binary packages for the
repository.

Pushing directly to master without making a merge request, because this
is a trivial change.
2019-09-18 21:56:34 +02:00
Luca Weiss f853c0da20
Update alpine_to_hostspec mappings (!1812)
The important part about this patch is the change for armhf, which
adjusts the hostspec to the one used by Alpine.

Fixes a part of postmarketOS/pmaports#363
2019-09-18 20:34:41 +02:00
Daniele Debernardi 54e51759ad
aportgen: add feature to fork upstream packages (!1811)
This MR add the --fork-alpine argument to the pmboostrap aportgen
command, which downloads the APKBUILD and related files and copies them
into the pmaports/temp folder.
2019-09-14 01:39:11 +02:00
Luca Weiss 30384b9083
kconfig check: support passing a file directly (!1802)
This allows for example for me to call the kconfig check function on the
.config file in my Linux tree: $ pmbootstrap kconfig check --file
.config and it reports me which kconfig options I need to enable.
2019-08-24 00:10:02 +02:00
Luca Weiss 2ad8b66ccc
Fix case sensitivity: Qemu => QEMU (!1800) 2019-07-05 20:27:12 +02:00
Idan Horo fd79a60383
pmbootstrap install: make --no-fde default (!1798)
Implement --fde, add deprecation warning to --no-fde, update related
help files and tests.
2019-07-05 20:04:10 +02:00
Oliver Smith f5db1b4306
apkbuild parser: support depends="$depends ..." (!1795)
Properly handle the following two cases in APKBUILDs:
* depends="$depends ..."
* depends="${depends} ..."

First I've attempted to refactor the parsing code to do this in a more
generic way. But I've realized that it would make more sense to retire
the python based APKBUILD parsing approach altogether and finally use a
shell script parser. Let's discuss this in #1801.
2019-07-03 23:55:21 +02:00
Oliver Smith d13997241c
kconfig check: properly test UEVENT_HELPER, LBDAF (!1796)
Add check for UEVENT_HELPER for kernels >= 4.0.0. Change LBDAF check to
be only required for kernels <5.2.0. Adjust the config format and
checking code to support such range specific checks.
2019-06-30 16:39:56 +02:00
Oliver Smith 22b49fe495
Add pmb.parse.version.check_string() (!1796)
Compare a version against a check string. This will be used in
"pmbootstrap kconfig check", to only require certain options if the
pkgver is in a specified range.
2019-06-30 16:39:56 +02:00
Luca Weiss 650ed4925e
Fix spelling mistakes (!1794)
codespell --skip="./.git,./aports*"
2019-06-25 09:20:05 +02:00
Oliver Smith b4c301974e
Add pmbootstrap checksum --verify (!1788)
Download all sources and verify their checksums.

This will be used in pmaports.git CI, if ci:skip-build is set in the
commit message (currently it just skips the build, and we don't test if
the source checksums are valid or not).
2019-05-19 22:07:20 +02:00
Luca Weiss c847dfde7c
Improve algorithm to pick package provider (!1775)
Pick the provider with the shortest name instead of always the first one
2019-04-22 20:33:03 +02:00
Zhuowei Zhang 1fffe83df3
Add "crossdirect": faster cross compiling (!1758)
Launch native cross compilers inside foreign chroot. Enable by default,
but allow disabling with --no-crossdirect for now. This option and the
distcc-sshd related code will be removed in the future.
2019-03-23 01:13:36 +01:00
Oliver Smith 7f9bfee722 Add "pmbootstrap build --no-depends" (!1769)
Aborts the build if any dependencies would have to be build first. This
is useful for build.postmarketos.org, because we want to build exactly
one package in one build job. If dependencies would need to be built, we
made a mistake earlier, and not aborting the build makes it harder to
find that orginal mistake.
2019-03-15 09:58:31 +01:00
Oliver Smith 2634bbea95
kconfig check: allow skip with !pmb:kconfigcheck (!1753)
Ignore APKBUILDs that have "!pmb:kconfigcheck" in their options by
default in "pmbootstrap kconfig check", but print a note that they have
been skipped. Check all kernels with "pmbootstrap kconfig check -f".

This is necessary, because the Librem 5 devboard kernel's config does
not have CONFIG_DM_CRYPT enabled in their config, and we check for
that. As the device is still under heavy development, we will make our
lives easier by just using the upstream kernel config without any
changes and ignoring it in our check by default.
2019-02-21 19:39:25 +01:00
Robert Yang a6db644f00
envkernel: Add build command to create an apk package from envkernel (!1747)
Provides a quick way to incrementally compile a kernel and push it to
device.

Example usage.

Compile the kernel:
$ cd /src/linux/
$ source /src/pmbootstrap/helpers/envkernel.sh
$ make tegra_postmarketos_defconfig
$ make -jX

Package kernel and flash to device:
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel

Modify kernel source then incremental compile, package, and flash:
$ make -jX
$ pmbootstrap build --envkernel linux-samsung-p4wifi
$ pmbootstrap flasher flash_kernel
2019-02-15 16:24:07 +01:00
Robert Yang 7d7a29d032 parser: Add function to read a function from an APKBUILD (!1747) 2019-02-15 16:21:47 +01:00
Robert Yang cd7d6a2431 parser: Common function to read the contents of an APKBUILD file (!1747) 2019-02-15 16:21:47 +01:00
Oliver Smith bda78e42ee
pmb/parse/arch.py: add armv6l -> armhf to mapping (!1756)
Makes it possible to run pmbootstrap on the Raspberry Pi Zero W for
example.
2019-02-13 10:06:47 +01:00
Robert Yang 67254b62c4 static code analysis: make it pass flake8 3.7.4 (!1748)
Test with flake8: *.py
./test/check_checksums.py:13:13: E117 over-indented
./pmb/config/init.py:97:8: F632 use ==/!= to compare str, bytes, and int literals
./pmb/parse/arguments.py:229:13: E117 over-indented
2019-02-02 17:36:19 -05:00
Grant Miller 6b4fd9b911 Add completion for `pmbootstrap kconfig` (!1745) 2019-01-23 00:26:45 -06:00
Grant Miller 28be44e657 package_completer: Filter by prefix (!1745) 2019-01-23 00:10:09 -06:00
Grant Miller 72f93f5dfc cosmetic: Change "packagecompleter" to "package_completer" (!1745) 2019-01-23 00:06:27 -06:00
Ion Agorria 39b3a489ef
pmbootstrap flasher flash_kernel --partition (!1741)
Allow changing the kernel partition for fastboot and heimdall in
deviceinfo and on the fly while doing "pmbootstrap flasher
flash_kernel". Also allow changing the partition for
"... flash_rootfs" with fastboot (this was only possible with
heimdall so far).

Introduce two new deviceinfo variables:
* flash_fastboot_partition_kernel
* flash_fastboot_partition_system

This is useful for devices with dual partitioning that have boot_a
and boot_b.
2019-01-20 01:14:30 +01:00
Oliver Smith f16bdaf0ca
Update copyright to 2019
Happy new year \o/
2019-01-02 09:31:20 +01:00
Oliver Smith 7f956e2f72 version parsing: fix errors with 3.0.0_pre1 (!1735)
Python < 3.6 randomized the order of keys in dictionaries, unless
OrderedDict was used. Use OrderedDict to store the version suffixes.

When the order was randomized, the valid version string 3.0.0_pre1 did
not always pass the validation check. The suffix "pre" should always be
detected as such, but with the random order, it was sometimes detected
as "p" suffix (see below). The following letters "re" are not a valid
suffix_no (the number expected to follow the suffix) and so it failed.

suffixes = {
    "pre": ["pre", ...],
    "post": ["p". ...]
}
2018-12-31 07:46:47 +01:00
Oliver Smith 96d4f2e6be
deviceinfo: resolve path for parser errors (!1732)
Display the resolved deviceinfo file path in exceptions from the
deviceinfo parser. Instead of messages like these:

RuntimeError: Please add 'deviceinfo_codename="lg-mako"' to: /home/user/.local/var/pmbootstrap/cache_git/pmaports/.gitlab-ci/testcases/../../device/device-lg-mako/deviceinfo

We get the more readable version:

RuntimeError: Please add 'deviceinfo_codename="lg-mako"' to: /home/user/.local/var/pmbootstrap/cache_git/pmaports/device/device-lg-mako/deviceinfo
2018-12-26 21:43:54 +01:00
Luca Weiss 747ccfdd75
deviceinfo: add and require _codename (!1732)
deviceinfo_codneame holds the device's code name, so we can easily look
it up in the finished postmarketOS installation by reading
/etc/deviceinfo.

Related: postmarketOS/pmaports#157
2018-12-26 21:43:38 +01:00
Oliver Smith 23df933db4
Run os.path.expanduser() on all args paths (!1731)
Properly replace ~ with $HOME in all paths. Fix tab completion when
passing a path starting with ~ to pmaports:

$ pmbootstrap --aports ~/src/pmaports/ build linux-<TAB>
2018-12-17 08:55:33 +01:00
Oliver Smith 1662fc1961
Fix packages tab completion (!1731)
Make tab completion work again for packages, if the default pmaports
dir is used. In the last "args" code refactor, the variable replacing
code was moved into its own function. We did not call it in the
packagecompleter() yet, so it could not replace the $WORK variable in
the default pmaports path.
2018-12-17 08:55:25 +01:00
Oliver Smith 60f12ae2df pmbootstrap config -h: show keys / tab complete (!1729)
Show all config keys that can be queried and set in the
'pmbootstrap config -h' output and make tab completion work for the
key names.

I've set "metavar" and placed the variables in the helpstring. That
way, argparse will not generate a huge "positional arguments" string
that blows up the layout of the help output:
[{ccache_size,device,extra_packages,hostname,jobs,kernel,keymap,...
2018-12-17 08:45:46 +01:00
Grant Miller dfaca1e06f Enable armv7 (!1730)
I need to do some more work on pmaports in order for this to actually
be useful.
2018-12-17 08:06:10 +01:00
Oliver Smith a5a64158e9
allow specifying multiple postmarketOS mirrors (!1718)
Multiple -mp arguments can be used to list multiple mirrors:
$ pmbootstrap -mp=first -mp=second chroot -- cat /etc/apk/repositories

This is needed for the new build infrastructure, so we can have a WIP
repository to which we push packages until all of them are up to date,
and then publish all of them at once. Software like KDE/Plasma Mobile,
which expect a lot of packages to be updated from one version to
another will not end up with a half-way through upgrade that way.
2018-12-06 07:31:43 +01:00
Oliver Smith 2c6c5a9df9
prettier --help output for -m, -mp arguments (!1718)
-m is the Alpine mirror, -mp is the postmarketOS mirror. Use "URL" as
metavar and add help text that explains how to disable the postmarketOS
mirror (so all pmaports get built locally).
2018-12-06 07:31:32 +01:00
Oliver Smith 933c4d0f0d new action: 'pmbootstrap repo_missing'
Add a new action that lists all aports, for which no binary packages
exist. Only list packages that can be built for the relevant arch
(specified with --arch). This works recursively: when a package can be
built for a certain arch, but one of its dependencies
(or their depends) can not be built for that arch, then don't list it.

This action will be used for the new sr.ht based build infrastructure,
to figure out which packages need to be built ahead of time (so we can
trigger each of them as single build job). Determining the order of the
packages to be built is not determined with pmbootstrap, the serverside
code of build.postmarketos.org takes care of that.

For testing purposes, a single package can also be specified and the
action will list if it can be built for that arch with its
dependencies, and what needs to be built exactly.

Add pmb/helpers/package.py to hold functions that work on both pmaports
and (binary package) repos - in contrary to the existing
pmb/helpers/pmaports.py (see previous commit) and pmb/helpers/repo.py,
which only work with one of those.

Refactoring:
* pmb/helpers/pmaports.py: add a get_list() function, which lists all
  aports and use it instead of writing the same glob loop over and over
* add pmb.helpers.pmaports.get(), which finds an APKBUILD and parses it
  in one step.
* rename pmb.build._package.check_arch to ...check_arch_abort to
  distinguish it from the other check_arch function
2018-12-01 21:30:59 +00:00
Oliver Smith a44b80b31d build.find_aport() -> helpers.pmaports.find()
Move find_aport() and find_aport_guess_main() from pmb/build/other.py
to the new file pmb/helpers/pmaports.py.

Finding aports is not only needed when building packages, hence it
makes sense to move it out of pmb.build. The pmb/helpers/pmaports.py
file will have more pmaports related functions in a follow up commit.
2018-12-01 21:30:59 +00:00
Luca Weiss 9c037831a0
Remove legacy kconfig_check and menuconfig args
Fixes #1690
2018-12-01 01:32:52 +01:00
Oliver Smith e458b1fdbc Add --offline flag
Allow working offline with pmbootstrap, as long as all packages that
are being used have already been downloaded.
2018-10-25 20:37:33 +00:00
Luca Weiss 6495fca971 Remove fuzzy versions for dependencies
Fixes #1344
2018-10-11 05:30:22 +00:00
Oliver Smith 1f8832c4cc install apk keys and binfmt data with setup.py 2018-09-17 10:06:57 +00:00
Oliver Smith 3ce00de710 Move aports into own repository (pmaports) 2018-09-05 05:57:38 +00:00
Grant Miller 7eaf9de000 Implement bash tab completion 2018-08-27 21:35:05 +00:00
Grant Miller 8c9c0ce999 Check if device was renamed 2018-08-22 21:30:16 +00:00
Oliver Smith 68e1feef17 pmb: qemu-user-static from aport, not from Debian
Overview:
In order to execute foreign arch binaries on the host system, we are
using the Linux kernel's binfmt_misc feature in combination with
static builds of QEMU. Before this patch, the statically compiled
QEMU binaries were taken from Debian (mostly because I did not realize
that Alpine ships them as well). Now we can use the ones from the aport.

Benefits:
This allows us to easily update and patch the QEMU executables, we
don't need to be in sync with Debian's versions anymore.

Alpine's package is more modular, so we can save some download,
install, zap time, as well as disk space: setting up an armhf chroot
with pmbootstrap took ~102 MB before, now it's ~18 MB.

Detailed changes:
* Remove `cross/qemu-user-static-repack` aport
* Add `data/qemu-user-binfmt.txt` with the binfmt_misc flags for ELF
  binaries of various arches (extracted from Debian's packaging)
* When parsing that file, don't write verbose messages to
  `pmbootstrap log` anymore, only to the verbose log (can be enabled
  with `pmbootstrap -v`)
* Rename `pmb.parse.arch.alpine_to_debian()` to ...`alpine_to_qemu()`
* Rename `arch_debian` to `arch_qemu`
2018-08-10 15:11:21 +00:00
Oliver Smith f75586208f pmb: unsupported arch: link to wiki page
The old message was outdated and incomplete, added a link to this wiki
page instead: <https://postmarketos.org/newarch>
2018-07-26 22:00:25 +00:00
Oliver Smith b0d4244b38 pmb: flasher: require action argument
With this patch, "pmbootstrap flasher" will fail with "the following
arguments are required: action_flasher". Without it, it just prints
"Done" and quits.
2018-07-26 10:09:35 +00:00
Oliver Smith 4844719b1d pmb: adjust to distcc 3.3 and wrap it with sshd
Overview:
Since Alpine updated to distcc 3.3 last week, pmbootstrap wasn't able to use
distcc for cross compilation anymore. It always falled back to running the
compiler in QEMU (which works, but is a lot slower). The reason for that is,
that distcc requires all compilers that are being used in a whitelist now.

This partially fixes CVE-2004-2687 in distccd, which allowed trivial remote
code execution by any process connecting to the distccd server. We only run
distccd on localhost, but still this can be used for privilege escalation of
sandboxed processes running on the host system (not part of pmbootstrap
chroots).

Because the CVE is only partially fixed (see the comment in
`pmb/chroot/distccd.py` for details), we make sure that only the building
chroots can talk to the distcc server by running distcc over ssh.

Details:
* Completely refactored `pmb/chroot/distccd.py` to run distcc over ssh
  * Store the running distcc server's arguments as JSON now, not as INI
* Make debugging distcc issues easy:
  * Set DISTCC_BACKOFF_PERIOD=0, so the distcc client will not ignore the
    server after errors happened (this masks the original error!)
  * New pmbootstrap parameters:
    * `--distcc-nofallback`: avoids falling back to compiling with QEMU and not
	   throwing an error
	* `--ccache-disable`: avoid ccache (when the compiler output is cached,
	  distcc does not get used)
  * `--verbose` prints verbose output of the distcc too
  * New test case, that uses the new pmbootstrap parameters to force
	compilation through distcc, and shows the output of distcc and distccd in
	verbose mode on error (as well as the log of sshd)
2018-07-26 09:47:23 +00:00
Oliver Smith 0d7802d7ef
s/system partition/rootfs: fix remaining mentions
Follow-up to !1373, where `pmbootstrap flasher flash_system` was
replaced with `pmbootstrap flasher flash_rootfs`. We still had used
terms like "system partition" in a lot of places.

This commit replaces it everywhere, so it's clear that we're talking
about the pmOS rootfs (which may or may not be installed to Android's
system partition).
2018-07-15 23:41:31 +02:00
Oliver Smith 8268dc0e3d pmbootstrap: kill process if silent for 5 minutes (rewrite logging) 2018-07-14 01:13:28 +00:00
Oliver Smith bbae995be8 APKBUILD parser: replace $pkgname in *depends*
Fixes #1613
2018-07-11 19:24:40 +00:00
Oliver Smith b8637a024a
add --force as well 2018-07-08 23:49:03 +02:00
Oliver Smith d4f4ea8488
pmb: add -f (force) parameter to bootimg_analyze
We are analyzing the `boot.img` with `file` before we send it to
`unpackbootimg`. File does not recognize all kinds of `boot.img` files,
which `unpackbootimg` can extract, so we need a way to skip this check.

Details:
* Add `-f` parameter, continues extraction with a warning if the file
  seems to be invalid
* Tell the user that `-f` can be used if the `boot.img` is invalid and
  it's not specified
* Consistent spelling of `boot.img` instead of `bootimg` in messages

Fixes #1608
2018-07-06 21:57:18 +02:00
Oliver Smith f6dcfbfe56 Use Alpine's QEMU rather than host system QEMU (v2) 2018-07-06 19:50:59 +00:00
Oliver Smith 6e21c44342 Revert "Merge branch 'alpine-qemu' into 'master'"
This reverts merge request !1612
2018-07-05 17:25:14 +00:00
ryang c650354fc3 qemu: Use Alpine's QEMU rather than host system QEMU
Use --host-qemu to use QEMU that is installed on the host system.
2018-07-01 19:39:02 -04:00
steamport 30df0725ca menuconfig: add gconfig/xconfig support (#1509)
* change "pmbootstrap kconfig_check" to "pmbootstrap kconfig check"
* change "pmbootstrap menuconfig" to "pmbootstrap kconfig edit [-x|-g]"
  (with legacy alias, because the first syntax was referenced to a lot)
* enable X11 interfaces: -x: xconfig, -g: gconfig
* new function to copy the xauthority file:
  pmb.chroot.other.copy_xauthority()
* remove menufconfig() function from the kernel template and all kernel
  aports ([skip ci] because it would rebuild all kernels and run out of
  time). Alpine has dropped this as well, and it wouldn't work with the
  new code anyway.
2018-06-09 06:52:24 +00:00
Marcin Mielniczuk 36c5ce3ca4 Be more verbose about the missing aports directory (#1454)
Show the expected path in the error message.
2018-05-01 00:49:20 +00:00
Daniele Debernardi 827a60cd25 Add install flag to generate separate boot and system images (#1442)
* Usage: pmbootstrap install --split
* Make obvious that export is the next step when split images are created
* Fix note for missing rootfs image on export
  * Change wording from "system image" to "rootfs image"
  * The idea was to show the note only when the rootfs image was not
    generated yet. But this was broken, because the path we checked for
    was missing the chroot path prefix (which is added now).
  * Also don't display the message, when the split image files exist
2018-05-01 00:18:40 +00:00
Oliver Smith a1598dc55d
Fix randomized pkg providers for python < 3.6 (#1446) 2018-04-28 21:32:00 +00:00
Oliver Smith 14e234264f
msm-fb-refresher: use initfs hook (#1438)
The `msm-fb-refresher` updates the screen for msm based devices. It is
not needed for all devices, so we had some extra code in the initramfs,
that would only add it when the `deviceinfo_msm_refresher` variable was
set. However, we are able now to add files to initramfs hooks, so this
hack can be removed and simplify everything.

Changes:
* Remove `deviceinfo_msm_refresher` from all deviceinfos
* Add sanity check for it
* Move all `deviceinfo` sanity checks to an extra function
* `postmarketos-mkinitfs`: remove code for msm refresher
* `msm-fb-refresher`: add initramfs hook
2018-04-24 21:09:53 +00:00
Oliver Smith 9c83cdcea1
Display wiki link on dependency error (#1410) 2018-04-23 20:51:06 +00:00
Oliver Smith a77a0dcc32
Add envkernel.sh: instant mainlining environment (#1424)
Changes:
* `helpers/envkernel.sh`:
  * installs everything needed for kernel compilation in the native
    chroot
  * mounts the kernel source to `/mnt/linux` inside the chroot
  * creates `/mnt/linux/.output` and chowns it to the `pmos` user, that
    folder will be used for the kernel build output
  * sets up aliases for `make`, `pmbootstrap`, `pmbroot`, `kernelroot`
* new action `pmbootstrap work_migrate`: does the interactive work
  folder migration if necessary, otherwise it doesn't output anything
  * when calling this first, we can safely use all other commands
    non-interactively without showing the output

Benefits:
* Fast setup (especially for people who are new to kernel
  compilation
  * No need to figure out distribution specific package names
    (cross compilers!)
  * No need to do a test build just to verify that the right
    packages are installed
* Less error prone
  * The right dependencies are always installed
  * `ARCH` and `CROSS_COMPILE` variables always get set automatically
    and based on `deviceinfo_arch`
  * If the build environment is broken for some reason, just zap and
    start over
* Easy to reproduce problems

Notes:
* `make menuconfig` works as well
* Sourcing was tested with `zsh`, `bash` and `fish`, it should be easy to
  extend for other shells
2018-04-19 21:27:38 +00:00
Oliver Smith b66b5dcc34
pmbootstrap init: kernel selection / remove linux-pmos-lts (#1363)
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
  for now. The kernel isn't used right now, and we save lots of
  maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
  `"downstream", "mainline", "stable"` (downstream is always
  `linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
  has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
  the `deviceinfo_dtb` file is missing (because we expect it to be
  missing for downstream kernels)
* device-sony-amami:
  * add kernel subpackages for downstream, mainline
  * set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
  * don't mix specifying kernels in depends *and* subpackages
  * 1 kernel in depends is maximum
  * kernel subpackages must have a valid name
  * Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
  `--ignore-depends` can be used with device packages to avoid building
  the kernel. The idea was to make the transition easier after a change
  we did months ago, and now the kernel doesn't always get built before
  building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
  * Add autoinstall=True to kernel_flavors_installed(). When the flag
    is set, the function makes sure that at least one kernel for the
    device is installed.
  * Remove kernel_flavor_autodetect() function, wherever it was used,
    it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
  kernel_flavors_installed() takes care of that now.
2018-04-03 23:50:09 +00:00
Oliver Smith 6bb8444fef
"...flasher flash_rootfs" instead of "..._system" (#1373)
* Change `pmbootstrap flasher flash_system` command to
  `pmbootstrap flasher flash_rootfs`
* The old command still works, but all references have been changed to
  the new command
* Remove obsolete `pmbootstrap flasher export` (that was changed to
  `pmbootstrap export` a few months ago)
* Update `README.md` and ZSH auto completion
* Change the description of the generated rootfs image (not talking
  about a system image anymore, mention that it has subpartitions)
* Better description of `pmbootstrap flasher flash_rootfs --partition`
2018-03-30 01:11:20 +00:00
Oliver Smith 33a09ea1c7
Fix dependency resolver being stuck after reboot (#1359)
When the native arch (e.g. `x86_64`) `APKINDEX` files are outdated, and
`pmbootstrap` gets instructed to build a linux package for a foreign
arch, then the `APKINDEX` cache did not get used anymore for the
current session. This means that every lookup of a package in an
`APKINDEX` caused the whole `APKINDEX` file to get parsed again instead
of using the cached version. This slowed it down so much that it felt
like `pmbootstrap` was looping forever.

How this happens in detail:
* Whenever pmbootstrap parses an `APKINDEX`, it fills up the
  `args.cache["apkindex"]` dict with the parsed information and the
  last modified date of the file.
* `pmbootstrap` checks the last modified date of the `APKINDEX` files
  and updates them if they are older than 4 hours.
* When the bug appeared, then the cache was already filled up, then an
  update happened and then `pmbootstrap` tried to read from the cache.
* So when reading from the `APKINDEX`, the cache gets ignored because
  the last modified date is different.
* Up to this commit, the cache does not get deleted and filled up
  again!

How to test:
Try these commands once without this commit, and then with this commit
applied:
```
$ sudo touch -m -t 201801010000 \
    ~/.local/var/pmbootstrap/cache_apk_x86_64/APKINDEX.*
$ pmbootstrap -v build linux-postmarketos-mainline --arch=armhf
```

Without the patch, you can see in `pmbootstrap log` that it is
resolving the dependencies properly, but very slowly. With the patch
the resolving happens almost instantly.
2018-03-24 16:31:31 +00:00
Oliver Smith 1ed51f83f1
Package resolving: Don't fail on fuzzy versions (#1355)
When parsing the depends of entries in the APKINDEX file, we ignore
all operators (<, =, >). (This is enough for our use case, was we only
do the dependency resolving to check which packages need to be built
and `apk` does the dependency resolving again before installing
anything).

We did not ignore the ~ character for fuzzy version compares, this is
fixed with this commit.
2018-03-24 16:29:32 +00:00
Oliver Smith 5ea00e0862
pmbootstrap newapkbuild: Properly parse arguments (#1320)
* pmbootstrap newapkbuild: Properly parse arguments

The `pmbootstrap newapkbuild` action wraps Alpine's `newapkbuild`. We
used to directly pass all arguments to `newapkbuild` without verifying
in Python whether they make sense or not. However, as `newpakbuild`
doesn't do strict sanity checks on the arguments, it is easy to end up
with unexpected behavior when using the command for the first time.

For example, `newapkbuild` allows either specifying a PKGNAME or SRCURL
as last parameter, and also allows setting a PKGNAME with the `-n`
parameter. It only makes sense to use that option when passing a
SRCURL.

With this commit, we duplicate the optins that should be passed through
to `newapkbuild` and use argparse to fully sanitize the options and
display a help page (`pmbootstrap newapkbuild -h`) that is consistent
with the other help pages.

Details:
* The `-f` (force) flag does not get passed through anymore. Instead we
  use it in Python to skip asking if an existing aport should be
  overwritten (the aports are outside of the chroot, so `newapkbuild`
  can't handle it in a way that makes sense for pmbootstrap).
* Output of `newapkbuild` gets redirected to the log file now, as we
  don't need it to display a help page.
* Don't verify the pkgver while creating the new APKBUILD. When passing
  a SRCURL, the pkgver gets extracted from the end of the URL and may
  not have a valid format yet (but we want the APKBUILD anyway).
* Stored options passed through in `pmb/config/__init__.py` and use it
  in both `pmb/parse/arguments.py` and `pmb/helpers/frontend.py`.
* Only allow `-n` with SRCURL
* The postmarketOS aports folder gets specified with `--folder` now.
  That way the generated help page is much closer to the original one
  from `newapkbuild`. The default is `main`.
* Made the package type flags (CMake, autotools, ...) exclusive so only
  one of them can be specified
2018-03-15 21:42:34 +00:00
Oliver Smith b8f35d45b8
aportgen: Gracefully handle old aports_upstream (#1291)
In order to get cross-compilers, we generate a few aports (e.g.
binutils-armhf, gcc-armhf) automatically from Alpine's aports.
pmbootstrap was already able to perform a git checkout of Alpine's
aports repository. But it needed to be manually updated. Otherwise
the `pmbootstrap aportgen` command could actually downgrade the aport
instead of updating it to the current version.

After thinking about adding a dedicated pmbootstrap command for
updating git repositories, I thought it would be better to not open
that can of worms (pmbootstrap as general git wrapper? no thanks).

The solution implemented here compares the upstream aport version of
the git checkout of a certain package (e.g. gcc for gcc-armhf) with the
version in Alpine's binary package APKINDEX. When the aport version is
lower than the binary package version, it shows the user how to update
the git repository with just one command:

    pmbootstrap chroot --add=git --user -- \
        git -C /mnt/pmbootstrap-git/aports_upstream pull

Changes:
* `pmb.aportgen.core.get_upstream_aport()`: new function, that returns
  the absolute path to the upstream aport on disk, after checking the
  version of the aport against the binary package.
* Use that new function in pmb.aportgen.gcc and pmb.aportgen.binutils
* New function `pmb.helpers.repo.alpine_apkindex_path()`: updates the
  APKINDEX if necessary and returns the absolute path to the APKINDEX.
  This code was basically present already, but not as function, so now
  we have a bit less overhead there.
* `pmbootstrap chroot`: new `--user` argument
* `pmb.parse.apkbuild`: make pkgname check optional, as it fails with
  the official gcc APKBUILD before we modify it (the current APKBUILD
  parser is not meant to be perfect, as this would require a full shell
  parsing implementation).
* Extended `test_aportgen.py` and enabled it by default in
  `testcases_fast.sh`.  Previously it was disabled due to traffic
  concerns (cloning the aports repo, but then again we do a full KDE
  plasma mobile installation in Travis now, so that shouldn't matter
  too much).
* `testcases_fast.sh`: With "test_aport_in_sync_with_git" removed
  from the disabled-by-default list (left over from timestamp based
  rebuilds), there were no more test cases disabled by default. I've
  changed it, so now the qemu_running_processes test case is disabled,
  and added an `--all` parameter to the script to disable no test
  cases. Travis runs with the `--all` parameter while it's useful to
  do a quick local test without `--all` in roughly 2 minutes instead of
  10.
* `aports/cross/binutils-*`: Fix `_mirror` variable to point to current
  default Alpine mirror (so the aportgen testcase runs through).
2018-03-11 14:18:21 +00:00
Oliver Smith af4b817c21
Skip virtual packages when parsing APKINDEX (#1278)
Since PR #1247 we are using the virtual package `.pmbootstrap` to mark
packages as not explcitly installed. In case `pmbootstrap` doesn't
finish the installation because of a bug, the `.pmbootstrap` virtual
package may not get uninstalled.

As virtual packages don't have the "timestamp" attribute set in the
package database (which indicates when the package was built), the
APKINDEX parser fails to parse them.

Changes:
* pmb.parse.apkindex.parse_next_block(): don't require the "timestamp"
  attribute to be set (but the arch attribute instead, which is always
  present)
* pmb.parse.apkindex.parse(): when a block does not have a `timestamp`
  attribute, skip it, because it must be a virtual package.
* add test cases for both functions with a package database that
  contains a virtual package.
2018-03-10 13:15:30 +00:00
Oliver Smith e72afc53fd
pmbootstrap zap -m: delete outdated packages too (#1306)
`-m` is for deleting local compiled packages, for which there is no
aport with the same version. Prior to this change, this only worked
for packages where no aport exists, or for packages that are newer
than the aports.

That is, because we used the usual APKINDEX parsing logic, which
ignores old packages in the APKINDEX and only returns the one with the
highest version (that makes sense during dependency resolution).

Changes:
* New `pmb.parse.apkindex.parse_blocks()` function that returns a raw
  list of blocks, instead of the dict with removed duplicates with
  lower version you get from the usual `.parse()` function.
* Renamed each of the zap flags and their descriptions to make clear
  what they are doing now.
  ```
  short  long (old)         long (new)
  -p     --packages         --pkgs-local
  -m     --mismatch-bins    --pkgs-local-mismatch
  -o,    --old-bins         --pkgs-online-mismatch
  ```
2018-03-10 12:08:02 +00:00
Oliver Smith 3510a4868f
Fix building packages by provides name (#1303)
Use case: `mkbootimg` provides the `unpackbootimg` package. When
running `pmb.chroot.apk.install(args,"unpackbootimg")`, it was not
able to properly build the package.

Reproducing the error:
```
sudo rm ~/.local/var/pmbootstrap/packages/x86_64/mkbootimg*
pmbootstrap index
pmbootstrap --mirror-pmOS="" chroot --add=unpackbootimg
```

Or alternatively (simpler but less illustrative):
```
pmbootstrap build unpackbootimg --force
```
2018-03-08 21:30:55 +00:00
Daniele Debernardi 94306b51ee deviceinfo: remove external_disk_install and external_disk, use external_storage instead (#1301)
* deviceinfo: remove external_disk_install and external_disk, use external_storage instead
* Complain when external_disk* is used
2018-03-07 22:35:02 +00:00
Oliver Smith 99127111a1
Only download APKINDEX for relevant architectures (#1201)
### Only download APKINDEX for relevant architectures
We're downloading the APKINDEX files for all architectures supported by
postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time,
we only need it for the native and device arch, so this PR reduces the
downloaded files to what is really necessary.

### Intuitive pmbootstrap update logic
* pmb.helpers.repo.update():
  * Default is updating all arches where the APKBUILD files exist
  * Add existing_only parameter
  * Return True when files have been downloaded
  * Properly print which arches will be updated
  * Print update reason only in verbose log
  * Add and improve comments
* pmb.parse.arguments(), update action:
  * Add --non-existing parameter
  * Default for --arch is None (instead of arch.native)
* pmb.helpers.frontend.update():
  * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
Luca Weiss 6b9e7ecf75 qemu: allow the user to specify the kernel flavor (#1256) 2018-02-25 19:20:22 +00:00
Oliver Smith ad5a0d4294
Make proprietary drivers optional (1/2): pmbootstrap changes (#1254)
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.

Changes:
* New config file options nonfree_firmware and nonfree_userland, which
  we ask for during "pmbootstrap init" if there are non-free components
  for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
  packages are called $pkgname-nonfree-firmware and
  $pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
  subpackages. Parsing that is implemented in
  pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
  the regular APKBUILD parsing, as this would need a change in the
  output format, and it is a lot *less* code if done like in this
  commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
  pmb/install/install.py to _install.py: needed to call the function in
  the usual way (e.g. pmb.parse.apkbuild()) but still being able to
  test the individual functions from these files in the test suite.
  We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
  packages that will be installed, based on the user's choice in
  "pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
  refactored test/test_questions.py to have multiple functions for
  testing the various questions / question types from
  "pmbootstrap init" instead of having it all in one big function.
  This allows to use another aport folder for testing the new
  non-free related questions in init.
2018-02-24 21:49:10 +00:00
Oliver Smith 8f5ffe0589
APKINDEX not found: Only show in verbose log (#1250)
The message appears 20 or so times in a row when setting up a new
chroot. Right now we log it into the debug log (which is what
"pmbootstrap log" and "pmbootstrap --details-to-stdout" show). This
is annoying for normal pmbootstrap usage, and really clutters up the
Travis output.

This commit changes the log level to verbose, so it only shows up
when "-v" is passed to pmbootstrap.
2018-02-24 08:03:34 +00:00
Oliver Smith db5e69630e
Index parser: support multiple package providers (#1202)
* The APKINDEX parser used to return a dictionary with one package for
  a given package name. This works for the installed packages database,
  because there can only be one provider for a package. But when
  parsing packages from binary repositories, we need to support
  multiple providers for one package. It is now possible to get a
  dictionary with either multiple providers, or just a single provider
  for each package.
* Dependency parsing logic has been adjusted, to support multiple
  providers. For multiple providers, the one with the same package
  name as the package we are looking up is prefered. If there is none
  (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages
  that will be installed anyway, and after that packages that are
  already installed. When all else fails, it just picks the first one
  and prints a note in the "pmbootstrap log".
* Added testcases for all functions in pmb.parse.apkindex and
  pmb.parse.depends
* pmbootstrap chroot has a new "--add" parameter to specify packages
  that pmbootstrap should build if neccessary, and install in the
  chroot. This can be used to quickly test the depencency resolution
  of pmbootstrap without doing a full "pmbootstrap install".

Fixes #1122.
2018-02-20 19:52:28 +00:00
Oliver Smith 0f371e426f
pmbootstrap build --src: override source for any package (#1210)
* New "pmbootstrap build --src=/local/source/path hello-world" syntax
* The local source path gets mounted inside the chroot
* From there, a copy of the source code gets created with rsync (so
  we can write into the source folder if necessary, for better
  compatibility with all kinds of APKBUILDs)
* After the aport gets copied into the chroot before building (as
  usually), we extend the APKBUILD with overrides to make it use
  mountpoint's source instead of downloading the package's source
  from the web as usually
* The package built with the local source gets _pYYYYMMDDHHMMSS
  appended to the pkgver
* linux-postmarketos-mainline: use $builddir, fix patch checksum
2018-02-19 22:04:01 +00:00
Alexander 3d35b45835 Check kernel config options based on the architecture (#1233)
The 'necessary_kconfig_options' dictionary in pmb/config/__init__.py
now has the different architectures (space separated) as the keys and
the dictionary, which matches kernel config options and their
expected value, as its value.
For that purpose, the 'check' function in pmb/parse/kconfig.py was
modified, so that it takes the architecture from the kconfig filename
and uses it to find the needed kernel config options.
Closes #1218.
2018-02-17 13:37:35 +00:00
Lucas Ramage 42018f8014 mkbootimg: replace Python version with osm0sis' C version (#1193)
Closes #441. Adjust bootimg_analyze code:
* Install mkbootimg (which now provides unpackbootimg) instead of
  unpackbootimg. In theory, pmbootstrap should recognize this
  automatically, however right now it does not yet handle this case.
* The file names of the extracted files have changed.
2018-02-11 11:41:41 +00:00
Oliver Smith e8c27795a8
Remove rest of 'pmbootstrap challenge' left overs (#1173)
Follow up to #1162.

* `pmb.build.buildinfo()`: Used to record the build environment. It is
  flawed because it scans the repo APKINDEX files instead of using the
  actually installed packages list. When it was implemented we were not
  able to do the latter. After this is removed, `pmb.parse.depends` can
  be simplified (it needs to be rewritten for #1122).
* `pmb.helpers.repo.diff()` and `pmb.helpers.repo.files()`: These were
  used exclusively by `pmb.build.buildinfo()`, to learn about which
  files have been changed in the local repository folder after a
  package was built. The idea was, that we could find subpackages that
  way. But this information is present in the installed package list as
  well, which is a much cleaner approach.
2018-02-01 22:03:21 +00:00
Oliver Smith 0f5056f6b9
Fix pmbootstrap zap -m / various zap improvements (#1166)
zap -m:
* APKINDEX parsing: parse the "origin" field as well, so we know
  where a subpackage comes from
* pmbootstrap zap -m: properly delete all packages, that do not
  have an aport or where the aport has another version. This also
  works with subpackages now,
  we use the origin field to resolve it.
* Only reindex when packages have been deleted in "zap -m"

zap in general:
* Show the amount of cleared up space after the deletion instead
  of "Done"
* Print "Shutdown complete" to "pmbootstrap log" instead of stdout
  (we need to call it twice during zap now to get the space
  calculation right)
* Add `--dry` argument to `pmbootstrap zap` (this was very useful
  for debugging) to list the packages/chroots that would get
  deleted
* Roughly output the command that would get executed to delete
  files, so it's obvious what's going on in --dry mode. (% rm ...)
2018-01-31 19:34:02 +00:00
Oliver Smith 3c59126bc1
Remove timestamp based rebuilds (#1174)
If you want to build a package without changing the version number,
please use `--force` from now on. For example:

    pmbootstrap build --force hello-world

Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close #1167, close #1156, close #1023 and
close #985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
2018-01-28 23:27:33 +00:00
Pavel Machek bdeec7a255 Optional --rsync parameter for pmbootstrap install only copies the diff (#1151)
Should reduce wear of sd card. Example usage:
pmbootstrap install --sdcard=/dev/mmcblk0 --no-fde --rsync
More rsync output with pmbootstrap -v.
2018-01-28 23:25:21 +00:00
Oliver Smith 566da1718d
version.py: fix validation of 6.0_0002 (#1160) 2018-01-25 18:23:55 +00:00
Oliver Smith 0479031f7e
Remove lots of legacy 'challenge' code (#1162)
Preparation for #1122.

* `pmb.parse.apkindex.parse()`, removed strict parameter: This used to raise
  an exception when two entries in the apkindex provided the same package.
  Turns out this is *not* invalid after all, two packages can provide the same
  soname for example (e.g. libhybris, mesa-egl). In an APKINDEX, sonames are
  listed as they were packages ("so:libjpeg.so.8" etc.).
* Remove `pmbootstrap challenge` leftover code from reproducible builds effort,
  which was a dead end. This code uses the broken strict feature.
2018-01-25 18:08:39 +00:00
Daniele Debernardi 845d946389 Enable running pmbootstrap on x86 architecture (#1155) 2018-01-24 00:56:11 +00:00
Oliver Smith 3c8a93fa7b
Improved pmbootstrap init (#1095)
This PR makes the workflow faster and pmbootstrap will
produce less traffic. Details:
* Check if it's possible to create and read from a device
  node directly when initializing a chroot (closes #472)
* Copy the Qemu binary into the forign-arch chroots
  before initializing them, so the post-install script
  directly work during the chroot setup and we don't need
  to call apk fix afterwards
* Use pmb.helpers.repo.update(), which only updates the
  APKINDEX files if they are older than 4 hours, instead
  of using apk's repo update function which always
  downloads the APKINDEX files
* Chroot initialization
  * Getting the initial APKINDEX to download apk-tools-static
  * Updating the APKINDEX at the start of pmbootstrap install
* Fixed a bug in from_chroot_suffix: the buildroot_x86_64 has
  architecture x86_64, not x86.
2018-01-23 00:54:48 +00:00
Oliver Smith 0ae23afa60
Fix #839: Check pkgver after parsing APKBUILD / various small improvements (#854)
Small improvements:
* Allow to specify multiple packages to `pmbootstrap parse_apkbuild`
* Specifying no package will parse all packages (like kconfig_check)
  (also `parse_apkbuild`)
* JSON output is sorted of `parse_apkbuild`
* Make pkgver check optional, so we can disable it in the device wizard test case
* Parse_apk* -> apk*_parse
* Don't let the user mess with globs (disallow '*' in pkgname)
2018-01-18 22:05:27 +00:00
Oliver Smith 9a3ce3ee70
Wrap Alpine's newapkbuild (#894)
Closes #836.
2018-01-15 22:00:11 +00:00
Oliver Smith 4715c0f1bf
pmbootstrap: Disallow running as root (#1120) 2018-01-14 08:13:35 +00:00
Oliver Smith 1992f37036
Gracefully handle packages breaking because of soname bumps (#1116)
Fixes #893. Changes:
* New action: "pmbootstrap pkgrel_bump"
* pmbootstrap detects missing soname depends when trying to install
  anyting, and suggests "pkgrel_bump --auto" to fix it
* Testcase test_soname_bump.py checks the pmOS binary package repo
  for soname breakage, so we see it when CI runs for new PRs
* libsamsung-ipc: bump pkgrel because of soname bump
2018-01-14 01:26:42 +00:00
Oliver Smith 12340fe5f6
Fix qemu-vexpress and qemu-aarch64 (#1029)
* Don't ask for the mesa driver when the Qemu arch is not the
  native arch and always use swrast in that case
* qemu-vexpress: use LTS kernel
* qemu-aarch64: use drm-backend for weston
2018-01-08 15:18:37 +00:00
Oliver Smith 7750c1dd40
Happy new year! (update copyright to 2018) 2018-01-04 04:53:35 +01:00
drebrez c0c197f57b Feature debug-shell (#1028)
* Changed usb-shell behavior, it wait for some user action before continue booting
* Rename usb-shell to debug-shell and changed port to 23
* Add `20-debug-shell.sh` script to static code analysis
* Enable eth0 interface in initramfs (qemu)
* Add additional script to run a shell in order to be able to kill it from a telnet session
2017-12-23 19:22:28 +00:00
Attila Szöllősi c6eb56c200 Rename deviceinfo variable flash_methods to flash_method (#1030)
* Rename deviceinfo variable flash_methods to flash_method
* Update pmb.config.deviceinfo_attributes / add sanity check
* Add test case that parses all deviceinfo files
2017-12-21 22:12:51 +00:00
Oliver Smith 567ac64e26
ccache: Fix for distcc cross-compiling / various improvements (#1026)
* ccache: Fix for distcc cross-compiling / various improvements

* Make ccache work when cross-compiling with distcc (fix #716)
* Allow to configure the ccache size in "pmbootstrap init"
* Moved ccache stats code from pmb/build/other.py to
  pmb/helpers/frontend.py
* Grouped job count, ccache size and timestamp based rebuilds
  together to "build options" and allow to skip them
* Sorted config options that had to be modified anyway
  alphabetically

* Improve comment in arch-bin-masquerade APKBUILD
2017-12-21 16:42:29 +00:00
Oliver Smith 071ec4c214
build necessary logic: look at the package with the highest version (#1004)
Fixes #955. Previously we did not look through all APKINDEXes while
looking for the package with a given name and the highest version.
This caused pmbootstrap to build packages even if they are in the
binary repo and up to date.
2017-12-20 16:04:28 +00:00
Oliver Smith 96eaf481d1
Cache which packages have been built (#972)
Fixes #968: Dependency calculation took a long time for `postmarketos-ui-plasma-mobile`
2017-12-05 23:17:39 +00:00
Oliver Smith e9ca68dc05
Fix #941: Use the right arch for foreign-arch-only packages (#943)
This is a follow-up to #935.

* fix regression #941: pmbootstrap doesn't automatically pick the
  right architecture for building when none is specified
* remove obsolete --noarch-arch parameter
2017-11-28 19:12:16 +00:00
Oliver Smith d3c77c39ac
Fix #824: Refactor pmb/build/package.py (make depends work like in abuild) (#935)
* Rename pmb/build/package.py to pmb/build/_package.py, so we can
  access the functions it contains in testcases, and still use
  pmb.build.package()
* Refactor the entire file. Instead of one big function that does
  too many things, we have many small ones now, that are tested
  in the testsuite and easier to modify
* Whenever building a package, pmbootstrap does not only build and
  install the "makedepends" (like we did before), now it does the
  same for the "depends". That's required to be compatible with
  abuild. The old behavior can still be used with 'pmbootstrap
  build --ignore-depends'.
* Because of that change, noarch packages can no longer be built in
  the native chroot if we need them for a foreign chroot. A device-
  package depending on a kernel would pull in the same kernel for
  the native architecture otherwise.
* Running 'pmbootstrap build device-...' without '--ignore-depends'
  and without a matching '--arch' displays a note that explains
  this change to the user and tells how to use it instead.
* Noarch packages no longer get symlinked. That was only
  implemented for packages built in the native chroot, and now that
  is not always the case anymore. Symlinking these packages creates
  packages with broken dependencies anyway (e.g.
  device-samsung-i9100 can't be installed in x86_64, because
  linux-samsung-i9100 is armhf only).
* Rename "carch" to "arch" wherever used. Naming it "carch"
  sometimes is confusing with no benefit.
* Add a testcase for the aarch64 qemu workaround (because it failed
  first and I needed to know for sure if it is working again).
* Improved some verbose logging, which helped with development of
  this feature.
* Removed the old "build" test case (which was disabled in
  testcases_fast.sh) as the new "build_package" test case covers its
  functionallity.
* Only build indexes if the packages folder exists for that arch (Travis
  couldn't run a test case otherwise)
2017-11-26 14:32:02 +00:00
Oliver Smith a7b881e4cc
Close #871: Enable binary repository (#887)
* add my own build key
* enable the repo in the config
* update the README file
* Adjust testcase, that validates the keys and enable it in testcases_fast.sh
* Only save/load keys to/from the config file, which we ask for during
  'pmbootstrap init', so the binary repo gets used even if a config file
  already exists (this also removes a workaround, that deletes the work
  folder path from the config dictionary before writing it)
* Download missing APKINDEX.tar.gz files with Python code, before
  attempting to build packages (so we know which ones aleady exist in
  the binary packages repository)
* Consider APKINDEX files older than 4 hours as outdated and download
  them again (also in Python code)
* Provide 'pmbootstrap update' to force-update the APKINDEX files
* Travis: more logging output on failure
* Only allow keys from config_keys to be used by "pmbootstrap config"
2017-11-19 15:04:08 +00:00
drebrez 94e2387af5 Add `pmbootstrap bootimg_analyze` / prompt during new device wizard (#905) 2017-11-19 14:35:23 +00:00
Oliver Smith 1bf892f5eb
Close #453: Support mesa-dri-virtio in Qemu (#861)
The mesa driver, which ends up in the installation image, needs to be known
before the installation is done (in other words: when running the qemu action,
it is to late as the image has already been generated). That's why one can
choose the Qemu mesa driver in `pmbootstrap init` now:

```
Device [qemu-amd64]:
Which mesa driver do you prefer for your Qemu device? Only select something other
than the default if you are having graphical problems (such as glitches).
Mesa driver (dri-swrast/dri-virtio) [dri-virtio]:
```

It is still possible to select `dri-swrast`, because `dri-virtio` may not work
in all cases, and that way we could easily debug it or experiment with other
mesa drivers (e.g. the "vmware" one, which is supported by mesa and Qemu).

Other changes:
* `pmbootstrap qemu` accepts a `--display` variable now, which passes the value
  directly to `qemu`'s `display` option. It defaults to `sdl,gl=on` (@PureTryOut
  reported that to work best with plasma mobile on his PC). `--display` and
  `--spice` (which is still working) are mutually exclusive.
* Removed obsolete telnet port pass-through: We only use the debug telnet port
  since osk-sdl has been merged.
* Add show-cursor to the Qemu command line, so it shows a cursor in X11
* Refactored the spice code (`command_spice` only returns the spice command,
  because it has all necessary information already) and the spice port can be
  specified on the commandline now (previously it was hardcoded in one place and
  then always looked up from there).
* Start comments with capital letters.
* Keep the log on the screen a bit shorter (e.g. Qemu command is written to the
  "pmbootstrap log" anyway, so there's no need to display it again).
* linux-postmarketos-stable: Adjust kernel configs
x86_64, armhf: enable as modules:
CONFIG_DRM_VIRTIO_GPU, CONFIG_VIRTIO_PCI, CONFIG_VIRTIO_BALLOON
aarch64: all 3 options were already enabled as built-in (no change)
* Set '-vga virtio' for mesa-dri-virtio
2017-11-05 13:48:49 +00:00
Oliver Smith 50faf5fe22
Close #865: Check line endings in APKBUILDs (#868) 2017-11-04 16:12:33 +00:00
drebrez ba3e2f6d26 Add argument to zap (-o) older versions of packages from all chroots (#841)
- also moved the mismatch clean (-m) before zapping the chroots
- added confirmation questions to -o and -m
2017-11-02 17:33:14 +00:00
drebrez 9c1ef821b3 Add architecture argument for the buildroot chroot (#832)
* Add architecture argument for the buildroot chroot, defaults to device architecture
* Output pmbootstrap log file after failure to debug Travis failure
* Travis: disable timestamp based rebuilds
2017-10-28 00:45:15 +00:00
drebrez 732a94fd45 Unregister all binfmt architectures during `pmbootstrap shutdown` (#822)
* Unregister all binfmt architectures during `pmbootstrap shutdown`
* Remove `-r, --rootfs` argument from `pmbootstrap build_init` command
2017-10-25 22:54:17 +00:00
drebrez 459f97ef1d Support multiple architectures in `pmbootstrap menuconfig` command (#814) 2017-10-24 16:18:42 +00:00
Oliver Smith 1285f74c5f Fix #731: Create symlinks for noarch-subpackages (#740)
* apkindex:
  * Also parse the architecture field
* symlink_noarch_package:
  * Renamed to symlink_noarch_packages
  * Always work on all packages (so we don't need to guess which
    subpackages have been generated after a certain build)
  * Get invoked when running 'pmbootstrap index'
  * Use 'apk index' to generate one index, where the architecture
    does not get rewritten (abuild does that by default, due to
    Alpine's repos not having a 'noarch' folder and diverging from
    that doesn't make things easier for us). That goes super fast,
    and then we know which packages are noarch packages and can
    create the symlinks.
* Made output less verbose:
  * Use -q for 'apk index' when calling it directly (when it gets
    called by abuild we can't control that)
  * Output that the APKINDEXes get reindexed only to the 'pmbootstrap
    log'.
2017-10-11 15:11:25 +00:00
Pablo Castellano 7a519a911e Fix: cannot run 'pmbootstrap config' when specified device doesn't exist (#689) 2017-10-03 18:37:50 +00:00
clayton craft d45c7d2d84 Add armv7l (armhf) as supported native arch. (#682) 2017-10-02 22:49:40 +00:00
Attila Szöllősi 3cbd0d19c2 Add support for isorec flashing in recovery installer (#609) 2017-09-28 22:05:00 +00:00
Pablo Castellano c855ee095b Qemu support for the QXL driver and SPICE (#481)
* pmb.helpers.run: support running processes in background
* enable QXL driver support in the linux kernel configurations so
that we can also use SPICE to connect to the VM.

QXL is a paravirtual graphics driver with 2D support

The SPICE project aims to provide a complete open source solution for remote
access to virtual machines in a seamless way.

Both DRM_QXL and DRM_BOCHS are enabled as modules.
According to [1], on Linux guests, the qxl and bochs_drm kernel modules
must be loaded in order to gain a decent performance

* qemu: add new option --spice to connect to VM using a SPICE client

If specified, 'pmbootstrap qemu' will look for some SPICE client in the
user's PATH and run qemu using the QXL driver.

Currently supported spice clients are 'spicy' and 'remote-viewer' but
adding support for more clients can be easily done.

qemu with qxl support will run on port 8077/tcp, which doesn't belong to
any well-known service and represents 'PM' in decimal.

References:
[0] https://www.linux-kvm.org/page/SPICE
[1] https://wiki.archlinux.org/index.php/QEMU#qxl
[2] https://wiki.archlinux.org/index.php/QEMU#SPICE
[3] https://github.com/postmarketOS/pmbootstrap/issues/453 (partially fixed)
2017-09-26 20:52:00 +00:00
Oliver Smith bf6f5cb884 Fix confusing error message for invalid makedepends (#540)
This happens currently, when a makedepend is invalid:
ERROR: UnboundLocalError: local variable 'pkgname' referenced before assignment

With this patch, you get the meaning full error that should have been printed instead:
ERROR: Could not find package 'invalid-package' in the aports folder and could not find it in any APKINDEX
2017-09-19 19:51:43 +00:00
clayton craft 558cd40fbf Adjust luksFormat options to reduce time to open luks device on boot (#430)
This is a workaround for #429, until the iteration count can be
specified directly in cryptsetup.

* Add default iter-time option, and option to override
* set SHA1 default hash for luksFormat, add option to override
* [RX51] load omap-sham in initramfs for HW accel. sha1
2017-09-19 19:46:18 +00:00
Attila Szöllősi ae6a58b6ed Close #554: kernel config checking (#589)
* Check kernel config
* Allow specifying multiple kernel packages, and also no packages
  which defaults to scanning all kernel configs (it is super fast
  anyway)
* Add the check to Travis CI
* Adjust existing kernel configs, so they pass the kconfig_check.
(We've had to put in a lot of defaults in the aarch64
linux-postmarketos configs, that's why the diff is a bit unclean.)
* Increase modified kernel pkgrels
2017-09-18 21:36:54 +00:00
Brian Otto 19531d518d Added a new option that allows you to increase the system image size for qemu (#568) 2017-09-15 15:41:40 +00:00
drebrez 207c200229 zap: add `-d`/`--distfiles` argument to clear downloaded files cache (#576) 2017-09-14 18:28:10 +00:00
drebrez a31cd0897b flasher flash_system: add possibility to select partition (#565)
Usage example:
pmbootstrap flasher flash_system --partition=userdata
2017-09-13 17:50:06 +00:00
Oliver Smith bed1eacbb5 Close #256: Implement strict package building mode (#532)
Contrary to abuild, pmbootstrap only installs makedepends, and
keeps the installed packages around - both hacks save lots of time.

However, they may introduce missing makedepends in the APKBUILDs,
that the authors of the APKBUILDs do not notice because it works
for them.

This PR adss a strict mode, which will always clean the chroots
before building a package, and also remove all installed dependencies
after the package was built. You can use the following syntax to
only zap once, but build many packages at once:
`pmbootstrap build --strict hello-world 0xffff heimdall`

It also builds dependencies properly without leaving makedepends
behind.
2017-09-08 23:50:59 +00:00
Oliver Smith b29cc877a7 Alias: "pmbootstrap export" for "pmbootstrap flasher export" (#417)
* moved export code to pmb/export and split it up
* added deprecation notice to "pmbootstrap flasher export"
* made "pmbootstrap export" work
* adjusted the "pmbootstrap flasher export" hints in the code
2017-09-06 20:14:03 +00:00
Yuval Adam b6003a2815 Close #327: Add initial setup.py (#443) 2017-09-02 19:30:40 +00:00
Attila Szöllősi c6fe6e134a Flavor instead of device name when creating recovery zip (#491)
* Flavor instead of device name when creating recovery zip.
* Move parse_flavor_arg function
2017-09-02 03:53:58 +00:00
clayton craft a9e5b362dc Implement zapping of apk packages with ver different than aports (466) (#474)
This adds a new option to `zap`: `-m / --mismatch-bins`

When set, any binary apks in the work directory packages folder will be
removed if their version differs from the version in the relevant
APKBUILD in aports.
2017-08-28 20:34:03 +00:00
clayton craft 3942a06a2d Add -y/--assume-yes option to install (#450) (#467) 2017-08-26 15:50:04 +00:00
Attila Szöllősi fbe968f1ab Recovery installer zip (#404)
Supports flashing with TWRP and other Android recovery OS through adb sideload,
as well as exporting a generated recovery zip file.
See also:
https://github.com/postmarketOS/pmbootstrap/wiki/deviceinfo_flash_methods#recovery-mode-adb
2017-08-24 21:07:36 +00:00
Oliver Smith c323f21ca5 (binary repo reated) Don't specify pkgnames from "provides" as dependencies (#416)
* Don't specify pkgnames from "provides" as dependencies

Always use the regular pkgname. That way, we avoid listing all
kinds of so: files as dependencies (because Alpine automatically
adds them as depends= to the package database). This fixes building
weston, and reproducing the build with `pmbootstrap challenge`.

Additional changes.
* Clear the parsed APKINDEX cache for the current pmbootstrap
  session after building a package
* Avoid rebuilding a package, in case it was already built due to
  circular dependencies
2017-08-19 12:52:11 +00:00
NotKit 689eb5c3c9 Fix APKBUILD attribute parsing when opening quote is followed by line break (#415)
his adds a check to continue attribute parsing when end_char is found
in the end of first line, but is present only once. Found in gtk+2.0 APKBUILD:
https://git.alpinelinux.org/cgit/aports/plain/main/gtk+2.0/APKBUILD
2017-08-19 01:15:28 +00:00
Oliver Smith 1c13ca4fd9 Fix #363: Omit /home/user when calculating system image size (#389)
* The system image size is now calculated as: root size - home size.
* New function in `pmb/helpers/other.py`: `folder_size()`, with a
  testcase.
* Instead of copying everything to the system image folder, and
  deleting the home folder afterwards, do not copy the home folder
  in the first place.
* Added `pmbootstrap -s` to skip generating the initramfs for faster
  debugging.
* Set the default value in the "are you sure, that your partition has
  at least..." to "y", so we can run `yes '' | pmbootstrap install`
  to make it run through the whole installation process.
* Increase full size to 120%, boot partition gets 15 MB free space now
2017-08-18 19:19:48 +00:00
Anton Lazarev fa3c21c1c3 Now with 100% more `pmbootstrap log`! (#406) 2017-08-17 23:27:06 +00:00
Pablo Castellano 401c29af76 Close #326: Implement command to retrieve and set configuration values (#359)
* Implement command to retrieve and set configuration values
* qemu: show advice to use "pmbootstrap config"
* Allow "pmbootstrap config" without positional arguments (prints the full config)
* Check valid variable names
2017-08-14 14:25:28 +00:00
Oliver Smith f3f21d3152 Fix #342: don't use distutils.version.LooseVersion anymore (#364)
Previously, distutils.version.LooseVersion was used, because it was
sort of close enough to how Alpine parses versions.

This new version uses the exact same algorithm, as `apk` does, and
it passes *all* of `apk`'s testcases for version checking (previously
we simply skipped the ones, that did not pass).

* Remove pmb/helpers/version.py left-over (it is in parse now)
* Make asserts consistent, do not use unnecessary parenthesis
2017-08-12 14:03:40 +00:00
Pablo Castellano 49b35ad32d Close #226: Launch postmarketOS in a qemu virtual machine (#350)
Thanks to Pablo Castellano and Martijn Braam!
In postmarketOS we are now able to generate system images with the
correct configuration so that they can boot already using qemu

This commit brings the `pmbootstrap qemu` action.
This command is very handy because you don't have to set all the
qemu parameters, pmbootstrap does it for you.

* device-qemu-vexpress: Added kernel command line according to wiki
* qemu: Added workaround for image writing permissions
* qemu: Added support to launch postmarketOS in a QEMU virtual machine

- Support for emulating these architectures in QEMU: arm, aarch64, x86_84
- Generate QEMU command correctly depending no guest architecture (arm/x86)
- Run QEMU in the same architecture as the host by default
- Refactoring in pmb.parse.arch and pmb.qemu.run
- Raise exception if DTB file or system image are not present
- Display more useful information when something fails (e.g. image not found)
- Run qemu version depending on arch (host or argument), not device configured

* device-qemu-amd64: set deviceinfo_kernel_cmdline to "PMOS_NO_OUTPUT_REDIRECT"
* qemu: added --memory argument to specific guest RAM
* device-qemu-amd64: adjusted deviceinfo_kernel_cmdline (console=tty1)
* Added /etc/network/interfaces for qemu-amd64
* qemu: Added KVM support if /dev/kvm if present
* Specify separate machines for architecture
* qemu: Check if QEMU is installed instead of crashing
* Added graphics driver to qemu-aarch64

- Use arm (as used in qemu) instead of armhf (used in Alpine)
- qemu argument is -dtb
- Follow same style to build the command + arguments

* qemu: Added SSH port redirection: ./pmbootstrap.py qemu -p 2222
2017-08-09 20:26:40 +00:00
Oliver Smith 21c09b3b3c Fix various build issues (fix #189, fix #341) (#345)
Changes:
* Removed the apkindex_files cache. That particular cache caused
  bug #189 and didn't bring any real-world performance improvements
  (tested 3x with that cache and 3x without, no significant speed
  difference). I decided to remove it instead of keeping it/adding
  even more code to resolve the bug.
* Fix the check for already built packages: always use the architecture,
  that the package should be built for instead of the architecture of
  the build environment (e.g. use armhf, even when building a noarch
  package in the x86_64 chroot). This partially resolves #341.
* Make pmb.chroot.apk.install_is_necessary() more robust: If the binary
  package is missing, although it should be there, print a warning and
  build it with force.
2017-08-09 17:59:21 +00:00
Oliver Smith ae64894278
Fix #349: Verify pkgname against folder name after replacing vars
This fixes a regression introduced in 7fe2bc18c3.
Sorry for the inconvenience!
2017-08-08 15:03:17 +02:00
Oliver Smith 7fe2bc18c3 Close #334: Check if pkgname is the aport folder name (#347)
...and remove an redundant sentence with broken grammar in another
error message.
2017-08-07 22:53:54 +00:00
drebrez 257666799e Add `-c, --clear` argument for `log` and `log_distccd` actions to clear log files (#344) 2017-08-07 18:56:35 +00:00
Oliver Smith 025d646e47 Close #314: Add --details-to-stdout parameter (#317)
When this parameter is set, `args.logfd` (the log file descriptor)
gets set to `sys.stdout`, so all logging gets directly sent to the
active pmbootstrap session, and not to a log file.

This will be used for Travis jobs, because they get killed after
10 minutes without any output (and builds may take longer).
2017-08-02 19:38:42 +00:00