Ask for release channel and switch pmaports branch to the related branch
defined in channels.cfg.
Store in pmbootstrap.cfg whether the user chose a channel (boolean). If
the user did not choose a channel yet, suggest the recommended channel
from channels.cfg (currently "edge").
The ui-extras questions will attempt to find a postmarketos-ui-<ui>
package in pmaports. If the package does not exist as "root" APKBUILD
it currently attempts to parse all APKBUILDs in case it is somewhere
defined as a subpackage. This is really slow (up to 2-3 seconds),
which feels weird during "pmbootstrap init".
For the UI packages we specifically look for the root UI package,
not the subpackage, so let's skip searching for subpackages in this
case. This makes selecting the "none" UI nice and fast again.
In the future, device ports will be located in a subdirectory
below device/... (e.g. device/testing/device-...).
Replace all occurrences of device/* with a glob that checks the
subdirectories instead.
Note: To ensure that this always works properly we should also add some
checks that all devices are indeed located under one of the supported
subdirectories (i.e. testing/community/main).
Change the glob for pmaports to <aports>/**/APKBUILD.
This allows using subdirectories for organization outside of device/
as well.
mesa-dri-swrast and mesa-dri-virtio are both provided by mesa-dri-gallium
now, so this option does not have much use anymore. With both selections,
exactly the same packages are installed.
Add new "pmbootstrap status" command, which does a quick health check
for the work dir. As first health check, verify that the chroots are not
too old. Replace the reminder text at the end of "pmbootstrap init" to
tell users to run "pmbootstrap status" instead of "pmbootstrap zap" once
a day before working with pmbootstrap.
Related: #1829
While at it, also remove unnecessary "#!/usr/bin/env python3" in files
that only get imported, and adjust other empty/comment lines in the
beginnings of the files for consistency.
This makes files easier to read, and makes the pmbootstrap codebase more
consistent with the build.postmarketos.org codebase.
At the moment we have a simple subpkgdesc() function that can only
parse "pkgdesc" from subpackages, without support for any variables.
But we have a quite nice variable parser now that can be extended
to work for subpackages.
Simply put this works by:
- Finding the lines that belong to the subpackage function
- Stripping indentation (tab)
- Parsing relevant attributes similar to the apkbuild() function
The "subpackages" in the parsed APKBUILD are replaced by a dict
of subpkgname: {"pkgdesc": "...", "depends": "..."} which are
parsed from the subpackage function (if found).
This makes it possible to get the "depends" of a subpackage.
Do not install git in the native chroot and use it from there. Remove the
chown_to_user argument from pmb.helpers.git.clone(), the resulting dir
is now always owned by the user. While at it, refactor the function and
display the clone URL.
Previously we had cloned aports_upstream (from Alpine) with
chown_to_user=False (legacy) and pmaports with chown_to_user=True.
pmb.helpers.git.rev_parse() would only work after chown_to_user=True.
Check if git is installed in "pmbootstrap init", and remove the same
check from rev_parse(). Add a new work dir version, that checks for git
and changes ownership of already checked out aports_upstream to the
host system's user.
When creating a new work dir, create cache_git instead of cache_http.
cache_http is created on demand already, with proper permissions. But
cache_git must be created, otherwise pmb.helpers.mount.bind will create it
as root.
This is in preparation for the "pmbootstrap pull" feature, as it allows
using the host system's git in all new code paths. We will be able to
handle repositories even if they were cloned outside of the work dir
(which we do in a few CI scripts for example).
Related: #1858
When running pmbootstrap init, first select device vendor, then device
codename. Also fixed tests for new behavior and added some new ones for
new scenarios.
From https://docs.python.org/3/library/re.html
"Backslashes are not handled in any special way
in a string literal prefixed with 'r'."
Signed-off-by: Steffen Pankratz <kratz00@gmx.de>
Test with flake8: *.py
./test/check_checksums.py:13:13: E117 over-indented
./pmb/config/init.py:97:8: F632 use ==/!= to compare str, bytes, and int literals
./pmb/parse/arguments.py:229:13: E117 over-indented
Rename pmb.helpers.devices.list() to
pmb.helpers.devices.list_codenames(). Python already has a list()
function, so we name our function to make the calls to the codenames
listing function inside pmb/helpers/devices.py less confusing.
Make it possible to point the pmaports dir to an existing repository
clone, while not having to give the --aports/-p parameter with every
command.
pmbootstrap --aports=/my/pmaports/path init
If the config file exists already, the dir can also be set with:
pmbootstrap config aports /my/pmaports/path
Prior to this commit, it was possible to type in packages with a
trailing comma in `pmbootstrap init` when asked for extra packages.
This leads to problems during `pmbootstrap install`, so now we disallow
it. Fixes#1540.
* As discussed in IRC/matrix, we're removing `linux-postmarketos-lts`
for now. The kernel isn't used right now, and we save lots of
maintenance effort with not updating it every week or so.
* new config option `"kernel"` with possible values:
`"downstream", "mainline", "stable"` (downstream is always
`linux-$devicename`)
* ask for the kernel during `pmbootstrap init` if the device package
has kernel subpackages and install it in `_install.py`
* postmarketos-mkinitfs: display note instead of exit with error when
the `deviceinfo_dtb` file is missing (because we expect it to be
missing for downstream kernels)
* device-sony-amami:
* add kernel subpackages for downstream, mainline
* set `deviceinfo_dtb`
* device-qemu-amd64: add kernel subpackages for stable, lts, mainline
* test cases and test data for new functions
* test case that checks all aports for right usage of the feature:
* don't mix specifying kernels in depends *and* subpackages
* 1 kernel in depends is maximum
* kernel subpackages must have a valid name
* Test if devices packages reference at least one kernel
* Remove `_build_device_depends_note()` which informs the user that
`--ignore-depends` can be used with device packages to avoid building
the kernel. The idea was to make the transition easier after a change
we did months ago, and now the kernel doesn't always get built before
building the device package so it's not relevant anymore.
* pmb/chroot/other.py:
* Add autoinstall=True to kernel_flavors_installed(). When the flag
is set, the function makes sure that at least one kernel for the
device is installed.
* Remove kernel_flavor_autodetect() function, wherever it was used,
it has been replaced with kernel_flavors_installed()[0].
* pmb.helpers.frontend.py: remove code to install at least one kernel,
kernel_flavors_installed() takes care of that now.
As noted in commit 255c715624
`/var/cache/distfiles` is writable by everyone. It is supposed to be
writable only by `root` and by the `abuild` group (in which we put the
`pmos` user already for building packages).
Changes:
* `pmb.build.init()`: make `/var/cache/distfiles` writable only by
members of the `abuild` group (and root)
* Increase workfolder version to 2
* Add migration code that fixes the permissions for existing work
folders
* Refactor the migration code a bit to make this possible
* Save "" (empty string) in the user's config as hostname if the user
let it default to the name of the device. That way, when the device
gets changed, the user won't get the old device's name as hostname
by accident.
* Add a test case
Here are the changes necessary in pmbootstrap to make proprietary
software installed onto the device (firmware and userspace drivers)
optional (#756). To full close the issue, we need to apply this concept
to all device packages we already have in a follow-up PR.
Changes:
* New config file options nonfree_firmware and nonfree_userland, which
we ask for during "pmbootstrap init" if there are non-free components
for the selected device.
* We find that out by checking the APKBUILD's subpakages: The non-free
packages are called $pkgname-nonfree-firmware and
$pkgname-nonfree-userland.
* During "pmbootstrap init" we also show the pkgdesc of these
subpackages. Parsing that is implemented in
pmb.parse._apkbuild.subpkgdesc(). It was not implemented as part of
the regular APKBUILD parsing, as this would need a change in the
output format, and it is a lot *less* code if done like in this
commit.
* pmb/parse/apkbuild.py was renamed to _apkbuild.py, and
pmb/install/install.py to _install.py: needed to call the function in
the usual way (e.g. pmb.parse.apkbuild()) but still being able to
test the individual functions from these files in the test suite.
We did the same thing for pmb/build/_package.py already.
* Install: New function get_nonfree_packages() returns the non-free
packages that will be installed, based on the user's choice in
"pmbootstrap init" and on the subpackages the device has.
* Added test cases and test data (APKBUILDs) for all new code,
refactored test/test_questions.py to have multiple functions for
testing the various questions / question types from
"pmbootstrap init" instead of having it all in one big function.
This allows to use another aport folder for testing the new
non-free related questions in init.
If you want to build a package without changing the version number,
please use `--force` from now on. For example:
pmbootstrap build --force hello-world
Prior to this commit, changes were detected automatically (timestamp
based rebuilds). However, that feature does not work as expected with
the binary package repository we have now, and depending on how you use
git, it has never worked. Close#1167, close#1156, close#1023 and
close#985. This commit also mentions --force when a package is up to date,
but the user requested to build it.
* Don't ask for the mesa driver when the Qemu arch is not the
native arch and always use swrast in that case
* qemu-vexpress: use LTS kernel
* qemu-aarch64: use drm-backend for weston
This fixes#1066, where pmbootstrap crashes with a permission error
when you run it for the first time (no work folder exists) and you
run the boot.img analyzer because you want to start a new device
port.
It also prints a more helpful message if pmbootstrap crashes before
the log file was generated (suggests to use --details-to-stdout).
* ccache: Fix for distcc cross-compiling / various improvements
* Make ccache work when cross-compiling with distcc (fix#716)
* Allow to configure the ccache size in "pmbootstrap init"
* Moved ccache stats code from pmb/build/other.py to
pmb/helpers/frontend.py
* Grouped job count, ccache size and timestamp based rebuilds
together to "build options" and allow to skip them
* Sorted config options that had to be modified anyway
alphabetically
* Improve comment in arch-bin-masquerade APKBUILD
* add my own build key
* enable the repo in the config
* update the README file
* Adjust testcase, that validates the keys and enable it in testcases_fast.sh
* Only save/load keys to/from the config file, which we ask for during
'pmbootstrap init', so the binary repo gets used even if a config file
already exists (this also removes a workaround, that deletes the work
folder path from the config dictionary before writing it)
* Download missing APKINDEX.tar.gz files with Python code, before
attempting to build packages (so we know which ones aleady exist in
the binary packages repository)
* Consider APKINDEX files older than 4 hours as outdated and download
them again (also in Python code)
* Provide 'pmbootstrap update' to force-update the APKINDEX files
* Travis: more logging output on failure
* Only allow keys from config_keys to be used by "pmbootstrap config"
The mesa driver, which ends up in the installation image, needs to be known
before the installation is done (in other words: when running the qemu action,
it is to late as the image has already been generated). That's why one can
choose the Qemu mesa driver in `pmbootstrap init` now:
```
Device [qemu-amd64]:
Which mesa driver do you prefer for your Qemu device? Only select something other
than the default if you are having graphical problems (such as glitches).
Mesa driver (dri-swrast/dri-virtio) [dri-virtio]:
```
It is still possible to select `dri-swrast`, because `dri-virtio` may not work
in all cases, and that way we could easily debug it or experiment with other
mesa drivers (e.g. the "vmware" one, which is supported by mesa and Qemu).
Other changes:
* `pmbootstrap qemu` accepts a `--display` variable now, which passes the value
directly to `qemu`'s `display` option. It defaults to `sdl,gl=on` (@PureTryOut
reported that to work best with plasma mobile on his PC). `--display` and
`--spice` (which is still working) are mutually exclusive.
* Removed obsolete telnet port pass-through: We only use the debug telnet port
since osk-sdl has been merged.
* Add show-cursor to the Qemu command line, so it shows a cursor in X11
* Refactored the spice code (`command_spice` only returns the spice command,
because it has all necessary information already) and the spice port can be
specified on the commandline now (previously it was hardcoded in one place and
then always looked up from there).
* Start comments with capital letters.
* Keep the log on the screen a bit shorter (e.g. Qemu command is written to the
"pmbootstrap log" anyway, so there's no need to display it again).
* linux-postmarketos-stable: Adjust kernel configs
x86_64, armhf: enable as modules:
CONFIG_DRM_VIRTIO_GPU, CONFIG_VIRTIO_PCI, CONFIG_VIRTIO_BALLOON
aarch64: all 3 options were already enabled as built-in (no change)
* Set '-vga virtio' for mesa-dri-virtio
* pmbootstrap init: Generate new port device- and linux-package
* adds `pmbootstrap aportgen device-*` and
`pmbootstrap aportgen linux-*`
* ask for confirmation when selecting a non-existing device
* generate the packages directly from init
* refactor aportgen code
* fixed some easy things in the linux- APKBUILD (more to come in
follow-up PRs!)
Testing:
* Test all questions to the user from pmb.config.init and pmb.aportgen.device
(except for the timezone question, because we would need to monkeypatch the
os.path.exists() function, which messes up pytest, so we'd need to refactor
the timezone function to be more testsuite friendly first)
* Run the device wizard in a testcase a few times and check the output, that
pmbootstrap.aportgen.device and pmbootstrap.aportgen.linux create by parsing
the resulting APKBUILDs and deviceinfo and checking its contents.
* Build the generated device package once in the same testcase
Thanks a lot to @drebrez for all the help with this one:
<https://github.com/postmarketOS/pmbootstrap/pull/821>
See also the updated porting guide:
<https://wiki.postmarketos.org/wiki/Porting_to_a_new_device>
This way we could give the user a rough idea what will be installed,
and also use this to display a short warning about long compile times
(e.g. until the plasma mobile stuff is upstreamed).
* Allow to specify a custom username in "pmbootstrap init"
* Build chroots have "pmos" instead of "user" as username now
* Installation user UID is 1000 now (as in all other Linux distributions)
* Adjust autologins
* postmarketos-base: enable wheel group for sudo, removed previous sudoers file
* Implement safe upgrade path:
We save the version of the work folder format now, in $WORK/version.
When this file does not exist, it defaults to 0.
In case it does not match the currently required version
(pmb.config.work_version), then ask the user if it should
automatically be upgraded.
The device chroot shouldn't exist yet anyways. Also add a 'device_exists'
variable that can be used to determine if the device entered earlier in
the init has a deviceinfo under aports/device/device-*
This extends zap() to add a 'no_confirm' option (False by default), and
zap() is now called by init with no_confirm=True to automatically zap
any existing chroots after the user runs init. This helps insure that
what is installed in the chroots is exactly what the user expects after
setting options in init.
Additionally, we create `cache_http` to verify write access to the work
folder instead of `chroot_native`. So we can ask for zapping only if
no chroot folder exists.
Oliver Smith
Minecrell
Daniele Debernardi
Alexey Min
Steffen Pankratz
Luca Weiss
Robert Yang
Oliver Smith
Craig Comstock
Grant Miller
clayton craft
Pablo Castellano
Brian Otto
Martijn Braam