This extends zap() to add a 'no_confirm' option (False by default), and
zap() is now called by init with no_confirm=True to automatically zap
any existing chroots after the user runs init. This helps insure that
what is installed in the chroots is exactly what the user expects after
setting options in init.
Additionally, we create `cache_http` to verify write access to the work
folder instead of `chroot_native`. So we can ask for zapping only if
no chroot folder exists.
I've replaced all instances in the code of `os.path.abspath`
with `os.path.realpath`, as this does the same as `abspath`
plus resolving symlinks.
See also: https://stackoverflow.com/a/40311142
* Implement command to retrieve and set configuration values
* qemu: show advice to use "pmbootstrap config"
* Allow "pmbootstrap config" without positional arguments (prints the full config)
* Check valid variable names
Thanks to Pablo Castellano and Martijn Braam!
In postmarketOS we are now able to generate system images with the
correct configuration so that they can boot already using qemu
This commit brings the `pmbootstrap qemu` action.
This command is very handy because you don't have to set all the
qemu parameters, pmbootstrap does it for you.
* device-qemu-vexpress: Added kernel command line according to wiki
* qemu: Added workaround for image writing permissions
* qemu: Added support to launch postmarketOS in a QEMU virtual machine
- Support for emulating these architectures in QEMU: arm, aarch64, x86_84
- Generate QEMU command correctly depending no guest architecture (arm/x86)
- Run QEMU in the same architecture as the host by default
- Refactoring in pmb.parse.arch and pmb.qemu.run
- Raise exception if DTB file or system image are not present
- Display more useful information when something fails (e.g. image not found)
- Run qemu version depending on arch (host or argument), not device configured
* device-qemu-amd64: set deviceinfo_kernel_cmdline to "PMOS_NO_OUTPUT_REDIRECT"
* qemu: added --memory argument to specific guest RAM
* device-qemu-amd64: adjusted deviceinfo_kernel_cmdline (console=tty1)
* Added /etc/network/interfaces for qemu-amd64
* qemu: Added KVM support if /dev/kvm if present
* Specify separate machines for architecture
* qemu: Check if QEMU is installed instead of crashing
* Added graphics driver to qemu-aarch64
- Use arm (as used in qemu) instead of armhf (used in Alpine)
- qemu argument is -dtb
- Follow same style to build the command + arguments
* qemu: Added SSH port redirection: ./pmbootstrap.py qemu -p 2222
Changes:
* Removed the apkindex_files cache. That particular cache caused
bug #189 and didn't bring any real-world performance improvements
(tested 3x with that cache and 3x without, no significant speed
difference). I decided to remove it instead of keeping it/adding
even more code to resolve the bug.
* Fix the check for already built packages: always use the architecture,
that the package should be built for instead of the architecture of
the build environment (e.g. use armhf, even when building a noarch
package in the x86_64 chroot). This partially resolves#341.
* Make pmb.chroot.apk.install_is_necessary() more robust: If the binary
package is missing, although it should be there, print a warning and
build it with force.
When this parameter is set, `args.logfd` (the log file descriptor)
gets set to `sys.stdout`, so all logging gets directly sent to the
active pmbootstrap session, and not to a log file.
This will be used for Travis jobs, because they get killed after
10 minutes without any output (and builds may take longer).
* Refactored `umount_all` to get the list of folders to be umounted from
`umount_all_list`, so we can test that function in a test case.
* Adjusted `umount_all_list` to return the deep folder levels first
* Added a testcase for that
* Remove redundant calls to `umount_all()` (which were from a time before
`pmbootstrap` was released, in which failing commands did not cause
`pmbootstrap` to raise an exception)
* postmarketos-splash: change arch from "all" to "noarch"
* lg-mako: use .tar.gz instead of .zip (because that's the reference
APKBUILD)
* #220: Allow specifying multiple packages for checksum, build,
aportgen
* #239: Add chroot shortcuts (--rootfs and --buildroot)
* Show chroot and command before entering chroot
* Validate all inputs from `pmbootstrap init`
* Add a new `confirm()` function, that validates input of yes/no
questions properly
* All questions loop until they have a valid answer now
Thanks, craftyguy!
- UI is selectable with `pmbootstrap init`. Currently only 'weston' and 'none' are options. It'll automatically pick up any new `postmarketos-ui-*` packages added at later dates.
- splits off weston packages install from postmarketos-base and puts
them into postmarketos-ui-weston. Also note that NO weston packages are
installed by "pmbootstrap install" by default unless the user selects a ui in the `init`
- configuration of weston is now in postmarketos-ui-weston.
- the demos have been spun off to `postmarketos-demos`, and `postmarketos-ui-weston` lists this package as a dependency.
* Update runtime error message thrown for grsec/hardened kernels, saying that they are not supported right now, #107
* Remove unnecessary check for grsec enable/disable status - the build will not work in either case
* Ignore `>`, `<`, `=` and `!` operators, when they are specified in
the dependencies. This was the desired behavior before, but it was
not implemented correctly (so it wouldn't ignore them everywhere).
Of course the real fix would be to honor these operators like apk
does. But this isn't feasible right now, and it should work for
most, if not all, our use-cases. I have documented this in the wiki
under build internals and if we happen to need correct operator
handling, we should do it then.
Minor other changes:
* `pmbootstrap parse_apkindex`: support optional package parameter to
only show the parsed content for one package.
* Support building most python APKBUILDs by replacing ${pkgname#py-}
properly
This is required for developing and testing the binary repository
scripts (see #64). Changes:
* When specified, the local folder gets mounted inside the chroots
as /mnt/postmarketos-mirror
* The apkindex_files() function outputs the correct path to the local
repository (it does *not* hash the URL in that case, which would
be wrong)
* /etc/apk/repositories: when the pmOS mirror is a local folder,
the path "/mnt/postmarketos-mirror" gets added to that file instead
of the outside path (so apk finds it properly inside the chroot)
TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution.
---
pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has.
I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository.
To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on.
In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option.
In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds".
This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before.
Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases).
The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
* New commandline parameter --mirror-pmOS, where the binary repository
URL for postmarketOS can be specified (empty by default as of now,
this will be filled with the real URL once the repo works)
* Do not build packages, when they are in the binary repository and
the version of the package in the binary repository is up-to-date.
* Add a testcase for pmb.build.is_necessary().
...even if the pkgver and pkgrel have *not* changed. This should
make development much more intuitive. The detection works by looking
at the last modified timestamps, just like `make` does it.
* Two new functions for getting a list of files and their timestamps
in the repo, and diffing that information to get a list of changed
files: pmb.helpers.repo.files() and pmb.helpers.repo.diff().
(I've put it in the helpers folder, because it is not specific to
one chroot, but to all chroots at once.)
* pmbootstrap challenge (new command introduced a few commits back to
verify, that the contents of an APK file are deterministic) uses
these functions to a) support subpackages and b) optionally
output a list of changed files (this gets used in the pmbuilder
script, which lives outside of this repository).
This commit is progress for #64 again.
* The check for valid sdcard paths has been removed (because it
doesn't make sense for /dev/sd*, this might as well be a real
harddrive)
* New check: abort if any partition of the sdcard is mounted
* Automatically detect if the partitions are called "p1", "p2" etc.
or just "1", "2" etc. (/dev/sda1 vs. /dev/mmcblk0p1)
* pmb.helpers.mount.ismount(): check the source (new) and the target
path in /proc/mounts
This reverts commit 04aabd7d70.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
This reverts commit 2a39d9c091.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
This reverts commit c15bf48f77.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
This reverts commit f39c1dae69.
* automatically find the chroot binary on Debian, even if it is not
in the user's PATH
* don't use subprocess.run anymore (remove related testcase, that explicitly
checked for subprocess.run usage, and used recursive globbing, another
post 3.4 Python feature, for the checks. A similar case can be added in the
future, but right now it's more important to get Debian 3.4 working and all
PRs are reviewed anyway.)
* pytest fixtures: don't use the newer "yield" feature, as this is only
supported in a newer version of pytest, than provided on Debian Jessie
From manually testing, most stuff works in Debian Jessie. However, the
testsuite does not run through - creating an empty .tar.gz with Python
fails for some reason (this is done in test_apk_static.py).
clayton craft
Oliver Smith
Pablo Castellano
drebrez
Martijn Braam
cmdr2