* apkindex:
* Also parse the architecture field
* symlink_noarch_package:
* Renamed to symlink_noarch_packages
* Always work on all packages (so we don't need to guess which
subpackages have been generated after a certain build)
* Get invoked when running 'pmbootstrap index'
* Use 'apk index' to generate one index, where the architecture
does not get rewritten (abuild does that by default, due to
Alpine's repos not having a 'noarch' folder and diverging from
that doesn't make things easier for us). That goes super fast,
and then we know which packages are noarch packages and can
create the symlinks.
* Made output less verbose:
* Use -q for 'apk index' when calling it directly (when it gets
called by abuild we can't control that)
* Output that the APKINDEXes get reindexed only to the 'pmbootstrap
log'.
* pmb.helpers.run: support running processes in background
* enable QXL driver support in the linux kernel configurations so
that we can also use SPICE to connect to the VM.
QXL is a paravirtual graphics driver with 2D support
The SPICE project aims to provide a complete open source solution for remote
access to virtual machines in a seamless way.
Both DRM_QXL and DRM_BOCHS are enabled as modules.
According to [1], on Linux guests, the qxl and bochs_drm kernel modules
must be loaded in order to gain a decent performance
* qemu: add new option --spice to connect to VM using a SPICE client
If specified, 'pmbootstrap qemu' will look for some SPICE client in the
user's PATH and run qemu using the QXL driver.
Currently supported spice clients are 'spicy' and 'remote-viewer' but
adding support for more clients can be easily done.
qemu with qxl support will run on port 8077/tcp, which doesn't belong to
any well-known service and represents 'PM' in decimal.
References:
[0] https://www.linux-kvm.org/page/SPICE
[1] https://wiki.archlinux.org/index.php/QEMU#qxl
[2] https://wiki.archlinux.org/index.php/QEMU#SPICE
[3] https://github.com/postmarketOS/pmbootstrap/issues/453 (partially fixed)
We check if origin/HEAD is present. In case that reference is
missing, we show a meaningful error message now, with an explanation
on how to add it. Also moved find_out_of_sync_files_tracked() to
pmb.helpers.git
* Check kernel config
* Allow specifying multiple kernel packages, and also no packages
which defaults to scanning all kernel configs (it is super fast
anyway)
* Add the check to Travis CI
* Adjust existing kernel configs, so they pass the kconfig_check.
(We've had to put in a lot of defaults in the aarch64
linux-postmarketos configs, that's why the diff is a bit unclean.)
* Increase modified kernel pkgrels
Contrary to abuild, pmbootstrap only installs makedepends, and
keeps the installed packages around - both hacks save lots of time.
However, they may introduce missing makedepends in the APKBUILDs,
that the authors of the APKBUILDs do not notice because it works
for them.
This PR adss a strict mode, which will always clean the chroots
before building a package, and also remove all installed dependencies
after the package was built. You can use the following syntax to
only zap once, but build many packages at once:
`pmbootstrap build --strict hello-world 0xffff heimdall`
It also builds dependencies properly without leaving makedepends
behind.
* moved export code to pmb/export and split it up
* added deprecation notice to "pmbootstrap flasher export"
* made "pmbootstrap export" work
* adjusted the "pmbootstrap flasher export" hints in the code
This adds a new option to `zap`: `-m / --mismatch-bins`
When set, any binary apks in the work directory packages folder will be
removed if their version differs from the version in the relevant
APKBUILD in aports.
* Write custom os-release (closes#324)
* Return empty string instead of crashing when the directory is not a git repository
* Use https in homepage URL
* The system image size is now calculated as: root size - home size.
* New function in `pmb/helpers/other.py`: `folder_size()`, with a
testcase.
* Instead of copying everything to the system image folder, and
deleting the home folder afterwards, do not copy the home folder
in the first place.
* Added `pmbootstrap -s` to skip generating the initramfs for faster
debugging.
* Set the default value in the "are you sure, that your partition has
at least..." to "y", so we can run `yes '' | pmbootstrap install`
to make it run through the whole installation process.
* Increase full size to 120%, boot partition gets 15 MB free space now
This extends zap() to add a 'no_confirm' option (False by default), and
zap() is now called by init with no_confirm=True to automatically zap
any existing chroots after the user runs init. This helps insure that
what is installed in the chroots is exactly what the user expects after
setting options in init.
Additionally, we create `cache_http` to verify write access to the work
folder instead of `chroot_native`. So we can ask for zapping only if
no chroot folder exists.
I've replaced all instances in the code of `os.path.abspath`
with `os.path.realpath`, as this does the same as `abspath`
plus resolving symlinks.
See also: https://stackoverflow.com/a/40311142
* Implement command to retrieve and set configuration values
* qemu: show advice to use "pmbootstrap config"
* Allow "pmbootstrap config" without positional arguments (prints the full config)
* Check valid variable names
Thanks to Pablo Castellano and Martijn Braam!
In postmarketOS we are now able to generate system images with the
correct configuration so that they can boot already using qemu
This commit brings the `pmbootstrap qemu` action.
This command is very handy because you don't have to set all the
qemu parameters, pmbootstrap does it for you.
* device-qemu-vexpress: Added kernel command line according to wiki
* qemu: Added workaround for image writing permissions
* qemu: Added support to launch postmarketOS in a QEMU virtual machine
- Support for emulating these architectures in QEMU: arm, aarch64, x86_84
- Generate QEMU command correctly depending no guest architecture (arm/x86)
- Run QEMU in the same architecture as the host by default
- Refactoring in pmb.parse.arch and pmb.qemu.run
- Raise exception if DTB file or system image are not present
- Display more useful information when something fails (e.g. image not found)
- Run qemu version depending on arch (host or argument), not device configured
* device-qemu-amd64: set deviceinfo_kernel_cmdline to "PMOS_NO_OUTPUT_REDIRECT"
* qemu: added --memory argument to specific guest RAM
* device-qemu-amd64: adjusted deviceinfo_kernel_cmdline (console=tty1)
* Added /etc/network/interfaces for qemu-amd64
* qemu: Added KVM support if /dev/kvm if present
* Specify separate machines for architecture
* qemu: Check if QEMU is installed instead of crashing
* Added graphics driver to qemu-aarch64
- Use arm (as used in qemu) instead of armhf (used in Alpine)
- qemu argument is -dtb
- Follow same style to build the command + arguments
* qemu: Added SSH port redirection: ./pmbootstrap.py qemu -p 2222
Changes:
* Removed the apkindex_files cache. That particular cache caused
bug #189 and didn't bring any real-world performance improvements
(tested 3x with that cache and 3x without, no significant speed
difference). I decided to remove it instead of keeping it/adding
even more code to resolve the bug.
* Fix the check for already built packages: always use the architecture,
that the package should be built for instead of the architecture of
the build environment (e.g. use armhf, even when building a noarch
package in the x86_64 chroot). This partially resolves#341.
* Make pmb.chroot.apk.install_is_necessary() more robust: If the binary
package is missing, although it should be there, print a warning and
build it with force.
When this parameter is set, `args.logfd` (the log file descriptor)
gets set to `sys.stdout`, so all logging gets directly sent to the
active pmbootstrap session, and not to a log file.
This will be used for Travis jobs, because they get killed after
10 minutes without any output (and builds may take longer).
* Refactored `umount_all` to get the list of folders to be umounted from
`umount_all_list`, so we can test that function in a test case.
* Adjusted `umount_all_list` to return the deep folder levels first
* Added a testcase for that
* Remove redundant calls to `umount_all()` (which were from a time before
`pmbootstrap` was released, in which failing commands did not cause
`pmbootstrap` to raise an exception)
* postmarketos-splash: change arch from "all" to "noarch"
* lg-mako: use .tar.gz instead of .zip (because that's the reference
APKBUILD)
* #220: Allow specifying multiple packages for checksum, build,
aportgen
* #239: Add chroot shortcuts (--rootfs and --buildroot)
* Show chroot and command before entering chroot
* Validate all inputs from `pmbootstrap init`
* Add a new `confirm()` function, that validates input of yes/no
questions properly
* All questions loop until they have a valid answer now
Thanks, craftyguy!
- UI is selectable with `pmbootstrap init`. Currently only 'weston' and 'none' are options. It'll automatically pick up any new `postmarketos-ui-*` packages added at later dates.
- splits off weston packages install from postmarketos-base and puts
them into postmarketos-ui-weston. Also note that NO weston packages are
installed by "pmbootstrap install" by default unless the user selects a ui in the `init`
- configuration of weston is now in postmarketos-ui-weston.
- the demos have been spun off to `postmarketos-demos`, and `postmarketos-ui-weston` lists this package as a dependency.
* Update runtime error message thrown for grsec/hardened kernels, saying that they are not supported right now, #107
* Remove unnecessary check for grsec enable/disable status - the build will not work in either case
* Ignore `>`, `<`, `=` and `!` operators, when they are specified in
the dependencies. This was the desired behavior before, but it was
not implemented correctly (so it wouldn't ignore them everywhere).
Of course the real fix would be to honor these operators like apk
does. But this isn't feasible right now, and it should work for
most, if not all, our use-cases. I have documented this in the wiki
under build internals and if we happen to need correct operator
handling, we should do it then.
Minor other changes:
* `pmbootstrap parse_apkindex`: support optional package parameter to
only show the parsed content for one package.
* Support building most python APKBUILDs by replacing ${pkgname#py-}
properly
This is required for developing and testing the binary repository
scripts (see #64). Changes:
* When specified, the local folder gets mounted inside the chroots
as /mnt/postmarketos-mirror
* The apkindex_files() function outputs the correct path to the local
repository (it does *not* hash the URL in that case, which would
be wrong)
* /etc/apk/repositories: when the pmOS mirror is a local folder,
the path "/mnt/postmarketos-mirror" gets added to that file instead
of the outside path (so apk finds it properly inside the chroot)
TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution.
---
pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has.
I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository.
To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on.
In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option.
In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds".
This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before.
Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases).
The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
* New commandline parameter --mirror-pmOS, where the binary repository
URL for postmarketOS can be specified (empty by default as of now,
this will be filled with the real URL once the repo works)
* Do not build packages, when they are in the binary repository and
the version of the package in the binary repository is up-to-date.
* Add a testcase for pmb.build.is_necessary().
...even if the pkgver and pkgrel have *not* changed. This should
make development much more intuitive. The detection works by looking
at the last modified timestamps, just like `make` does it.
* Two new functions for getting a list of files and their timestamps
in the repo, and diffing that information to get a list of changed
files: pmb.helpers.repo.files() and pmb.helpers.repo.diff().
(I've put it in the helpers folder, because it is not specific to
one chroot, but to all chroots at once.)
* pmbootstrap challenge (new command introduced a few commits back to
verify, that the contents of an APK file are deterministic) uses
these functions to a) support subpackages and b) optionally
output a list of changed files (this gets used in the pmbuilder
script, which lives outside of this repository).
This commit is progress for #64 again.
* The check for valid sdcard paths has been removed (because it
doesn't make sense for /dev/sd*, this might as well be a real
harddrive)
* New check: abort if any partition of the sdcard is mounted
* Automatically detect if the partitions are called "p1", "p2" etc.
or just "1", "2" etc. (/dev/sda1 vs. /dev/mmcblk0p1)
* pmb.helpers.mount.ismount(): check the source (new) and the target
path in /proc/mounts
This reverts commit 04aabd7d70.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
This reverts commit 2a39d9c091.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
This reverts commit c15bf48f77.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
I did not test this well enough, sorry! This introduced problems
such as interactive shell not working anymore (#40), cryptsetup partion
creating not working anymore etc.
This reverts commit f39c1dae69.