synit
/
pmbootstrap
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
pmbootstrap/pmb/chroot/apk.py

266 lines
10 KiB
Python
Raw Normal View History

Hello, there!
2017-05-26 20:08:45 +00:00
"""
Happy new year! (update copyright to 2018)
2018-01-04 03:53:35 +00:00
Copyright 2018 Oliver Smith
Hello, there!
2017-05-26 20:08:45 +00:00
This file is part of pmbootstrap.
pmbootstrap is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
pmbootstrap is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with pmbootstrap. If not, see <http://www.gnu.org/licenses/>.
"""
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
import os
Hello, there!
2017-05-26 20:08:45 +00:00
import logging
Properly update /etc/apk/repositories when the mirrors change This is important for the binary repository scripts, so it's feasible to test the binary package build and challenge process locally without setting up a new chroot whenever changing the repo URLs. Also it behaves a bit more intuitively, because it really uses the repo URL specified on the commandline, even when the chroot is already set up.
2017-07-11 17:19:39 +00:00
import shlex
Hello, there!
2017-05-26 20:08:45 +00:00
import pmb.chroot
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
import pmb.config
Hello, there!
2017-05-26 20:08:45 +00:00
import pmb.parse.apkindex
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
import pmb.parse.arch
import pmb.parse.depends
Fix #342: don't use distutils.version.LooseVersion anymore (#364) Previously, distutils.version.LooseVersion was used, because it was sort of close enough to how Alpine parses versions. This new version uses the exact same algorithm, as `apk` does, and it passes *all* of `apk`'s testcases for version checking (previously we simply skipped the ones, that did not pass). * Remove pmb/helpers/version.py left-over (it is in parse now) * Make asserts consistent, do not use unnecessary parenthesis
2017-08-12 14:03:40 +00:00
import pmb.parse.version
Hello, there!
2017-05-26 20:08:45 +00:00
Properly update /etc/apk/repositories when the mirrors change This is important for the binary repository scripts, so it's feasible to test the binary package build and challenge process locally without setting up a new chroot whenever changing the repo URLs. Also it behaves a bit more intuitively, because it really uses the repo URL specified on the commandline, even when the chroot is already set up.
2017-07-11 17:19:39 +00:00
def update_repository_list(args, suffix="native", check=False):
"""
Update /etc/apk/repositories, if it is outdated (when the user changed the
--mirror-alpine or --mirror-pmOS parameters).
:param check: This function calls it self after updating the
/etc/apk/repositories file, to check if it was successful.
Only for this purpose, the "check" parameter should be set to
True.
"""
# Skip if we already did this
if suffix in args.cache["apk_repository_list_updated"]:
return
# Read old entries or create folder structure
path = args.work + "/chroot_" + suffix + "/etc/apk/repositories"
lines_old = []
if os.path.exists(path):
# Read all old lines
lines_old = []
with open(path) as handle:
for line in handle:
lines_old.append(line[:-1])
else:
pmb.helpers.run.root(args, ["mkdir", "-p", os.path.dirname(path)],
suffix)
# Up to date: Save cache, return
lines_new = pmb.helpers.repo.urls(args)
if lines_old == lines_new:
args.cache["apk_repository_list_updated"].append(suffix)
return
# Check phase: raise error when still outdated
if check:
raise RuntimeError("Failed to update: " + path)
# Update the file
logging.debug("(" + suffix + ") update /etc/apk/repositories")
if os.path.exists(path):
pmb.helpers.run.root(args, ["rm", path])
for line in lines_new:
pmb.helpers.run.root(args, ["sh", "-c", "echo " +
shlex.quote(line) + " >> " + path])
update_repository_list(args, suffix, True)
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
def check_min_version(args, suffix="native"):
"""
Check the minimum apk version, before running it the first time in the
current session (lifetime of one pmbootstrap call).
"""
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
# Skip if we already did this
if suffix in args.cache["apk_min_version_checked"]:
return
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
# Skip if apk is not installed yet
if not os.path.exists(args.work + "/chroot_" + suffix + "/sbin/apk"):
logging.debug("NOTE: Skipped apk version check for chroot '" + suffix +
"', because it is not installed yet!")
return
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
# Compare
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
version_installed = installed(args, suffix)["apk-tools"]["version"]
version_min = pmb.config.apk_tools_static_min_version
Index parser: support multiple package providers (#1202) * The APKINDEX parser used to return a dictionary with one package for a given package name. This works for the installed packages database, because there can only be one provider for a package. But when parsing packages from binary repositories, we need to support multiple providers for one package. It is now possible to get a dictionary with either multiple providers, or just a single provider for each package. * Dependency parsing logic has been adjusted, to support multiple providers. For multiple providers, the one with the same package name as the package we are looking up is prefered. If there is none (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages that will be installed anyway, and after that packages that are already installed. When all else fails, it just picks the first one and prints a note in the "pmbootstrap log". * Added testcases for all functions in pmb.parse.apkindex and pmb.parse.depends * pmbootstrap chroot has a new "--add" parameter to specify packages that pmbootstrap should build if neccessary, and install in the chroot. This can be used to quickly test the depencency resolution of pmbootstrap without doing a full "pmbootstrap install". Fixes #1122.
2018-02-20 19:52:28 +00:00
if pmb.parse.version.compare(version_installed, version_min) == -1:
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
raise RuntimeError("You have an outdated version of the 'apk' package"
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
" manager installed (your version: " + version_installed +
", expected at least: " + version_min + "). Delete"
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
" your http cache and zap all chroots, then try again:"
" 'pmbootstrap zap -hc'")
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
# Mark this suffix as checked
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
args.cache["apk_min_version_checked"].append(suffix)
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
def install_is_necessary(args, build, arch, package, packages_installed):
"""
This function optionally builds an out of date package, and checks if the
version installed inside a chroot is up to date.
:param build: Set to true to build the package, if the binary packages are
out of date, and it is in the aports folder.
:param packages_installed: Return value from installed().
:returns: True if the package needs to be installed/updated, False otherwise.
"""
# Build package
if build:
pmb.build.package(args, package, arch)
# No further checks when not installed
if package not in packages_installed:
return True
Fix various build issues (fix #189, fix #341) (#345) Changes: * Removed the apkindex_files cache. That particular cache caused bug #189 and didn't bring any real-world performance improvements (tested 3x with that cache and 3x without, no significant speed difference). I decided to remove it instead of keeping it/adding even more code to resolve the bug. * Fix the check for already built packages: always use the architecture, that the package should be built for instead of the architecture of the build environment (e.g. use armhf, even when building a noarch package in the x86_64 chroot). This partially resolves #341. * Make pmb.chroot.apk.install_is_necessary() more robust: If the binary package is missing, although it should be there, print a warning and build it with force.
2017-08-09 17:59:21 +00:00
# Make sure, that we really have a binary package
Index parser: support multiple package providers (#1202) * The APKINDEX parser used to return a dictionary with one package for a given package name. This works for the installed packages database, because there can only be one provider for a package. But when parsing packages from binary repositories, we need to support multiple providers for one package. It is now possible to get a dictionary with either multiple providers, or just a single provider for each package. * Dependency parsing logic has been adjusted, to support multiple providers. For multiple providers, the one with the same package name as the package we are looking up is prefered. If there is none (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages that will be installed anyway, and after that packages that are already installed. When all else fails, it just picks the first one and prints a note in the "pmbootstrap log". * Added testcases for all functions in pmb.parse.apkindex and pmb.parse.depends * pmbootstrap chroot has a new "--add" parameter to specify packages that pmbootstrap should build if neccessary, and install in the chroot. This can be used to quickly test the depencency resolution of pmbootstrap without doing a full "pmbootstrap install". Fixes #1122.
2018-02-20 19:52:28 +00:00
data_repo = pmb.parse.apkindex.package(args, package, arch, False)
Fix various build issues (fix #189, fix #341) (#345) Changes: * Removed the apkindex_files cache. That particular cache caused bug #189 and didn't bring any real-world performance improvements (tested 3x with that cache and 3x without, no significant speed difference). I decided to remove it instead of keeping it/adding even more code to resolve the bug. * Fix the check for already built packages: always use the architecture, that the package should be built for instead of the architecture of the build environment (e.g. use armhf, even when building a noarch package in the x86_64 chroot). This partially resolves #341. * Make pmb.chroot.apk.install_is_necessary() more robust: If the binary package is missing, although it should be there, print a warning and build it with force.
2017-08-09 17:59:21 +00:00
if not data_repo:
logging.warning("WARNING: Internal error in pmbootstrap," +
" package '" + package + "' for " + arch +
" has not been built yet, but it should have"
" been. Rebuilding it with force. Please "
" report this, if there is no ticket about this"
" yet!")
pmb.build.package(args, package, arch, True)
return install_is_necessary(args, build, arch, package,
packages_installed)
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
# Compare the installed version vs. the version in the repos
data_installed = packages_installed[package]
Fix #342: don't use distutils.version.LooseVersion anymore (#364) Previously, distutils.version.LooseVersion was used, because it was sort of close enough to how Alpine parses versions. This new version uses the exact same algorithm, as `apk` does, and it passes *all* of `apk`'s testcases for version checking (previously we simply skipped the ones, that did not pass). * Remove pmb/helpers/version.py left-over (it is in parse now) * Make asserts consistent, do not use unnecessary parenthesis
2017-08-12 14:03:40 +00:00
compare = pmb.parse.version.compare(data_installed["version"],
data_repo["version"])
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
# a) Installed newer (should not happen normally)
if compare == 1:
logging.info("WARNING: " + arch + " package '" + package +
"' installed version " + data_installed["version"] +
" is newer, than the version in the repositories: " +
Close #296: Add wiki links to repository warnings (#473) Also update one wiki link, that pointed to the old wiki.
2017-08-27 14:00:53 +00:00
data_repo["version"] +
" See also: <https://postmarketos.org/warning-repo>")
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
return False
# b) Repo newer
elif compare == -1:
return True
# c) Same version, look at last modified
elif compare == 0:
time_installed = float(data_installed["timestamp"])
time_repo = float(data_repo["timestamp"])
return time_repo > time_installed
def replace_aports_packages_with_path(args, packages, suffix, arch):
"""
apk will only re-install packages with the same pkgname, pkgver and pkgrel,
when you give it the absolute path to the package. This function replaces
all packages, that were built locally, with the absolute path to the package.
"""
ret = []
for package in packages:
aport = pmb.build.find_aport(args, package, False)
if aport:
apkbuild = pmb.parse.apkbuild(args, aport + "/APKBUILD")
Don't mark all packages as explicitly installed (#1247) pmbootstrap does dependency resolving on its own, and passes the list of resolved packages to apk when we want it to install something. The reason was outlined in #129: > fixing #120: packages do not get updated in "pmbootstrap install" > after they have been rebuilt. For this to work, we specify all > packages explicitly for abuild, instead of letting abuild do the > resolving. This new PR fixes #1212 (which noted that all of these dependencies were explicitly marked for installation) by doing the following: 1. All packages and dependencies get attached to the virtual package .pmbootstrap instead of world 2. We install the packages (without depends) explcitly 3. .pmbootstrap gets removed, which means that all packages from 1. stay installed, but are no longer marked as explicitly installed. They will get removed automatically, when the depending packages get removed. In addition, the mechanism for replacing the package of locally built packages with their full path, was broken and has been fixed in this commit. This is necessary to update packages of the same version with apk.
2018-02-25 20:49:47 +00:00
apk_path = ("/mnt/pmbootstrap-packages/" + arch + "/" +
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
package + "-" + apkbuild["pkgver"] + "-r" +
apkbuild["pkgrel"] + ".apk")
if os.path.exists(args.work + "/chroot_" + suffix + apk_path):
package = apk_path
ret.append(package)
return ret
Hello, there!
2017-05-26 20:08:45 +00:00
def install(args, packages, suffix="native", build=True):
"""
:param build: automatically build the package, when it does not exist yet
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
or needs to be updated, and it is inside the pm-aports
folder. Checking this is expensive - if you know, that all
packages are provides by upstream repos, set this to False!
Hello, there!
2017-05-26 20:08:45 +00:00
"""
# Initialize chroot
Update min apk version/add more apk version checks * Minimum version: 2.7.2 (which fixes two CVEs) * Check the minimum apk version before doing something with apk and before entering the chroot manually (previously, it has just checked the apk-tools-static version, which gets used to set up the chroot) * Reword the message for an outdated APK version. Most likely it is just the outdated http cache, instead of a man-in-the-middle attack. See also: https://github.com/alpinelinux/aports/commit/b849b481a0446d7ef2458232a8819ce0b301e629
2017-06-23 15:04:49 +00:00
check_min_version(args, suffix)
Hello, there!
2017-05-26 20:08:45 +00:00
pmb.chroot.init(args, suffix)
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
# Add depends to packages
arch = pmb.parse.arch.from_chroot_suffix(args, suffix)
Index parser: support multiple package providers (#1202) * The APKINDEX parser used to return a dictionary with one package for a given package name. This works for the installed packages database, because there can only be one provider for a package. But when parsing packages from binary repositories, we need to support multiple providers for one package. It is now possible to get a dictionary with either multiple providers, or just a single provider for each package. * Dependency parsing logic has been adjusted, to support multiple providers. For multiple providers, the one with the same package name as the package we are looking up is prefered. If there is none (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages that will be installed anyway, and after that packages that are already installed. When all else fails, it just picks the first one and prints a note in the "pmbootstrap log". * Added testcases for all functions in pmb.parse.apkindex and pmb.parse.depends * pmbootstrap chroot has a new "--add" parameter to specify packages that pmbootstrap should build if neccessary, and install in the chroot. This can be used to quickly test the depencency resolution of pmbootstrap without doing a full "pmbootstrap install". Fixes #1122.
2018-02-20 19:52:28 +00:00
packages_with_depends = pmb.parse.depends.recurse(args, packages, suffix)
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
Fix #824: Refactor pmb/build/package.py (make depends work like in abuild) (#935) * Rename pmb/build/package.py to pmb/build/_package.py, so we can access the functions it contains in testcases, and still use pmb.build.package() * Refactor the entire file. Instead of one big function that does too many things, we have many small ones now, that are tested in the testsuite and easier to modify * Whenever building a package, pmbootstrap does not only build and install the "makedepends" (like we did before), now it does the same for the "depends". That's required to be compatible with abuild. The old behavior can still be used with 'pmbootstrap build --ignore-depends'. * Because of that change, noarch packages can no longer be built in the native chroot if we need them for a foreign chroot. A device- package depending on a kernel would pull in the same kernel for the native architecture otherwise. * Running 'pmbootstrap build device-...' without '--ignore-depends' and without a matching '--arch' displays a note that explains this change to the user and tells how to use it instead. * Noarch packages no longer get symlinked. That was only implemented for packages built in the native chroot, and now that is not always the case anymore. Symlinking these packages creates packages with broken dependencies anyway (e.g. device-samsung-i9100 can't be installed in x86_64, because linux-samsung-i9100 is armhf only). * Rename "carch" to "arch" wherever used. Naming it "carch" sometimes is confusing with no benefit. * Add a testcase for the aarch64 qemu workaround (because it failed first and I needed to know for sure if it is working again). * Improved some verbose logging, which helped with development of this feature. * Removed the old "build" test case (which was disabled in testcases_fast.sh) as the new "build_package" test case covers its functionallity. * Only build indexes if the packages folder exists for that arch (Travis couldn't run a test case otherwise)
2017-11-26 14:32:02 +00:00
# Filter outdated packages (build them if required)
Hello, there!
2017-05-26 20:08:45 +00:00
packages_installed = installed(args, suffix)
packages_todo = []
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
for package in packages_with_depends:
if install_is_necessary(
args, build, arch, package, packages_installed):
Hello, there!
2017-05-26 20:08:45 +00:00
packages_todo.append(package)
if not len(packages_todo):
return
# Sanitize packages: don't allow '--allow-untrusted' and other options
# to be passed to apk!
for package in packages_todo:
if package.startswith("-"):
raise ValueError("Invalid package name: " + package)
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
# Readable install message without dependencies
message = "(" + suffix + ") install"
for pkgname in packages:
if pkgname not in packages_installed:
message += " " + pkgname
logging.info(message)
Don't mark all packages as explicitly installed (#1247) pmbootstrap does dependency resolving on its own, and passes the list of resolved packages to apk when we want it to install something. The reason was outlined in #129: > fixing #120: packages do not get updated in "pmbootstrap install" > after they have been rebuilt. For this to work, we specify all > packages explicitly for abuild, instead of letting abuild do the > resolving. This new PR fixes #1212 (which noted that all of these dependencies were explicitly marked for installation) by doing the following: 1. All packages and dependencies get attached to the virtual package .pmbootstrap instead of world 2. We install the packages (without depends) explcitly 3. .pmbootstrap gets removed, which means that all packages from 1. stay installed, but are no longer marked as explicitly installed. They will get removed automatically, when the depending packages get removed. In addition, the mechanism for replacing the package of locally built packages with their full path, was broken and has been fixed in this commit. This is necessary to update packages of the same version with apk.
2018-02-25 20:49:47 +00:00
# Local packages: Using the path instead of pkgname makes apk update
# packages of the same version if the build date is different
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
packages_todo = replace_aports_packages_with_path(args, packages_todo,
suffix, arch)
Don't mark all packages as explicitly installed (#1247) pmbootstrap does dependency resolving on its own, and passes the list of resolved packages to apk when we want it to install something. The reason was outlined in #129: > fixing #120: packages do not get updated in "pmbootstrap install" > after they have been rebuilt. For this to work, we specify all > packages explicitly for abuild, instead of letting abuild do the > resolving. This new PR fixes #1212 (which noted that all of these dependencies were explicitly marked for installation) by doing the following: 1. All packages and dependencies get attached to the virtual package .pmbootstrap instead of world 2. We install the packages (without depends) explcitly 3. .pmbootstrap gets removed, which means that all packages from 1. stay installed, but are no longer marked as explicitly installed. They will get removed automatically, when the depending packages get removed. In addition, the mechanism for replacing the package of locally built packages with their full path, was broken and has been fixed in this commit. This is necessary to update packages of the same version with apk.
2018-02-25 20:49:47 +00:00
# Use a virtual package to mark only the explicitly requested packages as
# explicitly installed, not their dependencies or specific paths (#1212)
commands = [["add"] + packages]
if packages != packages_todo:
commands = [["add", "-u", "--virtual", ".pmbootstrap"] + packages_todo,
["add"] + packages,
["del", ".pmbootstrap"]]
for command in commands:
pmb.chroot.root(args, ["apk", "--no-progress"] + command, suffix)
Hello, there!
2017-05-26 20:08:45 +00:00
Improved pmbootstrap init (#1095) This PR makes the workflow faster and pmbootstrap will produce less traffic. Details: * Check if it's possible to create and read from a device node directly when initializing a chroot (closes #472) * Copy the Qemu binary into the forign-arch chroots before initializing them, so the post-install script directly work during the chroot setup and we don't need to call apk fix afterwards * Use pmb.helpers.repo.update(), which only updates the APKINDEX files if they are older than 4 hours, instead of using apk's repo update function which always downloads the APKINDEX files * Chroot initialization * Getting the initial APKINDEX to download apk-tools-static * Updating the APKINDEX at the start of pmbootstrap install * Fixed a bug in from_chroot_suffix: the buildroot_x86_64 has architecture x86_64, not x86.
2018-01-23 00:54:48 +00:00
def upgrade(args, suffix="native"):
Fix #68: properly update the device chroot in the install step * build all dependencies, that have a newer version * upgrade all packages inside the chroot
2017-06-08 15:54:53 +00:00
"""
Upgrade all packages installed in a chroot
"""
Only download APKINDEX for relevant architectures (#1201) ### Only download APKINDEX for relevant architectures We're downloading the APKINDEX files for all architectures supported by postmarketOS currently (x86, x86_64, armhf, aarch64). Most of the time, we only need it for the native and device arch, so this PR reduces the downloaded files to what is really necessary. ### Intuitive pmbootstrap update logic * pmb.helpers.repo.update(): * Default is updating all arches where the APKBUILD files exist * Add existing_only parameter * Return True when files have been downloaded * Properly print which arches will be updated * Print update reason only in verbose log * Add and improve comments * pmb.parse.arguments(), update action: * Add --non-existing parameter * Default for --arch is None (instead of arch.native) * pmb.helpers.frontend.update(): * Inform about --non-existing if no APKBUILDs have been updated
2018-03-04 13:44:27 +00:00
# Update APKINDEX files
arch = pmb.parse.arch.from_chroot_suffix(args, suffix)
pmb.helpers.repo.update(args, arch)
Hello, there!
2017-05-26 20:08:45 +00:00
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
# Rebuild and upgrade out-of-date packages
convert dict.keys() to list() (#1290)
2018-03-04 20:10:27 +00:00
packages = list(installed(args, suffix).keys())
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
install(args, packages, suffix)
Hello, there!
2017-05-26 20:08:45 +00:00
def installed(args, suffix="native"):
Fix #68: properly update the device chroot in the install step * build all dependencies, that have a newer version * upgrade all packages inside the chroot
2017-06-08 15:54:53 +00:00
"""
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
Read the list of installed packages (which has almost the same format, as
an APKINDEX, but with more keys).
Properly rebuild/install packages when something changed (Fix #120, #108, #131) (#129) TLDR: Always rebuild/install packages when something changed when executing "pmbootstrap install/initfs/flash", more speed in dependency resolution. --- pmbootstrap has already gotten some support for "timestamp based rebuilds", which modifies the logic for when packages should be rebuilt. It doesn't only consider packages outdated with old pkgver/pkgrel combinations, but also packages, where a source file has a newer timestamp, than the built package has. I've found out, that this can lead to more rebuilds than expected. For example, when you check out the pmbootstrap git repository again into another folder, although you have already built packages. Then all files have the timestamp of the checkout, and the packages will appear to be outdated. While this is not largely a concern now, this will become a problem once we have a binary package repository, because then the packages from the binary repo will always seem to be outdated, if you just freshly checked out the repository. To combat this, git gets asked if the files from the aport we're looking at are in sync with upstream, or not. Only when the files are not in sync with upstream and the timestamps of the sources are newer, a rebuild gets triggered from now on. In case this logic should fail, I've added an option during "pmbootstrap init" where you can enable or disable the "timestamp based rebuilds" option. In addition to that, this commit also works on fixing #120: packages do not get updated in "pmbootstrap install" after they have been rebuilt. For this to work, we specify all packages explicitly for abuild, instead of letting abuild do the resolving. This feature will also work with the "timestamp based rebuilds". This commit also fixes the working_dir argument in pmb.helpers.run.user, which was simply ignored before. Finally, the performance of the dependency resolution is faster again (when compared to the current version in master), because the parsed apkbuilds and finding the aport by pkgname gets cached during one pmbootstrap call (in args.cache, which also makes it easy to put fake data there in testcases). The new dependency resolution code can output lots of verbose messages for debugging by specifying the `-v` parameter. The meaning of that changed, it used to output the file names where log messages come from, but no one seemed to use that anyway.
2017-07-10 15:23:43 +00:00
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
:returns: a dictionary with the following structure:
{ "postmarketos-mkinitfs":
{
"pkgname": "postmarketos-mkinitfs"
"version": "0.0.4-r10",
"depends": ["busybox-extras", "lddtree", ...],
"provides": ["mkinitfs=0.0.1"]
}, ...
}
Fix #68: properly update the device chroot in the install step * build all dependencies, that have a newer version * upgrade all packages inside the chroot
2017-06-08 15:54:53 +00:00
"""
pmb.chroot.apk.installed(): use apk's internal database ...instead of running apk every time to get the list of installed packages and their versions. The internal package database from apk has the same format, as the extracted APKINDEX file (except that it has more key-value pairs, which we ignore/do not need right now). So the APKINDEX code has been extended to parse both tar-packed APKINDEX files and regular text files in the APKINDEX format. This is required for #108, for a better detection of outdated packages (because the internal package database saves the package's timestamp, too). A nice benefit is, that this is faster than calling apk every time and it doesn't fill up the log as much. I've also used this improved function for determining the apk version (for the outdated version check), and I've deleted pmb.parse.other.package_split(), as it is not needed anymore.
2017-06-27 15:56:15 +00:00
path = args.work + "/chroot_" + suffix + "/lib/apk/db/installed"
Index parser: support multiple package providers (#1202) * The APKINDEX parser used to return a dictionary with one package for a given package name. This works for the installed packages database, because there can only be one provider for a package. But when parsing packages from binary repositories, we need to support multiple providers for one package. It is now possible to get a dictionary with either multiple providers, or just a single provider for each package. * Dependency parsing logic has been adjusted, to support multiple providers. For multiple providers, the one with the same package name as the package we are looking up is prefered. If there is none (eg. "so:libEGL.so.1" is provided by "mesa-egl"), it prefers packages that will be installed anyway, and after that packages that are already installed. When all else fails, it just picks the first one and prints a note in the "pmbootstrap log". * Added testcases for all functions in pmb.parse.apkindex and pmb.parse.depends * pmbootstrap chroot has a new "--add" parameter to specify packages that pmbootstrap should build if neccessary, and install in the chroot. This can be used to quickly test the depencency resolution of pmbootstrap without doing a full "pmbootstrap install". Fixes #1122.
2018-02-20 19:52:28 +00:00
return pmb.parse.apkindex.parse(args, path, False)