pmb/aportgen/grub_efi.py: fix distfile collisions

Same fix as MR 1947, but for grub_efi.py instead of musl.py. While at it, order the imports alphabetically.
pmb/aportgen/grub_efi.py: alpine mirrordir
2020-06-21 15:09:05 +02:00 · 2020-06-21 15:09:05 +02:00 · 2020-06-21 15:09:05 +02:00 · 2020-06-21 15:09:05 +02:00 · 2020-06-21 15:09:05 +02:00 · 2020-06-21 15:09:05 +02:00
6 changed files with 110 additions and 138 deletions
--- a/pmb/aportgen/busybox_static.py
+++ b/pmb/aportgen/busybox_static.py
@ -1,18 +1,15 @@
 # Copyright 2020 Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
-import glob
-import os
-import pmb.helpers.run
 import pmb.aportgen.core
-import pmb.parse.apkindex
+import pmb.build
 import pmb.chroot.apk
 import pmb.chroot.apk_static
+import pmb.helpers.run
+import pmb.parse.apkindex


 def generate(args, pkgname):
-    # Install busybox-static in chroot to get verified apks
    arch = pkgname.split("-")[2]
-    pmb.chroot.apk.install(args, ["busybox-static"], "buildroot_" + arch)

    # Parse version from APKINDEX
    package_data = pmb.parse.apkindex.package(args, "busybox")
@ -20,43 +17,33 @@ def generate(args, pkgname):
    pkgver = version.split("-r")[0]
    pkgrel = version.split("-r")[1]

-    # Copy the apk file to the distfiles cache
-    pattern = (args.work + "/cache_apk_" + arch + "/busybox-static-" +
-               version + ".*.apk")
-    glob_result = glob.glob(pattern)
-    if not len(glob_result):
-        raise RuntimeError("Could not find aport " + pattern + "!"
-                           " Update your aports_upstream git repo"
-                           " to the latest version, delete your http cache"
-                           " (pmbootstrap zap -hc) and try again.")
-    path = glob_result[0]
-    path_target = (args.work + "/cache_distfiles/busybox-static-" +
-                   version + "-" + arch + ".apk")
-    if not os.path.exists(path_target):
-        pmb.helpers.run.root(args, ["cp", path, path_target])
-
-    # Hash the distfile
-    hashes = pmb.chroot.user(args, ["sha512sum",
-                                    "busybox-static-" + version + "-" + arch + ".apk"],
-                             "buildroot_" + arch, "/var/cache/distfiles",
-                             output_return=True)
+    # Prepare aportgen tempdir inside and outside of chroot
+    tempdir = "/tmp/aportgen"
+    pmb.chroot.root(args, ["rm", "-rf", tempdir])
+    pmb.helpers.run.user(args, ["mkdir", "-p", f"{args.work}/aportgen",
+                                f"{args.work}/chroot_native/{tempdir}"])

    # Write the APKBUILD
-    pmb.helpers.run.user(args, ["mkdir", "-p", args.work + "/aportgen"])
-    with open(args.work + "/aportgen/APKBUILD", "w", encoding="utf-8") as handle:
-        # Variables
-        handle.write("# Automatically generated aport, do not edit!\n"
-                     "# Generator: pmbootstrap aportgen " + pkgname + "\n"
-                     "\n"
-                     "pkgname=" + pkgname + "\n"
-                     "pkgver=" + pkgver + "\n"
-                     "pkgrel=" + pkgrel + "\n"
-                     "\n"
-                     "_arch=\"" + arch + "\"\n"
-                     "_mirror=\"" + args.mirror_alpine + "\"\n"
-                     )
-        # Static part
-        static = """
+    channel_cfg = pmb.config.pmaports.read_config_channel(args)
+    mirrordir = channel_cfg["mirrordir_alpine"]
+    apkbuild_path = f"{args.work}/chroot_native/{tempdir}/APKBUILD"
+    with open(apkbuild_path, "w", encoding="utf-8") as handle:
+        apkbuild = f"""\
+            # Automatically generated aport, do not edit!
+            # Generator: pmbootstrap aportgen {pkgname}
+
+            # Stub for apkbuild-lint
+            if [ -z "$(type -t arch_to_hostspec)" ]; then
+                arch_to_hostspec() {{ :; }}
+            fi
+
+            pkgname={pkgname}
+            pkgver={pkgver}
+            pkgrel={pkgrel}
+
+            _arch="{arch}"
+            _mirror="{args.mirror_alpine}"
+
            url="http://busybox.net"
            license="GPL2"
            arch="all"
@ -65,18 +52,21 @@ def generate(args, pkgname):
            _target="$(arch_to_hostspec $_arch)"

            source="
-                busybox-static-$pkgver-r$pkgrel-$_arch.apk::$_mirror/edge/main/$_arch/busybox-static-$pkgver-r$pkgrel.apk
+                busybox-static-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk::$_mirror/{mirrordir}/main/$_arch/busybox-static-$pkgver-r$pkgrel.apk
            "

-            package() {
+            package() {{
                mkdir -p "$pkgdir/usr/$_target"
                cd "$pkgdir/usr/$_target"
-                tar -xf $srcdir/busybox-static-$pkgver-r$pkgrel-$_arch.apk
+                tar -xf $srcdir/busybox-static-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk
                rm .PKGINFO .SIGN.*
-            }
+            }}
        """
-        for line in static.split("\n"):
-            handle.write(line[12:] + "\n")
+        for line in apkbuild.split("\n"):
+            handle.write(line[12:].replace(" " * 4, "\t") + "\n")

-        # Hashes
-        handle.write("sha512sums=\"" + hashes.rstrip() + "\"\n")
+    # Generate checksums
+    pmb.build.init(args)
+    pmb.chroot.root(args, ["chown", "-R", "pmos:pmos", tempdir])
+    pmb.chroot.user(args, ["abuild", "checksum"], working_dir=tempdir)
+    pmb.helpers.run.user(args, ["cp", apkbuild_path, f"{args.work}/aportgen"])
--- a/pmb/aportgen/core.py
+++ b/pmb/aportgen/core.py
@ -163,6 +163,19 @@ def get_upstream_aport(args, pkgname):
    pmb.helpers.git.clone(args, "aports_upstream")
    aports_upstream_path = args.work + "/cache_git/aports_upstream"

+    # Checkout branch
+    channel_cfg = pmb.config.pmaports.read_config_channel(args)
+    branch = channel_cfg["branch_aports"]
+    logging.info(f"Checkout aports.git branch: {branch}")
+    if pmb.helpers.run.user(args, ["git", "checkout", branch],
+                            aports_upstream_path, check=False):
+        logging.info("NOTE: run 'pmbootstrap pull' and try again")
+        logging.info("NOTE: if it still fails, your aports.git was cloned with"
+                     " an older version of pmbootstrap, as shallow clone."
+                     " Unshallow it, or remove it and let pmbootstrap clone it"
+                     f" again: {aports_upstream_path}")
+        raise RuntimeError("Branch checkout failed.")
+
    # Search package
    paths = glob.glob(aports_upstream_path + "/*/" + pkgname)
    if len(paths) > 1:
--- a/pmb/aportgen/grub_efi.py
+++ b/pmb/aportgen/grub_efi.py
@ -1,12 +1,11 @@
-# Copyright 2020 Nick Reitemeyer
+# Copyright 2020 Nick Reitemeyer, Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
-import glob
-import os
-import pmb.helpers.run
 import pmb.aportgen.core
-import pmb.parse.apkindex
+import pmb.build
 import pmb.chroot.apk
 import pmb.chroot.apk_static
+import pmb.helpers.run
+import pmb.parse.apkindex


 def generate(args, pkgname):
@ -18,54 +17,46 @@ def generate(args, pkgname):
    pkgver = version.split("-r")[0]
    pkgrel = version.split("-r")[1]

-    pmb.chroot.apk.install(args, ["grub-efi"], "buildroot_x86")
-    pattern = (args.work + "/cache_apk_" + arch + "/grub-efi-" +
-               version + ".*.apk")
-    glob_result = glob.glob(pattern)
-    if not len(glob_result):
-        raise RuntimeError("Could not find aport " + pattern + "!"
-                           " Update your aports_upstream git repo"
-                           " to the latest version, delete your http cache"
-                           " (pmbootstrap zap -hc) and try again.")
-    path = glob_result[0]
-    path_target = (args.work + "/cache_distfiles/grub-efi-" +
-                   version + "-" + arch + ".apk")
-    if not os.path.exists(path_target):
-        pmb.helpers.run.root(args, ["cp", path, path_target])
+    # Prepare aportgen tempdir inside and outside of chroot
+    tempdir = "/tmp/aportgen"
+    pmb.chroot.root(args, ["rm", "-rf", tempdir])
+    pmb.helpers.run.user(args, ["mkdir", "-p", f"{args.work}/aportgen",
+                                f"{args.work}/chroot_native/{tempdir}"])

-    # Hash the distfile
-    hashes = pmb.chroot.user(args, ["sha512sum",
-                                    "grub-efi-" + version + "-" + arch + ".apk"],
-                             "buildroot_" + arch, "/var/cache/distfiles",
-                             output_return=True)
+    # Write the APKBUILD
+    channel_cfg = pmb.config.pmaports.read_config_channel(args)
+    mirrordir = channel_cfg["mirrordir_alpine"]
+    apkbuild_path = f"{args.work}/chroot_native/{tempdir}/APKBUILD"
+    with open(apkbuild_path, "w", encoding="utf-8") as handle:
+        apkbuild = f"""\
+            # Automatically generated aport, do not edit!
+            # Generator: pmbootstrap aportgen {pkgname}
+
+            pkgname={pkgname}
+            pkgver={pkgver}
+            pkgrel={pkgrel}
+
+            _arch="{arch}"
+            _mirror="{args.mirror_alpine}"

-    pmb.helpers.run.user(args, ["mkdir", "-p", args.work + "/aportgen"])
-    with open(args.work + "/aportgen/APKBUILD", "w", encoding="utf-8") as handle:
-        handle.write("# Automatically generated aport, do not edit!\n"
-                     "# Generator: pmbootstrap aportgen " + pkgname + "\n"
-                     "\n"
-                     "pkgname=" + pkgname + "\n"
-                     "pkgver=" + pkgver + "\n"
-                     "pkgrel=" + pkgrel + "\n"
-                     "\n"
-                     "_arch=\"" + arch + "\"\n"
-                     "_mirror=\"" + args.mirror_alpine + "\"\n"
-                     )
-        static = """
            pkgdesc="GRUB $_arch EFI files for every architecture"
            url="https://www.gnu.org/software/grub/"
            license="GPL-3.0-or-later"
            arch="all"
-            source="grub-efi-$pkgver-r$pkgrel-$_arch.apk::$_mirror/edge/main/$_arch/grub-efi-$pkgver-r$pkgrel.apk"
+            source="grub-efi-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk::$_mirror/{mirrordir}/main/$_arch/grub-efi-$pkgver-r$pkgrel.apk"

-            package() {
+            package() {{
                mkdir -p "$pkgdir"
                cd "$pkgdir"
-                tar -xf "$srcdir/grub-efi-$pkgver-r$pkgrel-$_arch.apk"
+                tar -xf "$srcdir/grub-efi-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk"
                rm .PKGINFO .SIGN.*
-            }
+            }}
        """
-        for line in static.split("\n"):
-            handle.write(line[12:] + "\n")
+        for line in apkbuild.split("\n"):
+            handle.write(line[12:].replace(" " * 4, "\t") + "\n")

-        handle.write("sha512sums=\"" + hashes.rstrip() + "\"\n")
+    # Generate checksums
+    pmb.build.init(args)
+    pmb.chroot.root(args, ["chown", "-R", "pmos:pmos", tempdir])
+    pmb.chroot.user(args, ["abuild", "checksum"], working_dir=tempdir)
+    pmb.helpers.run.user(args, ["cp", apkbuild_path, f"{args.work}/aportgen"])
--- a/pmb/aportgen/musl.py
+++ b/pmb/aportgen/musl.py
@ -1,19 +1,14 @@
 # Copyright 2020 Oliver Smith
 # SPDX-License-Identifier: GPL-3.0-or-later
-import glob
-import os
-import pmb.helpers.run
 import pmb.aportgen.core
-import pmb.parse.apkindex
+import pmb.build
 import pmb.chroot.apk
 import pmb.chroot.apk_static
+import pmb.helpers.run
+import pmb.parse.apkindex


 def generate(args, pkgname):
-    # Install musl in chroot to get verified apks
-    arch = pkgname.split("-")[1]
-    pmb.chroot.apk.install(args, ["musl-dev"], "buildroot_" + arch)
-
    # Parse musl version from APKINDEX
    package_data = pmb.parse.apkindex.package(args, "musl")
    version = package_data["version"]
@ -21,37 +16,21 @@ def generate(args, pkgname):
    pkgrel = version.split("-r")[1]

    # Architectures to build this package for
+    arch = pkgname.split("-")[1]
    arches = list(pmb.config.build_device_architectures)
    arches.remove(arch)

-    # Copy the apk files to the distfiles cache
-    for subpkgname in ["musl", "musl-dev"]:
-        pattern = (args.work + "/cache_apk_" + arch + "/" + subpkgname +
-                   "-" + version + ".*.apk")
-        glob_result = glob.glob(pattern)
-        if not len(glob_result):
-            raise RuntimeError("Could not find aport " + pattern + "!"
-                               " Update your aports_upstream git repo"
-                               " to the latest version, delete your http cache"
-                               " (pmbootstrap zap -hc) and try again.")
-        path = glob_result[0]
-        path_target = (args.work + "/cache_distfiles/" + subpkgname + "-" +
-                       version + "-" + arch + ".apk")
-        if not os.path.exists(path_target):
-            pmb.helpers.run.root(args, ["cp", path, path_target])
-
-    # Hash the distfiles
-    hashes = pmb.chroot.user(args, ["sha512sum",
-                                    "musl-" + version + "-" + arch + ".apk",
-                                    "musl-dev-" + version + "-" + arch + ".apk"], "buildroot_" + arch,
-                             "/var/cache/distfiles", output_return=True)
+    # Prepare aportgen tempdir inside and outside of chroot
+    tempdir = "/tmp/aportgen"
+    pmb.chroot.root(args, ["rm", "-rf", tempdir])
+    pmb.helpers.run.user(args, ["mkdir", "-p", f"{args.work}/aportgen",
+                                f"{args.work}/chroot_native/{tempdir}"])

    # Write the APKBUILD
    channel_cfg = pmb.config.pmaports.read_config_channel(args)
    mirrordir = channel_cfg["mirrordir_alpine"]
-    pmb.helpers.run.user(args, ["mkdir", "-p", args.work + "/aportgen"])
-    with open(args.work + "/aportgen/APKBUILD", "w", encoding="utf-8") as handle:
-        # Variables
+    apkbuild_path = f"{args.work}/chroot_native/{tempdir}/APKBUILD"
+    with open(apkbuild_path, "w", encoding="utf-8") as handle:
        apkbuild = f"""\
            # Automatically generated aport, do not edit!
            # Generator: pmbootstrap aportgen {pkgname}
@ -78,8 +57,8 @@ def generate(args, pkgname):
            _target="$(arch_to_hostspec $_arch)"

            source="
-                musl-$pkgver-r$pkgrel-$_arch.apk::$_mirror/{mirrordir}/main/$_arch/musl-$pkgver-r$pkgrel.apk
-                musl-dev-$pkgver-r$pkgrel-$_arch.apk::$_mirror/{mirrordir}/main/$_arch/musl-dev-$pkgver-r$pkgrel.apk
+                musl-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk::$_mirror/{mirrordir}/main/$_arch/musl-$pkgver-r$pkgrel.apk
+                musl-dev-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk::$_mirror/{mirrordir}/main/$_arch/musl-dev-$pkgver-r$pkgrel.apk
            "

            package() {{
@ -87,7 +66,7 @@ def generate(args, pkgname):
                cd "$pkgdir/usr/$_target"
                # Use 'busybox tar' to avoid 'tar: Child returned status 141'
                # on some machines (builds.sr.ht, gitlab-ci). See pmaports#26.
-                busybox tar -xf $srcdir/musl-$pkgver-r$pkgrel-$_arch.apk
+                busybox tar -xf $srcdir/musl-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk
                rm .PKGINFO .SIGN.*
            }}
            package_dev() {{
@ -95,7 +74,7 @@ def generate(args, pkgname):
                cd "$subpkgdir/usr/$_target"
                # Use 'busybox tar' to avoid 'tar: Child returned status 141'
                # on some machines (builds.sr.ht, gitlab-ci). See pmaports#26.
-                busybox tar -xf $srcdir/musl-dev-$pkgver-r$pkgrel-$_arch.apk
+                busybox tar -xf $srcdir/musl-dev-$pkgver-r$pkgrel-$_arch-{mirrordir}.apk
                rm .PKGINFO .SIGN.*

                # symlink everything from /usr/$_target/usr/* to /usr/$_target/*
@ -113,5 +92,8 @@ def generate(args, pkgname):
        for line in apkbuild.split("\n"):
            handle.write(line[12:].replace(" " * 4, "\t") + "\n")

-        # Hashes
-        handle.write("sha512sums=\"" + hashes.rstrip() + "\"\n")
+    # Generate checksums
+    pmb.build.init(args)
+    pmb.chroot.root(args, ["chown", "-R", "pmos:pmos", tempdir])
+    pmb.chroot.user(args, ["abuild", "checksum"], working_dir=tempdir)
+    pmb.helpers.run.user(args, ["cp", apkbuild_path, f"{args.work}/aportgen"])
--- a/pmb/config/pmaports.py
+++ b/pmb/config/pmaports.py
@ -33,7 +33,7 @@ def clone(args):
                 " recipes (pmaports)...")

    # Set up the native chroot and clone pmaports
-    pmb.helpers.git.clone(args, "pmaports", False)
+    pmb.helpers.git.clone(args, "pmaports")


 def symlink(args):
--- a/pmb/helpers/git.py
+++ b/pmb/helpers/git.py
@ -21,15 +21,13 @@ def get_path(args, name_repo):
    return args.work + "/cache_git/" + name_repo


-def clone(args, name_repo, shallow=True):
+def clone(args, name_repo):
    """ Clone a git repository to $WORK/cache_git/$name_repo (or to the
        overridden path set in args, as with pmbootstrap --aports).

        :param name_repo: short alias used for the repository name, from
                          pmb.config.git_repos (e.g. "aports_upstream",
-                          "pmaports")
-        :param shallow: only clone the last revision of the repository, instead
-                        of the entire repository (faster, saves bandwith) """
+                          "pmaports") """
    # Check for repo name in the config
    if name_repo not in pmb.config.git_repos:
        raise ValueError("No git repository configured for " + name_repo)
@ -39,8 +37,6 @@ def clone(args, name_repo, shallow=True):
        # Build git command
        url = pmb.config.git_repos[name_repo]
        command = ["git", "clone"]
-        if shallow:
-            command += ["--depth=1"]
        command += [url, path]

        # Create parent dir and clone
Author	SHA1	Message	Date
Oliver Smith	abe54f8b76	pmb/aportgen/grub_efi.py: fix distfile collisions Same fix as MR 1947, but for grub_efi.py instead of musl.py. While at it, order the imports alphabetically.	2020-06-21 15:09:05 +02:00
Oliver Smith	d230b396b3	pmb/aportgen/grub_efi.py: alpine mirrordir On branch stable, we would use Alpine's mirrordir "v3.11" for example.	2020-06-21 15:09:05 +02:00
Oliver Smith	23d4bb7020	pmb/aportgen/grub_efi.py: pass apkbuild-lint	2020-06-21 15:09:05 +02:00
Oliver Smith	f7a154b61b	pmb/aportgen/grub_efi.py: use f-string in APKBUILD	2020-06-21 15:09:05 +02:00
Oliver Smith	8b7c9fb46a	pmb/aportgen/busybox_static.py: fix distfile collisions Same fix as MR 1947, but for busybox_static.py instead of musl.py. While at it, order the imports alphabetically.	2020-06-21 15:09:05 +02:00
Oliver Smith	98b28c6d84	pmb/aportgen/busybox_static.py: alpine mirrordir On branch stable, we would use Alpine's mirrordir "v3.11" for example.	2020-06-21 15:09:05 +02:00
Oliver Smith	ae7b02c918	pmb/aportgen/busybox_static.py: pass lint	2020-06-21 15:09:05 +02:00
Oliver Smith	61cf186566	pmb/aportgen/busybox_static.py: use f-string	2020-06-21 15:09:05 +02:00
Oliver Smith	130de05881	aportgen get_upstream_aport: checkout branch Checkout the aports.git branch for the current channel (e.g. 3.12-stable), before trying to find the APKBUILD. I had tried to auto-unshallow the git repository earlier, but then the tags were missing. I decided that it's not worth to provide a migration path: "pmbootstrap aportgen" is only used by few advanced users (to maintain the pmaports repo).	2020-06-21 15:09:04 +02:00
Oliver Smith	54938032af	pmb.helpers.git.clone: remove shallow option This made sense for Alpine's aports.git repository as we were only using the master branch. But now that we are using more branches, we need the entire git repository with all its branches cloned.	2020-06-21 15:09:04 +02:00
Oliver Smith	ccb5e323d7	pmb/aportgen/musl.py: fix distfile collisions Add the Alpine mirrordir (e.g. "edge", "v3.12") to the distfile, so musl-.apk files of the same version and arch but from different mirrordirs do not collide. Let "abuild checksum" download these apks and generate the checksums, instead of letting apk download them as side-effect of initializing foreign arch chroots. The latter did not work anymore, because we would copy the apk file with a glob that may matches the hash of either mirrordir. Essentially: glob.glob(f"{args.work}/cache_apk_{arch}/{subpkgname}-{version}..apk")[0] In the context of the on-device installer, I found that calculating this hash is not trivial, so let's just avoid it here as well. While at it, order the imports of musl.py alphabetically.	2020-06-21 15:08:59 +02:00