Commit Graph

174 Commits

Author SHA1 Message Date
Yuval Adam b6003a2815 Close #327: Add initial setup.py (#443) 2017-09-02 19:30:40 +00:00
Attila Szöllősi c6fe6e134a Flavor instead of device name when creating recovery zip (#491)
* Flavor instead of device name when creating recovery zip.
* Move parse_flavor_arg function
2017-09-02 03:53:58 +00:00
Oliver Smith db6491c679 Fix #475: Retry finding the install partitions (#479) 2017-08-30 18:58:51 +00:00
clayton craft a9e5b362dc Implement zapping of apk packages with ver different than aports (466) (#474)
This adds a new option to `zap`: `-m / --mismatch-bins`

When set, any binary apks in the work directory packages folder will be
removed if their version differs from the version in the relevant
APKBUILD in aports.
2017-08-28 20:34:03 +00:00
Oliver Smith 25bad18830 Close #296: Add wiki links to repository warnings (#473)
Also update one wiki link, that pointed to the old wiki.
2017-08-27 14:00:53 +00:00
clayton craft 3942a06a2d Add -y/--assume-yes option to install (#450) (#467) 2017-08-26 15:50:04 +00:00
Oliver Smith c33db6315e Fix #463: Ignore exit code from parted (#465)
Parted often succeeds, but then returns a non-zero exit code, because
it can not inform the kernel of the changes.
In most cases this is not even necessary, so it really should not
fail there.
When the error was fatal, pmbootstrap will crash shortly afterwards
when it tried to mount or run mkfs on the partition anyway.
2017-08-25 17:39:52 +00:00
Attila Szöllősi fbe968f1ab Recovery installer zip (#404)
Supports flashing with TWRP and other Android recovery OS through adb sideload,
as well as exporting a generated recovery zip file.
See also:
https://github.com/postmarketOS/pmbootstrap/wiki/deviceinfo_flash_methods#recovery-mode-adb
2017-08-24 21:07:36 +00:00
Pablo Castellano 3954883547 Write custom os-release (closes #324) (#439)
* Write custom os-release (closes #324)
* Return empty string instead of crashing when the directory is not a git repository
* Use https in homepage URL
2017-08-23 16:40:16 +00:00
indefini a0ba9895c3 If there is no resolv.conf, create an empty file. (#423) 2017-08-20 12:34:31 +00:00
Martijn Braam c536e4ea58 Add setting for device keymap for devices with keymaps (#379)
* Added rx51_us keymap to nokia n900
* Added keymap option to init
* Made install command run setup-keymap when neccesary
* Validate keymap on install
2017-08-19 21:40:20 +00:00
clayton craft 2d6f9befca Disable zap on init for new devices (#418)
The device chroot shouldn't exist yet anyways. Also add a 'device_exists'
variable that can be used to determine if the device entered earlier in
the init has a deviceinfo under aports/device/device-*
2017-08-19 16:04:53 +00:00
Oliver Smith c323f21ca5 (binary repo reated) Don't specify pkgnames from "provides" as dependencies (#416)
* Don't specify pkgnames from "provides" as dependencies

Always use the regular pkgname. That way, we avoid listing all
kinds of so: files as dependencies (because Alpine automatically
adds them as depends= to the package database). This fixes building
weston, and reproducing the build with `pmbootstrap challenge`.

Additional changes.
* Clear the parsed APKINDEX cache for the current pmbootstrap
  session after building a package
* Avoid rebuilding a package, in case it was already built due to
  circular dependencies
2017-08-19 12:52:11 +00:00
NotKit 689eb5c3c9 Fix APKBUILD attribute parsing when opening quote is followed by line break (#415)
his adds a check to continue attribute parsing when end_char is found
in the end of first line, but is present only once. Found in gtk+2.0 APKBUILD:
https://git.alpinelinux.org/cgit/aports/plain/main/gtk+2.0/APKBUILD
2017-08-19 01:15:28 +00:00
Oliver Smith 1c13ca4fd9 Fix #363: Omit /home/user when calculating system image size (#389)
* The system image size is now calculated as: root size - home size.
* New function in `pmb/helpers/other.py`: `folder_size()`, with a
  testcase.
* Instead of copying everything to the system image folder, and
  deleting the home folder afterwards, do not copy the home folder
  in the first place.
* Added `pmbootstrap -s` to skip generating the initramfs for faster
  debugging.
* Set the default value in the "are you sure, that your partition has
  at least..." to "y", so we can run `yes '' | pmbootstrap install`
  to make it run through the whole installation process.
* Increase full size to 120%, boot partition gets 15 MB free space now
2017-08-18 19:19:48 +00:00
clayton craft d0f09ca0d0 Resolve #361 by zapping existing chroots after init (#385)
This extends zap() to add a 'no_confirm' option (False by default), and
zap() is now called by init with no_confirm=True to automatically zap
any existing chroots after the user runs init. This helps insure that
what is installed in the chroots is exactly what the user expects after
setting options in init.

Additionally, we create `cache_http` to verify write access to the work
folder instead of `chroot_native`. So we can ask for zapping only if
no chroot folder exists.
2017-08-18 16:25:58 +00:00
Anton Lazarev fa3c21c1c3 Now with 100% more `pmbootstrap log`! (#406) 2017-08-17 23:27:06 +00:00
clayton craft 5144323137 Allow users to copy ssh pubkey to device (#387) 2017-08-16 17:30:38 +00:00
clayton craft 3800cd09f4 Do not modprobe loop if /dev/loop* exists (#391) 2017-08-15 18:10:04 +00:00
Oliver Smith ff9f2d620f Fix #271: properly resolve symlinks in all paths (#329)
I've replaced all instances in the code of `os.path.abspath`
with `os.path.realpath`, as this does the same as `abspath`
plus resolving symlinks.
See also: https://stackoverflow.com/a/40311142
2017-08-15 14:08:48 +00:00
drebrez a1370bafe1 Always use generated boot.img for `fastboot boot` command (#370) 2017-08-14 23:03:27 +00:00
Pablo Castellano 401c29af76 Close #326: Implement command to retrieve and set configuration values (#359)
* Implement command to retrieve and set configuration values
* qemu: show advice to use "pmbootstrap config"
* Allow "pmbootstrap config" without positional arguments (prints the full config)
* Check valid variable names
2017-08-14 14:25:28 +00:00
Oliver Smith c904ffc751 Make gcc-aarch64 reproducible (#366)
This fixes https://github.com/postmarketOS/binary-package-repo/issues/1

GCC generates hardlinks between files `A` and `B` in its `make install` step. The problem is, that `tar` randomly packages `A` as full binary, and links `B` to `A`, or the other way around! I was able to reproduce this issue consistently when re-building `gcc-aarch64` on Travis CI (interestingly, this did not appear for `gcc-armhf`).

The fix is, to delete `B` and create a symlink `B` that points to `A` instead.
2017-08-12 14:06:02 +00:00
Oliver Smith f3f21d3152 Fix #342: don't use distutils.version.LooseVersion anymore (#364)
Previously, distutils.version.LooseVersion was used, because it was
sort of close enough to how Alpine parses versions.

This new version uses the exact same algorithm, as `apk` does, and
it passes *all* of `apk`'s testcases for version checking (previously
we simply skipped the ones, that did not pass).

* Remove pmb/helpers/version.py left-over (it is in parse now)
* Make asserts consistent, do not use unnecessary parenthesis
2017-08-12 14:03:40 +00:00
Pablo Castellano 8d358837ee Minor fixes (#360)
* aports: Use $install variable inside $source

So these files also get fingerprinted

* qemu: Use shutil.which instead of pmb.helpers.run.user

* Fix typo in comment
2017-08-10 23:58:28 +00:00
Pablo Castellano 49b35ad32d Close #226: Launch postmarketOS in a qemu virtual machine (#350)
Thanks to Pablo Castellano and Martijn Braam!
In postmarketOS we are now able to generate system images with the
correct configuration so that they can boot already using qemu

This commit brings the `pmbootstrap qemu` action.
This command is very handy because you don't have to set all the
qemu parameters, pmbootstrap does it for you.

* device-qemu-vexpress: Added kernel command line according to wiki
* qemu: Added workaround for image writing permissions
* qemu: Added support to launch postmarketOS in a QEMU virtual machine

- Support for emulating these architectures in QEMU: arm, aarch64, x86_84
- Generate QEMU command correctly depending no guest architecture (arm/x86)
- Run QEMU in the same architecture as the host by default
- Refactoring in pmb.parse.arch and pmb.qemu.run
- Raise exception if DTB file or system image are not present
- Display more useful information when something fails (e.g. image not found)
- Run qemu version depending on arch (host or argument), not device configured

* device-qemu-amd64: set deviceinfo_kernel_cmdline to "PMOS_NO_OUTPUT_REDIRECT"
* qemu: added --memory argument to specific guest RAM
* device-qemu-amd64: adjusted deviceinfo_kernel_cmdline (console=tty1)
* Added /etc/network/interfaces for qemu-amd64
* qemu: Added KVM support if /dev/kvm if present
* Specify separate machines for architecture
* qemu: Check if QEMU is installed instead of crashing
* Added graphics driver to qemu-aarch64

- Use arm (as used in qemu) instead of armhf (used in Alpine)
- qemu argument is -dtb
- Follow same style to build the command + arguments

* qemu: Added SSH port redirection: ./pmbootstrap.py qemu -p 2222
2017-08-09 20:26:40 +00:00
drebrez 96cb52847d Increase minimal system image size by 15% to have some free space (#336) 2017-08-09 19:16:04 +00:00
Oliver Smith 21c09b3b3c Fix various build issues (fix #189, fix #341) (#345)
Changes:
* Removed the apkindex_files cache. That particular cache caused
  bug #189 and didn't bring any real-world performance improvements
  (tested 3x with that cache and 3x without, no significant speed
  difference). I decided to remove it instead of keeping it/adding
  even more code to resolve the bug.
* Fix the check for already built packages: always use the architecture,
  that the package should be built for instead of the architecture of
  the build environment (e.g. use armhf, even when building a noarch
  package in the x86_64 chroot). This partially resolves #341.
* Make pmb.chroot.apk.install_is_necessary() more robust: If the binary
  package is missing, although it should be there, print a warning and
  build it with force.
2017-08-09 17:59:21 +00:00
Oliver Smith ae64894278
Fix #349: Verify pkgname against folder name after replacing vars
This fixes a regression introduced in 7fe2bc18c3.
Sorry for the inconvenience!
2017-08-08 15:03:17 +02:00
Oliver Smith 7fe2bc18c3 Close #334: Check if pkgname is the aport folder name (#347)
...and remove an redundant sentence with broken grammar in another
error message.
2017-08-07 22:53:54 +00:00
drebrez 257666799e Add `-c, --clear` argument for `log` and `log_distccd` actions to clear log files (#344) 2017-08-07 18:56:35 +00:00
Pablo Castellano b41e75e662 Minor fixes (#316)
* Fixed typo and specify list order

* Simplify code for jobs default value
2017-08-06 12:50:38 +00:00
Oliver Smith 5eb9b92e7b Fix: gcc-armhf not reproducible (#64) (#333)
libstdc++.a from gcc-armhf was not reproducible on Travis (it was, when built locally!). These .a files are just archives of object files .o, and in this case it was caused by a random order of the .o files in the archive.

This PR patches the package() function of the APKBUILD when running pmbootstrap aportgen gcc-armhf (same for aarch64 of course), to extract all .a files, and repack them to be reproducible (by sorting the files before packing them).

As usually, we can still inherit everything from the upstream gcc aport from Alpine, and apply our changes on top of that.

Travis without the patch:
https://api.travis-ci.org/jobs/260402679/log.txt?deansi=true

> CHALLENGE FAILED for usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a:File 'usr/armv6-alpine-linux-muslgnueabihf/lib/libstdc++.a' is different!

Travis with the patch (I've instructed Travis to run off this branch to test it):
https://api.travis-ci.org/jobs/260806203/log.txt?deansi=true

> Done. Your build exited with 0.
2017-08-04 20:38:27 +00:00
Oliver Smith 742e4624e8 Fix #319: pmbootstrap menuconfig didn't work for mainline kernel (#328)
Also did some minor refactoring.
2017-08-03 16:11:23 +00:00
drebrez 147863d98a Automatically compute the minimum size for the partition and resize it during the boot process (#127)
* Automatically compute the minimum size for the partitions
* Automatically resize the pmOS_root partition during the boot process
* Resize root partition only if there is unallocated space at the end of the device.
* Added more echos to make debugging easier while looking at the pmOS_init.log.
* Updated static_code_analysis.sh script to run shellcheck with `-x` option.
2017-08-03 16:01:00 +00:00
Oliver Smith 025d646e47 Close #314: Add --details-to-stdout parameter (#317)
When this parameter is set, `args.logfd` (the log file descriptor)
gets set to `sys.stdout`, so all logging gets directly sent to the
active pmbootstrap session, and not to a log file.

This will be used for Travis jobs, because they get killed after
10 minutes without any output (and builds may take longer).
2017-08-02 19:38:42 +00:00
Pablo Castellano 0e4017105d Implement sparse system images (fix #299) (#303)
* Packaged libsparse

libsparse from the Android project provides multiple tools like img2simg
and simg2img.
These are used to split a large image for the system partition into
separate smaller chunks with sparse headers

This is required for several devices (at least bullhead, fp2 and titan)
because it fixes the "Invalid sparse file format at header magi" error

https://github.com/postmarketOS/pmbootstrap/issues/299

* Added new variable deviceinfo_flash_sparse (fixes #299)

Right after the system image is generated, pmbootstrap checks this
variable. In case it is true, run img2simg on it

* motorola-titan: enable deviceinfo_flash_sparse

* libsparse: use source from github: anestisb/android-simg2img

It is not that easy to use the upstream archive because everytime
you download it, the files have the current date as creation date
and that makes the file have a different checksum every download
https://github.com/postmarketOS/pmbootstrap/pull/303#issuecomment-319017197
2017-08-02 16:21:50 +00:00
Pablo Castellano 8389448bfd init: Allow to specify extra packages that will be always installed to rootfs (#304) 2017-08-01 15:31:33 +00:00
Oliver Smith fe385cad6d Fix #300: Properly unset environment in chroot (#302) 2017-07-30 22:08:36 +00:00
drebrez 6cb663eb6e Add possibility to export Odin flashable tar (#297)
Add possibility to export Odin flashable tar for devices where the
flasher method is heimdall-isorec or heimdall-bootimg
2017-07-30 21:15:59 +00:00
Pablo Castellano 915da12dd1 Allow 'pmbootstrap menuconfig' without specifying "linux-" (#295)
We can allow both ways:
* pmbootstrap menuconfig linux-motorola-titan
* pmbootstrap menuconfig motorola-titan

The former will show a tip about the second
This also prevents users from running menuconfig on aports like
'device-motorola-titan' or 'mkbootimg'
2017-07-30 18:41:49 +00:00
clayton craft ed8fba54c3 pmb.helpers.ui scan for UIs in new aport structure (#290)
This fixes #289.
2017-07-30 00:34:34 +02:00
Pablo Castellano 2cbf56a4c4 Usability improvements, fix #278 (#279)
* Quote architecture in logging message for easier reading
* Added shortcut arguments for --rootfs, --buildroot and --suffix
* Simply remove beforehand link to nowhere if exists (fix #278)
Fixed crash when symlink already existed but pointed to a non-existing location
2017-07-29 15:16:43 +00:00
Oliver Smith af007250d4
Bump pmbootstrap version to 0.2.0 2017-07-29 01:09:07 +02:00
Oliver Smith 314c17e03c Close #194: Aports subfolders! See migration guide in the wiki (#227)
Migration guide:
https://github.com/postmarketOS/pmbootstrap/wiki/Migration-to-aports-subfolders
2017-07-28 22:34:40 +00:00
Oliver Smith 8b17ac05ea
Fix #282: '/bin/ash': Permission denied
This was a regression from the improved input validation PR, which
created the chroot_native folder with the wrong permissions (to test
if we have write access to the work folder).

sjamaan found out, that this was the cause of the issue, and also
explained how to fix it, thanks a lot!
2017-07-28 22:15:51 +02:00
drebrez 7283f03315 Split up initramfs to make it smaller (Close #211) (#257)
Huge thank you to @drebrez for his amazing work on this PR!

* Add generation of initramfs-extra with additional binaries
Extract both initramfs with `pmbootstrap initfs extract`
Add new splashscreens for missing partitions/files errors
Changes in init script:
 - use busybox findfs applet to find boot partition
 - mount boot partion
 - extract initramfs-extra
 - show error splashscreens accordingly and loop forever
 - start usb unlock directly from unlock_root_partition (hook removed)

* Print out a text message for serial debugging in case of errors
Add initramfs-extra files to `pmbootstrap initfs ls` output

* Fix trailing whitespace in comment...

* ls: Indicate which initramfs we're looking at / add wiki link

I've rewritten the initramfs-development article to reflect the
changes made in this PR. It will be a good read for someone who
extracted the initramfs and wants to know why we have two files.
2017-07-28 18:51:21 +00:00
Oliver Smith 0ae6a150e8 Fix #269: Always use truncate to create empty image file (#276)
I've realized that truncate is provided by busybox anyway, so no
need to install coreutils for this.

This should fix creating system image files on ZFS, where fallocate
is not implemented.
2017-07-28 17:41:35 +00:00
Oliver Smith 67867a9c6e Fix #226: Building busybox recurses forever. (#252)
If you really want to build busybox, I recommend turning the
"timestamp based rebuilds" feature off, otherwise it will build
for all architectures all the time whenever you change something,
even if you do not increase the version number (that's the idea
of that feature). This is, because busybox is a dependency for
basiscally everything, so it must get updated whenever you install
something, in case it was out of date.

It is easier to simply rename the package.
2017-07-28 17:40:27 +00:00
Oliver Smith 10bf08dca1 Fix #166: pmbootstrap shutdown: umount deep folder levels first (#274)
* Refactored `umount_all` to get the list of folders to be umounted from
  `umount_all_list`, so we can test that function in a test case.
* Adjusted `umount_all_list` to return the deep folder levels first
* Added a testcase for that
* Remove redundant calls to `umount_all()` (which were from a time before
  `pmbootstrap` was released, in which failing commands did not cause
  `pmbootstrap` to raise an exception)
2017-07-27 18:14:02 +00:00