From bcf74324dbe1501707343a2c61b5ba471fbaa0b9 Mon Sep 17 00:00:00 2001 From: Tony Garnock-Jones Date: Fri, 11 Feb 2022 10:48:49 +0100 Subject: [PATCH] Progress from yesterday --- src/SUMMARY.md | 97 +++++++++++++++++++-- src/architecture.md | 15 ++-- src/glossary.md | 32 ++++--- src/guide/index.md | 1 + src/{ => guide}/preserves.md | 0 src/install.md | 110 +++++++++++++++++++++++- src/introduction.md | 31 ++++++- src/operation/builtin/config-watcher.md | 1 + src/operation/builtin/daemon.md | 1 + src/operation/builtin/gatekeeper.md | 1 + src/operation/builtin/index.md | 1 + src/operation/builtin/relay-listener.md | 1 + src/operation/index.md | 67 +++++++++++++++ src/operation/scripting.md | 1 + src/operation/system-bus.md | 43 +++++++++ 15 files changed, 371 insertions(+), 31 deletions(-) create mode 100644 src/guide/index.md rename src/{ => guide}/preserves.md (100%) create mode 100644 src/operation/builtin/config-watcher.md create mode 100644 src/operation/builtin/daemon.md create mode 100644 src/operation/builtin/gatekeeper.md create mode 100644 src/operation/builtin/index.md create mode 100644 src/operation/builtin/relay-listener.md create mode 100644 src/operation/index.md create mode 100644 src/operation/scripting.md create mode 100644 src/operation/system-bus.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index f391bcf..b3b4b9a 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -1,11 +1,98 @@ # Summary -[Overview](./introduction.md) +[Introduction](./introduction.md) + +# Preliminaries - [Architecture](./architecture.md) -- [Installation](./install.md) -- [Preserves](./preserves.md) -- [Protocol](./protocol.md) +- [Source code, Building, and Installation](./install.md) +- [Glossary](./glossary.md) -[Glossary](./glossary.md) +# User and Administration Guide +- [System overview](./operation/index.md) +- [The System Bus: syndicate-server](./operation/system-bus.md) + - [Bus scripting language](./operation/scripting.md) + - [Built-in services and service classes](./operation/builtin/index.md) + - [Gatekeeper](./operation/builtin/gatekeeper.md) + - [TCP/IP and Unix-socket Transports](./operation/builtin/relay-listener.md) + - [Configuration watcher](./operation/builtin/config-watcher.md) + - [Process supervision and management](./operation/builtin/daemon.md) +- [Configuration files and directories]() + - [The boot layer]() + - [Logging]() + - ["Machine" dataspace]() + - ["exec" of one-off tasks]() + - [The core layer]() + - [The services layer]() + - [Networking core]() + - [Wifi & Mobile Data]() + - [Simple daemons]() + - [docker]() + - [ntpd]() + - [sshd]() + - [User settings]() +- [Defining services and service classes]() +- [Scheduling tasks]() + - [Timer tolerances]() + - [One-off tasks]() + - [Repeating tasks]() +- [Managing user settings]() +- [Restarting services]() +- [Rebooting and powering off the machine]() +- [Suspending the machine]() +- [Standard tools]() + - [preserves-tools]() +- [Standard libraries]() + - [Python support libraries]() + - [Shell-scripting libraries]() +- [Standard protocols and schema definitions]() + - [Preserves schemas]() + - [Preserves Schema metaschema]() + - [Preserves Path schema]() + - [Syndicated Actor Model schemas]() + - ["Observe" assertions]() + - [Patterns over assertions]() + - [Gatekeeper and Sturdy-references]() + - [Wire-protocol]() + - [Service dependencies]() + - [Tracing]() + - [Transport addresses]() + - [Synit schemas]() + - [Audio control]() + - [Telephony (call and SMS) support]() + - [Modem support]() + - [Hayes]() + - [Samsung Galaxy S7]() + - [MIME type support]() + - [Network core status and configuration]() + - [Sound effects]() + - [Time stamps]() + - [User interface definitions and interaction]() + - [User settings]() + +# Programming Guide and Reference + +- [Overview](./guide/index.md) +- [Preserves](./guide/preserves.md) +- [Schemas and schema bundles]() +- [Language-neutral protocols]() +- [Rust]() +- [Python]() +- [Smalltalk]() +- [Bash (Shell)]() +- [Capturing and rendering interaction traces]() + +# Specifications and Theory + +- [Protocol specification](./protocol.md) + +- [Syndicated Actor Model]() + - [Actors, Entities, Assertions and Messages]() + - [Dataspaces]() + - [Object-capabilities]() + - [Privilege separation]() + - [Attenuation of authority for security]() + - [Attenuation of authority for privacy]() + +- [System layer analysis]() diff --git a/src/architecture.md b/src/architecture.md index 71bfb32..f685c82 100644 --- a/src/architecture.md +++ b/src/architecture.md @@ -7,15 +7,12 @@ > Giroux](https://twitter.com/__simt__/status/1489790925266059264), > Feb. 2022 -The **[Syndicated Actor Model -(SAM)](./glossary.md#syndicated-actor-model)** is the core -architecture of Synit. The SAM builds upon [E-style -actors](./glossary.md#e), replacing message-exchange with -eventually-consistent *state replication* as the fundamental building -block for interaction. Both E and the SAM are instances of the [Object -Capability (ocap) model](./glossary.md#object-capability-model), a -[compositional](./glossary.md#compositional) approach to system -security. +The **[Syndicated Actor Model (SAM)](./glossary.md#syndicated-actor-model)** is at the heart of +Synit. In turn, the SAM builds upon [E-style actors](./glossary.md#e), replacing +message-exchange with eventually-consistent *state replication* as the fundamental building +block for interaction. Both E and the SAM are instances of the [Object Capability (ocap) +model](./glossary.md#object-capability-model), a [compositional](./glossary.md#compositional) +approach to system security. The "feel" of the system is somewhere between Smalltalk-style object-orientation, publish-subscribe programming, E- or Erlang-style diff --git a/src/glossary.md b/src/glossary.md index 76b81cc..8298f29 100644 --- a/src/glossary.md +++ b/src/glossary.md @@ -1,21 +1,25 @@ # Glossary +## Actor +## Assertion +## Bus Scripting Language +## Capability +## Compositional +## Conversational State +## Dataspace +## E +## Facet +## Macaroon +## Message +## Object Capability Model +## Preserves +## S6 +## Schema +## Supervision tree +## Supervisor ## Syndicated Actor Model Often abbreviated **SAM**. -## Object Capability Model - -## Facet -## Assertion -## Dataspace ## System Layer -## Actor -## Message -## Conversational State -## Capability -## Preserves -## E -## Compositional -## Schema -## Macaroon +## System dataspace diff --git a/src/guide/index.md b/src/guide/index.md new file mode 100644 index 0000000..07dd0c5 --- /dev/null +++ b/src/guide/index.md @@ -0,0 +1 @@ +# Overview diff --git a/src/preserves.md b/src/guide/preserves.md similarity index 100% rename from src/preserves.md rename to src/guide/preserves.md diff --git a/src/install.md b/src/install.md index 25267fe..4f856c6 100644 --- a/src/install.md +++ b/src/install.md @@ -1 +1,109 @@ -# Installation +# Source code, Building, and Installation + +The initial application of Synit is to *mobile phones*. + +As such, in addition to regular [system layer](./glossary.md#system-layer) concepts, Synit +supports concepts from mobile telephony: calls, SMSes, mobile data, headsets, speakerphone, +hotspots, battery levels and charging status, and so on. + +Synit builds upon many existing technologies, but primarily relies on the following: + + - [PostmarketOS](https://postmarketos.org/). Synit builds on PostmarketOS, replacing only a + few core packages. All of PostmarketOS and Alpine Linux are available underneath Synit. + + - [Preserves](https://preserves.gitlab.io/). The Preserves data language and its associated + schema and query languages are central to Synit. + + - [Syndicate](https://syndicate-lang.org/). Syndicate is an umbrella project for tools and + specifications related to the [Syndicated Actor Model](./glossary.md#syndicate-actor-model) + (the SAM). + +## You will need + + - A Linux development system. (I use Debian testing/unstable.) + + - Rust nightly and Cargo (perhaps installed via [rustup](https://rustup.rs/)). + + - Python 3.9 or greater + + - `git`, `ssh`, `rsync` + + - Make, a C compiler, and so on; standard Unix programming tools. + + - Source code for Synit components (see below). + + - A standard PostmarketOS distribution for the target computer or mobile phone. If you don't + want to install on actual hardware, you can use a virtual machine. See the [instructions for + installing PostmarketOS](https://wiki.postmarketos.org/wiki/Installation_guide). + + - **Great tolerance for the possibility of soft-bricking your phone.** This is experimental + software! When it breaks, you'll often have to (at least) reinstall PostmarketOS from + absolute scratch on the machine. I do lots of development using `qemu-amd64` for this + reason. + +## Get the code + +The Synit codebase itself is contained in the [`syndicate-system` git +repository](https://git.syndicate-lang.org/synit/synit): + + git clone https://git.syndicate-lang.org/synit/synit + +See the [README](https://git.syndicate-lang.org/synit/synit/src/branch/main/README.md) for an +overview of the contents of the repository. + +Synit depends on published packages for Preserves and Syndicate support in each of the many +programming languages it uses. These will be automatically found and downloaded during the +Synit build process, but you can find details on the [Preserves](https://preserves.gitlab.io/) +and [Syndicate](https://syndicate-lang.org/) homepages, respectively. + +## Build the packages + +To build, type `make ARCH=`*\* in the root of your checkout, where +*\* is one of: + + - `aarch64` (default), for e.g. Pinephone or Samsung Galaxy S7 deployment + - `x86_64`, for e.g. `qemu-amd64` deployment + +The result of the build will be a collection of Alpine Linux `apk` packages in +`packaging/target/packages/`*\*`/`. At the time of writing, these include + + - `preserves-schemas`, common schema files for working with general Preserves data and schemas + - `preserves-tools`, standard command-line tools for working with Preserves documents + (pretty-printer, document query processor, etc.) + - `py3-preserves`, python support libraries for Preserves + - `py3-syndicate`, python support for the Syndicated Actor Model + - `squeak-cog-vm` and `squeak-stack-vm`, Squeak Smalltalk virtual machine for the + Smalltalk-based portions of the system + - `syndicate-schemas`, common schema files for working with the Syndicated Actor Model + - `syndicate-server`, package for the core system bus + - `synit-config`, main package for Synit, with configuration files, `init` scripts, system + daemons and so on. + - `synit-pid1`, PID1 program for Synit that starts the core system bus and then becomes + passive + +## Install PostmarketOS on your system + +Follow the instructions for your device on the [PostmarketOS +wiki](https://wiki.postmarketos.org/wiki/Devices). + +Boot and connect your device to your development machine. Make sure you can `ssh` into it. + +## Upload Synit packages + +Use `scripts/upload-bundle.sh` to rsync the ingredients for transforming stock PostmarketOS to +Synit to the phone. + +## Run the transformation script + +Use `ssh` to log into your phone. Run `./transmogrify.sh`. (If your user's password on the +phone is anything other than `user`, you will have to run `SUDOPASS=yourpassword +./transmogrify.sh`.) + +This will install the Synit packages. After this step is complete, the system will boot into +Synit; it may very well be unbootable, depending on the state of the codebase! Make sure you +know how to restore to a stock PostmarketOS installation. + +## Reboot and hope + +With luck, you'll see the Smalltalk user interface start up. From here, you can operate the +system normally, following the information in [the following chapter](./operation/index.md). diff --git a/src/introduction.md b/src/introduction.md index fdaa1d9..d20d504 100644 --- a/src/introduction.md +++ b/src/introduction.md @@ -1,4 +1,4 @@ -# The Synit Manual +# Synit: an Alternative System Layer ## Welcome! @@ -17,7 +17,22 @@ Linux's Unix heritage. It is currently **experimental** software. - Project homepage: - - Source code: + + - Source code: + +## Organisation of this manual + +This manual is organised into sections: + + 1. Preliminaries. **[Architecture](./architecture.md)**, **[installation + instructions](./install.md)**, and **[project glossary](./glossary.md)**. + + 2. **[User and Administration Guide.](./operation/index.md)** The system handbook for Synit. + + 3. **[Programming Guide and Reference.](./guide/index.md)** How to write programs for Synit. + + 4. **[Specifications and Theory.](./theory/index.md)** Deeper investigation of some of the + concepts underpinning Synit's architecture. ## Quickstart @@ -37,3 +52,15 @@ Zero](https://nlnet.nl/NGI0/) [PET](https://nlnet.nl/PET/) programme. Please see "[Structuring the System Layer with Dataspaces (2021)](https://syndicate-lang.org/projects/2021/system-layer/)" for details of the funded project. + +## Copyright and License + +Creative Commons License +This manual is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/). +Copyright © 2021–2022 Tony Garnock-Jones . + +The Synit programs and source code are separately licensed. Please see the source code for +details. diff --git a/src/operation/builtin/config-watcher.md b/src/operation/builtin/config-watcher.md new file mode 100644 index 0000000..7484ce4 --- /dev/null +++ b/src/operation/builtin/config-watcher.md @@ -0,0 +1 @@ +# Configuration watcher diff --git a/src/operation/builtin/daemon.md b/src/operation/builtin/daemon.md new file mode 100644 index 0000000..4cb35e0 --- /dev/null +++ b/src/operation/builtin/daemon.md @@ -0,0 +1 @@ +# Process supervision and management diff --git a/src/operation/builtin/gatekeeper.md b/src/operation/builtin/gatekeeper.md new file mode 100644 index 0000000..6d79d3f --- /dev/null +++ b/src/operation/builtin/gatekeeper.md @@ -0,0 +1 @@ +# Gatekeeper diff --git a/src/operation/builtin/index.md b/src/operation/builtin/index.md new file mode 100644 index 0000000..62c1525 --- /dev/null +++ b/src/operation/builtin/index.md @@ -0,0 +1 @@ +# Built-in services and service classes diff --git a/src/operation/builtin/relay-listener.md b/src/operation/builtin/relay-listener.md new file mode 100644 index 0000000..5dbb54b --- /dev/null +++ b/src/operation/builtin/relay-listener.md @@ -0,0 +1 @@ +# TCP/IP and Unix-socket Transports diff --git a/src/operation/index.md b/src/operation/index.md new file mode 100644 index 0000000..d2fb04e --- /dev/null +++ b/src/operation/index.md @@ -0,0 +1,67 @@ +# System overview + +Synit uses the Linux kernel as a hardware abstraction and virtualisation layer. + +All processes in the system are arranged into a [supervision +tree](../glossary.md#supervision-tree), conceptually rooted at the [system +bus](./system-bus.md) (NB. not at PID 1). + + PID TTY STAT TIME COMMAND + 1 ? Ssl 0:00 /sbin/synit-pid1 + 1034 ? Sl 0:00 /usr/bin/syndicate-server --inferior --config /etc/syndicate/boot + 1045 ttyS0 Ss+ 0:00 \_ getty 0 /dev/console + 1052 ? S 0:01 \_ /sbin/udevd --children-max=5 + 1113 ? Sl 0:11 \_ python3 /usr/lib/synit/interface-monitor + 1129 ? S 0:00 \_ sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups + 1286 ? S 0:00 \_ wpa_supplicant -Dnl80211,wext -C/run/wpa_supplicant -i wlan0 + 1287 ? S 0:00 \_ udhcpc -i usb0 -fR -s /usr/lib/synit/udhcpc.script + 1390 ? S 0:00 \_ ntpd -d -n -p pool.ntp.org + 1431 ? S 0:01 \_ python3 /usr/lib/synit/user-settings-daemon + 1497 ? Sl 0:01 \_ python3 /usr/lib/synit/wifi-daemon + 1516 ? S 0:00 \_ udhcpc -i wlan0 -fR -s /usr/lib/synit/udhcpc.script + 1035 ? S 0:00 s6-log t s999999 n500 /var/log/synit + +## Boot process + +The kernel first loads the stock PostmarketOS `initrd`, which performs a number of important +tasks and then delegates to `/sbin/init`. + +### /sbin/init = synit-init.sh + + - Source code: [`[synit]/packaging/packages/synit-config/files/sbin/synit-init.sh`](https://git.syndicate-lang.org/synit/synit/src/branch/main/packaging/packages/synit-config/files/sbin/synit-init.sh) + +The [`synit-config` package](../install.md#build-the-packages) overrides the usual contents of +`/sbin/init`, replacing it with a short shell script, `synit-init.sh`. This script, in turn, +takes care of a few boring tasks such as mounting `/dev`, `/proc`, `/run`, etc., ensuring that +a few important directories exist, and remounting `/` as read-write before `exec`ing +`/sbin/synit-pid1`. + +For the remainder of the lifetime of the system, `/sbin/synit-pid1` is the PID 1 `init` +process. + +### /sbin/synit-pid1 + + - Source code: [`[synit]/synit-pid1/`](https://git.syndicate-lang.org/synit/synit/src/branch/main/synit-pid1) + - Packaging: [`[synit]/packaging/packages/synit-pid1/`](https://git.syndicate-lang.org/synit/synit/src/branch/main/packaging/packages/synit-pid1) + +The `synit-pid1` program starts by spawning the [system bus](./system-bus.md) +(`syndicate-server` in the process tree above) and the program `/sbin/synit-log`, connecting +`stderr` of the former to `stdin` of the latter. + +It then goes on to perform two tasks concurrently: the first is the Unix +[`init`](https://en.wikipedia.org/wiki/Init) role, reaping zombie processes, and the second is +to interact with the system bus as an ordinary system service. + +The latter allows the system to treat `init` just like any other part of the system, accessing +its abilities to reboot or power off the system using messages and assertions in the system +dataspace as usual. + +Even though `synit-pid1` is, to the kernel, a *parent* process of `syndicate-server`, it is +logically a *child* process. + +### /sbin/synit-log + + - Source code: [`[synit]/packaging/packages/synit-pid1/synit-log`](https://git.syndicate-lang.org/synit/synit/src/branch/main/packaging/packages/synit-pid1/synit-log) + +This short shell script invokes the [S6](../glossary.md#s6) program `s6-log` to capture log +output from the system bus, directing it to files in `/var/log/synit/`. diff --git a/src/operation/scripting.md b/src/operation/scripting.md new file mode 100644 index 0000000..088b83f --- /dev/null +++ b/src/operation/scripting.md @@ -0,0 +1 @@ +# Bus scripting language diff --git a/src/operation/system-bus.md b/src/operation/system-bus.md new file mode 100644 index 0000000..0bb413c --- /dev/null +++ b/src/operation/system-bus.md @@ -0,0 +1,43 @@ +# The System Bus: syndicate-server + + - Source code: [`[syndicate-rs]/syndicate-server/`](https://git.syndicate-lang.org/syndicate-lang/syndicate-rs/src/branch/main/syndicate-server) + - Packaging: [`[synit]/packaging/packages/syndicate-server/`](https://git.syndicate-lang.org/synit/synit/src/branch/main/packaging/packages/syndicate-server) + +The `syndicate-server` program has a number of closely-related functions. In many ways, it is a +reification of the [system layer concept](../glossary.md#system-layer) itself. + + 1. It provides a root **system bus** service for use by other programs. In this way, it is + analogous to D-Bus. + + 2. It provides a [**gatekeeper** service](./builtin/gatekeeper.md), for exposing + [capabilities](../glossary.md#capability) to running objects as (potentially long-lived) + [macaroon](../glossary.md#macaroon)-style "sturdy references", plus TCP/IP- and + Unix-socket-based **[transports](./builtin/relay-listener.md)** for accessing capabilities + through the gatekeeper. + + 3. It provides a limited **[configuration scripting language](./scripting.md)** suitable for + programming [dataspaces](../glossary.md#dataspace) with simple reactive behaviours. + + 4. It provides an [`inotify`](https://en.wikipedia.org/wiki/Inotify)-based **[configuration + loader](./builtin/config-watcher.md)** which loads and executes configuration files written + in the scripting language. + + 5. It provides [**process supervision** and **dependency tracking**](./builtin/daemon.md) + services for starting other programs. + +## The main system bus + +The [`synit-pid1`](./index.md#sbinsynit-pid1) program invokes `syndicate-server` like this: + + /usr/bin/syndicate-server --inferior --config /etc/syndicate/boot + +The first flag, `--inferior`, tells the server to expect to be able to communicate on its +stdin/stdout using the [standard wire protocol](../protocol.md). This lets `synit-pid1` join +the community of actors running within the system dataspace. + +The second flag, `--config /etc/syndicate/boot`, tells the server to start monitoring the +directory tree rooted at `/etc/syndicate/boot` for changes. Files whose names end with `.pr` +within that tree are loaded as configuration script files. + +Almost all of Synit is a consequence of careful use of the configuration script files in +`/etc/syndicate`.