From dd67e2950eddc6e44869c4423ab8c6904d37625d Mon Sep 17 00:00:00 2001 From: Tony Garnock-Jones Date: Tue, 8 Mar 2022 06:58:12 +0100 Subject: [PATCH] Describe builtin services --- src/operation/builtin/config-watcher.md | 31 +++- src/operation/builtin/daemon.md | 181 +++++++++++++++++++++++- src/operation/builtin/gatekeeper.md | 17 +-- src/operation/builtin/logging.md | 22 +++ src/operation/builtin/relay-listener.md | 62 +++++++- src/operation/scripting.md | 3 +- src/operation/service.md | 8 +- 7 files changed, 299 insertions(+), 25 deletions(-) diff --git a/src/operation/builtin/config-watcher.md b/src/operation/builtin/config-watcher.md index af39700..241d573 100644 --- a/src/operation/builtin/config-watcher.md +++ b/src/operation/builtin/config-watcher.md @@ -1,7 +1,30 @@ # Configuration watcher - - Any number of [**configuration watchers**](./config-watcher.md) may be created to monitor - directories for changes to files written using the [server scripting - language](../scripting.md). (These can also be started from the `syndicate-server` - command-line with `-c` options.) + - Relevant schema: [[syndicate-rs]/syndicate-server/protocols/schemas/internalServices.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-rs/src/branch/main/syndicate-server/protocols/schemas/internalServices.prs) +Assertions [requiring](../service.md#require-service) a service with name matching +`ConfigWatcher` cause the server to start a configuration watcher service monitoring files in +and subdirectories of the given `path` for changes: + +```preserves-schema +ConfigWatcher = . +ConfigEnv = { symbol: any ...:... }. +``` + +The `path` may name either a file or directory. Any time the configuration watcher finds a file +matching the glob `*.pr` within the tree rooted at `path`, it loads the file. Each time a +`*.pr` file is loaded, it is interpreted as a [configuration scripting +language](../scripting.md) program, with a copy of `env` as the "initial environment" for the +script. + +Whenever a change to a `*.pr` file is detected, the configuration watcher reloads the file, +discarding previous internal state related to the file. + +Note that a quirk of the config language requires that there exist an entry in `env` with key +the symbol `config` and value an [entity reference](../../glossary.md#reference) (usually +denoting a [dataspace entity](../../glossary.md#dataspace)). However, the `config` entry need +not be the same as the surrounding `$config`! A useful pattern is to set up a new +`ConfigWatcher` with `env` containing a `config` binding pointing to an +[attenuated](../../glossary.md#attenuation) reference to the current `config` dataspace, or +even [an entirely fresh dataspace](../scripting.md#expr:dataspace) created specifically for the +task. diff --git a/src/operation/builtin/daemon.md b/src/operation/builtin/daemon.md index 3f4d07c..c581ad6 100644 --- a/src/operation/builtin/daemon.md +++ b/src/operation/builtin/daemon.md @@ -1,4 +1,181 @@ # Process supervision and management - - Finally, [**external programs**](./daemon.md) can be started, either as long-lived "daemon" - services or as one-off scripts. + - Relevant schema: [[syndicate-rs]/syndicate-server/protocols/schemas/externalServices.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-rs/src/branch/main/syndicate-server/protocols/schemas/externalServices.prs) + +Assertions [requiring](../service.md#require-service) a service with name matching +`DaemonService` cause the server to start a subprocess-based service: + +```preserves-schema +DaemonService = . +``` + +Each `daemon` service can have zero or more subprocesses associated with it. Subprocesses can +be long-lived services or short-lived, system-state-changing programs or scripts. + +## Adding process specifications to a service + +Each subprocess associated with a `DaemonService` is defined with a `DaemonProcess` assertion: + +```preserves-schema +DaemonProcess = . +DaemonProcessSpec = + / @simple CommandLine + / @oneShot + / @full FullDaemonProcess . +``` + +The simplest kind of subprocess specification is a `CommandLine`, either a string (sent to `sh +-c`) or an array of program name (looked up in the `$PATH`) and arguments: + +```preserves-schema +CommandLine = @shell string / @full FullCommandLine . +FullCommandLine = [@program string, @args string ...] . +``` + +The `simple` and `oneShot` variants of `DaemonProcessSpec` expand into `FullDaemonProcess` +values as follows: + + - a `simple` command-line *c* becomes `{ argv: `*c*` }`; and + - a record `` becomes `{ argv: `*c*`, readyOnStart: false, restart: on-error }`. + +## Subprocess specification + +The `FullDaemonProcess` type matches a Preserves dictionary having, at minimum, an `argv` key, +and optionally including many other parameters controlling various aspects of the subprocess to +be created.[^full-process-split-out] + +```preserves-schema +FullDaemonProcess = + & @process FullProcess + & @readyOnStart ReadyOnStart + & @restart RestartField + & @protocol ProtocolField . + +FullProcess = + & { argv: CommandLine } + & @env ProcessEnv + & @dir ProcessDir + & @clearEnv ClearEnv . +``` + +The `CommandLine` associated with `argv` specifies the program name to invoke and its +command-line arguments. The other options are described in the remainder of this section. + +### Ready-signalling + +If the key `readyOnStart` is present in a `FullDaemonProcess` dictionary, then if its +associated value is `#t` (the default), the service will be considered +[`ready`](../service.md#service-states) immediately after it has been spawned; if its value is +`#f`, some other arrangement is expected to be made to announce a `ready` `ServiceState` +against the service's name. + +```preserves-schema +ReadyOnStart = + / @present { readyOnStart: bool } + / @invalid { readyOnStart: any } + / @absent {} . +``` + +### Whether and when to restart + +The default restart policy is `always`. It can be overridden by providing the key `restart` a +`FullDaemonProcess` dictionary, mapping to a valid `RestartPolicy` value. + +```preserves-schema +RestartField = + / @present { restart: RestartPolicy } + / @invalid { restart: any } + / @absent {} . + +RestartPolicy = =always / =on-error / =all / =never . +``` + +The valid restart policies are: + + - `always`: Whether the process terminates normally or abnormally, restart it without affecting + any peer processes within the service. + + - `on-error`: If the process terminates normally, leave everything alone; if it terminates + abnormally, restart it without affecting peers. + + - `all`: If the process terminates normally, leave everything alone; if it terminates + abnormally, restart the whole daemon (all processes within the `Daemonservice`). + + - `never`: Treat both normal and abnormal termination as normal termination; that is, never + restart, and enter state [`complete`](../service.md#service-states) even if the process + fails. + +### Speaking Syndicate Network Protocol via stdin/stdout + +By default, the `syndicate-server` program assumes nothing about the information to be read and +written via a subprocess's standard input and standard output. This can be overridden with a +`protocol` entry in a `FullDaemonProcess` specification. (Standard error is always considered +to produce information to be put in the system logs, however.) + +```preserves-schema +ProtocolField = + / @present { protocol: Protocol } + / @invalid { protocol: any } + / @absent {} . + +Protocol = =none / =application/syndicate / =text/syndicate . +``` + +The available options for `protocol` are: + + - `none`: the standard input of the subprocess is connected to `/dev/null`, and the standard + output and standard error are logged. + + - `application/syndicate`: the subprocess standard input and output are used as a *binary* + syntax [Syndicate network protocol](../../protocol.md) relay. Standard error is logged. The + subprocess is expected to make some [entity](../../glossary.md#entity) available to the + server via [initial oid](../../glossary.md#initial-oid) 0. The server reflects this + expectation by automatically placing a [service object](../service.md#service-object) record + into the dataspace alongside the `daemon` record defining the subprocess. + + - `text/syndicate`: as for `application/syndicate`, but Preserves' *text* syntax is used + instead of binary syntax. + +### Specifying subprocess environment variables + +By default, the Unix process environment passed on to subprocesses is not changed. Supplying +`clearEnv` and/or `env` keys alters this behaviour. + +```preserves-schema +ClearEnv = + / @present { clearEnv: bool } + / @invalid { clearEnv: any } + / @absent {} . + +ProcessEnv = + / @present { env: { EnvVariable: EnvValue ...:... } } + / @invalid { env: any } + / @absent {} . + +EnvVariable = @string string / @symbol symbol / @invalid any . +EnvValue = @set string / @remove #f / @invalid any . +``` + +Setting `clearEnv` to `#t` causes the environment to be emptied before `env` is processed and +before the subprocess is started. The `env` key is expected to contain a dictionary whose keys +are strings or symbols and whose values are either a string, to set the variable to a new +value, or `#f`, to remove it from the environment. + +### Setting the Current Working Directory for a subprocess + +By default, each subprocess inherits the current working directory of the `syndicate-server` +program. Setting a `dir` key to a string value in a `FullDaemonProcess` overrides this. + +```preserves-schema +ProcessDir = + / @present { dir: string } + / @invalid { dir: any } + / @absent {} . +``` + +---- + +#### Notes + +[^full-process-split-out]: The `FullProcess` type is split out in order for it to be able to be + reused outside the specific context of a *daemon* process. diff --git a/src/operation/builtin/gatekeeper.md b/src/operation/builtin/gatekeeper.md index 5f2b4db..f697dca 100644 --- a/src/operation/builtin/gatekeeper.md +++ b/src/operation/builtin/gatekeeper.md @@ -1,15 +1,15 @@ # Gatekeeper - - Relevant schema: - - Gatekeepers: [[syndicate-protocol]/schemas/gatekeeper.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/branch/main/schemas/gatekeeper.prs) - - Sturdyrefs: [[syndicate-protocol]/schemas/sturdy.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/branch/main/schemas/sturdy.prs) - When `syndicate-server` starts, it creates a *gatekeeper service entity*, which accepts `resolve` assertions requesting conversion of a long-lived "sturdyref" to a [live reference](../../glossary.md#reference). The gatekeeper is the [default object](../../glossary.md#initial-ref), available as [OID](../../glossary.md#oid) 0 to peers at the other end of [relay listener](./relay-listener.md) connections. +## Gatekeeper protocol + + - Relevant schema: [[syndicate-protocol]/schemas/gatekeeper.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/branch/main/schemas/gatekeeper.prs) + ```preserves-schema Resolve = . ``` @@ -27,6 +27,8 @@ asserted to the `observer` in the `resolve`. ## Sturdyrefs + - Relevant schema: [[syndicate-protocol]/schemas/sturdy.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/branch/main/schemas/sturdy.prs) + A "sturdyref" is a long-lived certificate including a cryptographic signature that can be upgraded by a gatekeeper entity to a live reference to the entity named in the sturdyref. The current sturdyref implementation is based on the design of @@ -65,9 +67,8 @@ is shamelessly taken from [macaroons][], though our caveats presently embody onl Macaroons paper are called "first-party caveats" over assertion structure; future versions of the server may add "third-party caveats" and other, richer, predicates over assertions. -Each `Attenuation`'s `Caveat`s are run in *right to left* order. - -For details of the structure and interpretation of `Caveat`s, see [the relevant section of the -Syndicate network protocol specification](../../protocol.md#attenuation-of-authority). +Each `Attenuation`'s `Caveat`s are run in *right to left* order. The structure and +interpretation of `Caveat`s is described fully in [the relevant section of the Syndicate +network protocol specification](../../protocol.md#attenuation-of-authority). [Macaroons]: ../../glossary.md#macaroon diff --git a/src/operation/builtin/logging.md b/src/operation/builtin/logging.md index b921bbe..881ad79 100644 --- a/src/operation/builtin/logging.md +++ b/src/operation/builtin/logging.md @@ -1 +1,23 @@ # Logging + +The Synit logging infrastructure is still underdeveloped. + +At present, there is an actor created at `syndicate-server` startup time that monitors the +`$log` dataspace for messages of the form: + +```preserves-schema +LogEntry = . +``` + +When it receives a log entry, it looks for a few conventional and optional keys in the `detail` +field, each permitted to be any kind of [value](../../glossary.md#value): + + - `pid`, conventionally a Unix process ID; + - `line`, conventionally a string of free-form text intended for people to read; + - `service`, conventionally a service name in the sense of + [`require-service`](../service.md#require-service)/[`run-service`](../service.md#run-service); and + - `stream`, conventionally one of the symbols `stdout` or `stderr`. + +The timestamp and the special keys are then formatted, along with all other information in the +entry record, and printed to the `syndicate-server`'s standard error at `INFO` level using +[`tracing`](https://docs.rs/tracing/latest/tracing/). diff --git a/src/operation/builtin/relay-listener.md b/src/operation/builtin/relay-listener.md index 15d0923..b6586fc 100644 --- a/src/operation/builtin/relay-listener.md +++ b/src/operation/builtin/relay-listener.md @@ -1,11 +1,61 @@ -# TCP/IP, WebSocket and Unix-socket Transports +# Relay Listeners - - Any number of [**TCP/IP, WebSocket, and Unix socket transports**](./relay-listener.md) may - be configured to allow external access to the gatekeeper and its registered services. (These - can also be started from the `syndicate-server` command-line with `-p` and `-s` options.) + - Relevant schema: + - [[syndicate-rs]/syndicate-server/protocols/schemas/internalServices.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-rs/src/branch/main/syndicate-server/protocols/schemas/internalServices.prs) + - [[syndicate-protocol]/schemas/transportAddress.prs](https://git.syndicate-lang.org/syndicate-lang/syndicate-protocols/src/branch/main/schemas/transportAddress.prs) -## TCP/IP +The `syndicate-server` program can be configured to listen on TCP/IP ports and Unix +sockets[^sock-stream-only] for incoming connections speaking the [Syndicate network +protocol][protocol]. -## WebSockets +[protocol]: ../../protocol.md + +## TCP/IP and WebSockets + +Assertions [requiring](../service.md#require-service) a service with name matching +`TcpRelayListener` cause the server to start a TCP server socket on the given `addr`'s `host` +and `port`, exposing the `gatekeeper` [entity reference](../../glossary.md#reference) as the +[initial ref](../../glossary.md#initial-ref) of incoming connections: + +```preserves-schema +TcpRelayListener = . +Tcp = . +``` + +When a new connection arrives, the first byte is examined to see what kind of connection it is +and which Preserves syntax it will use. + + - If it is ASCII "`G`" (0x47), it cannot be the start of a [protocol][] packet, so it is + interpreted as the start of a WebSocket connection and handed off to the + [tokio_tungstenite](https://docs.rs/tokio-tungstenite/latest/tokio_tungstenite/) WebSocket + library. Within the WebSocket's context, each packet must be encoded as a binary packet + using Preserves binary syntax. + + - Otherwise, if it could start a valid UTF-8 sequence, the connection will be a plain TCP/IP + link using the Preserves text syntax. + + - Otherwise, it's a byte which cannot be part of a valid UTF-8 sequence, so it is interpreted + as a Preserves binary syntax tag: the connection will be a plain TCP/IP link using Preserves + binary syntax. ## Unix sockets + +Assertions [requiring](../service.md#require-service) a service with name matching +`UnixRelayListener` cause the server to start a Unix server socket on the given `addr`'s +`path`, exposing the `gatekeeper` [entity reference](../../glossary.md#reference) as the +[initial ref](../../glossary.md#initial-ref) of incoming connections: + +```preserves-schema +UnixRelayListener = . +Unix = . +``` + +Syntax autodetection is as for [TCP/IP](#tcpip), except that WebSockets are not supported over +Unix sockets. + +---- + +#### Notes + +[^sock-stream-only]: Only `SOCK_STREAM` Unix sockets are supported, at present. In future, + `SOCK_DGRAM` could be useful for e.g. file-descriptor passing. diff --git a/src/operation/scripting.md b/src/operation/scripting.md index 197df7c..1ee5c27 100644 --- a/src/operation/scripting.md +++ b/src/operation/scripting.md @@ -194,7 +194,8 @@ are introduced into the environment. The right-hand-side of a `let`, after the equals sign, is either a normal *ValueExpr* or one of the following special "convenience" expressions: - - `dataspace`: Evaluates to a fresh, empty [dataspace](../glossary.md#dataspace) entity. + - `dataspace`: Evaluates to a fresh, empty + [dataspace](../glossary.md#dataspace) entity. - `timestamp`: Evaluates to a string containing an [RFC-3339](https://datatracker.ietf.org/doc/html/rfc3339)-formatted timestamp. diff --git a/src/operation/service.md b/src/operation/service.md index c042495..0d84fb1 100644 --- a/src/operation/service.md +++ b/src/operation/service.md @@ -46,7 +46,7 @@ without use of variables or any reactive constructs. A few different kinds of assertions, all declared in [the `service.prs` schema][service.prs], form the heart of the system. -### Assert that a service and its dependencies should be started +### Assert that a service and its dependencies should be started ```preserves-schema RequireService = . @@ -55,7 +55,7 @@ RequireService = . Asserts that a service should begin (and stay) running after waiting for its dependencies and considering reverse-dependencies, blocks, and so on. -### Assert that a service should start *right now* +### Assert that a service should start *right now* ```preserves-schema RunService = . @@ -76,7 +76,7 @@ ServiceDependency = . Asserts that, when `depender` is `require-service`d, it should not be started until `dependee` has been asserted, and also that `dependee`'s `serviceName` should be `require-service`d. -### Convey the current state of a service +### Convey the current state of a service ```preserves-schema ServiceState = . @@ -101,7 +101,7 @@ A few built-in states are defined: In addition, any user-defined value is acceptable as a `State`. -### Make an entity representing a service instance available +### Make an entity representing a service instance available ```preserves-schema ServiceObject = .