33 KiB
Syndicate Protocol
Actors that share a local scope can communicate directly. To communicate further afield, scopes are connected using relay actors. Relays allow indirect communication: distant entities can be addressed as if they were local.
Relays exchange Syndicate Protocol messages across a transport. A transport is the underlying medium connecting one relay to its counterparts on a given network. For example, a TLS-on-TCP/IP socket may connect a pair of relays to one another, or a UDP multicast socket may connect an entire group of relays across an ethernet.1
| TLS/TCP/IP socket |<-->|Relay| . . . || +-----+ +------------------------+ +-----+ | | | | | +-------------+ +-------------+
-->
## Transport requirements
Transports must
- be able to carry [Preserves](./glossary.md#preserves) values back and forth,
- be reliable and in-order,
- have a well-defined session lifecycle (created → connected → disconnected), and
- assure confidentiality, integrity and replay-resistance.
This document focuses primarily on point-to-point transports, discussing multicast and
in-memory variations briefly toward the end.
## Roles and session lifecycle
The protocol is completely symmetric, aside from [certain conventions detailed
below](#well-known-oids) about the entities available for use immediately upon connection
establishment. It is *not* a client/server protocol.
**Session startup.** To begin a session on a newly-established point-to-point link, a relay
simply starts sending packets. Each peer starts the session with an empty entity reference map
([see below](#membranes)) and making no assertions in either the outbound (on behalf of local
entities) or inbound (on behalf of the remote peer) directions.
**Session teardown.** At the end of a session, terminated normally or abnormally, cleanly or
through involuntary transport disconnection, all published assertions are
retracted.[^automatic-when-implemented-with-sam] This is in keeping with the essence of the
[Syndicated Actor Model (SAM)](./glossary.md#syndicated-actor-model).
## Packet definitions
Packets exchanged by relays are [Preserves](./glossary.md#preserves) values defined using
Preserves [schema](./glossary.md#schema).
```preserves-schema
Packet = Turn / Error / Extension .
A packet may be a turn, an error, or an extension.
Packets are neither commands nor responses; they are events.
Extension packets
Extension = <<rec> @label any @fields [any ...]> .
An extension packet must be a Preserves record, but is otherwise unconstrained.
Handling. Peers MUST ignore extensions that they do not understand.2
Error packets
Error = <error @message string @detail any>.
Handling. An error packet describes something that went wrong on the other end of the connection. Error packets are primarily intended for debugging.
Receipt of an error packet denotes that the sender has terminated (crashed) and will not respond further; the connection will usually be closed shortly thereafter.
Error packets are optional: connections may simply be closed without comment.
Turn packets
Turn = [TurnEvent ...].
TurnEvent = [@oid Oid @event Event].
Event = Assert / Retract / Message / Sync .
Assert = <assert @assertion Assertion @handle Handle>.
Retract = <retract @handle Handle>.
Message = <message @body Assertion>.
Sync = <sync @peer #!#t>.
Assertion = any .
Handle = int .
Oid = int .
A Turn is the most important packet variant. It directly reflects the
SAM notion of a turn.
Handling. Each Turn carries events to be delivered to
entities residing in the scope at the receiving end of the transport.
Upon receipt of a Turn, the sequence of TurnEvents is examined. The
OID in each TurnEvent selects an entity known to the recipient. Each
Event is either publication of an assertion, retraction of a previously-published assertion,
delivery of a single message, or a synchronization event.
The assertion fields of Assert events and the body fields of Message events may contain
any Preserves value, including embedded entity references. On the wire, these will always be
formatted as described below. As each Assert or Message is
processed, embedded references are mapped to internal references. Symmetrically, internal
references are mapped to their external form prior to transmission. The mapping procedure to
follow is detailed below.
Turn boundaries. In the case that the receiving party is structured internally using the
SAM, it is important to preserve turn boundaries. Since turn boundaries are a
per-actor concept, but a Turn mentions only entities, the receiver
must map entities to actors, group TurnEvents into per-actor queues, and deliver those queues
to each actor in a single SAM turn for each actor.
Uniqueness. The Handles used to refer to published assertions MUST be unique within the
scope of the transport connection.
Capabilities on the wire
References embedded in Turn packets denote capabilities for interacting with some entity.
For example, assertion of a capability-bearing record could appear as the following Event:
<assert <please-reply-to #![0 555]>>
The #![0 555] is concrete Preserves text syntax for
an embedded (#!) value ([0 555]).
In the Syndicate Protocol, these embedded values MUST conform to the WireRef
schema:3
WireRef = @mine [0 @oid Oid] / @yours [1 @oid Oid @attenuation Caveat ...].
Oid = int .
The mine variant denotes capability references managed by the sender of a given packet; the
yours variant, the receiver of the packet. A relay receiving a packet mentioning #![0 555] will use #![1 555] in later responses that refer to that same entity, and vice versa.
Attenuation of authority
A yours-variant capability may include a request4 to impose
additional conditions on the receiver's use of its own capability, known as an
attenuation of the capability's authority.
An attenuation is a chain of Caveats.5 A Caveat acts as a
function that, given a Preserves value representing an assertion or message body, yields either
a possibly-rewritten value, or no value at all.6 In the latter case, the value
has been rejected. In the former case, the rewritten value is used as input to the next
Caveat in the chain, or as the final assertion or message body for delivery to the entity
backing the capability.
The chain of Caveats in an attenuation is written down in reverse order: newer Caveats
are appended to the sequence, and each Caveat's output is fed into the input of the next
leftward Caveat in the sequence. If no Caveats are present, the capability is unattenuated,
and inputs are passed through to the backing capability unmodified.
Caveat = Rewrite / Alts .
Rewrite = <rewrite @pattern Pattern @template Template>.
Alts = <or @alternatives [Rewrite ...]>.
A Caveat can be either a single Rewrite or a sequence of alternative possible rewrites,
tried in left-to-right order until one of them accepts the input or there are none left to try.
(A single Rewrite R is equivalent to <or [R]>.)
A Rewrite applies its Pattern to the input to the Caveat. If it matches, the bindings
captured by the pattern are gathered together and used in instantiation of the Rewrite's
Template, yielding the output from the Caveat. If the pattern does not match, the Rewrite
has rejected the input, and other alternatives are tried until none remain, at which point
the whole Caveat has rejected the input and processing of the triggering event stops.
Patterns
A Pattern within a rewrite can be any of the following variants:
Pattern = PDiscard / PAtom / PEmbedded / PBind / PAnd / PNot / Lit / PCompound .
Wildcard. PDiscard matches any value:
PDiscard = <_>.
Atomic type. PAtom requires that a matched value be a boolean, a single- or double-precision float, an
integer, a string, a binary blob, or a symbol, respectively:
PAtom = =Boolean / =Float / =Double / =SignedInteger / =String / =ByteString / =Symbol .
Embedded value. PEmbedded requires that a matched value be an embedded capability:
PEmbedded = =Embedded .
Binding. PBind first captures the matched value, adding it to the bindings vector, and then applies
the nested pattern. If the subpattern matches, the PBind succeeds; otherwise, it fails:
PBind = <bind @pattern Pattern>.
Conjunction. PAnd is a conjunction of patterns; every pattern in patterns must match for the PAnd to
match:
PAnd = <and @patterns [Pattern ...]>.
Negation. PNot is a pattern negation: if pattern matches, the PNot fails to match, and vice
versa. It is an error for pattern to include any PBind subpatterns.
PNot = <not @pattern Pattern>.
Literal. Lit is an exact match pattern. If the matched value is exactly equal to value (according to
Preserves' own built-in equivalence relation), the match succeeds; otherwise, it fails:
Lit = <lit @value any>.
Compound. Finally, PCompound patterns match compound data structures. The rec variant demands that a
matched value be a record, with label exactly equal to label and fields one-for-one matching
the Patterns in fields; the arr variant demands a sequence, with each element matching
the corresponding element of items; and dict demands a dictionary having at least entries
named by the keys of the entries dictionary, each matching the corresponding Pattern.
PCompound =
/ @rec <rec @label any @fields [Pattern ...]>
/ @arr <arr @items [Pattern ...]>
/ @dict <dict @entries { any: Pattern ...:... }> .
Bindings
Matching notionally produces a sequence of values, one for each PBind in the pattern.
When a PBind pattern is seen, the matcher first appends the matched value to the binding
sequence and then recurses on the nested subpattern. This makes binding indexes appear in
left-to-right order as a Pattern is read.
Example. Given the pattern <bind <arr [<bind <_>>, <bind <_>>]>> and the matched value
["a" "b"], the resulting captured values are, in order, ["a" "b"], "a", and "b"; the
template <ref 0> will be instantiated to ["1" "2"], <ref 1> to "a", and <ref 2> to
"b".
Templates
A Template within a rewrite produces a concrete Preserves value when instantiated with a
vector of captured binding values. Template instantiation may fail, yielding no value.
A given Template may be any of the following variants:
Template = TAttenuate / TRef / Lit / TCompound .
TAttenuate first instantiates the sub-template. If it yields a value, and if that value is
an embedded reference (i.e. a capability), the Caveats in attenuation are appended to the
(possibly-empty) sequence of Caveats already present in the embedded capability. The
resulting possibly-attenuated capability is the final result of instantiation of the
TAttenuate.
TAttenuate = <attenuate @template Template @attenuation [Caveat ...]>.
TRef retrieves the bindingth (0-based) index into the bindings vector, yielding the
associated captured value as the result of instantiation. It is an error if binding is less
than zero, or greater than or equal to the number of bindings in the bindings vector.
TRef = <ref @binding int>.
Lit (the same definition as used in the grammar for Pattern above) instantiates to exactly
its value argument:
Lit = <lit @value any>.
Finally, TCompound instantiates to compound data. The rec variant produces a record with
the given label and fields; arr produces an array; and dict a dictionary:
TCompound =
/ @rec <rec @label any @fields [Template ...]>
/ @arr <arr @items [Template ...]>
/ @dict <dict @entries { any: Template ...:... }> .
Validity of Caveats
The above definitions imply some validity constraints on Caveats.
-
All
TRefs must be bound: the index referred to must relate to the index associated with somePBindin the pattern corresponding to the template. -
Binding under negation is forbidden: a
patternwithin aPNotmay not include anyPBindconstructors. -
The value produced by instantiation of
templatewithin aTAttenuatemust be an embedded reference (a capability).
Implementations MUST enforce these constraints (either statically or dynamically).
Membranes
Every relay maintains two stateful objects called membranes. A membrane is a bidirectional mapping between OID and relay-internal entity pointer. Membranes connect embedded references on the wire to entity references local to the relay.
-
The import membrane connects OIDs managed by the remote peer to local relay entities which proxy access to an "imported" remote entity.
-
The export membrane connects OIDs managed by the local peer to any local "exported" entities accessible to the peer.
|
|
Export Membrane | Import Membrane
|
+-+ | +-+
Pointer | | ID | ID | |
0x1234 <-+-+-> "my 7" | "your 7"<-+-+-> 0x9abc
| | | | |
^ | | ^ | ^ | | ^
| | | | -+- | | | |
V | | | | | | V
/------\ | | \-------------/ | | /------\
|Entity| | | | | |Relay |<-- ...
\------/ | | | | |Entity|
0x1234 | | --------> | | \------/
=-------------+-+---= packets | | 0x9abc
0x462e | | <-------- =---+-+-------------=
/------\ | | | | 0xa043
|Relay | | | | | /------\
... -->|Entity| | | /-------------\ | | |Entity|
\------/ | | | | | | \------/
^ | | | -+- | | | ^
| | | | | | | | |
V | | V | V | | V
| | | | |
0x462e <-+-+->"your 3" | "my 3" <-+-+-> 0xa043
Pointer | | ID | ID | | Pointer
+-+ | +-+
|
Import Membrane | Export Membrane
|
|
+--0x7f10652fe7c0<-+-> 13 peer
/--\ | |
|A3|<----+ | +-+-------------------------
\--/ | |
|
|
----------------------------------------+
-->
<!--
Each relay rewrites the embedded references in the messages it sends and receives. It maps back
and forth between one scope's name for an entity and the other scope's name for the same
entity.
```ditaa protocol-scope-chains
(Illustrative Example)
Browser syndicate–server
+----------------+ WebSocket +-------+-----------------------+
|In–browser scope|<----------->| Relay |<-\ |
+----------------+ (2) +-------+ | +---------+ |
| ^ \->|Dataspace| |
| | +---------+ |
| V ^ ^ ^ (1) |
+----------------+ TCP/IP +-------+ | | | |
|Remote Syndicate|<----------->| Relay |<----+-/ | | |
|server/dataspace| (3) +-------+ | | | |
+----------------+ | V V V |
| +-------+ +-------+ |
| | Relay | | Relay | |
+----------+-------+-+-------+--+
^ ^
LAN multicast (4) | | UNIX
... <-------/---------/---------/---------/ | socket
| | | | (5)
v v v v
+-------+ +-------+ +-------+ +-------+
|. . . | |. . . | |. . . | |. . . |
+-------+ +-------+ +-------+ +-------+
In the diagram above, networks (scopes) 1 and 4 are multicast, while networks 2, 3 and 5 are point-to-point. Four relays bridge scope 1 to scopes 2 through 5. Within each scope, peers are able to interact with each other directly. Each point-to-point scope contains exactly two peers.
-->
Logically, a membrane's state can be represented as a set of WireSymbol structures: a
WireSymbol is a triple of an OID, a local reference pointer (its ref), and a reference
count. There is never more than one WireSymbol associated with an OID or a ref.
A WireSymbol exists only so long as some assertion mentioning its OID exists across the relay
link. When the last assertion mentioning an OID is retracted, its WireSymbol is deleted.
Assertions mentioning a particular OID can come from either side of the relay link:
initially, a local reference is sent to the peer in an assertion, but then the peer may assert
something back, either targeting or mentioning the same entity. Care must be taken not to
release an OID entry prematurely in such situations.
For example, at least the following contribute to a WireSymbol's reference count:
-
The initial entry mapping a local entity ref to an well-known OID for use at session startup (see below) contributes a permanent reference.
-
Mention of an OID in a received or sent
TurnEventadds one to the OID's reference count for the duration of processing of the event. ForAssertevents in either direction, the duration of processing is until the assertion is later retracted. For receivedMessageevents, the duration of processing is until the incoming message has been forwarded on to the target ref.
"Transient" references. Embedded references in Message event bodies are special. Because
messages, unlike assertions, have no notion of lifetime—they are forwarded and forgotten—it is
not possible for a message to cause establishment of a long-lived entry in a membrane's
WireSymbol set. Therefore, messages MUST NOT embed any reference not previously known to the
peer (a "transient reference"). In other words, only after using an assertion to introduce a
reference, associating a conversational context with its lifetime, is it permitted to discuss
the reference using messages. A relay receiving a message bearing a transient reference MUST
terminate the session with an error. A relay about to send such a message SHOULD preemptively
refuse to do so.
Rewriting embedded references upon receipt
When processing a Value v in a received Assert or Message event, embedded references in
v are decoded from their on-the-wire WireRef form to in-memory
ref-pointer form.
The value is recursively traversed. As the relay comes across each embedded WireRef,
-
If it is of
minevariant, it refers to an entity exported by the remote, sending peer. Its OID is looked up in the import membrane.-
If no
WireSymbolexists in the import membrane, one is created, mapping the OID to a fresh relay entity. -
If a
WireSymbolis already present, its associated ref is substituted into v.
-
-
If it is of
yoursvariant, it refers to an entity previously exported by the local, receiving peer. Its OID is looked up in the export membrane.-
If no
WireSymbolexists for the OID, one is created, associating the OID with a dummy inert entity ref. The dummy ref is substituted into v. -
If a
WireSymbolexists for the OID, and theWireRefis not attenuated, the associated ref is substituted into v. If theWireRefis attenuated, the associated ref is wrapped with theCaveats from theWireRefbefore its substitution into v.
-
-
In each case, the
WireSymbolassociated with the OID has its reference count incremented (if anAssertis being processed).
Rewriting embedded references for transmission
When transmitting a Value v in an Assert or Message event, embedded references in v
are encoded from their in-memory ref-pointer form to on-the-wire WireRef
form.
The value is recursively traversed. As the relay comes across each embedded reference:
-
The reference is first looked up in the export membrane. If an associated
WireSymbolis present in the export membrane, its OID is substituted as amine-variantWireRefinto v. -
Otherwise, it is looked up in the import membrane. If no associated
WireSymbolexists there, a fresh OID andWireSymbolare placed in the export membrane, and the new OID is substituted as amine-variantWireRefinto v. -
Otherwise, it refers to a previously-imported entity.
-
If the local entity reference has not been attenuated subsequent to its import, the OID it was imported under is substituted as a
yours-variantWireRefinto v with an empty attenuation. -
If it has been attenuated, the relay may choose whether to trust the remote party to enforce an attenuation request. If it trusts the peer to honour attenuation requests, it substitutes a
yours-variantWireRefwith non-empty attenuation into v. Otherwise, a fresh OID andWireSymbolare placed in the export membrane, with ref denoting to the attenuated local reference, and the new OID is substituted as amine-variantWireRefinto v.
-
Relay entities
Client and server roles
Well-known OIDs
OID 0, initial ref, initial oid
Security considerations
((Tease out into Related Work section?))
OIDs are locally-meaningful only, so if the transport is secure, so is the reference. Can't steal one and put it on a different transport: it's like taking fd 6 from another process and trying to use fd 6 locally to mean what the other process means. Extensive related work and prior art here.
http://www.erights.org/elib/distrib/captp/index.html
Relate terms here to captp terms:
- Hah,
NonceLocatorvsGatekeeper - well-known "positions" (??) (vs "OID"s?)
- OID = "index", "capability-list index", "c-list index"
- @cwebber says "c-list is the structure mapping descriptors to live-refs"
Secrecy
Privacy
Specific transport mappings
TCP/IP
TLS TCP/IP
WebSockets
Other kinds of medium
Multicast/broadcast, in-memory
Appendix: Complete schema of the protocol
The following is a consolidated form of the definitions from the text above.
Protocol packets
The authoritative
version of this schema is
[syndicate-protocols]/schemas/protocol.prs.
version 1 .
Packet = Turn / Error / Extension .
Extension = <<rec> @label any @fields [any ...]> .
Error = <error @message string @detail any>.
Assertion = any .
Handle = int .
Event = Assert / Retract / Message / Sync .
Oid = int .
Turn = [TurnEvent ...].
TurnEvent = [@oid Oid @event Event].
Assert = <assert @assertion Assertion @handle Handle>.
Retract = <retract @handle Handle>.
Message = <message @body Assertion>.
Sync = <sync @peer #!#t>.
Capabilities, WireRefs, and attenuations
The authoritative version of this schema is
[syndicate-protocols]/schemas/sturdy.prs.
version 1 .
Attenuation = [Caveat ...].
Caveat = Rewrite / Alts .
Rewrite = <rewrite @pattern Pattern @template Template>.
Alts = <or @alternatives [Rewrite ...]>.
Oid = int .
WireRef = @mine [0 @oid Oid] / @yours [1 @oid Oid @attenuation Caveat ...].
Lit = <lit @value any>.
Pattern = PDiscard / PAtom / PEmbedded / PBind / PAnd / PNot / Lit / PCompound .
PDiscard = <_>.
PAtom = =Boolean / =Float / =Double / =SignedInteger / =String / =ByteString / =Symbol .
PEmbedded = =Embedded .
PBind = <bind @pattern Pattern>.
PAnd = <and @patterns [Pattern ...]>.
PNot = <not @pattern Pattern>.
PCompound =
/ @rec <rec @label any @fields [Pattern ...]>
/ @arr <arr @items [Pattern ...]>
/ @dict <dict @entries { any: Pattern ...:... }> .
Template = TAttenuate / TRef / Lit / TCompound .
TAttenuate = <attenuate @template Template @attenuation Attenuation>.
TRef = <ref @binding int>.
TCompound =
/ @rec <rec @label any @fields [Template ...]>
/ @arr <arr @items [Template ...]>
/ @dict <dict @entries { any: Template ...:... }> .
Appendix: Pseudocode for attenuation, pattern matching, and template instantiation
Attenuation
def attenuate(attenuation, value):
for caveat in reversed(attenuation):
value = applyCaveat(caveat, value)
if value is None:
return None
return value
def applyCaveat(caveat, value):
if caveat is 'Alts' variant:
for rewrite in caveat.alternatives:
possibleResult = tryRewrite(rewrite, value);
if possibleResult is not None:
return possibleResult
return None
if caveat is 'Rewrite' variant:
return tryRewrite(caveat, value)
def tryRewrite(rewrite, value):
bindings = applyPattern(rewrite.pattern, value)
if bindings is None:
return None
else:
return instantiateTemplate(rewrite.template, bindings)
Pattern matching
def match(pattern, value, bindings):
if pattern is 'PDiscard' variant:
return True
if pattern is 'PAtom' variant:
return True if value is of the appropriate atomic class else False
if pattern is 'PEmbedded' variant:
return True if value is a capability else False
if pattern is 'PBind' variant:
append value to bindings
return match(pattern.pattern, value, bindings)
if pattern is 'PAnd' variant:
for p in pattern.patterns:
if not match(p, value, bindings):
return False
return True
if pattern is 'PNot' variant:
return False if match(pattern.pattern, value, bindings) else True
if pattern is 'Lit' variant:
return (pattern.value == value)
if pattern is 'PCompound' variant:
if pattern is 'rec' variant:
if value is not a record: return False
if value.label is not equal to pattern.label: return False
if value.fields.length is not equal to pattern.fields.length: return False
for i in [0 .. pattern.fields.length):
if not match(pattern.fields[i], value.fields[i], bindings):
return False
return True
if pattern is 'arr' variant:
if value is not a sequence: return False
if value.length is not equal to pattern.items.length: return False
for i in [0 .. pattern.items.length):
if not match(pattern.items[i], value[i], bindings):
return False
return True
if pattern is 'dict' variant:
if value is not a dictionary: return False
for k in keys of pattern.entries:
if k not in keys of value: return False
if not match(pattern.entries[k], value[k], bindings):
return False
return True
Template instantiation
def instantiate(template, bindings):
if template is 'TAttenuate' variant:
c = instantiate(template.template, bindings)
if c is not a capability: raise an exception
c′ = c with the caveats in template.attenuation appended to the existing
attenuation in c
return c′
if template is 'TRef' variant:
if 0 ≤ template.binding < bindings.length:
return bindings[template.binding]
else:
raise an exception
if template is 'Lit' variant:
return template.value
if template is 'TCompound' variant:
if template is 'rec' variant:
return Record(label=template.label,
fields=[instantiate(t, bindings) for t in template.fields])
if template is 'arr' variant:
return [instantiate(t, bindings) for t in template.items]
if template is 'dict' variant:
result = {}
for k in keys of template.entries:
result[k] = instantiate(template.entries[k], bindings)
return result
Notes
-
In fact, it makes perfect sense to run the relay protocol between actors that are already connected in some scope: this is like running a VPN, tunnelling IP over IP. A variation of the Syndicate Protocol like this gives federated dataspaces. ↩︎
-
This specification does not define any extensions, but future revisions could, for example, use extensions to perform version-negotiation. Another potential future use could be to propagate provenance information for tracing/debugging. ↩︎
-
The syntax for
WireRefs is slightly silly, using tuples as quasi-records with0and1acting as quasi-labels. It would probably be better to use real records, like<my @oid Oid>and<yours @oid Oid @attenuation [Caveat ...]>. Pros: less cryptic. Cons: slightly more verbose on the wire. TODO: should we revise the spec in this regard? ↩︎ -
Such conditions can only ever be requests: after all, every
yours-capability is already completely accessible to the recipient of the packet. Similarly, it does not make sense to include an attenuation description on amy-capability. However, in every case, if a party wishes to enforce an attenuation on amy- oryours-capability, it may record the attenuation against the underlying capability internally, issuing to its peers a freshmy-capability denoting the attenuated capability. ↩︎ -
This terminology, "caveat", is lifted from the excellent paper on Macaroons, where it is used to describe a more general mechanism. Future versions of this specification may opt to include some of this generality. ↩︎
-
TODO: It might be better to have a
Caveatyield zero or more values? That way they can act as filters. I've sometimes wanted the multiple-value case, though I've so far been able to work around its lack. TODO: Perhaps it would also make sense to have aCaveatmap an event to zero or more events, rather than to values? Tricky corners there include ensuring that carried authority isn't misused; macaroons are a very elegant solution to this problem, of course, so maybe the macaroon design idea could be adapted to this. For now,Value→Option<Value>is probably OK. ↩︎