synit
/
synit-manual
1
0
Fork 0
Code Issues 2 Pull Requests Projects Releases Wiki Activity
synit-manual/src/guide/preserves.md

13 KiB
Raw Blame History

Preserves

Synit makes extensive use of Preserves, a programming-language-independent language for data.

The Preserves data language is in many ways comparable to JSON, XML, S-expressions, CBOR, ASN.1 BER, and so on. From the specification document:

Preserves supports records with user-defined labels, embedded references, and the usual suite of atomic and compound data types, including binary data as a distinct type from text strings.

Why does Synit rely on Preserves?

There are five aspects of Preserves that make it particularly relevant to Synit:

  • the core Preserves data language has a robust semantics;
  • Preserves values may have [capability references() embedded within them;
  • Preserves has a schema language useful for specifying protocols among actors;
  • a canonical form exists for every Preserves value; and
  • Preserves has a query language for extracting portions of a Preserves value.

Grammar of values

The main reason Preserves is useful for Synit is that it has semantics: the specification defines a language-independent equivalence relation over Preserves values.1 This makes it a solid foundation for a multi-language, multi-process, potentially distributed system like Synit. 2

Abstract syntax: Values

The abstract syntax of Preserves values is as follows (from the specification):

                    Value = Atom           Atom = Boolean
                          | Compound            | Float
                          | Embedded            | Double
                                                | SignedInteger
                 Compound = Record              | String
                          | Sequence            | ByteString
                          | Set                 | Symbol
                          | Dictionary

Concrete syntax

Because Preserves has semantics independent of its syntax, we are free to define syntax appropriate for its use in different settings. Values can be automatically, losslessly translated from one syntax to another. The core Preserves specification defines both a text-based, human-readable, JSON-like syntax, that is a syntactic superset of JSON, and a completely equivalent compact binary syntax, crucial to the definition of canonical form for Preserves values.3

Here are a few example values, written using the text syntax (see the specification for the grammar):

Boolean    : #t, #f
Float      : 1.0f, 10.4e3f, -100.6f
Double     : 1.0, 10.4e3, -100.6
Integer    : 1, 0, -100
String     : "Hello, world!\n"
ByteString : #"bin\x00str\x00", #[YmluAHN0cgA], #x"62696e0073747200"
Symbol     : hello-world, |hello world|, =, !, hello?, ||, ...
Record     : <label field1 field2 ...>
Sequence   : [value1 value2 ...]
Set        : #{value1 value2 ...}
Dictionary : {key1: value1 key2: value2 ...: ...}
Embedded   : #!value

Commas are optional in sequences, sets, and dictionaries.

Canonical form

Every Preserves value can be serialized into a canonical form using the binary syntax along with a few simple rules about serialization ordering of elements in sets and keys in dictionaries.

Having a canonical form means that, for example, a SHA-512 (or other secure) digest of the canonical serialization of a value can be used as a unique, short name for the value.

For example, the value

<sms-delivery <address international "31653131313"> 
              <address international "31655512345">
              <rfc3339 "2022-02-09T08:18:29.88847+01:00">
              "This is a test SMS message">

serializes canonically to

00000000: b4b3 0c73 6d73 2d64 656c 6976 6572 79b4  ...sms-delivery.
00000010: b307 6164 6472 6573 73b3 0d69 6e74 6572  ..address..inter
00000020: 6e61 7469 6f6e 616c b10b 3331 3635 3331  national..316531
00000030: 3331 3331 3384 b4b3 0761 6464 7265 7373  31313....address
00000040: b30d 696e 7465 726e 6174 696f 6e61 6cb1  ..international.
00000050: 0b33 3136 3535 3531 3233 3435 84b4 b307  .31655512345....
00000060: 7266 6333 3333 39b1 1f32 3032 322d 3032  rfc3339..2022-02
00000070: 2d30 3954 3038 3a31 383a 3239 2e38 3838  -09T08:18:29.888
00000080: 3437 2b30 313a 3030 84b1 1a54 6869 7320  47+01:00...This 
00000090: 6973 2061 2074 6573 7420 534d 5320 6d65  is a test SMS me
000000a0: 7373 6167 6584                           ssage.

which has SHA-512 hash

bfea9bd5ddf7781e34b6ca7e146ba2e442ef8ce04fd5ff912f889359945d0e2967a77a13
c86b13959dcce7e8ba3950d303832b825648609447b3d147677163ce

Schemas

Preserves comes with a schema language suitable for defining protocols among actors/programs in Synit. Because Preserves is a superset of JSON, its schemas can be used for parsing JSON just as well as for native Preserves values. From the schema specification:

A Preserves schema connects Preserves Values to host-language data structures. Each definition within a schema can be processed by a compiler to produce

  • a host-language type definition;
  • a partial parsing function from Values to instances of the produced type; and
  • a total serialization function from instances of the type to Values.

Every parsed Value retains enough information to always be able to be serialized again, and every instance of a host-language data structure contains, by construction, enough information to be successfully serialized.

Instead of taking host-language data structure definitions as primary, in the way that systems like serde do, Preserves schemas take the shape of the serialized data as primary.

To see the difference, let's look at an example.

Example: Book Outline

Systems like Serde concentrate on defining (de)serializers for host-language type definitions.

Serde starts from definitions like the following4. It generates (de)serialization code for various different data languages (such as JSON, XML, CBOR, etc.) in a single programming language: Rust.

pub struct BookOutline {
    pub sections: Vec<BookItem>,
}
pub enum BookItem {
    Chapter(Chapter),
    Separator,
    PartTitle(String),
}
pub struct Chapter {
    pub name: String,
    pub sub_items: Vec<BookItem>,
}

The (de)serializers are able to produce and understand values such as the following JSON document, converting them to and from in-memory representations. The focus is on Rust: interpreting the produced documents from other languages is out-of-scope for Serde.

{
  "sections": [
    { "PartTitle": "Part I" },
    "Separator",
    {
      "Chapter": {
        "name": "Chapter One",
        "sub_items": []
      }
    },
    {
      "Chapter": {
        "name": "Chapter Two",
        "sub_items": []
      }
    }
  ]
}

By contrast, Preserves schemas focus on Preserves values5 only.

Each Preserves schema compiler generates type definitions and (de)serialization code for a single programming language able to understand common data. The grammar of the data itself is language-independent.

For example, a Preserves schema able to parse values compatible with those produced by Serde for the type definitions above is the following:

version 1 .

BookOutline = {
  "sections": @sections [BookItem ...],
} .

BookItem = @chapter { "Chapter": @value Chapter }
         / @separator "Separator"
         / @partTitle { "PartTitle": @value string } .

Chapter = {
  "name": @name string,
  "sub_items": @sub_items [BookItem ...],
} .

Using the Rust schema compiler, we see types such as the following, which are similar to but not the same as the original Rust types above:

pub struct BookOutline {
    pub sections: std::vec::Vec<BookItem>
}
pub enum BookItem {
    Chapter { value: std::boxed::Box<Chapter> },
    Separator,
    PartTitle { value: std::string::String }
}
pub struct Chapter {
    pub name: std::string::String,
    pub sub_items: std::vec::Vec<BookItem>
}

Using the TypeScript schema compiler, we see

export type BookOutline = {"sections": Array<BookItem>};

export type BookItem = (
    {"_variant": "chapter", "value": Chapter} |
    {"_variant": "separator"} |
    {"_variant": "partTitle", "value": string}
);

export type Chapter = {"name": string, "sub_items": Array<BookItem>};

Using the Racket schema compiler, we see

(struct BookOutline (sections))
(define (BookItem? p)
    (or (BookItem-chapter? p)
        (BookItem-separator? p)
        (BookItem-partTitle? p)))
(struct BookItem-chapter (value))
(struct BookItem-separator ())
(struct BookItem-partTitle (value))
(struct Chapter (name sub_items))

and so on.

Example: Book Outline redux, using Records

The schema for book outlines above accepts Preserves (JSON) documents compatible with the (de)serializers produced by Serde for a Rust-native type.

Instead, we might choose to define a Preserves-native data definition, and to work from that:6

version 1 .
BookOutline = <book-outline @sections [BookItem ...]> .
BookItem = Chapter / =separator / @partTitle string .
Chapter = <chapter @name string @sub_items [BookItem ...]> .

The schema compilers produce exactly the same type definitions for this variation. The differences are in the (de)serialization code only.

Here's the Preserves value equivalent to the example above, expressed using the Preserves-native schema:

<book-outline [
  "Part I"
  separator
  <chapter "Chapter One" []>
  <chapter "Chapter Two" []>
]>

Preserves Path

Notes

  1. The specification defines a total order relation over Preserves values as well. ↩︎

  2. In particular, dataspaces need the assertion data they contain to have a sensible equivalence predicate in order to be useful at all. If you can't reliably tell whether two values are the same or different, how are you supposed to use them to look things up in anything database-like? Languages like JSON, which don't have a well-defined equivalence relation, aren't good enough. When programs communicate with each other, they need to be sure that their peers will understand the information they receive exactly as it was sent. ↩︎

  3. Besides the two core syntaxes, other serialization syntaxes are in use in other systems. For example, the Spritely Goblins actor library uses a serialization syntax called Syrup, reminiscent of bencode. ↩︎

  4. This example is a simplified form of the preprocessor type definitions for mdBook, the system used to render this manual. I use a real Preserves schema definition for parsing and producing Serde's JSON representation of mdBook Book structures in order to preprocess the manual's source code. ↩︎

  5. Including JSON values, of course! ↩︎

  6. By doing so, we of course lose compatibility with the Serde structures. ↩︎