--- no_site_title: true title: "Preserves: an Expressive Data Language" --- Tony Garnock-Jones {{ site.version_date }}. Version {{ site.version }}. *Preserves* is a data model, with associated serialization formats. It supports *records* with user-defined *labels*, embedded *references*, and the usual suite of atomic and compound data types, including *binary* data as a distinct type from text strings. Its *annotations* allow separation of data from metadata such as [comments](conventions.html#comments), trace information, and provenance information. Preserves departs from many other data languages in defining how to *compare* two values. Comparison is based on the data model, not on syntax or on data structures of any particular implementation language. This document defines the core semantics and data model of Preserves and presents a handful of examples. Two other core documents define - a [human-readable text syntax](preserves-text.html), and - a [machine-oriented binary syntax](preserves-binary.html) for the Preserves data model. ## Values Preserves *values* are given meaning independent of their syntax. We will write "`Value`" when we mean the set of all Preserves values or an element of that set. `Value`s fall into two broad categories: *atomic* and *compound* data. Every `Value` is finite and non-cyclic. Embedded values, called `Embedded`s, are a third, special-case category. Value = Atom | Compound | Embedded Atom = Boolean | Float | Double | SignedInteger | String | ByteString | Symbol Compound = Record | Sequence | Set | Dictionary **Total order.** As we go, we will incrementally specify a total order over `Value`s. Two values of the same kind are compared using kind-specific rules. The ordering among values of different kinds is essentially arbitrary, but having a total order is convenient for many tasks, so we define it as follows: (Values) Atom < Compound < Embedded (Compounds) Record < Sequence < Set < Dictionary (Atoms) Boolean < Float < Double < SignedInteger < String < ByteString < Symbol **Equivalence.** Two `Value`s are equal if neither is less than the other according to the total order. ### Signed integers. A `SignedInteger` is an arbitrarily-large signed integer. `SignedInteger`s are compared as mathematical integers. ### Unicode strings. A `String` is a sequence of Unicode [code-point](http://www.unicode.org/glossary/#code_point)s.[^nul-permitted] `String`s are compared lexicographically, code-point by code-point.[^utf8-is-awesome] [^utf8-is-awesome]: Happily, the design of UTF-8 is such that this gives the same result as a lexicographic byte-by-byte comparison of the UTF-8 encoding of a string! [^nul-permitted]: All Unicode code-points are permitted, including NUL (code point zero). ### Binary data. A `ByteString` is a sequence of octets. `ByteString`s are compared lexicographically. ### Symbols. Programming languages like Lisp and Prolog frequently use string-like values called *symbols*. Here, a `Symbol` is, like a `String`, a sequence of Unicode code-points representing an identifier of some kind. `Symbol`s are also compared lexicographically by code-point. ### Booleans. There are two `Boolean`s, “false” and “true”. The “false” value is less-than the “true” value. ### IEEE floating-point values. `Float`s and `Double`s are single- and double-precision IEEE 754 floating-point values, respectively. `Float`s, `Double`s and `SignedInteger`s are disjoint; by the rules [above](#total-order), every `Float` is less than every `Double`, and every `SignedInteger` is greater than both. Two `Float`s or two `Double`s are to be ordered by the `totalOrder` predicate defined in section 5.10 of [IEEE Std 754-2008](https://dx.doi.org/10.1109/IEEESTD.2008.4610935). ### Records. A `Record` is a *labelled* tuple of `Value`s, the record's *fields*. A label can be any `Value`, but is usually a `Symbol`.[^extensibility] [^iri-labels] `Record`s are compared lexicographically: first by label, then by field sequence. [^extensibility]: The [Racket](https://racket-lang.org/) programming language defines “[prefab](http://docs.racket-lang.org/guide/define-struct.html#(part._prefab-struct))” structure types, which map well to our `Record`s. Racket supports record extensibility by encoding record supertypes into record labels as specially-formatted lists. [^iri-labels]: It is occasionally (but seldom) necessary to interpret such `Symbol` labels as UTF-8 encoded IRIs. Where a label can be read as a relative IRI, it is notionally interpreted with respect to the IRI `urn:uuid:6bf094a6-20f1-4887-ada7-46834a9b5b34`; where a label can be read as an absolute IRI, it stands for that IRI; and otherwise, it cannot be read as an IRI at all, and so the label simply stands for itself—for its own `Value`. ### Sequences. A `Sequence` is a sequence of `Value`s. `Sequence`s are compared lexicographically. ### Sets. A `Set` is an unordered finite set of `Value`s. It contains no duplicate values, following the [equivalence relation](#equivalence) induced by the total order on `Value`s. Two `Set`s are compared by sorting their elements ascending using the [total order](#total-order) and comparing the resulting `Sequence`s. ### Dictionaries. A `Dictionary` is an unordered finite collection of pairs of `Value`s. Each pair comprises a *key* and a *value*. Keys in a `Dictionary` are pairwise distinct. Instances of `Dictionary` are compared by lexicographic comparison of the sequences resulting from ordering each `Dictionary`'s pairs in ascending order by key. ### Embeddeds. An `Embedded` allows inclusion of *domain-specific*, potentially *stateful* or *located* data into a `Value`.[^embedded-rationale] `Embedded`s may be used to denote stateful objects, network services, object capabilities, file descriptors, Unix processes, or other possibly-stateful things. Because each `Embedded` is a domain-specific datum, comparison of two `Embedded`s is done according to domain-specific rules. [^embedded-rationale]: **Rationale.** Why include `Embedded`s as a special class, distinct from, say, a specially-labeled `Record`? First, a `Record` can only hold other `Value`s: in order to embed values such as live pointers to Java objects, some means of "escaping" from the `Value` data type must be provided. Second, `Embedded`s are meant to be able to denote stateful entities, for which comparison by address is appropriate; however, we do not wish to place restrictions on the *nature* of these entities: if we had used `Record`s instead of distinct `Embedded`s, users would have to invent an encoding of domain data into `Record`s that reflected domain ordering into `Value` ordering. This is often difficult and may not always be possible. Finally, because `Embedded`s are intended to be able to represent network and memory *locations*, they must be able to be rewritten at network and process boundaries. Having a distinct class allows generic `Embedded` rewriting without the quotation-related complications of encoding references as, say, `Record`s. *Examples.* In a Java or Python implementation, an `Embedded` may denote a reference to a Java or Python object; comparison would be done via the language's own rules for equivalence and ordering. In a Unix application, an `Embedded` may denote an open file descriptor or a process ID. In an HTTP-based application, each `Embedded` might be a URL, compared according to [RFC 6943](https://tools.ietf.org/html/rfc6943#section-3.3). When a `Value` is serialized for storage or transfer, `Embedded`s will usually be represented as ordinary `Value`s, in which case the ordinary rules for comparing `Value`s will apply. ## Examples The definitions above are independent of any particular concrete syntax. The examples of `Value`s that follow are written using [the Preserves text syntax](preserves-text.html), and the example encoded byte sequences use [the Preserves binary encoding](preserves-binary.html). ### Ordering. The total ordering specified [above](#total-order) means that the following statements are true: "bzz" < "c" < "caa" < #!"a" #t < 3.0f < 3.0 < 3 < "3" < |3| < [] < #!#t ### Simple examples. | Value | Encoded byte sequence | |-----------------------------------------------------|---------------------------------------------------------------------------------| | `>` | B4 B3 07 'c' 'a' 'p' 't' 'u' 'r' 'e' B4 B3 07 'd' 'i' 's' 'c' 'a' 'r' 'd' 84 84 | | `[1 2 3 4]` | B5 91 92 93 94 84 | | `[-2 -1 0 1]` | B5 9E 9F 90 91 84 | | `"hello"` (format B) | B1 05 'h' 'e' 'l' 'l' 'o' | | `["a" b #"c" [] #{} #t #f]` | B5 B1 01 'a' B3 01 'b' B2 01 'c' B5 84 B6 84 81 80 84 | | `-257` | A1 FE FF | | `-1` | 9F | | `0` | 90 | | `1` | 91 | | `255` | A1 00 FF | | `1.0f` | 82 3F 80 00 00 | | `1.0` | 83 3F F0 00 00 00 00 00 00 | | `-1.202e300` | 83 FE 3C B7 B7 59 BF 04 26 | | `#xf"7f800000"`, positive `Float` infinity | 82 7F 80 00 00 | | `#xd"fff0000000000000"`, negative `Double` infinity | 83 FF F0 00 00 00 00 00 00 | The next example uses a non-`Symbol` label for a record.[^extensibility2] The `Record` <[titled person 2 thing 1] 101 "Blackwell" "Dr"> encodes to B4 ;; Record B5 ;; Sequence B3 06 74 69 74 6C 65 64 ;; Symbol, "titled" B3 06 70 65 72 73 6F 6E ;; Symbol, "person" 92 ;; SignedInteger, "2" B3 05 74 68 69 6E 67 ;; Symbol, "thing" 91 ;; SignedInteger, "1" 84 ;; End (sequence) A0 65 ;; SignedInteger, "101" B1 09 42 6C 61 63 6B 77 65 6C 6C ;; String, "Blackwell" B4 ;; Record B3 04 64 61 74 65 ;; Symbol, "date" A1 07 1D ;; SignedInteger, "1821" 92 ;; SignedInteger, "2" 93 ;; SignedInteger, "3" 84 ;; End (record) B1 02 44 72 ;; String, "Dr" 84 ;; End (record) [^extensibility2]: It happens to line up with Racket's representation of a record label for an inheritance hierarchy where `titled` extends `person` extends `thing`: (struct date (year month day) #:prefab) (struct thing (id) #:prefab) (struct person thing (name date-of-birth) #:prefab) (struct titled person (title) #:prefab) For more detail on Racket's representations of record labels, see [the Racket documentation for `make-prefab-struct`](http://docs.racket-lang.org/reference/structutils.html#%28def._%28%28quote._~23~25kernel%29._make-prefab-struct%29%29). ### JSON examples. Preserves text syntax is a superset of JSON, so the examples from [RFC 8259](https://tools.ietf.org/html/rfc8259#section-13) read as valid Preserves. The JSON literals `true`, `false` and `null` all read as `Symbol`s, and JSON numbers read (unambiguously) either as `SignedInteger`s or as `Double`s.[^json-superset] [^json-superset]: The following [schema](./preserves-schema.html) definitions match exactly the JSON subset of a Preserves input: version 1 . JSON = @string string / @integer int / @double double / @boolean JSONBoolean / @null =null / @array [JSON ...] / @object { string: JSON ...:... } . JSONBoolean = =true / =false . The first RFC 8259 example: { "Image": { "Width": 800, "Height": 600, "Title": "View from 15th Floor", "Thumbnail": { "Url": "http://www.example.com/image/481989943", "Height": 125, "Width": 100 }, "Animated" : false, "IDs": [116, 943, 234, 38793] } } when read using the Preserves text syntax encodes via the binary syntax as follows: B7 B1 05 "Image" B7 B1 03 "IDs" B5 A0 74 A1 03 AF A1 00 EA A2 00 97 89 84 B1 05 "Title" B1 14 "View from 15th Floor" B1 05 "Width" A1 03 20 B1 06 "Height" A1 02 58 B1 08 "Animated" B3 05 "false" B1 09 "Thumbnail" B7 B1 03 "Url" B1 26 "http://www.example.com/image/481989943" B1 05 "Width" A0 64 B1 06 "Height" A0 7D 84 84 84 The second RFC 8259 example: [ { "precision": "zip", "Latitude": 37.7668, "Longitude": -122.3959, "Address": "", "City": "SAN FRANCISCO", "State": "CA", "Zip": "94107", "Country": "US" }, { "precision": "zip", "Latitude": 37.371991, "Longitude": -122.026020, "Address": "", "City": "SUNNYVALE", "State": "CA", "Zip": "94085", "Country": "US" } ] encodes to binary as follows: B5 B7 B1 03 "Zip" B1 05 "94107" B1 04 "City" B1 0D "SAN FRANCISCO" B1 05 "State" B1 02 "CA" B1 07 "Address" B1 00 B1 07 "Country" B1 02 "US" B1 08 "Latitude" 83 40 42 E2 26 80 9D 49 52 B1 09 "Longitude" 83 C0 5E 99 56 6C F4 1F 21 B1 09 "precision" B1 03 "zip" 84 B7 B1 03 "Zip" B1 05 "94085" B1 04 "City" B1 09 "SUNNYVALE" B1 05 "State" B1 02 "CA" B1 07 "Address" B1 00 B1 07 "Country" B1 02 "US" B1 08 "Latitude" 83 40 42 AF 9D 66 AD B4 03 B1 09 "Longitude" 83 C0 5E 81 AA 4F CA 42 AF B1 09 "precision" B1 03 "zip" 84 84 ## Notes