Python 0.994 docs

- @preserves/core@0.994.0
 - @preserves/schema-cli@0.994.0
 - @preserves/schema@0.994.0

Makefile

@@ -22,4 +22,3 @@ test-all:
 	(cd implementations/javascript; npm test)
 	(cd implementations/python; make test)
 	(cd implementations/racket/preserves; make testonly)
-	(cd implementations/rust; cargo test)

README.md

@@ -38,14 +38,14 @@ automatic, perfect-fidelity conversion between syntaxes.
 
 #### Implementations of the data model, plus Preserves textual and binary transfer syntax
 
-| Language[^pre-alpha-implementations]              | Code                                                                         | Package                                                                        | Docs                                      |
-|-----------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------|-------------------------------------------|
-| Nim                   | [git.syndicate-lang.org](https://git.syndicate-lang.org/ehmry/preserves-nim) |                                                                                |                                           |
-| Python                | [preserves.dev]({{page.projecttree}}/implementations/python/)                | [`pip install preserves`](https://pypi.org/project/preserves/)                 | [docs](python/latest/)                    |
-| Racket                | [preserves.dev]({{page.projecttree}}/implementations/racket/preserves/)      | [`raco pkg install preserves`](https://pkgs.racket-lang.org/package/preserves) |                                           |
-| Rust                  | [preserves.dev]({{page.projecttree}}/implementations/rust/)                  | [`cargo add preserves`](https://crates.io/crates/preserves)                    | [docs](https://docs.rs/preserves/latest/) |
-| Squeak Smalltalk      | [SqueakSource](https://squeaksource.com/Preserves.html)                      | `Installer ss project: 'Preserves';`<br>`  install: 'Preserves'`               |                                           |
-| TypeScript/JavaScript | [preserves.dev]({{page.projecttree}}/implementations/javascript/)            | [`yarn add @preserves/core`](https://www.npmjs.com/package/@preserves/core)    |                                           |
+| Language[^pre-alpha-implementations] | Code                                                                         | Package                                                                        | Docs                                      |
+|--------------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------|-------------------------------------------|
+| Nim                                  | [git.syndicate-lang.org](https://git.syndicate-lang.org/ehmry/preserves-nim) |                                                                                |                                           |
+| Python                               | [preserves.dev]({{page.projecttree}}/implementations/python/)                | [`pip install preserves`](https://pypi.org/project/preserves/)                 | [docs](python/latest/)                    |
+| Racket                               | [preserves.dev]({{page.projecttree}}/implementations/racket/preserves/)      | [`raco pkg install preserves`](https://pkgs.racket-lang.org/package/preserves) |                                           |
+| Rust                                 | [preserves.dev](https://gitlab.com/preserves/preserves-rs/)                  | [`cargo add preserves`](https://crates.io/crates/preserves)                    | [docs](https://docs.rs/preserves/latest/) |
+| Squeak Smalltalk                     | [SqueakSource](https://squeaksource.com/Preserves.html)                      | `Installer ss project: 'Preserves';`<br>`  install: 'Preserves'`               |                                           |
+| TypeScript/JavaScript                | [preserves.dev]({{page.projecttree}}/implementations/javascript/)            | [`yarn add @preserves/core`](https://www.npmjs.com/package/@preserves/core)    |                                           |
 
 [^pre-alpha-implementations]: Pre-alpha implementations also exist for
     [C]({{page.projecttree}}/implementations/c/) and
@@ -73,6 +73,7 @@ automatic, perfect-fidelity conversion between syntaxes.
 
 ## Additional resources
 
+ - [Preserves Expressions (P-expressions, pexprs)](preserves-expressions.html)
  - Some [conventions for common data types](conventions.html)
  - [Open questions](questions.html); see also the
    [issues list]({{page.projectpages}}/issues)

TUTORIAL.md

@@ -105,7 +105,7 @@ A few more interesting differences:
      {"dictionaries": "as keys???"}: "well, why not?"}
 ```
 
-Preserves technically provides a few types of numbers:
+Preserves technically provides various types of numbers:
 
 ```
     # Signed Integers
@@ -114,9 +114,6 @@ Preserves technically provides a few types of numbers:
     5907212309572059846509324862304968273468909473609826340
     -5907212309572059846509324862304968273468909473609826340
 
-    # Floats (Single-precision IEEE floats) (notice the trailing f)
-    3.1415927f
-
     # Doubles (Double-precision IEEE floats)
     3.141592653589793
 ```

_config.yml

@@ -13,5 +13,5 @@ defaults:
       layout: page
 
 title: "Preserves"
-version_date: "October 2023"
-version: "0.992.0"
+version_date: "February 2024"
+version: "0.994.0"

_data/python-versions.json

@@ -1,6 +1,10 @@
 [
+  {"version":"0.994.0","title":"0.994.0","aliases":["latest"]},
+  {"version":"0.993.0","title":"0.993.0","aliases":[]},
+  {"version":"0.992.2","title":"0.992.2","aliases":[]},
+  {"version":"0.992.1","title":"0.992.1","aliases":[]},
   {"version":"0.992.0","title":"0.992.0","aliases":[]},
-  {"version":"0.991.0","title":"0.991.0","aliases":["latest"]},
+  {"version":"0.991.0","title":"0.991.0","aliases":[]},
   {"version":"0.990.1","title":"0.990.1","aliases":[]},
   {"version":"0.990.0","title":"0.990.0","aliases":[]},
   {"version":"0.18.1","title":"0.18.1","aliases":[]},

_includes/cheatsheet-binary-plaintext.md

@@ -5,9 +5,8 @@ For a value `V`, we write `«V»` for the binary encoding of `V`.
                       «#t» = [0x81]
 
                     «@W V» = [0x85] ++ «W» ++ «V»
-                     «#!V» = [0x86] ++ «V»
+                     «#:V» = [0x86] ++ «V»
 
-  «V» if V ∈ Float         = [0x87, 0x04] ++ binary32(V)
   «V» if V ∈ Double        = [0x87, 0x08] ++ binary64(V)
 
   «V» if V ∈ SignedInteger = [0xB0] ++ varint(|intbytes(V)|) ++ intbytes(V)
@@ -29,5 +28,4 @@ For a value `V`, we write `«V»` for the binary encoding of `V`.
                              signedBigEndian(n >> 8) ++ [n & 255] otherwise
 ```
 
-The functions `binary32(F)` and `binary64(D)` yield big-endian 4- and
-8-byte IEEE 754 binary representations of `F` and `D`, respectively.
+The function `binary64(D)` yields the big-endian 8-byte IEEE 754 binary representation of `D`.

_includes/cheatsheet-binary.md

@@ -8,10 +8,9 @@ class="postcard-grammar binarysyntax">*V*</span>.
 
 {:.postcard-grammar.binarysyntax}
 «`@`*W* *V*» | = | `85` «*W*» «*V*»
-«`#!`*V*» | = | `86` «*V*»
+«`#:`*V*» | = | `86` «*V*»
 
 {:.postcard-grammar.binarysyntax}
-«*V*» | = | `87``04` **binary32**(*V*) | if *V* ∈ Float
 «*V*» | = | `87``08` **binary64**(*V*) | if *V* ∈ Double
 
 {:.postcard-grammar.binarysyntax}
@@ -37,10 +36,9 @@ class="postcard-grammar binarysyntax">*V*</span>.
 **signedBigEndian**(*n*) | = | <span class="outputish">*n* &amp; 255</span> | if −128 ≤ *n* ≤ 127
 | | **signedBigEndian**(*n* &gt;&gt; 8) <span class="outputish">*n* &amp; 255</span> | otherwise
 
-The functions <span class="postcard-grammar binarysyntax">**binary32**(*F*)</span> and <span
-class="postcard-grammar binarysyntax">**binary64**(*D*)</span> yield big-endian 4- and 8-byte
-IEEE 754 binary representations of <span class="postcard-grammar binarysyntax">*F*</span> and
-<span class="postcard-grammar binarysyntax">*D*</span>, respectively.
+The function <span class="postcard-grammar binarysyntax">**binary64**(*D*)</span> yields the
+big-endian 8-byte IEEE 754 binary representation of <span class="postcard-grammar
+binarysyntax">*D*</span>.
 
 <!--
 Together, <span class="postcard-grammar binarysyntax">**div**</span> and <span

_includes/cheatsheet-pexprs-plaintext.md

@@ -1,22 +1,21 @@
-The definition of `Atom` is as given in the Preserves text syntax.
+The definitions of `Atom`, `ws`, and `linecomment` are as given in the Preserves text syntax.
 
 ```text
-Document      :=  Expr* sp
-Expr          :=  sp (Atom | Compound | Punct | Embedded | Annotated)
+Document      :=  Expr* Trailer ws
+Expr          :=  ws (SimpleExpr | Punct)
+SimpleExpr    :=  Compound | Embedded | Annotated | Atom
 Compound      :=  Sequence | Record | Block | Group | Set
 Punct         :=  `,` | `;` | `:`+
 
-sp            :=  (space | tab | cr | lf)*
+Sequence      :=   `[` Expr* Trailer ws `]`
+Record        :=   `<` Expr* Trailer ws `>`
+Block         :=   `{` Expr* Trailer ws `}`
+Group         :=   `(` Expr* Trailer ws `)`
+Set           :=  `#{` Expr* Trailer ws `}`
 
-Sequence      :=   `[` Expr* Trailer sp `]`
-Record        :=   `<` Expr* Trailer sp `>`
-Block         :=   `{` Expr* Trailer sp `}`
-Group         :=   `(` Expr* Trailer sp `)`
-Set           :=  `#{` Expr* Trailer sp `}`
+Trailer       :=  (ws Annotation)*
 
-Trailer       :=  Annotation*
-
-Embedded      :=  `#!` Expr
-Annotated     :=  Annotation Expr
-Annotation    :=  `@` Expr | `#` ((space | tab) linecomment) (cr | lf)
+Embedded      :=  `#:` SimpleExpr
+Annotated     :=  Annotation SimpleExpr
+Annotation    :=  `@` SimpleExpr | `#` ((space | tab) linecomment) (cr | lf)
 ```

_includes/cheatsheet-pexprs.md

@@ -0,0 +1,23 @@
+The definitions of `Atom`, `ws`, and `linecomment` are as given in the Preserves text syntax.
+
+{:.postcard-grammar.textsyntax}
+| *Document*      | := |  *Expr*<sup>⋆</sup> *Trailer* **ws**
+| *Expr*          | := |  **ws** (*SimpleExpr* &#124; *Punct*)
+| *SimpleExpr*    | := |  *Compound* &#124; *Embedded* &#124; *Annotated* &#124; *Atom*
+| *Compound*      | := |  *Sequence* &#124; *Record* &#124; *Block* &#124; *Group* &#124; *Set*
+| *Punct*         | := |  `,` &#124; `;` &#124; `:`<sup>+</sup>
+
+{:.postcard-grammar.textsyntax}
+| *Sequence*      | := |   `[` *Expr*<sup>⋆</sup> *Trailer* **ws** `]`
+| *Record*        | := |   `<` *Expr*<sup>⋆</sup> *Trailer* **ws** `>`
+| *Block*         | := |   `{` *Expr*<sup>⋆</sup> *Trailer* **ws** `}`
+| *Group*         | := |   `(` *Expr*<sup>⋆</sup> *Trailer* **ws** `)`
+| *Set*           | := |  `#{` *Expr*<sup>⋆</sup> *Trailer* **ws** `}`
+
+{:.postcard-grammar.textsyntax}
+| *Trailer*       | := |  (**ws** *Annotation*)<sup>⋆</sup>
+
+{:.postcard-grammar.textsyntax}
+| *Embedded*      | := |  `#:` *SimpleExpr*
+| *Annotated*     | := |  *Annotation* *SimpleExpr*
+| *Annotation*    | := |  `@` *SimpleExpr* &#124; `#` ((**space** &#124; **tab**) *linecomment*) (**cr** &#124; **lf**)

_includes/cheatsheet-text-plaintext.md

@@ -1,18 +1,19 @@
 ```text
 Document      :=  Value ws
-Value         :=  ws (Record | Collection | Atom | Embedded | Annotated)
+Value         :=  ws (Record | Collection | Embedded | Annotated | Atom)
 Collection    :=  Sequence | Dictionary | Set
-Atom          :=  Boolean | ByteString | String | QuotedSymbol | Symbol | Number
-ws            :=  (space | tab | cr | lf)*
-commas        :=  (ws `,`)* ws
-delimiter     :=  ws | `<` | `>` | `[` | `]` | `{` | `}`
-                     | `#` | `:` | `"` | `|` | `@` | `;` | `,`
 
 Record        :=  `<` Value+ ws `>`
 Sequence      :=  `[` (commas Value)* commas `]`
 Set           :=  `#{` (commas Value)* commas `}`
 Dictionary    :=  `{` (commas Value ws `:` Value)* commas `}`
+commas        :=  (ws `,`)* ws
 
+Embedded      :=  `#:` Value
+Annotated     :=  Annotation Value
+Annotation    :=  `@` Value | `#` ((space | tab) linecomment) (cr | lf)
+
+Atom          :=  Boolean | ByteString | String | QuotedSymbol | Symbol | Number
 Boolean       :=  `#t` | `#f`
 ByteString    :=  `#"` binchar* `"`
                |  `#x"` (ws hex hex)* ws `"`
@@ -20,16 +21,10 @@ ByteString    :=  `#"` binchar* `"`
 String        :=  `"` («any unicode scalar except `\` or `"`» | escaped | `\"`)* `"`
 QuotedSymbol  :=  `|` («any unicode scalar except `\` or `|`» | escaped | `\|`)* `|`
 Symbol        :=  (`A`..`Z` | `a`..`z` | `0`..`9` | sympunct | symuchar)+
-Number        :=  Float | Double | SignedInteger
-Float         :=  flt (`f`|`F`) | `#xf"` (ws hex hex)4 ws `"`
+Number        :=  Double | SignedInteger
 Double        :=  flt | `#xd"` (ws hex hex)8 ws `"`
 SignedInteger :=  int
 
-Embedded      :=  `#!` Value
-Annotated     :=  Annotation Value
-Annotation    :=  `@` Value | `#` ((space | tab) linecomment) (cr | lf)
-linecomment   :=  «any unicode scalar except cr or lf»*
-
 escaped       :=  `\\` | `\/` | `\b` | `\f` | `\n` | `\r` | `\t` | `\u` hex hex hex hex
 binescaped    :=  `\\` | `\/` | `\b` | `\f` | `\n` | `\r` | `\t` | `\x` hex hex
 binchar       :=  «any scalar ≥32 and ≤126, except `\` or `"`» | binescaped | `\"`
@@ -45,4 +40,9 @@ int           :=  (`-`|`+`) (`0`..`9`)+
 frac          :=  `.` (`0`..`9`)+
 exp           :=  (`e`|`E`) (`-`|`+`) (`0`..`9`)+
 hex           :=  `A`..`F` | `a`..`f` | `0`..`9`
+
+ws            :=  (space | tab | cr | lf)*
+delimiter     :=  ws | `<` | `>` | `[` | `]` | `{` | `}`
+                     | `#` | `:` | `"` | `|` | `@` | `;` | `,`
+linecomment   :=  «any unicode scalar except cr or lf»*
 ```

_includes/cheatsheet-text.md

@@ -1,35 +1,31 @@
 {:.postcard-grammar.textsyntax}
 | *Document* | := | *Value* **ws** |
-| *Value* | := | **ws** (*Record* | *Collection* | *Atom* | *Embedded* | *Annotated*) |
+| *Value* | := | **ws** (*Record* | *Collection* | *Embedded* | *Annotated* | *Atom*) |
 | *Collection* | := | *Sequence* | *Dictionary* | *Set* |
-| *Atom* | := | *Boolean* | *ByteString* | *String* | *QuotedSymbol* | *Symbol* | *Number* |
-| **ws** | := | (**space** &#124; **tab** &#124; **cr** &#124; **lf**)<sup>⋆</sup> |
-| **commas** | := | (**ws** `,`)<sup>⋆</sup> **ws** |
-| **delimiter** | := | **ws** &#124; `<` &#124; `>` &#124; `[` &#124; `]` &#124; `{` &#124; `}` &#124; `#` &#124; `:` &#124; `"` &#124; `|` &#124; `@` &#124; `;` &#124; `,` |
 
 {:.postcard-grammar.textsyntax}
 | *Record* | := | `<`*Value*<sup>+</sup> **ws**`>` |
 | *Sequence* | := | `[`(**commas** *Value*)<sup>⋆</sup> **commas**`]` |
 | *Set* | := | `#{`(**commas** *Value*)<sup>⋆</sup> **commas**`}` |
 | *Dictionary* | := | `{` (**commas** *Value* **ws**`:`*Value*)<sup>⋆</sup> **commas**`}` |
+| **commas** | := | (**ws** `,`)<sup>⋆</sup> **ws** |
 
 {:.postcard-grammar.textsyntax}
+| *Embedded* | := | `#:`*Value* |
+| *Annotated* | := | *Annotation* *Value* |
+| *Annotation* | := | `@`*Value* &#124;`#` ((**space** &#124; **tab**) *linecomment*) (**cr** &#124; **lf**) |
+
+{:.postcard-grammar.textsyntax}
+| *Atom* | := | *Boolean* &#124; *ByteString* &#124; *String* &#124; *QuotedSymbol* &#124; *Symbol* &#124; *Number* |
 | *Boolean* | := | `#t`&#124;`#f` |
 | *ByteString* | := | `#"`*binchar*<sup>⋆</sup> `"`&#124;`#x"` (**ws** *hex* *hex*)<sup>⋆</sup> **ws**`"`&#124;`#[` (**ws** *base64char*)<sup>⋆</sup> **ws**`]` |
 | *String* | := | `"` (« any unicode scalar value except `\` or `"` » &#124; *escaped* &#124;`\"`)<sup>⋆</sup> `"` |
 | *QuotedSymbol* | := | `|` (« any unicode scalar value except `\` or `|` » &#124; *escaped* &#124;`\|`)<sup>⋆</sup> `|` |
 | *Symbol* | := | (`A`..`Z`&#124;`a`..`z`&#124;`0`..`9`&#124; *sympunct* &#124; *symuchar*)<sup>+</sup> |
-| *Number* | := | *Float* &#124; *Double* &#124; *SignedInteger* |
-| *Float* | := | *flt* (`f`&#124;`F`) &#124;`#xf"` (**ws** *hex* *hex*)<sup>4</sup> **ws**`"` |
+| *Number* | := | *Double* &#124; *SignedInteger* |
 | *Double* | := | *flt* &#124;`#xd"` (**ws** *hex* *hex*)<sup>8</sup> **ws**`"` |
 | *SignedInteger* | := | *int* |
 
-{:.postcard-grammar.textsyntax}
-| *Embedded* | := | `#!`*Value* |
-| *Annotated* | := | *Annotation* *Value* |
-| *Annotation* | := | `@`*Value* &#124;`#` ((**space** &#124; **tab**) *linecomment*) (**cr** &#124; **lf**) |
-| *linecomment* | := | « any unicode scalar value except **cr** or **lf** »<sup>⋆</sup> |
-
 {:.postcard-grammar.textsyntax}
 | *escaped* | := | `\\`&#124;`\/`&#124;`\b`&#124;`\f`&#124;`\n`&#124;`\r`&#124;`\t`&#124;`\u`*hex* *hex* *hex* *hex* |
 | *binescaped* | := | `\\`&#124;`\/`&#124;`\b`&#124;`\f`&#124;`\n`&#124;`\r`&#124;`\t`&#124;`\x`*hex* *hex* |
@@ -44,3 +40,8 @@
 | *frac* | := | `.` (`0`..`9`)<sup>+</sup> |
 | *exp* | := | (`e`&#124;`E`) (`-`&#124;`+`) (`0`..`9`)<sup>+</sup> |
 | *hex* | := | `A`..`F`&#124;`a`..`f`&#124;`0`..`9` |
+
+{:.postcard-grammar.textsyntax}
+| **ws** | := | (**space** &#124; **tab** &#124; **cr** &#124; **lf**)<sup>⋆</sup> |
+| **delimiter** | := | **ws** &#124; `<` &#124; `>` &#124; `[` &#124; `]` &#124; `{` &#124; `}` &#124; `#` &#124; `:` &#124; `"` &#124; `|` &#124; `@` &#124; `;` &#124; `,` |
+| *linecomment* | := | « any unicode scalar value except **cr** or **lf** »<sup>⋆</sup> |

_includes/python-representation.md

@@ -1,5 +1,5 @@
 Python's strings, byte strings, integers, booleans, and double-precision floats stand directly
-for their Preserves counterparts. Wrapper objects for [Float][preserves.values.Float] and
+for their Preserves counterparts. Wrapper objects for
 [Symbol][preserves.values.Symbol] complete the suite of atomic types.
 
 Python's lists and tuples correspond to Preserves `Sequence`s, and dicts and sets to

_includes/text-examples.md

@@ -2,7 +2,6 @@ Here are a few example values, written using the [text
 syntax](https://preserves.dev/preserves-text.html):
 
     Boolean    : #t #f
-    Float      : 1.0f 10.4e3f -100.6f
     Double     : 1.0 10.4e3 -100.6
     Integer    : 1 0 -100
     String     : "Hello, world!\n"
@@ -12,6 +11,6 @@ syntax](https://preserves.dev/preserves-text.html):
     Sequence   : [value1 value2 ...]
     Set        : #{value1 value2 ...}
     Dictionary : {key1: value1 key2: value2 ...: ...}
-    Embedded   : #!value
+    Embedded   : #:value
 
 Commas are optional in sequences, sets, and dictionaries.

_includes/value-grammar.md

@@ -4,7 +4,6 @@
                             | Embedded
 
                        Atom = Boolean
-                            | Float
                             | Double
                             | SignedInteger
                             | String

canonical-binary.md

@@ -38,7 +38,7 @@ representations of their keys.[^no-need-for-by-value]
 **Other kinds of `Value`.**
 There are no special canonicalization restrictions on
 `SignedInteger`s, `String`s, `ByteString`s, `Symbol`s, `Boolean`s,
-`Float`s, `Double`s, `Record`s, `Sequence`s, or `Embedded`s. The
+`Double`s, `Record`s, `Sequence`s, or `Embedded`s. The
 constraints given for these `Value`s in the [specification][spec]
 suffice to ensure canonicity.
 

cheatsheet-plaintext.md

@@ -6,6 +6,20 @@ title: "Preserves Quick Reference (Plaintext)"
 Tony Garnock-Jones <tonyg@leastfixedpoint.com>  
 {{ site.version_date }}. Version {{ site.version }}.
 
+These are non-normative "reference card" definitions. See also the normative [semantics]({{
+site.baseurl }}/preserves.html), [text syntax specification]({{ site.baseurl
+}}/preserves-text.html), and [machine-oriented syntax specification]({{ site.baseurl
+}}/preserves-binary.html), and the experimental [P-expressions definition]({{ site.baseurl
+}}/preserves-expressions.html).
+
+## <a id="binary"></a>Machine-Oriented Binary Syntax
+
 {% include cheatsheet-binary-plaintext.md %}
 
+## <a id="text"></a>Human-Oriented Text Syntax
+
 {% include cheatsheet-text-plaintext.md %}
+
+## <a id="pexprs"></a>P-expression Syntax
+
+{% include cheatsheet-pexprs-plaintext.md %}

cheatsheet.md

@@ -8,8 +8,9 @@ Tony Garnock-Jones <tonyg@leastfixedpoint.com>
 
 These are non-normative "reference card" definitions. See also the normative [semantics]({{
 site.baseurl }}/preserves.html), [text syntax specification]({{ site.baseurl
-}}/preserves-text.html) and [machine-oriented syntax specification]({{ site.baseurl
-}}/preserves-binary.html).
+}}/preserves-text.html), and [machine-oriented syntax specification]({{ site.baseurl
+}}/preserves-binary.html), and the experimental [P-expressions definition]({{ site.baseurl
+}}/preserves-expressions.html).
 
 ## <a id="binary"></a>Machine-Oriented Binary Syntax
 
@@ -18,3 +19,7 @@ site.baseurl }}/preserves.html), [text syntax specification]({{ site.baseurl
 ## <a id="text"></a>Human-Oriented Text Syntax
 
 {% include cheatsheet-text.md %}
+
+## <a id="pexprs"></a>P-expression Syntax
+
+{% include cheatsheet-pexprs.md %}

conventions.md

@@ -23,10 +23,10 @@ Appropriately-labelled `Record`s denote these domain-specific data
 types.[^why-dictionaries]
 
   [^why-dictionaries]: Given `Record`'s existence, it may seem odd
-    that `Dictionary`, `Set`, `Float`, etc. are given special
+    that `Dictionary`, `Set`, `Double`, etc. are given special
     treatment. Preserves aims to offer a useful basic equivalence
     predicate to programmers, and so if a data type demands a special
-    equivalence predicate, as `Dictionary`, `Set` and `Float` all do,
+    equivalence predicate, as `Dictionary`, `Set` and `Double` all do,
     then the type should be included in the base language. Otherwise,
     it can be represented as a `Record` and treated separately.
     `Boolean`, `String` and `Symbol` are seeming exceptions. The first

doc/preserves-tool.md

@@ -5,20 +5,29 @@ title: preserves-tool
 The `preserves-tool` program is a swiss army knife for working with
 Preserves documents.
 
-    preserves-tools 1.0.0
+```
+preserves-tool 4.992.0
+Swiss-army knife tool for working with Preserves data.
+See https://preserves.dev/. If no subcommand is specified, the default
+subcommand will be `convert`.
 
-    USAGE:
-        preserves-tool <SUBCOMMAND>
+USAGE:
+    preserves-tool [OPTIONS]
+    preserves-tool <SUBCOMMAND>
 
-    FLAGS:
-        -h, --help       Print help information
-        -V, --version    Print version information
+OPTIONS:
+    -h, --help       Print help information
+    -V, --version    Print version information
 
-    SUBCOMMANDS:
-        completions
-        convert
-        help           Print this message or the help of the given subcommand(s)
-        quote
+OPTIONS FOR DEFAULT SUBCOMMAND convert:
+    [...]
+
+SUBCOMMANDS:
+    completions
+    convert
+    help           Print this message or the help of the given subcommand(s)
+    quote
+```
 
 ## Installation
 
@@ -31,9 +40,10 @@ Then, `cargo install preserves-tools`.
 
 The tool includes three subcommands.
 
-### `preserves-tool convert`
+### `preserves-tool convert`, `preserves-tool`
 
-This is the main tool. It can
+This is the main tool, and is also the default if no subcommand is
+explicitly specified. It can
 
  - translate between the various Preserves text and binary document
    syntaxes;
@@ -48,38 +58,45 @@ This is the main tool. It can
     USAGE:
         preserves-tool convert [FLAGS] [OPTIONS]
 
-    FLAGS:
-            --collect
-            --escape-spaces
-        -h, --help             Prints help information
-        -V, --version          Prints version information
-
     OPTIONS:
-            --bundle <filename>...
-        -c, --commas <commas>
+            --bundle <filename>
+
+
+        -c, --commas <COMMAS>
                 [default: none] [possible values: none, separating, terminating]
 
-            --indent <on/off>
-                [default: on] [possible values: disabled, no, n, off, 0, false,
-                enabled, yes, y, on, 1, true]
+            --collect
 
-        -i, --input-format <input-format>
+
+            --escape-spaces
+
+
+        -h, --help
+                Print help information
+
+        -i, --input-format <INPUT_FORMAT>
                 [default: auto-detect] [possible values: auto-detect, text, binary]
 
-            --limit <limit>
-        -o, --output-format <output-format>
-                [default: text] [possible values: text, binary, unquoted]
-            --read-annotations <on/off>
-                [default: on] [possible values: disabled, no, n, off, 0, false,
-                enabled, yes, y, on, 1, true]
+            --indent <on/off>
+                [default: on] [possible values: disabled, enabled]
 
-            --schema <schema>
-            --select <select-expr>             [default: *]
-            --select-output <select-output>
+            --limit <LIMIT>
+
+
+        -o, --output-format <OUTPUT_FORMAT>
+                [default: text] [possible values: text, binary, unquoted]
+
+            --read-annotations <on/off>
+                [default: on] [possible values: disabled, enabled]
+
+            --select <SELECT_EXPR>
+                [default: *]
+
+            --select-output <SELECT_OUTPUT>
                 [default: sequence] [possible values: sequence, set]
+
             --write-annotations <on/off>
-                [default: on] [possible values: disabled, no, n, off, 0, false,
-                enabled, yes, y, on, 1, true]
+                [default: on] [possible values: disabled, enabled]
 
 ### `preserves-tool quote`
 
@@ -101,12 +118,10 @@ preserves-tool-quote
 USAGE:
     preserves-tool quote [OPTIONS] <SUBCOMMAND>
 
-FLAGS:
-    -h, --help       Print help information
-    -V, --version    Print version information
-
 OPTIONS:
-    -o, --output-format <OUTPUT_FORMAT>    [default: text] [possible values: text, binary, unquoted]
+    -h, --help                             Print help information
+    -o, --output-format <OUTPUT_FORMAT>    [default: text] [possible values: text,
+                                           binary, unquoted]
 
 SUBCOMMANDS:
     byte-string
@@ -119,32 +134,28 @@ SUBCOMMANDS:
 preserves-tool-quote-string
 
 USAGE:
-    preserves-tool quote string [FLAGS] [OPTIONS]
-
-FLAGS:
-        --escape-spaces
-    -h, --help                  Print help information
-        --include-terminator
-    -V, --version               Print version information
+    preserves-tool quote string [OPTIONS]
 
 OPTIONS:
-        --input-terminator <INPUT_TERMINATOR>    [default: eof] [possible values: eof, newline, nul]
+        --escape-spaces
+    -h, --help                                   Print help information
+        --include-terminator
+        --input-terminator <INPUT_TERMINATOR>    [default: eof] [possible values:
+                                                 eof, newline, nul]
 ```
 
 ```
 preserves-tool-quote-symbol
 
 USAGE:
-    preserves-tool quote symbol [FLAGS] [OPTIONS]
-
-FLAGS:
-        --escape-spaces
-    -h, --help                  Print help information
-        --include-terminator
-    -V, --version               Print version information
+    preserves-tool quote symbol [OPTIONS]
 
 OPTIONS:
-        --input-terminator <INPUT_TERMINATOR>    [default: eof] [possible values: eof, newline, nul]
+        --escape-spaces
+    -h, --help                                   Print help information
+        --include-terminator
+        --input-terminator <INPUT_TERMINATOR>    [default: eof] [possible values:
+                                                 eof, newline, nul]
 ```
 
 ```
@@ -153,9 +164,8 @@ preserves-tool-quote-byte-string
 USAGE:
     preserves-tool quote byte-string
 
-FLAGS:
-    -h, --help       Print help information
-    -V, --version    Print version information
+OPTIONS:
+    -h, --help    Print help information
 ```
 
 ### `preserves-tool completions`
@@ -176,12 +186,11 @@ Multiple shell dialects are supported (courtesy of
 preserves-tool-completions
 
 USAGE:
-    preserves-tool completions <dialect>
+    preserves-tool completions <SHELL>
 
 ARGS:
-    <dialect>    [possible values: bash, zsh, power-shell, fish, elvish]
+    <SHELL>    [possible values: bash, elvish, fish, powershell, zsh]
 
-FLAGS:
-    -h, --help       Print help information
-    -V, --version    Print version information
+OPTIONS:
+    -h, --help    Print help information
 ```

git-hooks/pre-commit

@@ -31,7 +31,6 @@ fi
 # Ensure that various copies of schema.prs, schema.bin, path.bin,
 # samples.pr and samples.bin are in fact identical.
 ${COMMAND} path/path.bin implementations/python/preserves/path.prb
-${COMMAND} path/path.bin implementations/rust/preserves-path/path.bin
 
 ${COMMAND} schema/schema.bin implementations/python/preserves/schema.prb
 ${COMMAND} schema/schema.prs implementations/racket/preserves/preserves-schema/schema.prs
@@ -40,11 +39,4 @@ ${COMMAND} tests/samples.bin implementations/python/tests/samples.bin
 ${COMMAND} tests/samples.pr implementations/python/tests/samples.pr
 ${COMMAND} tests/samples.pr implementations/racket/preserves/preserves/tests/samples.pr
 
-${COMMAND} _includes/what-is-preserves.md implementations/rust/preserves/doc/what-is-preserves.md
-${COMMAND} _includes/cheatsheet-binary-plaintext.md implementations/rust/preserves/doc/cheatsheet-binary-plaintext.md
-${COMMAND} _includes/cheatsheet-text-plaintext.md implementations/rust/preserves/doc/cheatsheet-text-plaintext.md
-${COMMAND} _includes/value-grammar.md implementations/rust/preserves/doc/value-grammar.md
-
-${COMMAND} _includes/what-is-preserves-schema.md implementations/rust/preserves-schema/doc/what-is-preserves-schema.md
-
 [ -z "$failed" ]

implementations/README.md

@@ -13,9 +13,9 @@ Here you may find:
  - [racket](racket/), an implementation for Racket 7.x and newer
    (though older Rackets may also work with it).
 
- - [rust](rust/), an implementation for Rust that interoperates with
-   serde.
-
 Other implementations are also available:
 
+ - [Preserves for Rust](https://gitlab.com/preserves/preserves-rs/), an implementation for Rust
+   that interoperates with serde.
+
  - [Preserves for Squeak Smalltalk](https://squeaksource.com/Preserves.html)

implementations/c/preserves.h

@@ -261,7 +261,6 @@ PRESERVES_OUTOFLINE
 
 typedef enum preserves_type_tag {
   PRESERVES_BOOLEAN = 0,
-  PRESERVES_FLOAT,
   PRESERVES_DOUBLE,
 
   PRESERVES_SIGNED_INTEGER,
@@ -283,7 +282,6 @@ typedef enum preserves_type_tag {
 PRESERVES_OUTOFLINE(char const *preserves_type_tag_name(preserves_type_tag_t type), {
     switch (type) {
       case PRESERVES_BOOLEAN: return "BOOLEAN";
-      case PRESERVES_FLOAT: return "FLOAT";
       case PRESERVES_DOUBLE: return "DOUBLE";
       case PRESERVES_SIGNED_INTEGER: return "SIGNED_INTEGER";
       case PRESERVES_STRING: return "STRING";
@@ -366,7 +364,6 @@ PRESERVES_OUTOFLINE
 
 /*
   PRESERVES_BOOLEAN: repr==PRESERVES_REPR_NONE, len=0, data._boolean
-  PRESERVES_FLOAT: repr=PRESERVES_REPR_NONE, len=0, data._float
   PRESERVES_DOUBLE: repr=PRESERVES_REPR_NONE, len=0, data._double
 
   PRESERVES_SIGNED_INTEGER:
@@ -418,7 +415,6 @@ typedef struct preserves_index_entry {
 
   union {
     bool _boolean;
-    float _float;
     double _double;
     int64_t _signed;
     uint64_t _unsigned;
@@ -818,18 +814,6 @@ PRESERVES_OUTOFLINE
          uint8_t *bs = _preserves_reader_next_bytes(r, len);
          if (bs == NULL) return _preserves_reader_finish(r, PRESERVES_END_INCOMPLETE_INPUT);
          switch (len) {
-           case 4: {
-             uint32_t i;
-             memcpy(&i, bs, 4);
-             i = ntohl(i);
-             float f;
-             memcpy(&f, &i, 4);
-             RETURN_ON_FAIL(_preserves_reader_emit_entry(r, &count, (preserves_index_entry_t) {
-                   .type = PRESERVES_FLOAT, .repr = PRESERVES_REPR_NONE, .len = 0, .data = {
-                     ._float = f
-                   }}));
-             break;
-           }
            case 8: {
              uint32_t lo, hi;
              memcpy(&hi, bs, 4);
@@ -995,10 +979,6 @@ PRESERVES_IMPLEMENTATION_CHUNK
        fprintf(f, i->data._boolean ? " #t" : " #f");
        break;
 
-     case PRESERVES_FLOAT:
-       fprintf(f, " %f", i->data._float);
-       break;
-
      case PRESERVES_DOUBLE:
        fprintf(f, " %f", i->data._double);
        break;

implementations/cpp/preserves_binary_reader.hpp

@@ -41,15 +41,6 @@ namespace Preserves {
             decodeEmbedded(decodeEmbedded)
         {}
 
-        boost::optional<float> next_float() {
-            uint8_t buf[4];
-            if (!next_chunk(buf, sizeof(buf))) return boost::none;
-            uint32_t n = buf[0] << 24 | buf[1] << 16 | buf[2] << 8 | buf[3];
-            float f;
-            memcpy(&f, &n, sizeof(f));
-            return f;
-        }
-
         boost::optional<double> next_double() {
             uint8_t buf[8];
             if (!next_chunk(buf, sizeof(buf))) return boost::none;
@@ -113,7 +104,6 @@ namespace Preserves {
                     return BinaryReader<>(i).next().map(decodeEmbedded).map(Value<T>::from_embedded);
                 case BinaryTag::Ieee754: return varint(i).flat_map([&](size_t len)-> boost::optional<Value<T>> {
                     switch (len) {
-                        case 4: return next_float().map(Value<T>::from_float);
                         case 8: return next_double().map(Value<T>::from_double);
                         default: return boost::none;
                     }

implementations/cpp/preserves_binary_writer.hpp

@@ -151,19 +151,6 @@ namespace Preserves {
             return (*this) << (b ? BinaryTag::True : BinaryTag::False);
         }
 
-        BinaryWriter& operator<<(float f) {
-            uint32_t n;
-            memcpy(&n, &f, sizeof(f));
-            uint8_t buf[4];
-            buf[0] = (n >> 24) & 0xff;
-            buf[1] = (n >> 16) & 0xff;
-            buf[2] = (n >> 8) & 0xff;
-            buf[3] = (n) & 0xff;
-            (*this) << BinaryTag::Ieee754;
-            put(uint8_t(sizeof(buf)));
-            return write(buf, sizeof(buf));
-        }
-
         BinaryWriter& operator<<(double d) {
             uint64_t n;
             memcpy(&n, &d, sizeof(d));

implementations/cpp/preserves_impl.hpp

@@ -35,13 +35,6 @@ namespace Preserves {
         BinaryWriter& write(BinaryWriter& w) const override {
             return w << this->_value();
         });
-    PRESERVES_ATOMIC_VALUE_CLASS(Float, float, float, ValueKind::Float, as_float,
-        BinaryWriter& write(BinaryWriter& w) const override {
-            return w << this->_value();
-        }
-        boost::optional<double> as_double() const override {
-            return this->value;
-        });
     PRESERVES_ATOMIC_VALUE_CLASS(Double, double, double, ValueKind::Double, as_double,
         BinaryWriter& write(BinaryWriter& w) const override {
             return w << this->_value();
@@ -57,13 +50,6 @@ namespace Preserves {
                 return boost::none;
             }
         }
-        boost::optional<float> as_float() const override {
-            if (uint64_t(float(this->value)) == this->value) {
-                return float(this->value);
-            } else {
-                return boost::none;
-            }
-        }
         boost::optional<double> as_double() const override {
             if (uint64_t(double(this->value)) == this->value) {
                 return double(this->value);
@@ -82,13 +68,6 @@ namespace Preserves {
                 return boost::none;
             }
         }
-        boost::optional<float> as_float() const override {
-            if (int64_t(float(this->value)) == this->value) {
-                return float(this->value);
-            } else {
-                return boost::none;
-            }
-        }
         boost::optional<double> as_double() const override {
             if (int64_t(double(this->value)) == this->value) {
                 return double(this->value);
@@ -295,7 +274,6 @@ namespace Preserves {
         bool is_mutable() const override { return underlying.is_mutable(); }
 
         boost::optional<bool> as_bool() const override { return underlying.as_bool(); }
-        boost::optional<float> as_float() const override { return underlying.as_float(); }
         boost::optional<double> as_double() const override { return underlying.as_double(); }
         boost::optional<uint64_t> as_unsigned() const override { return underlying.as_unsigned(); }
         boost::optional<int64_t> as_signed() const override { return underlying.as_signed(); }
@@ -355,12 +333,6 @@ namespace Preserves {
         return Value<T>(new Boolean<T>(b));
     }
 
-    template <typename T>
-    Value<T> Value<T>::from_float(float f)
-    {
-        return Value<T>(new Float<T>(f));
-    }
-
     template <typename T>
     Value<T> Value<T>::from_double(double d)
     {

implementations/cpp/preserves_value.hpp

@@ -14,7 +14,6 @@ namespace Preserves {
 
     enum class ValueKind {
         Boolean,
-        Float,
         Double,
         SignedInteger,
         String,
@@ -43,7 +42,6 @@ namespace Preserves {
         std::shared_ptr<ValueImpl<T>> _impl() const { return p; }
 
         static Value from_bool(bool b);
-        static Value from_float(float f);
         static Value from_double(double d);
         static Value from_int(uint64_t i);
         static Value from_int(int64_t i);
@@ -67,11 +65,9 @@ namespace Preserves {
 
         static Value from_number(uint64_t i) { return from_int(i); }
         static Value from_number(int64_t i) { return from_int(i); }
-        static Value from_number(float f) { return from_float(f); }
         static Value from_number(double d) { return from_double(d); }
 
         static Value from(bool b) { return from_bool(b); }
-        static Value from(float f) { return from_float(f); }
         static Value from(double d) { return from_double(d); }
         static Value from(uint64_t i) { return from_int(i); }
         static Value from(unsigned i) { return from_int(uint64_t(i)); }
@@ -95,7 +91,6 @@ namespace Preserves {
         bool is_mutable() const;
 
         bool is_bool() const { return value_kind() == ValueKind::Boolean; }
-        bool is_float() const { return value_kind() == ValueKind::Float; }
         bool is_double() const { return value_kind() == ValueKind::Double; }
         bool is_int() const { return value_kind() == ValueKind::SignedInteger; }
         bool is_string() const { return value_kind() == ValueKind::String; }
@@ -109,9 +104,6 @@ namespace Preserves {
         boost::optional<bool> as_bool() const;
         bool to_bool() const { return as_bool().value(); }
 
-        boost::optional<float> as_float() const;
-        float to_float() const { return as_float().value(); }
-
         boost::optional<double> as_double() const;
         double to_double() const { return as_double().value(); }
 
@@ -175,7 +167,6 @@ namespace Preserves {
         virtual bool is_mutable() const { return false; }
 
         virtual boost::optional<bool> as_bool() const { return boost::none; }
-        virtual boost::optional<float> as_float() const { return boost::none; }
         virtual boost::optional<double> as_double() const { return boost::none; }
         virtual boost::optional<uint64_t> as_unsigned() const { return boost::none; }
         virtual boost::optional<int64_t> as_signed() const { return boost::none; }
@@ -219,7 +210,6 @@ namespace Preserves {
 #define PRESERVES_DELEGATE_CAST(t, name) \
     template <typename T> boost::optional<t> Value<T>::name() const { return p->name(); }
     PRESERVES_DELEGATE_CAST(bool, as_bool);
-    PRESERVES_DELEGATE_CAST(float, as_float);
     PRESERVES_DELEGATE_CAST(double, as_double);
     PRESERVES_DELEGATE_CAST(uint64_t, as_unsigned);
     PRESERVES_DELEGATE_CAST(int64_t, as_signed);
@@ -265,7 +255,6 @@ namespace Preserves {
         if (bKind < aKind) return false;
         switch (aKind) {
             case ValueKind::Boolean: return a.to_bool() < b.to_bool();
-            case ValueKind::Float: return a.to_float() < b.to_float();
             case ValueKind::Double: return a.to_double() < b.to_double();
             case ValueKind::SignedInteger: {
                 if (auto av = a.as_signed()) {

implementations/dhall/render.dhall

@@ -41,7 +41,7 @@ let render
                       )
                       m
                 ++  " }"
-          , embedded = λ(value : Text) → "#!${value}"
+          , embedded = λ(value : Text) → "#:${value}"
           }
 
 let Preserves/boolean = ./boolean.dhall
@@ -94,7 +94,7 @@ let example0 =
               )}
           ''
         ≡ ''
-          { a: 1 b: [ 2 3 ] c: { d: 1.0 e: -1.0 } d: #!#t e: <capture <_>> }
+          { a: 1 b: [ 2 3 ] c: { d: 1.0 e: -1.0 } d: #:#t e: <capture <_>> }
           ''
 
 in  render

implementations/javascript/packages/core/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@preserves/core",
-  "version": "0.991.0",
+  "version": "0.994.0",
   "description": "Preserves data serialization format",
   "homepage": "https://gitlab.com/preserves/preserves",
   "license": "Apache-2.0",

implementations/javascript/packages/core/src/base64.ts

@@ -0,0 +1,46 @@
+const BASE64_DEC: {[key: string]: number} = {};
+[... 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'].forEach(
+    (c, i) => BASE64_DEC[c] = i);
+BASE64_DEC['+'] = BASE64_DEC['-'] = 62;
+BASE64_DEC['/'] = BASE64_DEC['_'] = 63;
+
+export function decodeBase64(s: string): Uint8Array {
+    const bs = new Uint8Array(Math.floor(s.length * 3/4));
+    let i = 0;
+    let j = 0;
+    while (i < s.length) {
+        const v1 = BASE64_DEC[s[i++]];
+        const v2 = BASE64_DEC[s[i++]];
+        const v3 = BASE64_DEC[s[i++]];
+        const v4 = BASE64_DEC[s[i++]];
+        const v = (v1 << 18) | (v2 << 12) | (v3 << 6) | v4;
+        bs[j++] = (v >> 16) & 255;
+        if (v3 === void 0) break;
+        bs[j++] = (v >> 8) & 255;
+        if (v4 === void 0) break;
+        bs[j++] = v & 255;
+    }
+    return bs.subarray(0, j);
+}
+
+const BASE64_ENC = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
+
+export function encodeBase64(bs: Uint8Array): string {
+    let s = '';
+    let buffer = 0;
+    let bitcount = 0;
+    for (let b of bs) {
+        buffer = ((buffer & 0x3f) << 8) | b;
+        bitcount += 8;
+        while (bitcount >= 6) {
+            bitcount -= 6;
+            const v = (buffer >> bitcount) & 0x3f;
+            s = s + BASE64_ENC[v];
+        }
+    }
+    if (bitcount > 0) {
+        const v = (buffer << (6 - bitcount)) & 0x3f;
+        s = s + BASE64_ENC[v];
+    }
+    return s;
+}

implementations/javascript/packages/core/src/bytes.ts

@@ -2,10 +2,11 @@ import { Tag } from './constants';
 import { GenericEmbedded } from './embedded';
 import { Encoder, Preservable } from './encoder';
 import { Value } from './values';
-import { Writer, PreserveWritable } from './writer';
+import type { Writer, PreserveWritable } from './writer';
+import { decodeBase64, encodeBase64 } from './base64';
 
 const textEncoder = new TextEncoder();
-const textDecoder = new TextDecoder();
+const textDecoder = new TextDecoder('utf-8', { fatal: true });
 
 export const IsPreservesBytes = Symbol.for('IsPreservesBytes');
 
@@ -51,6 +52,10 @@ export class Bytes implements Preservable<any>, PreserveWritable<any> {
         return new Bytes(Uint8Array.of(...bytes));
     }
 
+    static fromBase64(s: string): Bytes {
+        return new Bytes(decodeBase64(s));
+    }
+
     static fromHex(s: string): Bytes {
         if (s.length & 1) throw new Error("Cannot decode odd-length hexadecimal string");
         const result = new Bytes(s.length >> 1);
@@ -139,6 +144,10 @@ export class Bytes implements Preservable<any>, PreserveWritable<any> {
         return Bytes.isBytes(v) ? v : void 0;
     }
 
+    toBase64(): string {
+        return encodeBase64(this._view);
+    }
+
     toHex(digit = hexDigit): string {
         var nibbles = [];
         for (let i = 0; i < this.length; i++) {

implementations/javascript/packages/core/src/decoder.ts

@@ -2,7 +2,7 @@ import { Annotated } from "./annotated";
 import { DecodeError, ShortPacket } from "./codec";
 import { Tag } from "./constants";
 import { Set, Dictionary } from "./dictionary";
-import { DoubleFloat, SingleFloat } from "./float";
+import { DoubleFloat } from "./float";
 import { Record } from "./record";
 import { Bytes, BytesLike, underlying, hexDigit } from "./bytes";
 import { Value } from "./values";
@@ -31,7 +31,6 @@ export interface TypedDecoder<T> {
         body: (d: TypedDecoder<S>) => R): R;
 
     nextBoolean(): boolean | undefined;
-    nextFloat(): SingleFloat | undefined;
     nextDouble(): DoubleFloat | undefined;
     nextEmbedded(): Embedded<T> | undefined;
     nextSignedInteger(): number | bigint | undefined;
@@ -241,7 +240,6 @@ export class Decoder<T = never> implements TypedDecoder<T> {
             case Tag.Embedded: return this.state.wrap<T>(embed(this.embeddedDecode.decode(this.state)));
             case Tag.Ieee754:
                 switch (this.state.varint()) {
-                    case 4: return this.state.wrap<T>(SingleFloat.fromBytes(this.state.nextbytes(4)));
                     case 8: return this.state.wrap<T>(DoubleFloat.fromBytes(this.state.nextbytes(8)));
                     default: throw new DecodeError("Invalid IEEE754 size");
                 }
@@ -308,14 +306,6 @@ export class Decoder<T = never> implements TypedDecoder<T> {
         });
     }
 
-    nextFloat(): SingleFloat | undefined {
-        return this.skipAnnotations((reset) => {
-            if (this.state.nextbyte() !== Tag.Ieee754) return reset();
-            if (this.state.nextbyte() !== 4) return reset();
-            return SingleFloat.fromBytes(this.state.nextbytes(4));
-        });
-    }
-
     nextDouble(): DoubleFloat | undefined {
         return this.skipAnnotations((reset) => {
             if (this.state.nextbyte() !== Tag.Ieee754) return reset();

implementations/javascript/packages/core/src/dictionary.ts

@@ -1,6 +1,6 @@
-import { Encoder, canonicalEncode, canonicalString } from "./encoder";
+import { Encoder, canonicalString } from "./encoder";
 import { Tag } from "./constants";
-import { FlexMap, FlexSet, _iterMap } from "./flex";
+import { FlexMap, FlexSet, _iterMap, IdentitySet } from "./flex";
 import { Value } from "./values";
 import { Bytes } from './bytes';
 import { GenericEmbedded } from "./embedded";
@@ -11,21 +11,51 @@ import { annotations, Annotated } from "./annotated";
 export type DictionaryType = 'Dictionary' | 'Set';
 export const DictionaryType = Symbol.for('DictionaryType');
 
-export class KeyedDictionary<K extends Value<T>, V, T = GenericEmbedded> extends FlexMap<K, V>
+export type CompoundKey<T> = Value<T> | (Preservable<T> & PreserveWritable<T>);
+
+export class EncodableDictionary<K, V, T = GenericEmbedded> extends FlexMap<K, V>
     implements Preservable<T>, PreserveWritable<T>
+{
+    constructor(
+        public readonly encodeK: (k: K) => CompoundKey<T>,
+        public readonly encodeV: (v: V) => CompoundKey<T>,
+        items?: readonly [K, V][] | Iterable<readonly [K, V]>
+    ) {
+        super((k: K) => canonicalString(encodeK(k)), items);
+    }
+
+    __preserve_on__(encoder: Encoder<T>) {
+        encodeDictionaryOn(this,
+                           encoder,
+                           (k, e) => e.push(this.encodeK(k)),
+                           (v, e) => e.push(this.encodeV(v)));
+    }
+
+    __preserve_text_on__(w: Writer<T>) {
+        writeDictionaryOn(this,
+                          w,
+                          (k, w) => w.push(this.encodeK(k)),
+                          (v, w) => w.push(this.encodeV(v)));
+    }
+}
+
+export class KeyedDictionary<K extends CompoundKey<T>, V, T = GenericEmbedded>
+    extends EncodableDictionary<K, V, T>
 {
     get [DictionaryType](): DictionaryType {
         return 'Dictionary';
     }
 
-    static isKeyedDictionary<K extends Value<T>, V, T = GenericEmbedded>(x: any): x is KeyedDictionary<K, V, T> {
+    static isKeyedDictionary<K extends CompoundKey<T>, V, T = GenericEmbedded>(
+        x: any,
+    ): x is KeyedDictionary<K, V, T> {
         return x?.[DictionaryType] === 'Dictionary';
     }
 
-    constructor(items?: readonly [K, V][]);
-    constructor(items?: Iterable<readonly [K, V]>);
-    constructor(items?: Iterable<readonly [K, V]>) {
-        super(canonicalString, items);
+    constructor(items?: readonly [K, V][] | Iterable<readonly [K, V]>) {
+        // The cast in encodeV is suuuuuuuper unsound, since V may not in fact be Encodable and Writable.
+        // Don't try to encode/write dictionaries holding non-encodable/non-writable values.
+        super(k => k, v => v as CompoundKey<T>, items);
     }
 
     mapEntries<W, S extends Value<R>, R = GenericEmbedded>(f: (entry: [K, V]) => [S, W]): KeyedDictionary<S, W, R> {
@@ -42,44 +72,6 @@ export class KeyedDictionary<K extends Value<T>, V, T = GenericEmbedded> extends
     }
 
     get [Symbol.toStringTag]() { return 'Dictionary'; }
-
-    __preserve_on__(encoder: Encoder<T>) {
-        if (encoder.canonical) {
-            const entries = Array.from(this);
-            const pieces = entries.map<[Bytes, number]>(([k, _v], i) => [canonicalEncode(k), i]);
-            pieces.sort((a, b) => Bytes.compare(a[0], b[0]));
-            encoder.state.emitbyte(Tag.Dictionary);
-            pieces.forEach(([_encodedKey, i]) => {
-                const [k, v] = entries[i];
-                encoder.push(k);
-                encoder.push(v as unknown as Value<T>); // Suuuuuuuper unsound
-            });
-            encoder.state.emitbyte(Tag.End);
-        } else {
-            encoder.state.emitbyte(Tag.Dictionary);
-            this.forEach((v, k) => {
-                encoder.push(k);
-                encoder.push(v as unknown as Value<T>); // Suuuuuuuper unsound
-            });
-            encoder.state.emitbyte(Tag.End);
-        }
-    }
-
-    __preserve_text_on__(w: Writer<T>) {
-        w.state.writeSeq('{', '}', this.entries(), ([k, v]) => {
-            w.push(k);
-            if (Annotated.isAnnotated<T>(v) && (annotations(v).length > 1) && w.state.isIndenting) {
-                w.state.pieces.push(':');
-                w.state.indentCount++;
-                w.state.writeIndent();
-                w.push(v);
-                w.state.indentCount--;
-            } else {
-                w.state.pieces.push(': ');
-                w.push(v as unknown as Value<T>); // Suuuuuuuper unsound
-            }
-        });
-    }
 }
 
 export class Dictionary<T = GenericEmbedded, V = Value<T>> extends KeyedDictionary<Value<T>, V, T> {
@@ -92,19 +84,91 @@ export class Dictionary<T = GenericEmbedded, V = Value<T>> extends KeyedDictiona
     }
 }
 
-export class KeyedSet<K extends Value<T>, T = GenericEmbedded> extends FlexSet<K>
+export function encodeDictionaryOn<K, V, T>(
+    dict: Map<K, V>,
+    encoder: Encoder<T>,
+    encodeK: (k: K, encoder: Encoder<T>) => void,
+    encodeV: (v: V, encoder: Encoder<T>) => void,
+) {
+    if (encoder.canonical) {
+        const entries = Array.from(dict);
+        const canonicalEncoder = new Encoder<T>({
+            canonical: true,
+            embeddedEncode: encoder.embeddedEncode,
+        });
+        const pieces = entries.map<[Bytes, number]>(([k, _v], i) => {
+            encodeK(k, canonicalEncoder);
+            return [canonicalEncoder.contents(), i];
+        });
+        pieces.sort((a, b) => Bytes.compare(a[0], b[0]));
+        encoder.grouped(Tag.Dictionary, () => pieces.forEach(([_encodedKey, i]) => {
+            const [k, v] = entries[i];
+            encodeK(k, encoder);
+            encodeV(v, encoder);
+        }));
+    } else {
+        encoder.grouped(Tag.Dictionary, () => dict.forEach((v, k) => {
+            encodeK(k, encoder);
+            encodeV(v, encoder);
+        }));
+    }
+}
+
+export function writeDictionaryOn<K, V, T>(
+    dict: Map<K, V>,
+    w: Writer<T>,
+    writeK: (k: K, w: Writer<T>) => void,
+    writeV: (v: V, w: Writer<T>) => void,
+) {
+    w.state.writeSeq('{', '}', dict.entries(), ([k, v]) => {
+        writeK(k, w);
+        if (Annotated.isAnnotated<T>(v) && (annotations(v).length > 1) && w.state.isIndenting) {
+            w.state.pieces.push(':');
+            w.state.indentCount++;
+            w.state.writeIndent();
+            writeV(v, w);
+            w.state.indentCount--;
+        } else {
+            w.state.pieces.push(': ');
+            writeV(v, w);
+        }
+    });
+}
+
+export class EncodableSet<V, T = GenericEmbedded> extends FlexSet<V>
     implements Preservable<T>, PreserveWritable<T>
+{
+    constructor(
+        public readonly encodeV: (v: V) => CompoundKey<T>,
+        items?: Iterable<V>,
+    ) {
+        super((v: V) => canonicalString(encodeV(v)), items);
+    }
+
+    __preserve_on__(encoder: Encoder<T>) {
+        encodeSetOn(this, encoder, (v, e) => e.push(this.encodeV(v)));
+    }
+
+    __preserve_text_on__(w: Writer<T>) {
+        writeSetOn(this, w, (v, w) => w.push(this.encodeV(v)));
+    }
+}
+
+export class KeyedSet<K extends CompoundKey<T>, T = GenericEmbedded>
+    extends EncodableSet<K, T>
 {
     get [DictionaryType](): DictionaryType {
         return 'Set';
     }
 
-    static isKeyedSet<K extends Value<T>, T = GenericEmbedded>(x: any): x is KeyedSet<K, T> {
+    static isKeyedSet<K extends CompoundKey<T>, T = GenericEmbedded>(
+        x: any,
+    ): x is KeyedSet<K, T> {
         return x?.[DictionaryType] === 'Set';
     }
 
     constructor(items?: Iterable<K>) {
-        super(canonicalString, items);
+        super(k => k, items);
     }
 
     map<S extends Value<R>, R = GenericEmbedded>(f: (value: K) => S): KeyedSet<S, R> {
@@ -122,20 +186,6 @@ export class KeyedSet<K extends Value<T>, T = GenericEmbedded> extends FlexSet<K
     }
 
     get [Symbol.toStringTag]() { return 'Set'; }
-
-    __preserve_on__(encoder: Encoder<T>) {
-        if (encoder.canonical) {
-            const pieces = Array.from(this).map<[Bytes, K]>(k => [canonicalEncode(k), k]);
-            pieces.sort((a, b) => Bytes.compare(a[0], b[0]));
-            encoder.encodevalues(Tag.Set, pieces.map(e => e[1]));
-        } else {
-            encoder.encodevalues(Tag.Set, this);
-        }
-    }
-
-    __preserve_text_on__(w: Writer<T>) {
-        w.state.writeSeq('#{', '}', this, vv => w.push(vv));
-    }
 }
 
 export class Set<T = GenericEmbedded> extends KeyedSet<Value<T>, T> {
@@ -147,3 +197,32 @@ export class Set<T = GenericEmbedded> extends KeyedSet<Value<T>, T> {
         return Set.isSet<T>(v) ? v : void 0;
     }
 }
+
+export function encodeSetOn<V, T>(
+    s: IdentitySet<V>,
+    encoder: Encoder<T>,
+    encodeV: (v: V, encoder: Encoder<T>) => void,
+) {
+    if (encoder.canonical) {
+        const canonicalEncoder = new Encoder<T>({
+            canonical: true,
+            embeddedEncode: encoder.embeddedEncode,
+        });
+        const pieces = Array.from(s).map<[Bytes, V]>(v => {
+            encodeV(v, canonicalEncoder);
+            return [canonicalEncoder.contents(), v];
+        });
+        pieces.sort((a, b) => Bytes.compare(a[0], b[0]));
+        encoder.grouped(Tag.Set, () => pieces.forEach(([_e, v]) => encodeV(v, encoder)));
+    } else {
+        encoder.grouped(Tag.Set, () => s.forEach(v => encodeV(v, encoder)));
+    }
+}
+
+export function writeSetOn<V, T>(
+    s: IdentitySet<V>,
+    w: Writer<T>,
+    writeV: (v: V, w: Writer<T>) => void,
+) {
+    w.state.writeSeq('#{', '}', s, vv => writeV(vv, w));
+}

implementations/javascript/packages/core/src/embedded.ts

@@ -26,7 +26,7 @@ export class Embedded<T> {
     }
 
     toString(): string {
-        return '#!' + (this.embeddedValue as any).toString();
+        return '#:' + (this.embeddedValue as any).toString();
     }
 
     __as_preserve__<R>(): T extends R ? Value<R> : never {

implementations/javascript/packages/core/src/encoder.ts

@@ -3,7 +3,7 @@ import { Bytes, unhexDigit } from "./bytes";
 import { Value } from "./values";
 import { EncodeError } from "./codec";
 import { Record, Tuple } from "./record";
-import { GenericEmbedded, EmbeddedTypeEncode } from "./embedded";
+import { EmbeddedTypeEncode } from "./embedded";
 import type { Embedded } from "./embedded";
 
 export type Encodable<T> =
@@ -242,9 +242,9 @@ export class Encoder<T = object> {
         return this.state.contentsString();
     }
 
-    encodevalues(tag: Tag, items: Iterable<Value<T>>) {
+    grouped(tag: Tag, f: () => void) {
         this.state.emitbyte(tag);
-        for (let i of items) { this.push(i); }
+        f();
         this.state.emitbyte(Tag.End);
     }
 
@@ -284,7 +284,9 @@ export class Encoder<T = object> {
             this.state.emitbyte(Tag.End);
         }
         else if (isIterable<Value<T>>(v)) {
-            this.encodevalues(Tag.Sequence, v);
+            this.grouped(Tag.Sequence, () => {
+                for (let i of v) this.push(i);
+            });
         }
         else {
             ((v: Embedded<T>) => {

implementations/javascript/packages/core/src/float.ts

@@ -1,12 +1,12 @@
 import { Tag } from "./constants";
-import { stringify } from "./text";
 import { Value } from "./values";
 import type { GenericEmbedded } from "./embedded";
 import type { Encoder, Preservable } from "./encoder";
 import type { Writer, PreserveWritable } from "./writer";
-import { Bytes, dataview, underlying } from "./bytes";
+import { Bytes, dataview } from "./bytes";
 
-export type FloatType = 'Single' | 'Double';
+// v Previously included 'Single'; may again in future. Also, 'Half', 'Quad'?
+export type FloatType = 'Double';
 export const FloatType = Symbol.for('FloatType');
 
 export abstract class Float {
@@ -16,8 +16,8 @@ export abstract class Float {
         this.value = typeof value === 'number' ? value : value.value;
     }
 
-    toString() {
-        return stringify(this);
+    __preserve_text_on__(w: Writer<any>) {
+        w.state.pieces.push(this.toString());
     }
 
     abstract toBytes(): Bytes;
@@ -38,7 +38,6 @@ export abstract class Float {
     abstract get [FloatType](): FloatType;
 
     static isFloat = (x: any): x is Float => x?.[FloatType] !== void 0;
-    static isSingle = (x: any): x is SingleFloat => x?.[FloatType] === 'Single';
     static isDouble = (x: any): x is DoubleFloat => x?.[FloatType] === 'Double';
 }
 
@@ -59,76 +58,39 @@ export function floatlikeString(f: number): string {
     return s + '.0';
 }
 
-export class SingleFloat extends Float implements Preservable<any>, PreserveWritable<any> {
-    __as_preserve__<T = GenericEmbedded>(): Value<T> {
-        return this;
-    }
-
-    static fromBytes(bs: Bytes | DataView): SingleFloat {
-        const view = dataview(bs);
-        const vf = view.getInt32(0, false);
-        if ((vf & 0x7f800000) === 0x7f800000) {
-            // NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
-            const sign = vf >> 31;
-            const payload = vf & 0x007fffff;
-            const dbs = new Bytes(8);
-            const dview = dataview(dbs);
-            dview.setInt16(0, (sign << 15) | 0x7ff0 | (payload >> 19), false);
-            dview.setInt32(2, (payload & 0x7ffff) << 13, false);
-            return new SingleFloat(dview.getFloat64(0, false));
-        } else {
-            return new SingleFloat(dataview(bs).getFloat32(0, false));
-        }
-    }
-
-    static __from_preserve__<T>(v: Value<T>): undefined | SingleFloat {
-        return Float.isSingle(v) ? v : void 0;
-    }
-
-    __w(v: DataView, offset: number) {
-        if (Number.isNaN(this.value)) {
-            const dbs = new Bytes(8);
-            const dview = dataview(dbs);
-            dview.setFloat64(0, this.value, false);
-            const sign = dview.getInt8(0) >> 7;
-            const payload = (dview.getInt32(1, false) >> 5) & 0x007fffff;
-            const vf = (sign << 31) | 0x7f800000 | payload;
-            v.setInt32(offset, vf, false);
-        } else {
-            v.setFloat32(offset, this.value, false);
-        }
-    }
-
-    __preserve_on__(encoder: Encoder<any>) {
-        encoder.state.emitbyte(Tag.Ieee754);
-        encoder.state.emitbyte(4);
-        encoder.state.makeroom(4);
-        this.__w(encoder.state.view, encoder.state.index);
-        encoder.state.index += 4;
-    }
-
-    toBytes(): Bytes {
-        const bs = new Bytes(4);
-        this.__w(bs.dataview(), 0);
-        return bs;
-    }
-
-    __preserve_text_on__(w: Writer<any>) {
-        if (Number.isFinite(this.value)) {
-            w.state.pieces.push(floatlikeString(this.value) + 'f');
-        } else {
-            w.state.pieces.push('#xf"', this.toBytes().toHex(), '"');
-        }
-    }
-
-    get [FloatType](): 'Single' {
-        return 'Single';
-    }
-}
-
-export function Single(value: number | Float): SingleFloat {
-    return new SingleFloat(value);
-}
+// -- These snippets are useful to keep in mind for promoting 4-byte, single-precision floats
+// -- to 8-byte, double-precision floats *while preserving NaN bit-patterns*:
+//
+// static fromBytes(bs: Bytes | DataView): SingleFloat {
+//     const view = dataview(bs);
+//     const vf = view.getInt32(0, false);
+//     if ((vf & 0x7f800000) === 0x7f800000) {
+//         // NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
+//         const sign = vf >> 31;
+//         const payload = vf & 0x007fffff;
+//         const dbs = new Bytes(8);
+//         const dview = dataview(dbs);
+//         dview.setInt16(0, (sign << 15) | 0x7ff0 | (payload >> 19), false);
+//         dview.setInt32(2, (payload & 0x7ffff) << 13, false);
+//         return new SingleFloat(dview.getFloat64(0, false));
+//     } else {
+//         return new SingleFloat(dataview(bs).getFloat32(0, false));
+//     }
+// }
+//
+// __w(v: DataView, offset: number) {
+//     if (Number.isNaN(this.value)) {
+//         const dbs = new Bytes(8);
+//         const dview = dataview(dbs);
+//         dview.setFloat64(0, this.value, false);
+//         const sign = dview.getInt8(0) >> 7;
+//         const payload = (dview.getInt32(1, false) >> 5) & 0x007fffff;
+//         const vf = (sign << 31) | 0x7f800000 | payload;
+//         v.setInt32(offset, vf, false);
+//     } else {
+//         v.setFloat32(offset, this.value, false);
+//     }
+// }
 
 export class DoubleFloat extends Float implements Preservable<any>, PreserveWritable<any> {
     __as_preserve__<T = GenericEmbedded>(): Value<T> {
@@ -157,11 +119,11 @@ export class DoubleFloat extends Float implements Preservable<any>, PreserveWrit
         return bs;
     }
 
-    __preserve_text_on__(w: Writer<any>) {
+    toString(): string {
         if (Number.isFinite(this.value)) {
-            w.state.pieces.push(floatlikeString(this.value));
+            return floatlikeString(this.value);
         } else {
-            w.state.pieces.push('#xd"', this.toBytes().toHex(), '"');
+            return '#xd"' + this.toBytes().toHex() + '"';
         }
     }
 

implementations/javascript/packages/core/src/fold.ts

@@ -3,12 +3,11 @@ import { Bytes } from "./bytes";
 import { Value } from "./values";
 import { Set, Dictionary } from "./dictionary";
 import { annotate, Annotated } from "./annotated";
-import { Double, Float, Single } from "./float";
+import { Double, Float } from "./float";
 import { Embedded } from "./embedded";
 
 export enum ValueClass {
     Boolean,
-    Float,
     Double,
     SignedInteger,
     String,
@@ -26,7 +25,6 @@ export type Fold<T, R = Value<T>> = (v: Value<T>) => R;
 
 export interface FoldMethods<T, R> {
     boolean(b: boolean): R;
-    single(f: number): R;
     double(f: number): R;
     integer(i: number | bigint): R;
     string(s: string): R;
@@ -45,7 +43,6 @@ export interface FoldMethods<T, R> {
 
 export class VoidFold<T> implements FoldMethods<T, void> {
     boolean(b: boolean): void {}
-    single(f: number): void {}
     double(f: number): void {}
     integer(i: number | bigint): void {}
     string(s: string): void {}
@@ -73,9 +70,6 @@ export abstract class ValueFold<T, R = T> implements FoldMethods<T, Value<R>> {
     boolean(b: boolean): Value<R> {
         return b;
     }
-    single(f: number): Value<R> {
-        return Single(f);
-    }
     double(f: number): Value<R> {
         return Double(f);
     }
@@ -134,7 +128,7 @@ export function valueClass<T>(v: Value<T>): ValueClass {
             return ValueClass.Boolean;
         case 'number':
             if (!Number.isInteger(v)) {
-                throw new Error("Non-integer number in Preserves valueClass; missing SingleFloat/DoubleFloat wrapper?");
+                throw new Error("Non-integer number in Preserves valueClass; missing Float wrapper?");
             } else {
                 return ValueClass.SignedInteger;
             }
@@ -157,8 +151,6 @@ export function valueClass<T>(v: Value<T>): ValueClass {
                 return ValueClass.Annotated;
             } else if (Bytes.isBytes(v)) {
                 return ValueClass.ByteString;
-            } else if (Float.isSingle(v)) {
-                return ValueClass.Float;
             } else if (Float.isDouble(v)) {
                 return ValueClass.Double;
             } else {
@@ -202,8 +194,6 @@ export function fold<T, R>(v: Value<T>, o: FoldMethods<T, R>): R {
                     return o.annotated(v, walk);
                 } else if (Bytes.isBytes(v)) {
                     return o.bytes(v);
-                } else if (Float.isSingle(v)) {
-                    return o.single(v.value);
                 } else if (Float.isDouble(v)) {
                     return o.double(v.value);
                 } else {

implementations/javascript/packages/core/src/fromjs.ts

@@ -9,7 +9,7 @@ export function fromJS<T = GenericEmbedded>(x: any): Value<T> {
         case 'number':
             if (!Number.isInteger(x)) {
                 // We require that clients be explicit about integer vs. non-integer types.
-                throw new TypeError("Refusing to autoconvert non-integer number to Single or Double");
+                throw new TypeError("Refusing to autoconvert non-integer number to Double");
             }
             // FALL THROUGH
         case 'bigint':

implementations/javascript/packages/core/src/merge.ts

@@ -32,7 +32,6 @@ export function merge<T>(
         }
         return fold<T, Value<T>>(a, {
             boolean: die,
-            single(_f: number) { return is(a, b) ? a : die(); },
             double(_f: number) { return is(a, b) ? a : die(); },
             integer: die,
             string: die,

implementations/javascript/packages/core/src/order.ts

@@ -0,0 +1,103 @@
+import { is, isAnnotated } from './is';
+import { Bytes } from './bytes';
+import { Set, Dictionary } from './dictionary';
+import { Embedded, isEmbedded } from './embedded';
+import { Float } from './float';
+import { Value } from './values';
+import { Record } from './record';
+
+export function typeCode<V>(v: Value<V>): number {
+    if (isAnnotated<V>(v)) v = v.item;
+    switch (typeof v) {
+        case 'boolean':
+            return 0;
+        case 'number':
+        case 'bigint':
+            return 3;
+        case 'string':
+            return 4;
+        case 'symbol':
+            return 6;
+        case 'object':
+            if (Float.isFloat(v)) return 2; // 1 was for single-precision floats
+            if (Bytes.isBytes(v)) return 5;
+            if (Array.isArray(v)) {
+                return ('label' in v) ? 7 : 8;
+            }
+            if (Set.isSet<V>(v)) return 9;
+            if (Dictionary.isDictionary<V>(v)) return 10;
+            if (isEmbedded(v)) return 11;
+            /* fall through */
+        default:
+            throw new Error("Invalid Value<V> in typeCode");
+    }
+}
+
+export function compare<V>(
+    a: Value<V>,
+    b: Value<V>,
+    compare_embedded: (a: V, b: V) => number = (a, b) => is(a, b) ? 0 : a < b ? -1 : 1,
+): number {
+    function cmp(a: Value<V>, b: Value<V>): number {
+        if (isAnnotated<V>(a)) a = a.item;
+        if (isAnnotated<V>(b)) b = b.item;
+        const ta = typeCode(a);
+        const tb = typeCode(b);
+        if (ta < tb) return -1;
+        if (ta > tb) return 1;
+        switch (ta) {
+            case 0:
+            case 3:
+            case 4: {
+                const va = a as any;
+                const vb = b as any;
+                return va < vb ? -1 : va > vb ? 1 : 0;
+            }
+            // case 1: // was single-precision
+            case 2: {
+                const va = (a as Float).value;
+                const vb = (b as Float).value;
+                return va < vb ? -1 : va > vb ? 1 : 0;
+            }
+            case 5:
+                return Bytes.compare(a as Bytes, b as Bytes);
+            case 6: {
+                const va = (a as symbol).description!;
+                const vb = (b as symbol).description!;
+                return va < vb ? -1 : va > vb ? 1 : 0;
+            }
+            case 7: {
+                const lr = cmp((a as Record<Value<V>, Value<V>[], V>).label,
+                               (b as Record<Value<V>, Value<V>[], V>).label);
+                if (lr !== 0) return lr;
+                /* fall through */
+            }
+            case 8: {
+                const va = a as Value<V>[];
+                const vb = b as Value<V>[];
+                const l = Math.min(va.length, vb.length)
+                for (let i = 0; i < l; i++) {
+                    const c = cmp(va[i], vb[i]);
+                    if (c !== 0) return c;
+                }
+                return va.length < vb.length ? -1 : va.length > vb.length ? 1 : 0;
+            }
+            case 9: {
+                const va = Array.from(a as Set<V>).sort(cmp);
+                const vb = Array.from(b as Set<V>).sort(cmp);
+                return cmp(va, vb);
+            }
+            case 10: {
+                const va = Array.from(a as Dictionary<V>).sort(cmp);
+                const vb = Array.from(b as Dictionary<V>).sort(cmp);
+                return cmp(va, vb);
+            }
+            case 11:
+                return compare_embedded((a as Embedded<V>).embeddedValue,
+                                        (b as Embedded<V>).embeddedValue);
+            default:
+                throw new Error("Invalid typeCode: " + ta);
+        }
+    }
+    return cmp(a, b);
+}

implementations/javascript/packages/core/src/reader.ts

@@ -8,7 +8,7 @@ import { Bytes, unhexDigit } from './bytes';
 import { Decoder, DecoderState, neverEmbeddedTypeDecode } from './decoder';
 import { Record } from './record';
 import { Annotated, newPosition, Position, updatePosition } from './annotated';
-import { Double, DoubleFloat, FloatType, Single, SingleFloat } from './float';
+import { Double, DoubleFloat, FloatType } from './float';
 import { stringify } from './text';
 import { embed, GenericEmbedded, EmbeddedTypeDecode } from './embedded';
 
@@ -24,12 +24,10 @@ export interface ReaderOptions<T> extends ReaderStateOptions {
 const MAX_SAFE_INTEGERn = BigInt(Number.MAX_SAFE_INTEGER);
 const MIN_SAFE_INTEGERn = BigInt(Number.MIN_SAFE_INTEGER);
 
-export const NUMBER_RE: RegExp = /^([-+]?\d+)(((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))([fF]?))?$/;
+export const NUMBER_RE: RegExp = /^([-+]?\d+)((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))?$/;
 // Groups:
 //  1 - integer part and sign
-//  2 - decimal part, exponent and Float marker
-//  3 - decimal part and exponent
-//  7 - Float marker
+//  2 - decimal part and exponent
 
 export class ReaderState {
     buffer: string;
@@ -131,20 +129,14 @@ export class ReaderState {
         }
     }
 
-    readHexFloat(precision: FloatType): SingleFloat | DoubleFloat {
+    readHexFloat(): DoubleFloat {
         const pos = this.copyPos();
         if (this.nextchar() !== '"') {
             this.error("Missing open-double-quote in hex-encoded floating-point number", pos);
         }
         const bs = this.readHexBinary();
-        switch (precision) {
-            case 'Single':
-                if (bs.length !== 4) this.error("Incorrect number of bytes in hex-encoded Float", pos);
-                return SingleFloat.fromBytes(bs);
-            case 'Double':
-                if (bs.length !== 8) this.error("Incorrect number of bytes in hex-encoded Double", pos);
-                return DoubleFloat.fromBytes(bs);
-        }
+        if (bs.length !== 8) this.error("Incorrect number of bytes in hex-encoded Double", pos);
+        return DoubleFloat.fromBytes(bs);
     }
 
     readBase64Binary(): Bytes {
@@ -155,7 +147,7 @@ export class ReaderState {
             if (c === ']') break;
             acc = acc + c;
         }
-        return decodeBase64(acc);
+        return Bytes.fromBase64(acc);
     }
 
     requireDelimiter(prefix: string): void {
@@ -180,10 +172,8 @@ export class ReaderState {
                 } else {
                     return Number(v);
                 }
-            } else if (m[7] === '') {
-                return Double(parseFloat(m[1] + m[3]));
             } else {
-                return Single(parseFloat(m[1] + m[3]));
+                return Double(parseFloat(acc));
             }
         } else {
             return Symbol.for(acc);
@@ -360,12 +350,11 @@ export class Reader<T> {
                         case '"': return this.state.readLiteralBinary();
                         case 'x': switch (this.state.nextchar()) {
                             case '"': return this.state.readHexBinary();
-                            case 'f': return this.state.readHexFloat('Single');
-                            case 'd': return this.state.readHexFloat('Double');
+                            case 'd': return this.state.readHexFloat();
                             default: this.state.error('Invalid #x syntax', startPos);
                         }
                         case '[': return this.state.readBase64Binary();
-                        case '!': return embed(this.embeddedType.fromValue(
+                        case ':': return embed(this.embeddedType.fromValue(
                             new Reader<GenericEmbedded>(this.state, genericEmbeddedTypeDecode).next(),
                             this.state.options));
                         default:
@@ -436,31 +425,6 @@ export class Reader<T> {
     }
 }
 
-const BASE64: {[key: string]: number} = {};
-[... 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789'].forEach(
-    (c, i) => BASE64[c] = i);
-BASE64['+'] = BASE64['-'] = 62;
-BASE64['/'] = BASE64['_'] = 63;
-
-export function decodeBase64(s: string): Bytes {
-    const bs = new Uint8Array(Math.floor(s.length * 3/4));
-    let i = 0;
-    let j = 0;
-    while (i < s.length) {
-        const v1 = BASE64[s[i++]];
-        const v2 = BASE64[s[i++]];
-        const v3 = BASE64[s[i++]];
-        const v4 = BASE64[s[i++]];
-        const v = (v1 << 18) | (v2 << 12) | (v3 << 6) | v4;
-        bs[j++] = (v >> 16) & 255;
-        if (v3 === void 0) break;
-        bs[j++] = (v >> 8) & 255;
-        if (v4 === void 0) break;
-        bs[j++] = v & 255;
-    }
-    return Bytes.from(bs.subarray(0, j));
-}
-
 function isSpace(s: string): boolean {
     return ' \t\n\r'.indexOf(s) !== -1;
 }

implementations/javascript/packages/core/src/runtime.ts

@@ -1,4 +1,5 @@
 export * from './annotated';
+export * from './base64';
 export * from './bytes';
 export * from './codec';
 export * from './compound';
@@ -13,6 +14,7 @@ export * from './fold';
 export * from './fromjs';
 export * from './is';
 export * from './merge';
+export * from './order';
 export * from './reader';
 export * from './record';
 export * from './strip';

implementations/javascript/packages/core/src/text.ts

@@ -43,7 +43,7 @@ export function stringify<T = GenericEmbedded>(x: any, options?: WriterOptions<T
     return Writer.stringify(fromJS<T>(x), options);
 }
 
-export function preserves<T>(pieces: TemplateStringsArray, ...values: any[]): string {
+export function preserves(pieces: TemplateStringsArray, ...values: any[]): string {
     const result = [pieces[0]];
     values.forEach((v, i) => {
         result.push(stringify(v));

implementations/javascript/packages/core/src/values.ts

@@ -1,7 +1,7 @@
 // Preserves Values.
 
 import type { Bytes } from './bytes';
-import type { DoubleFloat, SingleFloat } from './float';
+import type { DoubleFloat } from './float';
 import type { Annotated } from './annotated';
 import type { Set, Dictionary } from './dictionary';
 import type { Embedded, GenericEmbedded } from './embedded';
@@ -13,7 +13,6 @@ export type Value<T = GenericEmbedded> =
     | Annotated<T>;
 export type Atom =
     | boolean
-    | SingleFloat
     | DoubleFloat
     | number | bigint
     | string

implementations/javascript/packages/core/src/writer.ts

@@ -4,6 +4,7 @@ import type { GenericEmbedded, Embedded, EmbeddedTypeEncode } from "./embedded";
 import { Encoder, EncoderState } from "./encoder";
 import type { Value } from "./values";
 import { NUMBER_RE } from './reader';
+import { encodeBase64 } from './base64';
 
 export type Writable<T> =
     Value<T> | PreserveWritable<T> | Iterable<Value<T>> | ArrayBufferView;
@@ -205,28 +206,6 @@ export class WriterState {
     }
 }
 
-const BASE64 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'
-
-export function encodeBase64(bs: Uint8Array): string {
-    let s = '';
-    let buffer = 0;
-    let bitcount = 0;
-    for (let b of bs) {
-        buffer = ((buffer & 0x3f) << 8) | b;
-        bitcount += 8;
-        while (bitcount >= 6) {
-            bitcount -= 6;
-            const v = (buffer >> bitcount) & 0x3f;
-            s = s + BASE64[v];
-        }
-    }
-    if (bitcount > 0) {
-        const v = (buffer << (6 - bitcount)) & 0x3f;
-        s = s + BASE64[v];
-    }
-    return s;
-}
-
 export class Writer<T> {
     state: WriterState;
     embeddedWrite: EmbeddedWriter<T>;
@@ -318,7 +297,7 @@ export class Writer<T> {
                 }
                 else {
                     ((v: Embedded<T>) => {
-                        this.state.pieces.push('#!');
+                        this.state.pieces.push('#:');
                         if ('write' in this.embeddedWrite) {
                             this.embeddedWrite.write(this.state, v.embeddedValue);
                         } else {

implementations/javascript/packages/core/test/bytes.test.ts

@@ -82,37 +82,39 @@ describe('immutable byte arrays', () => {
 });
 
 describe('base64 decoder', () => {
-    describe('RFC4648 tests', () => {
-        it('10.0', () => expect(decodeBase64("")).is(Bytes.of()));
-        it('10.1', () => expect(decodeBase64("Zg==")).is(Bytes.of(102)));
-        it('10.2', () => expect(decodeBase64("Zm8=")).is(Bytes.of(102, 111)));
-        it('10.3', () => expect(decodeBase64("Zm9v")).is(Bytes.of(102, 111, 111)));
-        it('10.4', () => expect(decodeBase64("Zm9vYg==")).is(Bytes.of(102, 111, 111, 98)));
-        it('10.5', () => expect(decodeBase64("Zm9vYmE=")).is(Bytes.of(102, 111, 111, 98, 97)));
-        it('10.6', () => expect(decodeBase64("Zm9vYmFy")).is(Bytes.of(102, 111, 111, 98, 97, 114)));
+    const d64 = (s: string) => Bytes.from(decodeBase64(s));
 
-        it('10.1b', () => expect(decodeBase64("Zg")).is(Bytes.of(102)));
-        it('10.2b', () => expect(decodeBase64("Zm8")).is(Bytes.of(102, 111)));
-        it('10.4b', () => expect(decodeBase64("Zm9vYg")).is(Bytes.of(102, 111, 111, 98)));
-        it('10.5b', () => expect(decodeBase64("Zm9vYmE")).is(Bytes.of(102, 111, 111, 98, 97)));
+    describe('RFC4648 tests', () => {
+        it('10.0', () => expect(d64("")).is(Bytes.of()));
+        it('10.1', () => expect(d64("Zg==")).is(Bytes.of(102)));
+        it('10.2', () => expect(d64("Zm8=")).is(Bytes.of(102, 111)));
+        it('10.3', () => expect(d64("Zm9v")).is(Bytes.of(102, 111, 111)));
+        it('10.4', () => expect(d64("Zm9vYg==")).is(Bytes.of(102, 111, 111, 98)));
+        it('10.5', () => expect(d64("Zm9vYmE=")).is(Bytes.of(102, 111, 111, 98, 97)));
+        it('10.6', () => expect(d64("Zm9vYmFy")).is(Bytes.of(102, 111, 111, 98, 97, 114)));
+
+        it('10.1b', () => expect(d64("Zg")).is(Bytes.of(102)));
+        it('10.2b', () => expect(d64("Zm8")).is(Bytes.of(102, 111)));
+        it('10.4b', () => expect(d64("Zm9vYg")).is(Bytes.of(102, 111, 111, 98)));
+        it('10.5b', () => expect(d64("Zm9vYmE")).is(Bytes.of(102, 111, 111, 98, 97)));
     });
 
     describe('RFC4648 examples', () => {
         it('example0', () =>
-            expect(decodeBase64('FPucA9l+')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03, 0xd9, 0x7e)));
+            expect(d64('FPucA9l+')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03, 0xd9, 0x7e)));
         it('example1', () =>
-            expect(decodeBase64('FPucA9k=')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03, 0xd9)));
+            expect(d64('FPucA9k=')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03, 0xd9)));
         it('example1b', () =>
-            expect(decodeBase64('FPucA9k')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03, 0xd9)));
+            expect(d64('FPucA9k')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03, 0xd9)));
         it('example2', () =>
-            expect(decodeBase64('FPucAw==')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03)));
+            expect(d64('FPucAw==')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03)));
         it('example2b', () =>
-            expect(decodeBase64('FPucAw=')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03)));
+            expect(d64('FPucAw=')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03)));
         it('example2c', () =>
-            expect(decodeBase64('FPucAw')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03)));
+            expect(d64('FPucAw')).is(Bytes.of(0x14, 0xfb, 0x9c, 0x03)));
     });
 
     describe('Misc test cases', () => {
-        it('gQ==', () => expect(decodeBase64('gQ==')).is(Bytes.of(0x81)));
+        it('gQ==', () => expect(d64('gQ==')).is(Bytes.of(0x81)));
     });
 });

implementations/javascript/packages/core/test/values.test.ts

@@ -1,12 +1,6 @@
-import { Single, Double, fromJS, Dictionary, IDENTITY_FOLD, fold, mapEmbeddeds, Value, embed, preserves } from '../src/index';
+import { Double, fromJS, Dictionary, IDENTITY_FOLD, fold, mapEmbeddeds, Value, embed, preserves } from '../src/index';
 import './test-utils';
 
-describe('Single', () => {
-    it('should print reasonably', () => {
-        expect(Single(123.45).toString()).toEqual("123.45f");
-  });
-});
-
 describe('Double', () => {
     it('should print reasonably', () => {
         expect(Double(123.45).toString()).toEqual("123.45");
@@ -21,7 +15,7 @@ describe('fold', () => {
             new Dictionary([[[3, 4], fromJS([5, 6])],
                             ['a', 1],
                             ['b', true]]),
-            Single(3.4),
+            Double(3.4),
             t,
         ]);
     }

implementations/javascript/packages/schema-cli/bin/preserves-schema-ts.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-require('../dist/bin/preserves-schema-ts.js').main(process.argv.slice(2));
+require('../lib/bin/preserves-schema-ts.js').main(process.argv.slice(2));

implementations/javascript/packages/schema-cli/bin/preserves-schemac.js

@@ -1,2 +1,2 @@
 #!/usr/bin/env node
-require('../dist/bin/preserves-schemac.js').main(process.argv.slice(2));
+require('../lib/bin/preserves-schemac.js').main(process.argv.slice(2));

implementations/javascript/packages/schema-cli/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@preserves/schema-cli",
-  "version": "0.991.0",
+  "version": "0.994.0",
   "description": "Command-line tools for Preserves Schema",
   "homepage": "https://gitlab.com/preserves/preserves",
   "license": "Apache-2.0",
@@ -11,11 +11,9 @@
   "author": "Tony Garnock-Jones <tonyg@leastfixedpoint.com>",
   "scripts": {
     "clean": "rm -rf lib dist",
-    "prepare": "yarn compile && yarn rollup",
+    "prepare": "yarn compile",
     "compile": "tsc",
     "compile:watch": "yarn compile -w",
-    "rollup": "rollup -c",
-    "rollup:watch": "yarn rollup -w",
     "test": "true",
     "veryclean": "yarn run clean && rm -rf node_modules"
   },
@@ -28,8 +26,8 @@
     "@types/minimatch": "^3.0"
   },
   "dependencies": {
-    "@preserves/core": "^0.991.0",
-    "@preserves/schema": "^0.991.0",
+    "@preserves/core": "^0.994.0",
+    "@preserves/schema": "^0.994.0",
     "chalk": "^4.1",
     "chokidar": "^3.5",
     "commander": "^7.2",

implementations/javascript/packages/schema-cli/rollup.config.mjs

@@ -1,17 +0,0 @@
-import terser from '@rollup/plugin-terser';
-
-function cli(name) {
-  return {
-    input: `lib/bin/${name}.js`,
-    output: [{file: `dist/bin/${name}.js`, format: 'commonjs'}],
-    external: [
-      '@preserves/core',
-      '@preserves/schema',
-    ],
-  };
-}
-
-export default [
-  cli('preserves-schema-ts'),
-  cli('preserves-schemac'),
-];

implementations/javascript/packages/schema-cli/src/bin/preserves-schemac.ts

@@ -24,10 +24,9 @@ export function run(options: CommandLineArguments): void {
 
     if (failures.length === 0) {
         if (options.bundle) {
-            fs.writeSync(1, underlying(canonicalEncode(M.fromBundle({
-                modules: new KeyedDictionary<M.ModulePath, M.Schema, M.InputEmbedded>(
-                    inputFiles.map(i => [i.modulePath, i.schema])),
-            }))));
+            fs.writeSync(1, underlying(canonicalEncode(M.fromBundle(M.Bundle(
+                new KeyedDictionary<M.ModulePath, M.Schema, M.InputEmbedded>(
+                    inputFiles.map(i => [i.modulePath, i.schema])))))));
         } else {
             fs.writeSync(1, underlying(canonicalEncode(M.fromSchema(inputFiles[0].schema))));
         }

implementations/javascript/packages/schema-cli/tsconfig.json

@@ -9,6 +9,7 @@
     "declarationDir": "./lib",
     "esModuleInterop": true,
     "moduleResolution": "node",
+    "module": "commonjs",
     "sourceMap": true,
     "strict": true
   },

implementations/javascript/packages/schema/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@preserves/schema",
-  "version": "0.991.0",
+  "version": "0.994.0",
   "description": "Schema support for Preserves data serialization format",
   "homepage": "https://gitlab.com/preserves/preserves",
   "license": "Apache-2.0",
@@ -13,19 +13,19 @@
   "types": "lib/index.d.ts",
   "author": "Tony Garnock-Jones <tonyg@leastfixedpoint.com>",
   "scripts": {
-    "regenerate": "rm -rf ./src/gen && yarn copy-schema && ../schema-cli/bin/preserves-schema-ts.js --output ./src/gen ./dist:schema.prs",
+    "regenerate": "rm -rf ./src/gen && yarn copy-schema && ../schema-cli/bin/preserves-schema-ts.js --output ./src/gen ./dist:*.prs",
     "clean": "rm -rf lib dist",
-    "prepare": "yarn compile && yarn rollup && yarn copy-schema",
+    "prepare": "yarn compile && yarn rollup && yarn copy-schema && cp preserves-schema-browser.js dist/",
     "compile": "tsc",
     "compile:watch": "yarn compile -w",
     "rollup": "rollup -c",
     "rollup:watch": "yarn rollup -w",
-    "copy-schema": "mkdir -p ./dist && cp -a ../../../../schema/schema.prs ./dist",
+    "copy-schema": "mkdir -p ./dist && cp -a ../../../../schema/schema.prs ../../../../schema/host.prs ./dist",
     "test": "jest",
     "test:watch": "jest --watch",
     "veryclean": "yarn run clean && rm -rf node_modules"
   },
   "dependencies": {
-    "@preserves/core": "^0.991.0"
+    "@preserves/core": "^0.994.0"
   }
 }

implementations/javascript/packages/schema/preserves-schema-browser.js

@@ -0,0 +1,100 @@
+(() => {
+  const I = new PreservesSchema.SchemaInterpreter();
+
+  globalThis.Schema = { __interpreter: I };
+  let schemaReady;
+  globalThis.SchemaReady = new Promise(res => schemaReady = res);
+
+  async function translateScripts() {
+
+    const schemaScripts =
+          Array.from(document.getElementsByTagName('script'))
+          .filter(s => (s.type === 'text/preserves+schema' ||
+                        s.type === 'schema'));
+
+    for (const script of schemaScripts) {
+      function complain(message, detail) {
+        const e = new Error(message);
+        e.script = script;
+        e.detail = detail;
+        console.error(e);
+      }
+
+      let sourceCodeBlob;
+      const sourceUrl = script.src || script.getAttribute('data-src') || false;
+      if (sourceUrl) {
+        const res = await fetch(sourceUrl);
+        if (res.ok) {
+          sourceCodeBlob = new Uint8Array(await res.arrayBuffer());
+        } else {
+          complain(`Failed to retrieve schema from ${sourceUrl}`, { res });
+          continue;
+        }
+      } else {
+        sourceCodeBlob = new TextEncoder().encode(script.innerHTML);
+      }
+
+      const schemaName = () => {
+        const n = script.getAttribute('name');
+        if (n === null) complain(`<script type="schema"> must have name attribute`);
+        return n;
+      };
+
+      if (sourceCodeBlob[0] >= 128) {
+        // Binary Preserves blob
+        const value = Preserves.decode(sourceCodeBlob);
+        const bundle = PreservesSchema.Meta.toBundle(value);
+        if (bundle !== void 0) {
+          const prefixStr = script.getAttribute('prefix');
+          const bundlePrefix = (prefixStr ? prefixStr.split('.') : []).map(Symbol.for);
+          bundle.modules.forEach((schema, path) => {
+            const modulePath = [... bundlePrefix, ... path];
+            I.env.set(modulePath, schema);
+          });
+        } else {
+          const schema = PreservesSchema.Meta.toSchema(value);
+          if (schema !== void 0) {
+            const modulePath = schemaName().split('.').map(Symbol.for);
+            I.env.set(modulePath, schema);
+          }
+        }
+      } else {
+        // Presumably text
+        const sourceCode = new TextDecoder('utf-8', { fatal: true }).decode(sourceCodeBlob);
+        const name = schemaName();
+        const schema = PreservesSchema.readSchema(sourceCode, { name });
+        const modulePath = name.split('.').map(Symbol.for);
+        I.env.set(modulePath, schema);
+      }
+    }
+
+    // I.env.forEach((_schema, path) => console.log(PreservesSchema.Meta.formatModulePath(path)));
+
+    for (const [modulePath, schema] of I.env) {
+      let container = Schema;
+      modulePath.slice(0, -1).forEach(n => {
+        if (!(n.description in container)) container[n.description] = {};
+        container = container[n.description];
+      });
+      container[modulePath[modulePath.length - 1].description] = moduleFor(modulePath, schema);
+    }
+
+    schemaReady();
+  }
+
+  function moduleFor(modulePath, schema) {
+    const { Meta, GenType, Type } = PreservesSchema;
+    const mod = {};
+    schema.definitions.forEach((d, n) => {
+      const definitionName = n.description;
+      const definitionId = Meta.jsId(definitionName);
+      mod[`${definitionId}`] = I.definitionConstructor(modulePath, n);
+      mod[`from${definitionId}`] = I.unparser(modulePath, n);
+      mod[`to${definitionId}`] = v => I.tryParse(modulePath, n, v);
+      mod[`as${definitionId}`] = v => I.parse(modulePath, n, v);
+    });
+    return mod;
+  }
+
+  window.addEventListener('DOMContentLoaded', translateScripts);
+})();

implementations/javascript/packages/schema/src/compiler.ts

@@ -1,10 +1,10 @@
 import { encode, stringify } from "@preserves/core";
 import * as M from "./meta";
 import { CompilerOptions, ModuleContext } from "./compiler/context";
-import { Formatter, block, seq, braces, opseq } from "./compiler/block";
+import { Formatter, block, seq, braces } from "./compiler/block";
 import { typeForDefinition } from "./compiler/gentype";
 import { converterForDefinition } from "./compiler/genconverter";
-import { renderType } from "./compiler/rendertype";
+import { renderTypeWithConversionMixins } from "./compiler/rendertype";
 import { genConstructor } from "./compiler/genctor";
 import { unconverterForDefinition } from "./compiler/genunconverter";
 import { sourceCodeFor } from "./compiler/value";
@@ -32,10 +32,10 @@ export function compile(
     for (const [name, def] of schema.definitions) {
         const t = typeForDefinition(mod.resolver(), def);
         const nameStr = stringify(name);
-        const resultTypeItem = mod.withAsPreserveMixinType(nameStr, t);
+        const resultTypeItem = seq(nameStr, mod.genericArgsFor(t));
 
         mod.defineType(seq(`export type ${nameStr}`, mod.genericParametersFor(t),
-                           ` = `, renderType(mod, t), `;`));
+                           ` = `, renderTypeWithConversionMixins(mod, t), `;`));
 
         if (t.kind === 'union') {
             mod.defineFunctions(nameStr, _ctx =>
@@ -53,7 +53,7 @@ export function compile(
         const t = typeForDefinition(mod.resolver(), def);
         const name = name0 as symbol;
         const nameStr = name0.description!;
-        const resultTypeItem = mod.withAsPreserveMixinType(nameStr, t);
+        const resultTypeItem = seq(nameStr, mod.genericArgsFor(t));
 
         mod.defineFunctions(nameStr, ctx =>
             [seq(`export function as${name.description!}`, mod.genericParameters(),

implementations/javascript/packages/schema/src/compiler/context.ts

@@ -1,7 +1,7 @@
 import { Dictionary, KeyedSet, FlexSet, Position, stringify } from "@preserves/core";
 import { refPosition } from "../reader";
 import * as M from "../meta";
-import { anglebrackets, block, braces, commas, formatItems, Item, keyvalue, seq, opseq } from "./block";
+import { anglebrackets, block, braces, commas, formatItems, Item, keyvalue, seq } from "./block";
 import { ANY_TYPE, RefType, Type } from "./type";
 import { renderType, variantInitFor } from "./rendertype";
 import { typeForDefinition } from "./gentype";
@@ -97,7 +97,7 @@ export class ModuleContext {
         return (ref) => this.lookup(
             ref,
             (_p, _t) => Type.ref(ref.name.description!, ref),
-            (modPath, modId, modFile, modExpr, _p, t) => {
+            (modPath, modId, modFile, modExpr, _p, _t) => {
                 this.imports.add([modPath, modId, modFile, modExpr]);
                 return Type.ref(`${modId}${modExpr}.${ref.name.description!}`, ref);
             },
@@ -209,19 +209,6 @@ export class ModuleContext {
 
         return walk(t);
     }
-
-    withAsPreserveMixinType(name: string, t: Type): Item {
-        if (t.kind === 'unit' || t.kind === 'record' || t.kind === 'union') {
-            return opseq('any', ' & ',
-                         seq(name, this.genericArgsFor(t)),
-                         braces(seq('__as_preserve__',
-                                    this.hasEmbedded(t) ? '' : this.genericParameters(),
-                                    '()',
-                                    ': _.Value', this.genericArgs())));
-        } else {
-            return seq(name, this.genericArgsFor(t));
-        }
-    }
 }
 
 export class FunctionContext {
@@ -285,16 +272,25 @@ export class FunctionContext {
     }
 
     buildCapturedCompound(dest: string): Item {
-        const fields = [
-            ... variantInitFor(this.variantName),
-            ... this.captures.map(({ fieldName, sourceExpr }) =>
-                keyvalue(fieldName, sourceExpr)),
-            seq(`__as_preserve__() `, block(`return from${this.definitionName}(this)`))
-        ];
-        return seq(`${dest} = `, braces(... fields));
+        return seq(`${dest} = `, buildProduct(
+            this.definitionName, this.variantName, this.captures));
     }
 }
 
+export function buildProduct(
+    definitionName: string,
+    variant: string | undefined,
+    initializers: Capture[],
+): Item {
+    return braces(
+        ... variantInitFor(variant),
+        ... initializers.map(({ fieldName, sourceExpr }) => keyvalue(fieldName, sourceExpr)),
+        seq(`__as_preserve__() `, block(`return from${M.jsId(definitionName)}(this)`)),
+        seq(`__preserve_on__(e) { e.push(from${M.jsId(definitionName)}(this)); }`),
+        seq(`__preserve_text_on__(w) { w.push(from${M.jsId(definitionName)}(this)); }`),
+    );
+}
+
 export class WalkState {
     modulePath: M.ModulePath;
     readonly seen: FlexSet<M.Ref>;

implementations/javascript/packages/schema/src/compiler/genconverter.ts

@@ -1,8 +1,9 @@
 import { FunctionContext } from "./context";
 import * as M from '../meta';
-import { Item, seq } from "./block";
+import { Item, seq, parens, anglebrackets } from "./block";
 import { simpleType, typeFor } from "./gentype";
 import { ANY_TYPE, Type } from "./type";
+import { renderType } from "./rendertype";
 
 export function converterForDefinition(
     ctx: FunctionContext,
@@ -94,6 +95,31 @@ function converterForTuple(ctx: FunctionContext,
         : [seq(`if (_.isSequence(${src})`, lengthCheck, `) `, ctx.block(() => loop(0)))];
 }
 
+function encoderForSimplePattern(
+    ctx: FunctionContext,
+    p: M.SimplePattern,
+): Item | null {
+    switch (p._variant) {
+        case 'Ref':
+            return ctx.mod.lookup(
+                p.value,
+                (_p, t) => `from${M.jsId(p.value.name.description!)}${ctx.mod.genericArgsFor(t())}`,
+                (modPath, modId, modFile, modExpr, _p, t) => {
+                    ctx.mod.imports.add([modPath, modId, modFile, modExpr]);
+                    return `${modId}${modExpr}.from${M.jsId(p.value.name.description!)}${t ? ctx.mod.genericArgsFor(t()) : ''}`;
+                });
+        case 'embedded':
+            return `_.embed`;
+        case 'seqof': {
+            const e = encoderForSimplePattern(ctx, p.pattern);
+            if (e === null) return null;
+            return seq(`vs => vs.map`, parens(e));
+        }
+        default:
+            return null;
+    }
+}
+
 function converterFor(
     ctx: FunctionContext,
     np: M.NamedPattern,
@@ -128,7 +154,6 @@ export function converterForSimple(
             let valexp: Item = `${src}`;
             switch (p.atomKind._variant) {
                 case 'Boolean': test = `typeof ${src} === 'boolean'`; break;
-                case 'Float': test = `_.Float.isSingle(${src})`; valexp = `${src}.value`; break;
                 case 'Double': test =`_.Float.isDouble(${src})`; valexp = `${src}.value`; break;
                 case 'SignedInteger': test = `typeof ${src} === 'number'`; break;
                 case 'String': test = `typeof ${src} === 'string'`; break;
@@ -163,9 +188,12 @@ export function converterForSimple(
         case 'setof':
             return [`${dest} = void 0`,
                     seq(`if (_.Set.isSet<_embedded>(${src})) `, ctx.block(() => {
+                        const vt = simpleType(ctx.mod.resolver(), p.pattern);
                         const v = ctx.gentempname();
                         return [
-                            seq(`${dest} = new _.KeyedSet()`),
+                            seq(`${dest} = new _.EncodableSet`,
+                                anglebrackets(renderType(ctx.mod, vt), '_embedded'),
+                                parens(encoderForSimplePattern(ctx, p.pattern) ?? `v => v`)),
                             seq(`for (const ${v} of ${src}) `, ctx.block(() => [
                                 ... converterFor(ctx, M.anonymousSimplePattern(p.pattern), v, vv =>
                                     [`${dest}.add(${vv})`, `continue`]),
@@ -175,10 +203,16 @@ export function converterForSimple(
         case 'dictof':
             return [`${dest} = void 0`,
                     seq(`if (_.Dictionary.isDictionary<_embedded>(${src})) `, ctx.block(() => {
-                        const v = ctx.gentempname();
+                        const resolver = ctx.mod.resolver();
+                        const kt = simpleType(resolver, p.key);
+                        const vt = simpleType(resolver, p.value);
                         const k = ctx.gentempname();
+                        const v = ctx.gentempname();
                         return [
-                            seq(`${dest} = new _.KeyedDictionary()`),
+                            seq(`${dest} = new _.EncodableDictionary`,
+                                anglebrackets(renderType(ctx.mod, kt), renderType(ctx.mod, vt), '_embedded'),
+                                parens(encoderForSimplePattern(ctx, p.key) ?? `k => k`,
+                                       encoderForSimplePattern(ctx, p.value) ?? `v => v`)),
                             seq(`for (const [${k}, ${v}] of ${src}) `, ctx.block(() => [
                                 ... converterFor(ctx, M.anonymousSimplePattern(p.key), k, kk =>
                                     converterFor(ctx, M.anonymousSimplePattern(p.value), v, vv =>

implementations/javascript/packages/schema/src/compiler/genctor.ts

@@ -1,8 +1,8 @@
 import * as M from '../meta';
-import { block, braces, Item, keyvalue, parens, seq } from "./block";
+import { block, braces, Item, parens, seq } from "./block";
 import { FieldType, SimpleType, Type } from "./type";
 import { renderType } from "./rendertype";
-import { ModuleContext } from './context';
+import { ModuleContext, buildProduct } from './context';
 
 export function genConstructor(
     mod: ModuleContext,
@@ -29,12 +29,7 @@ export function genConstructor(
         simpleValue = (variant === void 0) && (arg.kind !== 'unit');
     }
 
-    const initializers: Item[] = (variant !== void 0)
-        ? [keyvalue('_variant', JSON.stringify(variant))]
-        : [];
-    formals.forEach(([n, _t]) => initializers.push(seq(JSON.stringify(n), ': ', M.jsId(n))));
-
-    initializers.push(seq(`__as_preserve__() `, block(`return from${M.jsId(definitionName)}(this)`)));
+    const initializers = formals.map(([n, _t]) => ({ fieldName: n, sourceExpr: M.jsId(n) }));
 
     const declArgs: Array<Item> = (formals.length > 1)
         ? [seq(braces(...formals.map(f => M.jsId(f[0]))), ': ',
@@ -48,7 +43,7 @@ export function genConstructor(
                 seq(`return `,
                     (simpleValue
                         ? 'value'
-                        : braces(...initializers))))),
+                        : buildProduct(definitionName, variant, initializers))))),
         seq(`${M.jsId(name)}.schema = function () `, block(
             seq(`return `, braces(
                 `schema: _schema()`,

implementations/javascript/packages/schema/src/compiler/gentype.ts

@@ -37,7 +37,6 @@ export function simpleType(resolver: RefResolver, p: M.SimplePattern): FieldType
         case 'atom':
             switch (p.atomKind._variant) {
                 case 'Boolean': return Type.ref(`boolean`, null);
-                case 'Float': return Type.ref(`number`, null);
                 case 'Double': return Type.ref(`number`, null);
                 case 'SignedInteger': return Type.ref(`number`, null);
                 case 'String': return Type.ref(`string`, null);

implementations/javascript/packages/schema/src/compiler/genunconverter.ts

@@ -40,7 +40,6 @@ function unconverterFor(ctx: FunctionContext, p: M.Pattern, src: string): Item {
                         return `${src}`;
                     case 'atom':
                         switch (p.atomKind._variant) {
-                            case 'Float': return `_.Single(${src})`;
                             case 'Double': return `_.Double(${src})`;
                             default: return `${src}`;
                         }

implementations/javascript/packages/schema/src/compiler/rendertype.ts

@@ -10,7 +10,7 @@ export function variantFor(variantName: string): Item {
     return keyvalue('_variant', JSON.stringify(variantName));
 }
 
-function simpleTypeFields(ctxt: ModuleContext, baseType: Type, t: SimpleType): Item[] {
+function simpleTypeFields(ctxt: ModuleContext, t: SimpleType): Item[] {
     switch (t.kind) {
         case 'unit':
             return [];
@@ -29,35 +29,49 @@ function simpleTypeFields(ctxt: ModuleContext, baseType: Type, t: SimpleType): I
 
 export function renderVariant(
     ctxt: ModuleContext,
-    baseType: Type,
     [variantName, t]: [string, SimpleType],
 ): Item {
-    let fields = simpleTypeFields(ctxt, baseType, t);
+    let fields = simpleTypeFields(ctxt, t);
     return braces(variantFor(variantName), ... fields);
 }
 
 export function renderType(ctxt: ModuleContext, t: Type): Item {
     switch (t.kind) {
         case 'union': return opseq('never', ' | ', ...
-            Array.from(t.variants).flatMap(entry => renderVariant(ctxt, t, entry)));
-        case 'unit': return braces(... simpleTypeFields(ctxt, t, t));
+            Array.from(t.variants).flatMap(entry => renderVariant(ctxt, entry)));
+        case 'unit': return braces(... simpleTypeFields(ctxt, t));
         case 'ref':
             if (t.ref === null && t.typeName === '_embedded') {
                 return t.typeName;
             } else {
                 return seq(t.typeName, ctxt.genericArgsFor(t));
             }
-        case 'set': return seq('_.KeyedSet', anglebrackets(
+        case 'set': return seq('_.EncodableSet', anglebrackets(
             renderType(ctxt, t.type),
             '_embedded'));
-        case 'dictionary': return seq('_.KeyedDictionary', anglebrackets(
+        case 'dictionary': return seq('_.EncodableDictionary', anglebrackets(
             renderType(ctxt, t.key),
             renderType(ctxt, t.value),
             '_embedded'));
         case 'array': return seq('Array', anglebrackets(renderType(ctxt, t.type)));
-        case 'record': return braces(... simpleTypeFields(ctxt, t, t));
+        case 'record': return braces(... simpleTypeFields(ctxt, t));
         default:
             ((_: never) => {})(t);
             throw new Error("Unreachable");
     }
 }
+
+export function renderTypeWithConversionMixins(ctxt: ModuleContext, t: Type): Item {
+    if (t.kind === 'unit' || t.kind === 'record' || t.kind === 'union') {
+        return opseq('any', ' & ',
+                     renderType(ctxt, t),
+                     seq('_.Preservable', ctxt.hasEmbedded(t) ? ctxt.genericArgs() : '<any>'),
+                     seq('_.PreserveWritable', ctxt.hasEmbedded(t) ? ctxt.genericArgs() : '<any>'),
+                     braces(seq('__as_preserve__',
+                                ctxt.hasEmbedded(t) ? '' : ctxt.genericParameters(),
+                                '()',
+                                ': _.Value', ctxt.genericArgs())));
+    } else {
+        return renderType(ctxt, t);
+    }
+}

implementations/javascript/packages/schema/src/compiler/value.ts

@@ -5,7 +5,6 @@ import * as M from '../meta';
 export function sourceCodeFor(v: Value<M.InputEmbedded>): Item {
     return fold(v, {
         boolean(b: boolean): Item { return b.toString(); },
-        single(f: number): Item { return f.toString(); },
         double(f: number): Item { return f.toString(); },
         integer(i: number): Item { return i.toString(); },
         string(s: string): Item { return JSON.stringify(s); },

implementations/javascript/packages/schema/src/gen/host.ts

@@ -0,0 +1,704 @@
+import * as _ from "@preserves/core";
+import * as _i_schema from "./schema";
+
+export const $any = _.Symbol.for("any");
+export const $array = _.Symbol.for("array");
+export const $embedded = _.Symbol.for("embedded");
+export const $map = _.Symbol.for("map");
+export const $rec = _.Symbol.for("rec");
+export const $ref = _.Symbol.for("ref");
+export const $set = _.Symbol.for("set");
+export const $union = _.Symbol.for("union");
+export const $unit = _.Symbol.for("unit");
+
+let __schema: _.Value | null = null;
+
+export function _schema() {
+    if (__schema === null) {
+        __schema = _.decode<_.GenericEmbedded>(_.Bytes.fromHex("b4b306736368656d61b7b30776657273696f6eb00101b30b646566696e6974696f6e73b7b3054669656c64b4b3026f72b5b5b104756e6974b4b3036c6974b304756e69748484b5b103616e79b4b3036c6974b303616e798484b5b108656d626564646564b4b3036c6974b308656d6265646465648484b5b1056172726179b4b303726563b4b3036c6974b305617272617984b4b3057475706c65b5b4b3056e616d6564b307656c656d656e74b4b303726566b584b3054669656c64848484848484b5b103736574b4b303726563b4b3036c6974b30373657484b4b3057475706c65b5b4b3056e616d6564b307656c656d656e74b4b303726566b584b3054669656c64848484848484b5b1036d6170b4b303726563b4b3036c6974b3036d617084b4b3057475706c65b5b4b3056e616d6564b3036b6579b4b303726566b584b3054669656c648484b4b3056e616d6564b30576616c7565b4b303726566b584b3054669656c64848484848484b5b103726566b4b303726563b4b3036c6974b30372656684b4b3057475706c65b5b4b3056e616d6564b3046e616d65b4b303726566b5b306736368656d6184b303526566848484848484b5b10841746f6d4b696e64b4b303726566b5b306736368656d6184b30841746f6d4b696e6484848484b3065265636f7264b4b303726563b4b3036c6974b30372656384b4b3057475706c65b5b4b3056e616d6564b3066669656c6473b4b3057365716f66b4b303726566b584b30a4e616d65644669656c64848484848484b30653696d706c65b4b3026f72b5b5b1054669656c64b4b303726566b584b3054669656c648484b5b1065265636f7264b4b303726566b584b3065265636f726484848484b30756617269616e74b4b3057475706c65b5b4b3056e616d6564b3056c6162656cb4b30461746f6db30653796d626f6c8484b4b3056e616d6564b30474797065b4b303726566b584b30653696d706c6584848484b30a446566696e6974696f6eb4b3026f72b5b5b105756e696f6eb4b303726563b4b3036c6974b305756e696f6e84b4b3057475706c65b5b4b3056e616d6564b30876617269616e7473b4b3057365716f66b4b303726566b584b30756617269616e7484848484848484b5b10653696d706c65b4b303726566b584b30653696d706c6584848484b30a4e616d65644669656c64b4b3057475706c65b5b4b3056e616d6564b3046e616d65b4b30461746f6db30653796d626f6c8484b4b3056e616d6564b30474797065b4b303726566b584b3054669656c648484848484b30c656d62656464656454797065808484"));
+    };
+    return __schema;
+}
+
+export const _imports = {"schema": _i_schema}
+
+
+export type Definition = (
+    (
+        {"_variant": "union", "variants": Array<Variant>} |
+        {"_variant": "Simple", "value": Simple}
+    ) &
+    _.Preservable<any> &
+    _.PreserveWritable<any> &
+    {__as_preserve__<_embedded = _.GenericEmbedded>(): _.Value<_embedded>}
+);
+
+export type Variant = (
+    {"label": symbol, "type": Simple} &
+    _.Preservable<any> &
+    _.PreserveWritable<any> &
+    {__as_preserve__<_embedded = _.GenericEmbedded>(): _.Value<_embedded>}
+);
+
+export type Simple = (
+    (
+        {"_variant": "Field", "value": Field} |
+        {"_variant": "Record", "value": Record}
+    ) &
+    _.Preservable<any> &
+    _.PreserveWritable<any> &
+    {__as_preserve__<_embedded = _.GenericEmbedded>(): _.Value<_embedded>}
+);
+
+export type Record = (
+    {"fields": Array<NamedField>} &
+    _.Preservable<any> &
+    _.PreserveWritable<any> &
+    {__as_preserve__<_embedded = _.GenericEmbedded>(): _.Value<_embedded>}
+);
+
+export type NamedField = (
+    {"name": symbol, "type": Field} &
+    _.Preservable<any> &
+    _.PreserveWritable<any> &
+    {__as_preserve__<_embedded = _.GenericEmbedded>(): _.Value<_embedded>}
+);
+
+export type Field = (
+    (
+        {"_variant": "unit"} |
+        {"_variant": "any"} |
+        {"_variant": "embedded"} |
+        {"_variant": "array", "element": Field} |
+        {"_variant": "set", "element": Field} |
+        {"_variant": "map", "key": Field, "value": Field} |
+        {"_variant": "ref", "name": _i_schema.Ref} |
+        {"_variant": "AtomKind", "value": _i_schema.AtomKind}
+    ) &
+    _.Preservable<any> &
+    _.PreserveWritable<any> &
+    {__as_preserve__<_embedded = _.GenericEmbedded>(): _.Value<_embedded>}
+);
+
+
+export namespace Definition {
+    export function union(variants: Array<Variant>): Definition {
+        return {
+            "_variant": "union",
+            "variants": variants,
+            __as_preserve__() {return fromDefinition(this);},
+            __preserve_on__(e) { e.push(fromDefinition(this)); },
+            __preserve_text_on__(w) { w.push(fromDefinition(this)); }
+        };
+    };
+    union.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Definition"),
+            variant: _.Symbol.for("union")
+        };
+    };
+    export function Simple(value: Simple): Definition {
+        return {
+            "_variant": "Simple",
+            "value": value,
+            __as_preserve__() {return fromDefinition(this);},
+            __preserve_on__(e) { e.push(fromDefinition(this)); },
+            __preserve_text_on__(w) { w.push(fromDefinition(this)); }
+        };
+    };
+    Simple.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Definition"),
+            variant: _.Symbol.for("Simple")
+        };
+    };
+}
+
+export function Variant({label, type}: {label: symbol, type: Simple}): Variant {
+    return {
+        "label": label,
+        "type": type,
+        __as_preserve__() {return fromVariant(this);},
+        __preserve_on__(e) { e.push(fromVariant(this)); },
+        __preserve_text_on__(w) { w.push(fromVariant(this)); }
+    };
+}
+
+Variant.schema = function () {
+    return {schema: _schema(), imports: _imports, definitionName: _.Symbol.for("Variant")};
+}
+
+export namespace Simple {
+    export function Field(value: Field): Simple {
+        return {
+            "_variant": "Field",
+            "value": value,
+            __as_preserve__() {return fromSimple(this);},
+            __preserve_on__(e) { e.push(fromSimple(this)); },
+            __preserve_text_on__(w) { w.push(fromSimple(this)); }
+        };
+    };
+    Field.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Simple"),
+            variant: _.Symbol.for("Field")
+        };
+    };
+    export function Record(value: Record): Simple {
+        return {
+            "_variant": "Record",
+            "value": value,
+            __as_preserve__() {return fromSimple(this);},
+            __preserve_on__(e) { e.push(fromSimple(this)); },
+            __preserve_text_on__(w) { w.push(fromSimple(this)); }
+        };
+    };
+    Record.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Simple"),
+            variant: _.Symbol.for("Record")
+        };
+    };
+}
+
+export function Record(fields: Array<NamedField>): Record {
+    return {
+        "fields": fields,
+        __as_preserve__() {return fromRecord(this);},
+        __preserve_on__(e) { e.push(fromRecord(this)); },
+        __preserve_text_on__(w) { w.push(fromRecord(this)); }
+    };
+}
+
+Record.schema = function () {
+    return {schema: _schema(), imports: _imports, definitionName: _.Symbol.for("Record")};
+}
+
+export function NamedField({name, type}: {name: symbol, type: Field}): NamedField {
+    return {
+        "name": name,
+        "type": type,
+        __as_preserve__() {return fromNamedField(this);},
+        __preserve_on__(e) { e.push(fromNamedField(this)); },
+        __preserve_text_on__(w) { w.push(fromNamedField(this)); }
+    };
+}
+
+NamedField.schema = function () {
+    return {
+        schema: _schema(),
+        imports: _imports,
+        definitionName: _.Symbol.for("NamedField")
+    };
+}
+
+export namespace Field {
+    export function unit(): Field {
+        return {
+            "_variant": "unit",
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    unit.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("unit")
+        };
+    };
+    export function any(): Field {
+        return {
+            "_variant": "any",
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    any.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("any")
+        };
+    };
+    export function embedded(): Field {
+        return {
+            "_variant": "embedded",
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    embedded.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("embedded")
+        };
+    };
+    export function array(element: Field): Field {
+        return {
+            "_variant": "array",
+            "element": element,
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    array.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("array")
+        };
+    };
+    export function set(element: Field): Field {
+        return {
+            "_variant": "set",
+            "element": element,
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    set.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("set")
+        };
+    };
+    export function map({key, value}: {key: Field, value: Field}): Field {
+        return {
+            "_variant": "map",
+            "key": key,
+            "value": value,
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    map.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("map")
+        };
+    };
+    export function ref(name: _i_schema.Ref): Field {
+        return {
+            "_variant": "ref",
+            "name": name,
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    ref.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("ref")
+        };
+    };
+    export function AtomKind(value: _i_schema.AtomKind): Field {
+        return {
+            "_variant": "AtomKind",
+            "value": value,
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    AtomKind.schema = function () {
+        return {
+            schema: _schema(),
+            imports: _imports,
+            definitionName: _.Symbol.for("Field"),
+            variant: _.Symbol.for("AtomKind")
+        };
+    };
+}
+
+export function asDefinition<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): Definition {
+    let result = toDefinition(v);
+    if (result === void 0) throw new TypeError(`Invalid Definition: ${_.stringify(v)}`);
+    return result;
+}
+
+export function toDefinition<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): undefined | Definition {
+    let result: undefined | Definition;
+    if (_.Record.isRecord<_.Value<_embedded>, _.Tuple<_.Value<_embedded>>, _embedded>(v)) {
+        let _tmp0: ({}) | undefined;
+        _tmp0 = _.is(v.label, $union) ? {} : void 0;
+        if (_tmp0 !== void 0) {
+            let _tmp1: (Array<Variant>) | undefined;
+            _tmp1 = void 0;
+            if (_.isSequence(v[0])) {
+                _tmp1 = [];
+                for (const _tmp2 of v[0]) {
+                    let _tmp3: (Variant) | undefined;
+                    _tmp3 = toVariant(_tmp2);
+                    if (_tmp3 !== void 0) {_tmp1.push(_tmp3); continue;};
+                    _tmp1 = void 0;
+                    break;
+                };
+            };
+            if (_tmp1 !== void 0) {
+                result = {
+                    "_variant": "union",
+                    "variants": _tmp1,
+                    __as_preserve__() {return fromDefinition(this);},
+                    __preserve_on__(e) { e.push(fromDefinition(this)); },
+                    __preserve_text_on__(w) { w.push(fromDefinition(this)); }
+                };
+            };
+        };
+    };
+    if (result === void 0) {
+        let _tmp4: (Simple) | undefined;
+        _tmp4 = toSimple(v);
+        if (_tmp4 !== void 0) {
+            result = {
+                "_variant": "Simple",
+                "value": _tmp4,
+                __as_preserve__() {return fromDefinition(this);},
+                __preserve_on__(e) { e.push(fromDefinition(this)); },
+                __preserve_text_on__(w) { w.push(fromDefinition(this)); }
+            };
+        };
+    };
+    return result;
+}
+
+export namespace Definition {export const __from_preserve__ = toDefinition;}
+
+export function fromDefinition<_embedded = _.GenericEmbedded>(_v: Definition): _.Value<_embedded> {
+    switch (_v._variant) {
+        case "union": {return _.Record($union, [_v["variants"].map(v => fromVariant<_embedded>(v))]);};
+        case "Simple": {return fromSimple<_embedded>(_v.value);};
+    };
+}
+
+export function asVariant<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): Variant {
+    let result = toVariant(v);
+    if (result === void 0) throw new TypeError(`Invalid Variant: ${_.stringify(v)}`);
+    return result;
+}
+
+export function toVariant<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): undefined | Variant {
+    let result: undefined | Variant;
+    if (_.isSequence(v) && v.length === 2) {
+        let _tmp0: (symbol) | undefined;
+        _tmp0 = typeof v[0] === 'symbol' ? v[0] : void 0;
+        if (_tmp0 !== void 0) {
+            let _tmp1: (Simple) | undefined;
+            _tmp1 = toSimple(v[1]);
+            if (_tmp1 !== void 0) {
+                result = {
+                    "label": _tmp0,
+                    "type": _tmp1,
+                    __as_preserve__() {return fromVariant(this);},
+                    __preserve_on__(e) { e.push(fromVariant(this)); },
+                    __preserve_text_on__(w) { w.push(fromVariant(this)); }
+                };
+            };
+        };
+    };
+    return result;
+}
+
+Variant.__from_preserve__ = toVariant;
+
+export function fromVariant<_embedded = _.GenericEmbedded>(_v: Variant): _.Value<_embedded> {return [_v["label"], fromSimple<_embedded>(_v["type"])];}
+
+export function asSimple<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): Simple {
+    let result = toSimple(v);
+    if (result === void 0) throw new TypeError(`Invalid Simple: ${_.stringify(v)}`);
+    return result;
+}
+
+export function toSimple<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): undefined | Simple {
+    let _tmp0: (Field) | undefined;
+    let result: undefined | Simple;
+    _tmp0 = toField(v);
+    if (_tmp0 !== void 0) {
+        result = {
+            "_variant": "Field",
+            "value": _tmp0,
+            __as_preserve__() {return fromSimple(this);},
+            __preserve_on__(e) { e.push(fromSimple(this)); },
+            __preserve_text_on__(w) { w.push(fromSimple(this)); }
+        };
+    };
+    if (result === void 0) {
+        let _tmp1: (Record) | undefined;
+        _tmp1 = toRecord(v);
+        if (_tmp1 !== void 0) {
+            result = {
+                "_variant": "Record",
+                "value": _tmp1,
+                __as_preserve__() {return fromSimple(this);},
+                __preserve_on__(e) { e.push(fromSimple(this)); },
+                __preserve_text_on__(w) { w.push(fromSimple(this)); }
+            };
+        };
+    };
+    return result;
+}
+
+export namespace Simple {export const __from_preserve__ = toSimple;}
+
+export function fromSimple<_embedded = _.GenericEmbedded>(_v: Simple): _.Value<_embedded> {
+    switch (_v._variant) {
+        case "Field": {return fromField<_embedded>(_v.value);};
+        case "Record": {return fromRecord<_embedded>(_v.value);};
+    };
+}
+
+export function asRecord<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): Record {
+    let result = toRecord(v);
+    if (result === void 0) throw new TypeError(`Invalid Record: ${_.stringify(v)}`);
+    return result;
+}
+
+export function toRecord<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): undefined | Record {
+    let result: undefined | Record;
+    if (_.Record.isRecord<_.Value<_embedded>, _.Tuple<_.Value<_embedded>>, _embedded>(v)) {
+        let _tmp0: ({}) | undefined;
+        _tmp0 = _.is(v.label, $rec) ? {} : void 0;
+        if (_tmp0 !== void 0) {
+            let _tmp1: (Array<NamedField>) | undefined;
+            _tmp1 = void 0;
+            if (_.isSequence(v[0])) {
+                _tmp1 = [];
+                for (const _tmp2 of v[0]) {
+                    let _tmp3: (NamedField) | undefined;
+                    _tmp3 = toNamedField(_tmp2);
+                    if (_tmp3 !== void 0) {_tmp1.push(_tmp3); continue;};
+                    _tmp1 = void 0;
+                    break;
+                };
+            };
+            if (_tmp1 !== void 0) {
+                result = {
+                    "fields": _tmp1,
+                    __as_preserve__() {return fromRecord(this);},
+                    __preserve_on__(e) { e.push(fromRecord(this)); },
+                    __preserve_text_on__(w) { w.push(fromRecord(this)); }
+                };
+            };
+        };
+    };
+    return result;
+}
+
+Record.__from_preserve__ = toRecord;
+
+export function fromRecord<_embedded = _.GenericEmbedded>(_v: Record): _.Value<_embedded> {return _.Record($rec, [_v["fields"].map(v => fromNamedField<_embedded>(v))]);}
+
+export function asNamedField<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): NamedField {
+    let result = toNamedField(v);
+    if (result === void 0) throw new TypeError(`Invalid NamedField: ${_.stringify(v)}`);
+    return result;
+}
+
+export function toNamedField<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): undefined | NamedField {
+    let result: undefined | NamedField;
+    if (_.isSequence(v) && v.length === 2) {
+        let _tmp0: (symbol) | undefined;
+        _tmp0 = typeof v[0] === 'symbol' ? v[0] : void 0;
+        if (_tmp0 !== void 0) {
+            let _tmp1: (Field) | undefined;
+            _tmp1 = toField(v[1]);
+            if (_tmp1 !== void 0) {
+                result = {
+                    "name": _tmp0,
+                    "type": _tmp1,
+                    __as_preserve__() {return fromNamedField(this);},
+                    __preserve_on__(e) { e.push(fromNamedField(this)); },
+                    __preserve_text_on__(w) { w.push(fromNamedField(this)); }
+                };
+            };
+        };
+    };
+    return result;
+}
+
+NamedField.__from_preserve__ = toNamedField;
+
+export function fromNamedField<_embedded = _.GenericEmbedded>(_v: NamedField): _.Value<_embedded> {return [_v["name"], fromField<_embedded>(_v["type"])];}
+
+export function asField<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): Field {
+    let result = toField(v);
+    if (result === void 0) throw new TypeError(`Invalid Field: ${_.stringify(v)}`);
+    return result;
+}
+
+export function toField<_embedded = _.GenericEmbedded>(v: _.Value<_embedded>): undefined | Field {
+    let _tmp0: ({}) | undefined;
+    let result: undefined | Field;
+    _tmp0 = _.is(v, $unit) ? {} : void 0;
+    if (_tmp0 !== void 0) {
+        result = {
+            "_variant": "unit",
+            __as_preserve__() {return fromField(this);},
+            __preserve_on__(e) { e.push(fromField(this)); },
+            __preserve_text_on__(w) { w.push(fromField(this)); }
+        };
+    };
+    if (result === void 0) {
+        let _tmp1: ({}) | undefined;
+        _tmp1 = _.is(v, $any) ? {} : void 0;
+        if (_tmp1 !== void 0) {
+            result = {
+                "_variant": "any",
+                __as_preserve__() {return fromField(this);},
+                __preserve_on__(e) { e.push(fromField(this)); },
+                __preserve_text_on__(w) { w.push(fromField(this)); }
+            };
+        };
+        if (result === void 0) {
+            let _tmp2: ({}) | undefined;
+            _tmp2 = _.is(v, $embedded) ? {} : void 0;
+            if (_tmp2 !== void 0) {
+                result = {
+                    "_variant": "embedded",
+                    __as_preserve__() {return fromField(this);},
+                    __preserve_on__(e) { e.push(fromField(this)); },
+                    __preserve_text_on__(w) { w.push(fromField(this)); }
+                };
+            };
+            if (result === void 0) {
+                if (_.Record.isRecord<_.Value<_embedded>, _.Tuple<_.Value<_embedded>>, _embedded>(v)) {
+                    let _tmp3: ({}) | undefined;
+                    _tmp3 = _.is(v.label, $array) ? {} : void 0;
+                    if (_tmp3 !== void 0) {
+                        let _tmp4: (Field) | undefined;
+                        _tmp4 = toField(v[0]);
+                        if (_tmp4 !== void 0) {
+                            result = {
+                                "_variant": "array",
+                                "element": _tmp4,
+                                __as_preserve__() {return fromField(this);},
+                                __preserve_on__(e) { e.push(fromField(this)); },
+                                __preserve_text_on__(w) { w.push(fromField(this)); }
+                            };
+                        };
+                    };
+                };
+                if (result === void 0) {
+                    if (_.Record.isRecord<_.Value<_embedded>, _.Tuple<_.Value<_embedded>>, _embedded>(v)) {
+                        let _tmp5: ({}) | undefined;
+                        _tmp5 = _.is(v.label, $set) ? {} : void 0;
+                        if (_tmp5 !== void 0) {
+                            let _tmp6: (Field) | undefined;
+                            _tmp6 = toField(v[0]);
+                            if (_tmp6 !== void 0) {
+                                result = {
+                                    "_variant": "set",
+                                    "element": _tmp6,
+                                    __as_preserve__() {return fromField(this);},
+                                    __preserve_on__(e) { e.push(fromField(this)); },
+                                    __preserve_text_on__(w) { w.push(fromField(this)); }
+                                };
+                            };
+                        };
+                    };
+                    if (result === void 0) {
+                        if (_.Record.isRecord<_.Value<_embedded>, _.Tuple<_.Value<_embedded>>, _embedded>(v)) {
+                            let _tmp7: ({}) | undefined;
+                            _tmp7 = _.is(v.label, $map) ? {} : void 0;
+                            if (_tmp7 !== void 0) {
+                                let _tmp8: (Field) | undefined;
+                                _tmp8 = toField(v[0]);
+                                if (_tmp8 !== void 0) {
+                                    let _tmp9: (Field) | undefined;
+                                    _tmp9 = toField(v[1]);
+                                    if (_tmp9 !== void 0) {
+                                        result = {
+                                            "_variant": "map",
+                                            "key": _tmp8,
+                                            "value": _tmp9,
+                                            __as_preserve__() {return fromField(this);},
+                                            __preserve_on__(e) { e.push(fromField(this)); },
+                                            __preserve_text_on__(w) { w.push(fromField(this)); }
+                                        };
+                                    };
+                                };
+                            };
+                        };
+                        if (result === void 0) {
+                            if (_.Record.isRecord<_.Value<_embedded>, _.Tuple<_.Value<_embedded>>, _embedded>(v)) {
+                                let _tmp10: ({}) | undefined;
+                                _tmp10 = _.is(v.label, $ref) ? {} : void 0;
+                                if (_tmp10 !== void 0) {
+                                    let _tmp11: (_i_schema.Ref) | undefined;
+                                    _tmp11 = _i_schema.toRef<_embedded>(v[0]);
+                                    if (_tmp11 !== void 0) {
+                                        result = {
+                                            "_variant": "ref",
+                                            "name": _tmp11,
+                                            __as_preserve__() {return fromField(this);},
+                                            __preserve_on__(e) { e.push(fromField(this)); },
+                                            __preserve_text_on__(w) { w.push(fromField(this)); }
+                                        };
+                                    };
+                                };
+                            };
+                            if (result === void 0) {
+                                let _tmp12: (_i_schema.AtomKind) | undefined;
+                                _tmp12 = _i_schema.toAtomKind<_embedded>(v);
+                                if (_tmp12 !== void 0) {
+                                    result = {
+                                        "_variant": "AtomKind",
+                                        "value": _tmp12,
+                                        __as_preserve__() {return fromField(this);},
+                                        __preserve_on__(e) { e.push(fromField(this)); },
+                                        __preserve_text_on__(w) { w.push(fromField(this)); }
+                                    };
+                                };
+                            };
+                        };
+                    };
+                };
+            };
+        };
+    };
+    return result;
+}
+
+export namespace Field {export const __from_preserve__ = toField;}
+
+export function fromField<_embedded = _.GenericEmbedded>(_v: Field): _.Value<_embedded> {
+    switch (_v._variant) {
+        case "unit": {return $unit;};
+        case "any": {return $any;};
+        case "embedded": {return $embedded;};
+        case "array": {return _.Record($array, [fromField<_embedded>(_v["element"])]);};
+        case "set": {return _.Record($set, [fromField<_embedded>(_v["element"])]);};
+        case "map": {
+            return _.Record($map, [fromField<_embedded>(_v["key"]), fromField<_embedded>(_v["value"])]);
+        };
+        case "ref": {return _.Record($ref, [_i_schema.fromRef<_embedded>(_v["name"])]);};
+        case "AtomKind": {return _i_schema.fromAtomKind<_embedded>(_v.value);};
+    };
+}
+

implementations/javascript/packages/schema/src/host.ts

@@ -0,0 +1,83 @@
+import { compare } from '@preserves/core';
+
+import * as M from './meta';
+import * as H from './gen/host';
+
+export * from './gen/host';
+
+export function definitionType<V>(p: M.Definition<V>): H.Definition {
+    switch (p._variant) {
+        case 'or': return H.Definition.union([p.pattern0, p.pattern1, ... p.patternN].map(p =>
+            H.Variant({ label: Symbol.for(p.variantLabel), type: patternType(p.pattern) })));
+        case 'and':
+            return H.Definition.Simple(productType([p.pattern0, p.pattern1, ... p.patternN]));
+        case 'Pattern':
+            return H.Definition.Simple(patternType(p.value));
+    }
+}
+
+export function patternType<V>(p: M.Pattern<V>): H.Simple {
+    switch (p._variant) {
+        case 'SimplePattern':
+            return H.Simple.Field(fieldType(p.value));
+        case 'CompoundPattern':
+            return productType([M.NamedPattern.anonymous(p)]);
+    }
+}
+
+export function fieldType<V>(p: M.SimplePattern<V>): H.Field {
+    switch (p._variant) {
+        case 'any': return H.Field.any();
+        case 'atom': return H.Field.AtomKind(p.atomKind);
+        case 'embedded': return H.Field.embedded();
+        case 'lit': return H.Field.unit();
+        case 'seqof': return H.Field.array(fieldType(p.pattern));
+        case 'setof': return H.Field.set(fieldType(p.pattern));
+        case 'dictof': return H.Field.map({ key: fieldType(p.key), value: fieldType(p.value) });
+        case 'Ref': return H.Field.ref(p.value);
+    }
+}
+
+export function productType<V>(ps: M.NamedPattern<V>[]): H.Simple {
+    const gathered: H.NamedField[] = [];
+    ps.forEach(p => gather(p, gathered));
+    if (gathered.length === 0) return H.Simple.Field(H.Field.unit());
+    return H.Simple.Record(H.Record(gathered));
+}
+
+function promote<V>(p: M.NamedSimplePattern<V>): M.NamedPattern<V> {
+    if (p._variant === 'named') return p;
+    return M.NamedPattern.anonymous(M.Pattern.SimplePattern(p.value));
+}
+
+function gather<V>(p: M.NamedPattern<V>, into: H.NamedField[]) {
+    switch (p._variant) {
+        case 'named': {
+            const t = fieldType(p.value.pattern);
+            if (t._variant !== 'unit') into.push(H.NamedField({ name: p.value.name, type: t }));
+            break;
+        }
+        case 'anonymous': {
+            if (p.value._variant === 'SimplePattern') return;
+            const q = p.value.value;
+            switch (q._variant) {
+                case 'rec':
+                    gather(q.label, into);
+                    gather(q.fields, into);
+                    break;
+                case 'tuple':
+                    q.patterns.forEach(p => gather(p, into));
+                    break;
+                case 'tuplePrefix':
+                    q.fixed.forEach(p => gather(p, into));
+                    gather(promote(q.variable), into);
+                    break;
+                case 'dict': {
+                    const items = Array.from(q.entries.entries()).sort((a, b) => compare(a[0], b[0]));
+                    items.forEach(([_key, p]) => gather(promote(p), into));
+                    break;
+                }
+            }
+        }
+    }
+}

implementations/javascript/packages/schema/src/index.ts

@@ -3,6 +3,11 @@ export * from './error';
 export * from './reader';
 export * from './compiler';
 export * from './reflection';
+
+export { SchemaInterpreter } from './interpreter';
+export * as Interpreter from './interpreter';
+
+export * as Host from './host';
 export * as Meta from './meta';
 export * as Type from './compiler/type';
 export * as GenType from './compiler/gentype';

implementations/javascript/packages/schema/src/interpreter.ts

@@ -0,0 +1,534 @@
+import { EncodableDictionary, KeyedDictionary, Dictionary, Value, is, Record, Float, Bytes, isEmbedded, isSequence, Set, Atom, Embedded, merge as plainMerge, Preservable, PreserveWritable, _iterMap, stringify, fromJS } from '@preserves/core';
+import { SchemaDefinition } from './reflection';
+import * as M from './meta';
+import * as H from './host';
+
+export const UNIT: true = true;
+
+export type Parsed<V> = Atom | V | Parsed<V>[] | DictOf<V> | Bindings<V>;
+export type TopParsed<V> = Atom | V | Parsed<V>[] | DictOf<V> | TopBindings<V>;
+
+export type Top<V> = Preservable<V> & PreserveWritable<V> & { __as_preserve__(): Value<V> };
+
+export type DictOf<V> = EncodableDictionary<Parsed<V>, Parsed<V>, V>;
+
+export type BindingName = string;
+export type Bindings<V> = { [key: BindingName]: Parsed<V> };
+export type TopBindings<V> = Bindings<V> & Top<V>;
+
+export type SingleConstructor<V> = ((input: any) => Parsed<V>) & { schema(): SchemaDefinition };
+export type MultipleConstructors<V> = { [key: string]: SingleConstructor<V> };
+export type DefinitionConstructors<V> = SingleConstructor<V> | MultipleConstructors<V>;
+
+export namespace Bindings {
+    export function empty<V>(): Bindings<V> {
+        return {};
+    }
+    export function single<V>(k: BindingName, v: Parsed<V>): Bindings<V> {
+        const bs = empty<V>();
+        bs[k] = v;
+        return bs;
+    }
+    export function merge<V>(... vs: Bindings<V>[]): Bindings<V> {
+        const acc = empty<V>();
+        for (const v of vs) {
+            Object.entries(v).forEach(([kw, vw]) => acc[kw] = vw);
+        }
+        return acc;
+    }
+}
+
+export type DynField<V> =
+    | { type: 'simple', value: Parsed<V> }
+    | { type: 'compound', values: Bindings<V> }
+;
+export namespace DynField {
+    export function unwrap<V>(f: DynField<V>): Parsed<V> {
+        if (f.type === 'simple') return f.value;
+        return f.values;
+    }
+    export function unwrap_compound<V>(f: DynField<V>): Bindings<V> {
+        if (f.type === 'simple') throw new Error("Cannot unwrap DynField.simple to compound fields");
+        return f.values;
+    }
+    export function simple<V>(value: Parsed<V>): DynField<V> {
+        return { type: 'simple', value };
+    }
+    export function maybeSimple<V>(value: Parsed<V> | null): DynField<V> {
+        return value === null ? compound(Bindings.empty()) : simple(value);
+    }
+    export function compound<V>(values: Bindings<V>): DynField<V> {
+        return { type: 'compound', values };
+    }
+    export function promote<V>(f: DynField<V>, key?: symbol): Bindings<V> {
+        if (f.type === 'compound') return f.values;
+        return key ? Bindings.single(M.jsId(key.description!), f.value) : Bindings.empty();
+    }
+}
+
+function optmap<A,B>(a: A | undefined, f: (a: A) => B): B | undefined {
+    if (a === void 0) return void 0;
+    return f(a);
+}
+
+export type Unparseable<V> = TopParsed<V>;
+export type Unparser<V> = (v: Parsed<V>) => Value<V>;
+export type UnparserCompound<V> = (v: Bindings<V>) => Value<V>;
+
+export class SchemaInterpreter<V> {
+    activeModule: M.ModulePath = [];
+    unparserCache: { [key: string]: [Unparser<V>] } = {};
+
+    constructor (
+        public env: M.Modules<V> = new KeyedDictionary(),
+        public mergeEmbeddeds: (a: V, b: V) => V | undefined = (_a, _b) => void 0,
+    ) {}
+
+    _withModule<R>(modulePath: M.ModulePath, f: () => R): R {
+        const saved = this.activeModule;
+        if (modulePath.length > 0) this.activeModule = modulePath;
+        try {
+            return f();
+        } finally {
+            if (modulePath.length > 0) this.activeModule = saved;
+        }
+    }
+
+    _findModule(modulePath: M.ModulePath): { resolved: M.ModulePath, schema: M.Schema<V> } {
+        const prefix = this.activeModule.slice();
+        while (true) {
+            const probe = [... prefix, ... modulePath];
+            const schema = this.env.get(probe);
+            if (schema !== void 0) {
+                return { resolved: probe, schema };
+            }
+            if (prefix.length === 0) {
+                throw new Error(`No such preserves-schema module: ${M.formatModulePath(modulePath)}, referred to in module ${M.formatModulePath(this.activeModule)}`);
+            }
+            prefix.pop();
+        }
+    }
+
+    _lookup<R>(
+        modulePath: M.ModulePath,
+        name: symbol,
+        f: (d: M.Definition<V>, schema: M.Schema<V>) => R,
+    ): R {
+        const { resolved, schema } = this._findModule(modulePath);
+        return this._withModule(resolved, () => {
+            const definition = schema.definitions.get(name);
+            if (definition === void 0) {
+                throw new Error(`No such preserves-schema definition: ${[... modulePath, name].map(s => s.description!).join('.')}`);
+            }
+            return f(definition, schema);
+        });
+    }
+
+    makeTop(modulePath: M.ModulePath, name: symbol, fields: Bindings<V>): TopBindings<V> {
+        const result = fields as TopBindings<V>;
+        result.__as_preserve__ = () => this.unparser(modulePath, name)(result);
+        result.__preserve_on__ = function (e) { e.push(this.__as_preserve__()); };
+        result.__preserve_text_on__ = function (w) { w.push(this.__as_preserve__()); };
+        return result;
+    }
+
+    definitionConstructor(
+        modulePath: M.ModulePath,
+        name: symbol,
+    ): DefinitionConstructors<V> {
+        return this._lookup(modulePath, name, (definition, schema): DefinitionConstructors<V> => {
+            function attachSchema(
+                f: (input: any) => Parsed<V>,
+                variant?: symbol,
+            ): SingleConstructor<V> {
+                const g = f as SingleConstructor<V>;
+                g.schema = () => ({
+                    schema: fromJS(schema),
+                    imports: {}, // TODO
+                    definitionName: name,
+                    variant,
+                });
+                return g;
+            }
+            const ty = H.definitionType(definition);
+            if (ty._variant === 'union') {
+                const multiple: MultipleConstructors<V> = {};
+                ty.variants.forEach(v => {
+                    const _variant = v.label.description!;
+                    switch (v.type._variant) {
+                        case 'Field':
+                            multiple[_variant] = attachSchema(
+                                (value: any) => this.makeTop(modulePath, name, { _variant, value }),
+                                v.label);
+                            break;
+                        case 'Record':
+                            multiple[_variant] = attachSchema(
+                                (fields: object) => this.makeTop(modulePath, name, { _variant, ... fields }),
+                                v.label);
+                            break;
+                    }
+                });
+                return multiple;
+            } else {
+                const flatName = M.formatModulePath([... modulePath, name]);
+                switch (ty.value._variant) {
+                    case 'Field': {
+                        const tmp = { [flatName]: (value: any) => value };
+                        return attachSchema(tmp[flatName]);
+                    }
+                    case 'Record': {
+                        const rec = ty.value.value;
+                        if (rec.fields.length > 1) {
+                            const tmp = { [flatName]: (fields: Bindings<V>) =>
+                                this.makeTop(modulePath, name, fields) };
+                            return attachSchema(tmp[flatName]);
+                        } else {
+                            const tmp = { [flatName]: (field: Parsed<V>) =>
+                                this.makeTop(modulePath, name, {
+                                    [M.jsId(rec.fields[0].name.description!)]: field,
+                                }) };
+                            return attachSchema(tmp[flatName]);
+                        }
+                    }
+                }
+            }
+        });
+    }
+
+    parse(
+        modulePath: M.ModulePath,
+        name: symbol,
+        input: Value<V>,
+    ): Unparseable<V> {
+        const v = this.tryParse(modulePath, name, input);
+        if (v === void 0) {
+            throw new TypeError(
+                `Invalid ${M.formatModulePath([... modulePath, name])}: ${stringify(input)}`)
+        }
+        return v;
+    }
+
+    tryParse(
+        modulePath: M.ModulePath,
+        name: symbol,
+        input: Value<V>,
+    ): Unparseable<V> | undefined {
+        return this._lookup(modulePath, name, definition =>
+            optmap(this.parseDefinition(definition, input), result0 => {
+                const ty = H.definitionType(definition);
+                if (ty._variant === 'union' || ty.value._variant === 'Record') {
+                    return this.makeTop(modulePath, name, result0 as Bindings<V>);
+                } else {
+                    return result0 as Exclude<Parsed<V>, Bindings<V>>;
+                }
+            }));
+    }
+
+    parseDefinition(d: M.Definition<V>, input: Value<V>): Parsed<V> | undefined {
+        switch (d._variant) {
+            case 'or':
+                return this.parseNamedAlternative(d.pattern0, input) ??
+                    this.parseNamedAlternative(d.pattern1, input) ??
+                    (() => {
+                        for (const p of d.patternN) {
+                            const r = this.parseNamedAlternative(p, input);
+                            if (r !== void 0) return r;
+                        }
+                        return void 0;
+                    })();
+            case 'and': {
+                const rs = [this.parseNamedPattern(d.pattern0, input),
+                            this.parseNamedPattern(d.pattern1, input),
+                            ... d.patternN.map(p => this.parseNamedPattern(p, input))];
+                for (const r of rs) {
+                    if (r === void 0) return void 0;
+                }
+                return Bindings.merge(... rs as Bindings<V>[]);
+            }
+            case 'Pattern':
+                return optmap(this.parsePattern(d.value, input), DynField.unwrap);
+        }
+    }
+
+    parseNamedAlternative(p: M.NamedAlternative<V>, input: Value<V>): Bindings<V> | undefined {
+        return optmap(this.parsePattern(p.pattern, input), w => {
+            const result = DynField.promote(w, Symbol.for('value'));
+            result._variant = p.variantLabel;
+            return result;
+        });
+    }
+
+    parseNamedPattern(p: M.NamedPattern<V>, input: Value<V>): Bindings<V> | undefined {
+        switch (p._variant) {
+            case 'named':
+                return optmap(this.parseSimplePattern(p.value.pattern, input),
+                              w => DynField.promote(DynField.maybeSimple(w), p.value.name));
+            case 'anonymous':
+                return optmap(this.parsePattern(p.value, input),
+                              w => DynField.promote(w));
+        }
+    }
+
+    parseNamedSimplePattern(p: M.NamedSimplePattern<V>, input: Value<V>): DynField<V> | undefined {
+        switch (p._variant) {
+            case 'named':
+                return optmap(this.parseSimplePattern(p.value.pattern, input),
+                              w => DynField.compound(DynField.promote(DynField.maybeSimple(w), p.value.name)));
+            case 'anonymous':
+                return optmap(this.parseSimplePattern(p.value, input), DynField.maybeSimple<V>);
+        }
+    }
+
+    parseSimplePattern(p: M.SimplePattern<V>, input: Value<V>): Parsed<V> | null | undefined {
+        const inputIf = (b: boolean) => b ? input : void 0;
+        switch (p._variant) {
+            case 'any': return input;
+            case 'atom': switch (p.atomKind._variant) {
+                case 'Boolean': return inputIf(typeof input === 'boolean');
+                case 'Double': return inputIf(Float.isDouble(input));
+                case 'SignedInteger': return inputIf(typeof input === 'number' || typeof input === 'bigint');
+                case 'String': return inputIf(typeof input === 'string');
+                case 'ByteString': return inputIf(Bytes.isBytes(input));
+                case 'Symbol': return inputIf(typeof input === 'symbol');
+            }
+            case 'embedded': return isEmbedded(input) ? input.embeddedValue : void 0;
+            case 'lit': return is(input, p.value) ? null : void 0;
+            case 'seqof': {
+                if (!isSequence(input)) return void 0;
+                const result: Parsed<V>[] = [];
+                for (const v of input) {
+                    const w = this.parseSimplePattern(p.pattern, v);
+                    if (w === void 0) return void 0;
+                    if (w !== null) result.push(w);
+                }
+                return result;
+            }
+            case 'setof': {
+                if (!Set.isSet<V>(input)) return void 0;
+                const result: Parsed<V>[] = [];
+                for (const v of input) {
+                    const w = this.parseSimplePattern(p.pattern, v);
+                    if (w === void 0) return void 0;
+                    if (w !== null) result.push(w);
+                }
+                return result;
+            }
+            case 'dictof': {
+                if (!Dictionary.isDictionary<V>(input)) return void 0;
+                const result: DictOf<V> = new EncodableDictionary(
+                    this.unparserSimplePattern(p.key),
+                    this.unparserSimplePattern(p.value));
+                for (const [k, v] of input) {
+                    const kw = this.parseSimplePattern(p.key, k);
+                    if (kw === void 0) return void 0;
+                    const vw = this.parseSimplePattern(p.value, v);
+                    if (vw === void 0) return void 0;
+                    result.set(kw === null ? UNIT : kw, vw === null ? UNIT : vw);
+                }
+                return result;
+            }
+            case 'Ref': return this.tryParse(p.value.module, p.value.name, input);
+        }
+    }
+
+    parseCompoundPattern(p: M.CompoundPattern<V>, input: Value<V>): Bindings<V> | undefined {
+        switch (p._variant) {
+            case 'rec':
+                if (!Record.isRecord<Value<V>, Array<Value<V>>, V>(input)) return void 0;
+                return optmap(this.parseNamedPattern(p.label, input.label),
+                              lw => optmap(this.parseNamedPattern(p.fields, Array.from(input)),
+                                           fsw => Bindings.merge(lw, fsw)));
+            case 'tuple': {
+                if (!isSequence(input)) return void 0;
+                if (input.length !== p.patterns.length) return void 0;
+                let results: Bindings<V>[] = [];
+                for (let i = 0; i < p.patterns.length; i++) {
+                    const w = this.parseNamedPattern(p.patterns[i], input[i]);
+                    if (w === void 0) return void 0;
+                    results.push(w);
+                }
+                return Bindings.merge(... results);
+            }
+            case 'tuplePrefix': {
+                if (!isSequence(input)) return void 0;
+                if (input.length < p.fixed.length) return void 0;
+                let fixed_results: Bindings<V>[] = [];
+                for (let i = 0; i < p.fixed.length; i++) {
+                    const w = this.parseNamedPattern(p.fixed[i], input[i]);
+                    if (w === void 0) return void 0;
+                    fixed_results.push(w);
+                }
+                const remainder = input.slice(p.fixed.length);
+                return optmap(this.parseNamedSimplePattern(p.variable, remainder), vw => {
+                    const variable_results = DynField.unwrap_compound(vw);
+                    return Bindings.merge(variable_results, ... fixed_results);
+                });
+            }
+            case 'dict': {
+                if (!Dictionary.isDictionary<V>(input)) return void 0;
+                const results: Bindings<V>[] = [];
+                for (const [key, vp] of p.entries) {
+                    const v = input.get(key);
+                    if (v === void 0) return void 0;
+                    const vw = this.parseNamedSimplePattern(vp, v);
+                    if (vw === void 0) return void 0;
+                    results.push(DynField.unwrap_compound(vw));
+                }
+                return Bindings.merge(... results);
+            }
+        }
+    }
+
+    parsePattern(p: M.Pattern<V>, input: Value<V>): DynField<V> | undefined {
+        switch (p._variant) {
+            case 'SimplePattern':
+                return optmap(this.parseSimplePattern(p.value, input), DynField.maybeSimple);
+            case 'CompoundPattern':
+                return optmap(this.parseCompoundPattern(p.value, input), DynField.compound);
+        }
+    }
+
+    unparse(
+        modulePath: M.ModulePath,
+        name: symbol,
+        v: Unparseable<V>,
+    ): Value<V> {
+        return this.unparser(modulePath, name)(v);
+    }
+
+    unparser(modulePath: M.ModulePath, name: symbol): Unparser<V> {
+        return this._unparser(modulePath, name)[0];
+    }
+
+    _unparser(modulePath: M.ModulePath, name: symbol): [Unparser<V>] {
+        const key = [... modulePath.map(n => n.description!), name.description!].join('.');
+        if (!(key in this.unparserCache)) {
+            const cell: [Unparser<V>] = [null!];
+            this.unparserCache[key] = cell;
+            cell[0] = this._lookup(modulePath, name, p => this.unparserDefinition(p));
+        }
+        return this.unparserCache[key];
+    }
+
+    unparserDefinition(p: M.Definition<V>): Unparser<V> {
+        switch (p._variant) {
+            case 'or': {
+                const ups = [p.pattern0, p.pattern1, ... p.patternN].map(
+                    p => this.unparserNamedAlternative(p));
+                return v => {
+                    const bs = v as Bindings<V>;
+                    return ups.find(up => up[0] === bs._variant)![1](bs);
+                };
+            }
+            case 'and': {
+                const ups = [p.pattern0, p.pattern1, ... p.patternN].map(
+                    p => this.unparserNamedPattern(p));
+                return v => plainMerge(this.mergeEmbeddeds,
+                                       ups[0](v), ... ups.slice(1).map(up => up(v)));
+            }
+            case 'Pattern':
+                return this.unparserPattern(p.value);
+        }
+    }
+
+    unparserNamedAlternative(p: M.NamedAlternative<V>): [string, UnparserCompound<V>] {
+        const up = this.unparserPattern(p.pattern);
+        const ty = H.patternType(p.pattern);
+        switch (ty._variant) {
+            case 'Field': return [p.variantLabel, bs => up(bs['value'])];
+            case 'Record': return [p.variantLabel, up];
+        }
+    }
+
+    unparserNamedPattern(p: M.NamedPattern<V>): Unparser<V> {
+        switch (p._variant) {
+            case 'named': {
+                const up = this.unparserSimplePattern(p.value.pattern);
+                const key = M.jsId(p.value.name.description!);
+                return v => up((v as Bindings<V>)[key]);
+            }
+            case 'anonymous':
+                return this.unparserPattern(p.value);
+        }
+    }
+
+    unparserPattern(p: M.Pattern<V>): Unparser<V> {
+        switch (p._variant) {
+            case 'CompoundPattern': {
+                const up = this.unparserCompoundPattern(p.value);
+                return v => up(v as Bindings<V>);
+            }
+            case 'SimplePattern':
+                return this.unparserSimplePattern(p.value);
+        }
+    }
+
+    unparserSimplePattern(p: M.SimplePattern<V>): Unparser<V> {
+        switch (p._variant) {
+            case 'any': return v => v as Value<V>; // ?!
+            case 'atom': return v => v as Atom;
+            case 'embedded': return v => new Embedded(v as V);
+            case 'lit': return _v => p.value;
+            case 'seqof': {
+                const up = this.unparserSimplePattern(p.pattern);
+                return vs => (vs as Parsed<V>[]).map(up);
+            }
+            case 'setof': {
+                const up = this.unparserSimplePattern(p.pattern);
+                return vs => new Set<V>((vs as Parsed<V>[]).map(up));
+            }
+            case 'dictof': {
+                const kp = this.unparserSimplePattern(p.key);
+                const vp = this.unparserSimplePattern(p.value);
+                return vs => new Dictionary<V>(_iterMap(
+                    (vs as DictOf<V>).entries(), ([k, v]) => [kp(k), vp(v)]));
+            }
+            case 'Ref': {
+                const up = this._unparser(p.value.module, p.value.name);
+                return v => up[0](v as Bindings<V>);
+            }
+        }
+    }
+
+    unparserCompoundPattern(p: M.CompoundPattern<V>): UnparserCompound<V> {
+        switch (p._variant) {
+            case 'rec': {
+                const lp = this.unparserNamedPattern(p.label);
+                const fp = this.unparserNamedPattern(p.fields);
+                return bs => Record(lp(bs), fp(bs) as Value<V>[]);
+            }
+            case 'tuple': {
+                const ups = p.patterns.map(p => this.unparserNamedPattern(p));
+                return bs => ups.map(up => up(bs));
+            }
+            case 'tuplePrefix': {
+                const fixed = p.fixed.map(p => this.unparserNamedPattern(p));
+                const variable = this.unparserNamedSimplePattern(p.variable);
+                return bs => [... fixed.map(up => up(bs)), ... variable(bs) as Value<V>[]];
+            }
+            case 'dict': {
+                const ups: [Value<V>, Unparser<V>][] = Array.from(p.entries.entries()).map(
+                    ([key, vp]) => [key, this.unparserNamedSimplePattern(vp)]);
+                return bs => {
+                    const result = new Dictionary<V>();
+                    for (const [key, up] of ups) {
+                        result.set(key, up(bs));
+                    }
+                    return result;
+                };
+            }
+        }
+    }
+
+    unparserNamedSimplePattern(p: M.NamedSimplePattern<V>): Unparser<V> {
+        switch (p._variant) {
+            case 'named': {
+                const up = this.unparserSimplePattern(p.value.pattern);
+                const key = M.jsId(p.value.name.description!);
+                return v => up((v as Bindings<V>)[key]);
+            }
+            case 'anonymous':
+                return this.unparserSimplePattern(p.value);
+        }
+    }
+}

implementations/javascript/packages/schema/src/meta.ts

@@ -95,7 +95,6 @@ export function anonymousSimplePattern(p: M.SimplePattern): M.NamedPattern {
 export function namelike(x: Input): string | undefined {
     if (typeof x === 'string') return x;
     if (typeof x === 'symbol') return x.description!;
-    if (typeof x === 'number') return '' + x;
-    if (typeof x === 'boolean') return '' + x;
+    if (typeof x === 'boolean') return x ? 'true' : 'false';
     return void 0;
 }

implementations/javascript/packages/schema/src/reader.ts

@@ -1,4 +1,4 @@
-import { Reader, Annotated, Dictionary, is, peel, preserves, Record, strip, Tuple, Position, position, stringify, isCompound, KeyedDictionary, annotate, annotations, isEmbedded, GenericEmbedded, genericEmbeddedTypeDecode } from '@preserves/core';
+import { Reader, Annotated, Dictionary, is, peel, preserves, Record, strip, Tuple, Position, position, stringify, isCompound, EncodableDictionary, annotate, annotations, isEmbedded, GenericEmbedded, genericEmbeddedTypeDecode } from '@preserves/core';
 import { Input, Pattern, Schema, Definition, CompoundPattern, SimplePattern } from './meta';
 import * as M from './meta';
 import { SchemaSyntaxError } from './error';
@@ -70,7 +70,7 @@ export function parseSchema(toplevelTokens: Array<Input>, options: SchemaReaderO
 {
     let version: M.Version | undefined = void 0;
     let embeddedType: M.EmbeddedTypeName = M.EmbeddedTypeName.$false();
-    let definitions = new KeyedDictionary<symbol, Definition, M.InputEmbedded>();
+    let definitions: M.Definitions = new EncodableDictionary(k => k, M.fromDefinition);
 
     function process(toplevelTokens: Array<Input>): void {
         const toplevelClauses = splitBy(peel(toplevelTokens) as Array<Input>, M.DOT);
@@ -161,7 +161,12 @@ function parseDefinition(name: symbol, pos: Position | null, body: Array<Input>)
             p.value._variant === 'lit')
         {
             const s = M.namelike(p.value.value);
-            if (s !== void 0) return M.NamedAlternative({ variantLabel: s, pattern: p });
+            if (s !== void 0) {
+                if (M.isValidToken(s)) {
+                    return M.NamedAlternative({ variantLabel: s, pattern: p });
+                }
+                throw new SchemaSyntaxError(preserves`Invalid name ${s} inferred for alternative: ${input}`, pos);
+            }
         }
         throw new SchemaSyntaxError(preserves`Name missing for alternative: ${input}`, pos);
     }
@@ -220,7 +225,6 @@ function parsePattern(name: symbol, body0: Array<Input>): Pattern {
             switch (str) {
                 case 'any': return ks(M.SimplePattern.any());
                 case 'bool': return ks(M.SimplePattern.atom(M.AtomKind.Boolean()));
-                case 'float': return ks(M.SimplePattern.atom(M.AtomKind.Float()));
                 case 'double': return ks(M.SimplePattern.atom(M.AtomKind.Double()));
                 case 'int': return ks(M.SimplePattern.atom(M.AtomKind.SignedInteger()));
                 case 'string': return ks(M.SimplePattern.atom(M.AtomKind.String()));
@@ -338,11 +342,11 @@ function parsePattern(name: symbol, body0: Array<Input>): Pattern {
         return (b: Input) => {
             let name = findName(b);
             if (name === false) {
-                if (literalName !== void 0 &&
-                    typeof literalName === 'symbol' &&
-                    M.isValidToken(literalName.description!))
-                {
-                    name = literalName;
+                if (literalName !== void 0) {
+                    const s = M.namelike(literalName);
+                    if (s !== void 0 && M.isValidToken(s)) {
+                        name = Symbol.for(s);
+                    }
                 }
             }
             if (name === false) {

implementations/javascript/packages/schema/test/reader.test.ts

@@ -9,12 +9,16 @@ describe('reader schema', () => {
     });
     it('is OK with an empty schema correctly versioned', () => {
         const s = readSchema('version 1 .');
-        expect(Object.getOwnPropertyNames(s.version)).toEqual(['__as_preserve__']);
+        expect(Object.getOwnPropertyNames(s.version)).toEqual([
+            '__as_preserve__',
+            '__preserve_on__',
+            '__preserve_text_on__',
+        ]);
         expect(s.definitions.size).toBe(0);
         expect(s.embeddedType._variant).toBe('false');
     });
     it('understands patterns under embed', () => {
-        const s = readSchema('version 1 . X = #!0 .');
+        const s = readSchema('version 1 . X = #:0 .');
         const def: Meta.Definition = s.definitions.get(Symbol.for('X'))!;
         if (def._variant !== 'Pattern') fail('bad definition 1');
         if (def.value._variant !== 'SimplePattern') fail ('bad definition 2');

implementations/python/.envrc

@@ -2,10 +2,7 @@ if ! [ -d .venv ]
 then
     python -m venv .venv
     . .venv/bin/activate
-    pip install -U coverage setuptools setuptools_scm wheel \
-        mkdocs 'mkdocstrings[python]' mkdocs-material mkdocs-macros-plugin \
-        mkdocs-git-revision-date-localized-plugin
-    pip install -e .
+    pip install -e '.[dev]'
 else
     . .venv/bin/activate
 fi

implementations/python/Makefile

@@ -1,7 +1,12 @@
-PACKAGEVERSION="`python3 setup.py --version`"
+# This used to just be
+# PACKAGEVERSION := "`python3 setup.py --version`"
+PACKAGEVERSION := $(shell python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')
 
 all: test build-docs
 
+test-in-docker: build-only
+	docker run --rm -v `pwd`/dist:/dist python /bin/bash -c 'pip install dist/preserves-$(PACKAGEVERSION)*.whl && python -c "import preserves.schema; print(preserves.schema)"'
+
 test: update-test-data
 	python3 -m unittest discover -s tests
 
@@ -34,5 +39,9 @@ clean:
 publish: clean build
 	twine upload dist/*
 
-build: build-docs
-	python3 setup.py sdist bdist_wheel
+build: build-docs build-only
+
+build-only: dist/preserves-$(PACKAGEVERSION).tar.gz
+
+dist/preserves-$(PACKAGEVERSION).tar.gz:
+	python3 -m build

implementations/python/preserves/__init__.py

@@ -7,7 +7,6 @@ The main package re-exports a subset of the exports of its constituent modules:
 - From [preserves.values][]:
     - [Annotated][preserves.values.Annotated]
     - [Embedded][preserves.values.Embedded]
-    - [Float][preserves.values.Float]
     - [ImmutableDict][preserves.values.ImmutableDict]
     - [Record][preserves.values.Record]
     - [Symbol][preserves.values.Symbol]
@@ -56,7 +55,7 @@ Finally, it provides a few utility aliases for common tasks:
 
 '''
 
-from .values import Float, Symbol, Record, ImmutableDict, Embedded, preserve
+from .values import Symbol, Record, ImmutableDict, Embedded, preserve
 
 from .values import Annotated, is_annotated, strip_annotations, annotate
 
@@ -69,7 +68,7 @@ from .text import Parser, Formatter, parse, parse_with_annotations, stringify
 
 from .merge import merge
 
-from . import fold, compare
+from . import fold, compare, schema
 
 loads = parse
 '''

implementations/python/preserves/binary.py

@@ -206,7 +206,6 @@ class Decoder(BinaryCodec):
             return self.wrap(Embedded(self.decode_embedded(self.next())))
         if tag == 0x87:
             count = self.nextbyte()
-            if count == 4: return self.wrap(Float.from_bytes(self.nextbytes(4)))
             if count == 8: return self.wrap(struct.unpack('>d', self.nextbytes(8))[0])
             raise DecodeError('Invalid IEEE754 size')
         if tag == 0xb0: return self.wrap(self.nextint(self.varint()))

implementations/python/preserves/compare.py

@@ -33,12 +33,12 @@ import numbers
 from enum import Enum
 from functools import cmp_to_key
 
-from .values import preserve, Float, Embedded, Record, Symbol, cmp_floats, _unwrap
+from .values import preserve, Embedded, Record, Symbol, cmp_floats, _unwrap
 from .compat import basestring_
 
 class TypeNumber(Enum):
     BOOL = 0
-    FLOAT = 1
+    # FLOAT = 1  # single-precision
     DOUBLE = 2
     SIGNED_INTEGER = 3
     STRING = 4
@@ -57,7 +57,6 @@ def type_number(v):
         raise ValueError('type_number expects Preserves value; use preserve()')
 
     if isinstance(v, bool): return TypeNumber.BOOL
-    if isinstance(v, Float): return TypeNumber.FLOAT
     if isinstance(v, float): return TypeNumber.DOUBLE
     if isinstance(v, numbers.Number): return TypeNumber.SIGNED_INTEGER
     if isinstance(v, basestring_): return TypeNumber.STRING

implementations/python/preserves/fold.py

@@ -9,7 +9,7 @@ def map_embeddeds(f, v):
 
     ```python
     >>> map_embeddeds(lambda w: Embedded(f'w={w}'), ['a', Embedded(123), {'z': 6.0}])
-    ('a', #!'w=123', {'z': 6.0})
+    ('a', #:'w=123', {'z': 6.0})
 
     ```
     """

implementations/python/preserves/merge.py

@@ -52,7 +52,7 @@ def merge2(a, b, merge_embedded=None):
       ...
     ValueError: Cannot merge items
     >>> merge2(Record('a', [1, {'x': 2}]), Record('a', [1, {'y': 3}]))
-    a(1, {'x': 2, 'y': 3})
+    'a'(1, {'x': 2, 'y': 3})
 
     ```
 

implementations/python/preserves/path.prb

@@ -1,5 +1,5 @@
 ´³schema·³version°

³definitions·³Axis´³orµµ±values´³rec´³lit³values„´³tupleµ„„„„µ±descendants´³rec´³lit³descendants„´³tupleµ„„„„µ±at´³rec´³lit³at„´³tupleµ´³named³key³any„„„„„µ±label´³rec´³lit³label„´³tupleµ„„„„µ±keys´³rec´³lit³keys„´³tupleµ„„„„µ±length´³rec´³lit³length„´³tupleµ„„„„µ±annotations´³rec´³lit³annotations„´³tupleµ„„„„µ±embedded´³rec´³lit³embedded„´³tupleµ„„„„µ±parse´³rec´³lit³parse„´³tupleµ´³named³module´³seqof´³atom³Symbol„„„´³named³name´³atom³Symbol„„„„„„µ±unparse´³rec´³lit³unparse„´³tupleµ´³named³module´³seqof´³atom³Symbol„„„´³named³name´³atom³Symbol„„„„„„„„³Step´³orµµ±Axis´³refµ„³Axis„„µ±Filter´³refµ„³Filter„„µ±Function´³refµ„³Function„„„„³Filter´³orµµ±nop´³rec´³lit³nop„´³tupleµ„„„„µ±compare´³rec´³lit³compare„´³tupleµ´³named³op´³refµ„³
-Comparison„„´³named³literal³any„„„„„µ±regex´³rec´³lit³regex„´³tupleµ´³named³regex´³atom³String„„„„„„µ±test´³rec´³lit³test„´³tupleµ´³named³pred´³refµ„³	Predicate„„„„„„µ±real´³rec´³lit³real„´³tupleµ„„„„µ±int´³rec´³lit³int„´³tupleµ„„„„µ±kind´³rec´³lit³kind„´³tupleµ´³named³kind´³refµ„³	ValueKind„„„„„„„„³Function´³rec´³lit³count„´³tupleµ´³named³selector´³refµ„³Selector„„„„„³Selector´³seqof´³refµ„³Step„„³	Predicate´³orµµ±Selector´³refµ„³Selector„„µ±not´³rec´³lit³not„´³tupleµ´³named³pred´³refµ„³	Predicate„„„„„„µ±or´³rec´³lit³or„´³tupleµ´³named³preds´³seqof´³refµ„³	Predicate„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³named³preds´³seqof´³refµ„³	Predicate„„„„„„„„„³	ValueKind´³orµµ±Boolean´³lit³Boolean„„µ±Float´³lit³Float„„µ±Double´³lit³Double„„µ±
SignedInteger´³lit³
SignedInteger„„µ±String´³lit³String„„µ±
+Comparison„„´³named³literal³any„„„„„µ±regex´³rec´³lit³regex„´³tupleµ´³named³regex´³atom³String„„„„„„µ±test´³rec´³lit³test„´³tupleµ´³named³pred´³refµ„³	Predicate„„„„„„µ±real´³rec´³lit³real„´³tupleµ„„„„µ±int´³rec´³lit³int„´³tupleµ„„„„µ±kind´³rec´³lit³kind„´³tupleµ´³named³kind´³refµ„³	ValueKind„„„„„„„„³Function´³rec´³lit³count„´³tupleµ´³named³selector´³refµ„³Selector„„„„„³Selector´³seqof´³refµ„³Step„„³	Predicate´³orµµ±Selector´³refµ„³Selector„„µ±not´³rec´³lit³not„´³tupleµ´³named³pred´³refµ„³	Predicate„„„„„„µ±or´³rec´³lit³or„´³tupleµ´³named³preds´³seqof´³refµ„³	Predicate„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³named³preds´³seqof´³refµ„³	Predicate„„„„„„„„„³	ValueKind´³orµµ±Boolean´³lit³Boolean„„µ±Double´³lit³Double„„µ±
SignedInteger´³lit³
SignedInteger„„µ±String´³lit³String„„µ±
 ByteString´³lit³
 ByteString„„µ±Symbol´³lit³Symbol„„µ±Record´³lit³Record„„µ±Sequence´³lit³Sequence„„µ±Set´³lit³Set„„µ±
 Dictionary´³lit³

implementations/python/preserves/path.py

@@ -170,7 +170,6 @@ FILTER_RE2 = Symbol('=r')
 FILTER_LABEL = Symbol('^')
 
 FILTER_BOOL = Symbol('bool')
-FILTER_FLOAT = Symbol('float')
 FILTER_DOUBLE = Symbol('double')
 FILTER_INT = Symbol('int')
 FILTER_STRING = Symbol('string')
@@ -226,7 +225,6 @@ def parse_step(tokens):
     if t == TRANSFORM_REAL: return syntax.Step.Filter(syntax.Filter.real)
     if t == TRANSFORM_INT: return syntax.Step.Filter(syntax.Filter.int)
     if t == FILTER_BOOL: return kind_filter(syntax.ValueKind.Boolean())
-    if t == FILTER_FLOAT: return kind_filter(syntax.ValueKind.Float())
     if t == FILTER_DOUBLE: return kind_filter(syntax.ValueKind.Double())
     if t == FILTER_INT: return kind_filter(syntax.ValueKind.SignedInteger())
     if t == FILTER_STRING: return kind_filter(syntax.ValueKind.String())
@@ -448,8 +446,6 @@ def exec(self, v):
 @extend(syntax.Filter.real)
 def exec(self, v):
     v = preserve(_unwrap(v))
-    if isinstance(v, Float):
-        return (v.value,)
     if type(v) == float:
         return (v,)
     if type(v) == int:
@@ -459,8 +455,6 @@ def exec(self, v):
 @extend(syntax.Filter.int)
 def exec(self, v):
     v = preserve(_unwrap(v))
-    if isinstance(v, Float):
-        return (int(v.value()),)
     if type(v) == float:
         return (int(v),)
     if type(v) == int:
@@ -476,10 +470,6 @@ def exec(self, v):
 def exec(self, v):
     return (v,) if type(v) == bool else ()
 
-@extend(syntax.ValueKind.Float)
-def exec(self, v):
-    return (v,) if isinstance(v, Float) else ()
-
 @extend(syntax.ValueKind.Double)
 def exec(self, v):
     return (v,) if type(v) == float else ()

implementations/python/preserves/schema.prb

@@ -1,6 +1,6 @@
 ´³schema·³version°

³definitions·³Ref´³rec´³lit³ref„´³tupleµ´³named³module´³refµ„³
 ModulePath„„´³named³name´³atom³Symbol„„„„„³Bundle´³rec´³lit³bundle„´³tupleµ´³named³modules´³refµ„³Modules„„„„„³Schema´³rec´³lit³schema„´³tupleµ´³dict·³version´³named³version´³refµ„³Version„„³definitions´³named³definitions´³refµ„³Definitions„„³embeddedType´³named³embeddedType´³refµ„³EmbeddedTypeName„„„„„„„³Binding´³rec´³lit³named„´³tupleµ´³named³name´³atom³Symbol„„´³named³pattern´³refµ„³
SimplePattern„„„„„³Modules´³dictof´³refµ„³
-ModulePath„´³refµ„³Schema„„³Pattern´³orµµ±
SimplePattern´³refµ„³
SimplePattern„„µ±CompoundPattern´³refµ„³CompoundPattern„„„„³Version´³lit°

„³AtomKind´³orµµ±Boolean´³lit³Boolean„„µ±Float´³lit³Float„„µ±Double´³lit³Double„„µ±
SignedInteger´³lit³
SignedInteger„„µ±String´³lit³String„„µ±
+ModulePath„´³refµ„³Schema„„³Pattern´³orµµ±
SimplePattern´³refµ„³
SimplePattern„„µ±CompoundPattern´³refµ„³CompoundPattern„„„„³Version´³lit°

„³AtomKind´³orµµ±Boolean´³lit³Boolean„„µ±Double´³lit³Double„„µ±
SignedInteger´³lit³
SignedInteger„„µ±String´³lit³String„„µ±
 ByteString´³lit³
 ByteString„„µ±Symbol´³lit³Symbol„„„„³
 Definition´³orµµ±or´³rec´³lit³or„´³tupleµ´³tuplePrefixµ´³named³pattern0´³refµ„³NamedAlternative„„´³named³pattern1´³refµ„³NamedAlternative„„„´³named³patternN´³seqof´³refµ„³NamedAlternative„„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³tuplePrefixµ´³named³pattern0´³refµ„³NamedPattern„„´³named³pattern1´³refµ„³NamedPattern„„„´³named³patternN´³seqof´³refµ„³NamedPattern„„„„„„„„µ±Pattern´³refµ„³Pattern„„„„³

implementations/python/preserves/schema.py

@@ -279,7 +279,6 @@ DICT = Symbol('dict')
 DICTOF = Symbol('dictof')
 DOUBLE = Symbol('Double')
 EMBEDDED = Symbol('embedded')
-FLOAT = Symbol('Float')
 LIT = Symbol('lit')
 NAMED = Symbol('named')
 OR = Symbol('or')
@@ -469,7 +468,6 @@ class SchemaObject:
         if p.key == ATOM:
             k = p[0]
             if k == BOOLEAN and isinstance(v, bool): return v
-            if k == FLOAT and isinstance(v, Float): return v
             if k == DOUBLE and isinstance(v, float): return v
             if k == SIGNED_INTEGER and isinstance(v, int): return v
             if k == STRING and isinstance(v, str): return v

implementations/python/preserves/text.py

@@ -28,7 +28,7 @@ from .binary import Decoder
 class TextCodec(object):
     pass
 
-NUMBER_RE = re.compile(r'^([-+]?\d+)(((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))([fF]?))?$')
+NUMBER_RE = re.compile(r'^([-+]?\d+)((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))?$')
 
 class Parser(TextCodec):
     """Parser for the human-readable Preserves text syntax.
@@ -251,15 +251,13 @@ class Parser(TextCodec):
             if c == '=': continue
             acc.append(c)
 
-    def read_hex_float(self, bytecount):
+    def read_hex_float(self):
         if self.nextchar() != '"':
             raise DecodeError('Missing open-double-quote in hex-encoded floating-point number')
         bs = self.read_hex_binary()
-        if len(bs) != bytecount:
+        if len(bs) != 8:
             raise DecodeError('Incorrect number of bytes in hex-encoded floating-point number')
-        if bytecount == 4: return Float.from_bytes(bs)
-        if bytecount == 8: return struct.unpack('>d', bs)[0]
-        raise DecodeError('Unsupported byte count in hex-encoded floating-point number')
+        return struct.unpack('>d', bs)[0]
 
     def upto(self, delimiter, skip_commas):
         vs = []
@@ -308,10 +306,8 @@ class Parser(TextCodec):
         if m:
             if m[2] is None:
                 return int(m[1])
-            elif m[7] == '':
-                return float(m[1] + m[3])
             else:
-                return Float(float(m[1] + m[3]))
+                return float(acc)
         else:
             return Symbol(acc)
 
@@ -357,11 +353,10 @@ class Parser(TextCodec):
             if c == 'x':
                 c = self.nextchar()
                 if c == '"': return self.wrap(self.read_hex_binary())
-                if c == 'f': return self.wrap(self.read_hex_float(4))
-                if c == 'd': return self.wrap(self.read_hex_float(8))
+                if c == 'd': return self.wrap(self.read_hex_float())
                 raise DecodeError('Invalid #x syntax')
             if c == '[': return self.wrap(self.read_base64_binary())
-            if c == '!':
+            if c == ':':
                 if self.parse_embedded is None:
                     raise DecodeError('No parse_embedded function supplied')
                 return self.wrap(Embedded(self.parse_embedded(self.next())))

implementations/python/preserves/values.py

@@ -39,135 +39,6 @@ def cmp_floats(a, b):
     if b & 0x8000000000000000: b = b ^ 0x7fffffffffffffff
     return a - b
 
-class Float(object):
-    """Wrapper for treating a Python double-precision floating-point value as a
-    single-precision (32-bit) float, from Preserves' perspective. (Python lacks native
-    single-precision floating point support.)
-
-    ```python
-    >>> Float(3.45)
-    Float(3.45)
-    >>> import preserves
-    >>> preserves.stringify(Float(3.45))
-    '3.45f'
-    >>> preserves.stringify(3.45)
-    '3.45'
-    >>> preserves.parse('3.45f')
-    Float(3.45)
-    >>> preserves.parse('3.45')
-    3.45
-    >>> preserves.encode(Float(3.45))
-    b'\\x87\\x04@\\\\\\xcc\\xcd'
-    >>> preserves.encode(3.45)
-    b'\\x87\\x08@\\x0b\\x99\\x99\\x99\\x99\\x99\\x9a'
-
-    ```
-
-    Attributes:
-        value (float): the double-precision representation of intended single-precision value
-    """
-    def __init__(self, value):
-        self.value = value
-
-    def __eq__(self, other):
-        other = _unwrap(other)
-        if other.__class__ is self.__class__:
-            return cmp_floats(self.value, other.value) == 0
-
-    def __lt__(self, other):
-        other = _unwrap(other)
-        if other.__class__ is self.__class__:
-            return cmp_floats(self.value, other.value) < 0
-
-    def __ne__(self, other):
-        return not self.__eq__(other)
-
-    def __hash__(self):
-        return hash(self.value)
-
-    def __repr__(self):
-        return 'Float(' + repr(self.value) + ')'
-
-    def to_bytes(self):
-        """Converts this 32-bit single-precision floating point value to its binary32 format,
-        taking care to preserve the quiet/signalling bit-pattern of NaN values, unlike its
-        `struct.pack('>f', ...)` equivalent.
-
-        ```python
-        >>> Float.from_bytes(b'\\x7f\\x80\\x00{')
-        Float(nan)
-        >>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
-        b'\\x7f\\x80\\x00{'
-
-        >>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
-        nan
-        >>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
-        b'\\x7f\\xc0\\x00{'
-        >>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
-        b'\\x7f\\xc0\\x00{'
-
-        ```
-
-        (Note well the difference between `7f80007b` and `7fc0007b`!)
-
-        """
-
-        if math.isnan(self.value) or math.isinf(self.value):
-            dbs = struct.pack('>d', self.value)
-            vd = struct.unpack('>Q', dbs)[0]
-            sign = vd >> 63
-            payload = (vd >> 29) & 0x007fffff
-            vf = (sign << 31) | 0x7f800000 | payload
-            return struct.pack('>I', vf)
-        else:
-            return struct.pack('>f', self.value)
-
-    def __preserve_write_binary__(self, encoder):
-        encoder.buffer.append(0x87)
-        encoder.buffer.append(4)
-        encoder.buffer.extend(self.to_bytes())
-
-    def __preserve_write_text__(self, formatter):
-        if math.isnan(self.value) or math.isinf(self.value):
-            formatter.chunks.append('#xf"' + self.to_bytes().hex() + '"')
-        else:
-            formatter.chunks.append(repr(self.value) + 'f')
-
-    @staticmethod
-    def from_bytes(bs):
-        """Converts a 4-byte-long byte string to a 32-bit single-precision floating point value
-        wrapped in a [Float][preserves.values.Float] instance. Takes care to preserve the
-        quiet/signalling bit-pattern of NaN values, unlike its `struct.unpack('>f', ...)`
-        equivalent.
-
-        ```python
-        >>> Float.from_bytes(b'\\x7f\\x80\\x00{')
-        Float(nan)
-        >>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
-        b'\\x7f\\x80\\x00{'
-
-        >>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
-        nan
-        >>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
-        b'\\x7f\\xc0\\x00{'
-        >>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
-        b'\\x7f\\xc0\\x00{'
-
-        ```
-
-        (Note well the difference between `7f80007b` and `7fc0007b`!)
-
-        """
-        vf = struct.unpack('>I', bs)[0]
-        if (vf & 0x7f800000) == 0x7f800000:
-            # NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
-            sign = vf >> 31
-            payload = vf & 0x007fffff
-            dbs = struct.pack('>Q', (sign << 63) | 0x7ff0000000000000 | (payload << 29))
-            return Float(struct.unpack('>d', dbs)[0])
-        else:
-            return Float(struct.unpack('>f', bs)[0])
-
 # FIXME: This regular expression is conservatively correct, but Anglo-chauvinistic.
 RAW_SYMBOL_RE = re.compile(r'^[-a-zA-Z0-9~!$%^&*?_=+/.]+$')
 
@@ -183,6 +54,10 @@ class Symbol(object):
     #xyz
     >>> Symbol('xyz').name
     'xyz'
+    >>> repr(Symbol('xyz'))
+    '#xyz'
+    >>> str(Symbol('xyz'))
+    'xyz'
     >>> import preserves
     >>> preserves.stringify(Symbol('xyz'))
     'xyz'
@@ -228,6 +103,9 @@ class Symbol(object):
     def __repr__(self):
         return '#' + self.name
 
+    def __str__(self):
+        return self.name
+
     def __preserve_write_binary__(self, encoder):
         bs = self.name.encode('utf-8')
         encoder.buffer.append(0xb3)
@@ -289,7 +167,7 @@ class Record(object):
         return self.__hash
 
     def __repr__(self):
-        return str(self.key) + '(' + ', '.join((repr(f) for f in self.fields)) + ')'
+        return repr(self.key) + '(' + ', '.join((repr(f) for f in self.fields)) + ')'
 
     def __preserve_write_binary__(self, encoder):
         encoder.buffer.append(0xb4)
@@ -432,7 +310,7 @@ class RecordConstructorInfo(object):
         return self.__hash
 
     def __repr__(self):
-        return str(self.key) + '/' + str(self.arity)
+        return repr(self.key) + '/' + str(self.arity)
 
 # Blub blub blub
 class ImmutableDict(dict):
@@ -697,7 +575,7 @@ class Embedded:
     >>> import io
     >>> e = Embedded(io.StringIO('some text'))
     >>> e                                        # doctest: +ELLIPSIS
-    #!<_io.StringIO object at ...>
+    #:<_io.StringIO object at ...>
     >>> e.embeddedValue                          # doctest: +ELLIPSIS
     <_io.StringIO object at ...>
 
@@ -710,7 +588,7 @@ class Embedded:
       ...
     TypeError: Cannot preserves-format: None
     >>> print(preserves.stringify(Embedded(None), format_embedded=lambda x: 'abcdef'))
-    #!"abcdef"
+    #:"abcdef"
 
     ```
 
@@ -732,12 +610,12 @@ class Embedded:
         return hash(self.embeddedValue)
 
     def __repr__(self):
-        return '#!%r' % (self.embeddedValue,)
+        return '#:%r' % (self.embeddedValue,)
 
     def __preserve_write_binary__(self, encoder):
         encoder.buffer.append(0x86)
         encoder.append(encoder.encode_embedded(self.embeddedValue))
 
     def __preserve_write_text__(self, formatter):
-        formatter.chunks.append('#!')
+        formatter.chunks.append('#:')
         formatter.append(formatter.format_embedded(self.embeddedValue))

implementations/python/pyproject.toml

@@ -0,0 +1,45 @@
+[project]
+name = "preserves"
+version = "0.994.0"
+description = "Data serialization format"
+readme = "README.md"
+requires-python = ">=3.6, <4"
+license = {text = "Apache-2.0"}
+
+authors = [
+    {name = "Tony Garnock-Jones", email = "tonyg@leastfixedpoint.com"},
+]
+
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+]
+
+[project.urls]
+Homepage = "https://preserves.dev/"
+Issues = "https://gitlab.com/preserves/preserves/-/issues"
+
+[project.optional-dependencies]
+dev = [
+    "build",
+    "coverage",
+    "mkdocs",
+    "mkdocstrings[python]",
+    "mkdocs-material",
+    "mkdocs-macros-plugin",
+    "mkdocs-git-revision-date-localized-plugin",
+    "twine",
+]
+
+[tool.setuptools]
+packages = ["preserves"]
+
+[tool.setuptools.package-data]
+preserves = ["*"]
+
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"

implementations/python/setup.py

@@ -1,25 +0,0 @@
-from setuptools import setup
-
-setup(
-    name="preserves",
-    version="0.992.0",
-    author="Tony Garnock-Jones",
-    author_email="tonyg@leastfixedpoint.com",
-    license="Apache Software License",
-    classifiers=[
-        "Development Status :: 4 - Beta",
-        "Intended Audience :: Developers",
-        "Topic :: Software Development :: Libraries :: Python Modules",
-        "License :: OSI Approved :: Apache Software License",
-        "Programming Language :: Python :: 3",
-    ],
-    packages=["preserves"],
-    url="https://preserves.dev/",
-    description="Data serialization format",
-    long_description=open("README.md").read(),
-    long_description_content_type="text/markdown",
-    install_requires=[],
-    python_requires=">=3.6, <4",
-    setup_requires=['setuptools_scm'],
-    include_package_data=True,
-)

implementations/python/single-precision.md

@@ -0,0 +1,73 @@
+The code below deals with expansion of single-precision IEEE 754 floating point values to
+double-precision (and vice-versa) without losing detail of NaN bit-patterns.
+
+```python
+def to_bytes(self):
+    """Converts this 32-bit single-precision floating point value to its binary32 format,
+    taking care to preserve the quiet/signalling bit-pattern of NaN values, unlike its
+    `struct.pack('>f', ...)` equivalent.
+
+    ```python
+    >>> Float.from_bytes(b'\\x7f\\x80\\x00{')
+    Float(nan)
+    >>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
+    b'\\x7f\\x80\\x00{'
+
+    >>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
+    nan
+    >>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
+    b'\\x7f\\xc0\\x00{'
+    >>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
+    b'\\x7f\\xc0\\x00{'
+
+    ```
+
+    (Note well the difference between `7f80007b` and `7fc0007b`!)
+
+    """
+
+    if math.isnan(self.value) or math.isinf(self.value):
+        dbs = struct.pack('>d', self.value)
+        vd = struct.unpack('>Q', dbs)[0]
+        sign = vd >> 63
+        payload = (vd >> 29) & 0x007fffff
+        vf = (sign << 31) | 0x7f800000 | payload
+        return struct.pack('>I', vf)
+    else:
+        return struct.pack('>f', self.value)
+
+@staticmethod
+def from_bytes(bs):
+    """Converts a 4-byte-long byte string to a 32-bit single-precision floating point value
+    wrapped in a [Float][preserves.values.Float] instance. Takes care to preserve the
+    quiet/signalling bit-pattern of NaN values, unlike its `struct.unpack('>f', ...)`
+    equivalent.
+
+    ```python
+    >>> Float.from_bytes(b'\\x7f\\x80\\x00{')
+    Float(nan)
+    >>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
+    b'\\x7f\\x80\\x00{'
+
+    >>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
+    nan
+    >>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
+    b'\\x7f\\xc0\\x00{'
+    >>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
+    b'\\x7f\\xc0\\x00{'
+
+    ```
+
+    (Note well the difference between `7f80007b` and `7fc0007b`!)
+
+    """
+    vf = struct.unpack('>I', bs)[0]
+    if (vf & 0x7f800000) == 0x7f800000:
+        # NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
+        sign = vf >> 31
+        payload = vf & 0x007fffff
+        dbs = struct.pack('>Q', (sign << 63) | 0x7ff0000000000000 | (payload << 29))
+        return Float(struct.unpack('>d', dbs)[0])
+    else:
+        return Float(struct.unpack('>f', bs)[0])
+```

implementations/python/tests/samples.pr

@@ -56,6 +56,12 @@
   annotation8a: @"Commas forbidden between @ and annotation in a collection" <ParseError "[@,a b]">
   annotation9: @"Commas forbidden between annotation and underlying" <ParseError "@a, b">
   annotation9a: @"Commas forbidden between annotation and underlying in a collection" <ParseError "[@a, b]">
+  annotation10: @"Line comment that's just hash-newline" <Test #x"85 B100 B000" #
+0>
+  annotation11: @"Line comments: one hash-newline, one normal"
+    <Test #x"85 B100 85 B1066e6f726d616c B000" #
+# normal
+0>
   bytes2: <Test #x"B20568656c6c6f" #"hello">
   bytes2a: <Test @"Internal whitespace is allowed, not including commas" #x"B2 05 68 65 6c 6c 6f" #"hello">
   bytes2b: @"Commas forbidden in internal whitespace" <ParseError "#x\"B2, 05, 68, 65, 6c, 6c, 6f\"">
@@ -91,6 +97,12 @@
   double+0: <Test #x"87080000000000000000" +0.0>
   double-0: <Test #x"87088000000000000000" -0.0>
   double1: <Test #x"87083ff0000000000000" 1.0>
+  double1a: <Test #x"87083ff0000000000000" 1e0>
+  double1b: <Test #x"87083ff0000000000000" 1.0e0>
+  double1c: <Test #x"87083ff0000000000000" 1e-0>
+  double1d: <Test #x"87083ff0000000000000" 1.0e-0>
+  double1e: <Test #x"87083ff0000000000000" 1e+0>
+  double1f: <Test #x"87083ff0000000000000" 1.0e+0>
   double2: <Test #x"8708fe3cb7b759bf0426" -1.202e300>
   double3: <Test #x"8708123456789abcdef0" #xd"12 34 56 78  9a bc de f0">
   double4: @"Fewer than 16 digits" <ParseError "#xd\"12345678\"">
@@ -107,25 +119,6 @@
   double15: @"-sNaN" <Test #x"8708fff8000000000111" #xd"fff8000000000111">
   double16: @"+sNaN" <Test #x"87087ff8000000000001" #xd"7ff8000000000001">
   double17: @"+sNaN" <Test #x"87087ff8000000000111" #xd"7ff8000000000111">
-  float0: <Test #x"870400000000" 0.0f>
-  float+0: <Test #x"870400000000" +0.0f>
-  float-0: <Test #x"870480000000" -0.0f>
-  float1: <Test #x"87043f800000" 1.0f>
-  float2: <Test #x"870412345678" #xf"12 34  56 78">
-  float3: @"Fewer than 8 digits" <ParseError "#xf\"123456\"">
-  float4: @"More than 8 digits" <ParseError "#xf\"123456789a\"">
-  float5: @"Invalid chars" <ParseError "#xf\"12zz5678\"">
-  float6: @"Positive infinity" <Test #x"87047f800000" #xf"7f800000">
-  float7: @"Negative infinity" <Test #x"8704ff800000" #xf"ff800000">
-  float8: @"+sNaN" <Test #x"87047f800001" #xf"7f800001">
-  float9: @"+sNaN" <Test #x"87047f800111" #xf"7f800111">
-  float10: @"-sNaN" <Test #x"8704ff800001" #xf"ff800001">
-  float11: @"-sNaN" <Test #x"8704ff800111" #xf"ff800111">
-  float12: @"Bad spacing" <ParseError "#xf\"12345 678\"">
-  float13: @"+qNaN" <Test #x"87047fc00001" #xf"7fc00001">
-  float14: @"+qNaN" <Test #x"87047fc00111" #xf"7fc00111">
-  float15: @"-qNaN" <Test #x"8704ffc00001" #xf"ffc00001">
-  float16: @"-qNaN" <Test #x"8704ffc00111" #xf"ffc00111">
   int-98765432109876543210987654321098765432109: <Test #x"b012feddc125aed4226c770369269596ce3f0ad3" -98765432109876543210987654321098765432109>
   int-12345678123456781234567812345678: <Test #x"b00eff642cf6684f11d1dad08c4a10b2" -12345678123456781234567812345678>
   int-1234567812345678123456781234567: <Test #x"b00df06ae570d4b4fb62ae746dce79" -1234567812345678123456781234567>
@@ -174,9 +167,9 @@
   list11: <Test #x"b5b0010184" [01]>
   list12: <Test #x"b5b0010c84" [12]>
   noinput0: @"No input at all" <DecodeEOF #x"">
-  embed0: <Test #x"86b000" #!0>
-  embed1: <Test #x"8686b000" #!#!0>
-  embed2: <Test #x"b586b00086b10568656c6c6f84" [#!0 #!"hello"]>
+  embed0: <Test #x"86b000" #:0>
+  embed1: <Test #x"8686b000" #:#:0>
+  embed2: <Test #x"b586b00086b10568656c6c6f84" [#:0 #:"hello"]>
   record1: <Test #x"b4 b30763617074757265 b4 b30764697363617264 84 84" <capture <discard>>>
   record2: <Test #x"b4 b3076f627365727665 b4 b305737065616b b4 b30764697363617264 84 b4 b30763617074757265 b4 b30764697363617264 84 84 84 84" <observe <speak <discard> <capture <discard>>>>>
   record2a: @"Commas not allowed in records" <ParseError "<observe <speak <discard>, <capture <discard>>>>">

implementations/python/tests/test_preserves.py

@@ -135,7 +135,6 @@ class BinaryCodecTests(PreservesTestCase):
         self._roundtrip(131072, _buf(0xb0, 0x03, 0x02, 0x00, 0x00))
 
     def test_floats(self):
-        self._roundtrip(Float(1.0), _buf(0x87, 0x04, 0x3f, 0x80, 0, 0))
         self._roundtrip(1.0, _buf(0x87, 0x08, 0x3f, 0xf0, 0, 0, 0, 0, 0, 0))
         self._roundtrip(-1.202e300, _buf(0x87, 0x08, 0xfe, 0x3c, 0xb7, 0xb7, 0x59, 0xbf, 0x04, 0x26))
 

implementations/racket/preserves/preserves-schema/gen/schema.rkt

@@ -14,7 +14,6 @@
   (define :encode-embedded values)
   (define (AtomKind? p)
     (or (AtomKind-Boolean? p)
-        (AtomKind-Float? p)
         (AtomKind-Double? p)
         (AtomKind-SignedInteger? p)
         (AtomKind-String? p)
@@ -29,15 +28,6 @@
    ((define/generic *->preserve ->preserve)
     (define (->preserve preservable)
       (match preservable ((AtomKind-Boolean) 'Boolean)))))
-  (struct
-   AtomKind-Float
-   ()
-   #:transparent
-   #:methods
-   gen:preservable
-   ((define/generic *->preserve ->preserve)
-    (define (->preserve preservable)
-      (match preservable ((AtomKind-Float) 'Float)))))
   (struct
    AtomKind-Double
    ()
@@ -87,7 +77,6 @@
     (match
      input
      ((and dest 'Boolean) (AtomKind-Boolean))
-     ((and dest 'Float) (AtomKind-Float))
      ((and dest 'Double) (AtomKind-Double))
      ((and dest 'SignedInteger) (AtomKind-SignedInteger))
      ((and dest 'String) (AtomKind-String))

implementations/racket/preserves/preserves-schema/methods.rkt

@@ -27,8 +27,7 @@
                     (define (->preserve preservable)
                       (for/hash [((k v) (in-hash preservable))]
                         (values (*->preserve k) (*->preserve v))))])
-  #:defaults ([float? (define (->preserve preservable) preservable)]
-              [embedded? (define (->preserve preservable) preservable)]
+  #:defaults ([embedded? (define (->preserve preservable) preservable)]
               [record?
                (define/generic *->preserve ->preserve)
                (define (->preserve preservable)

implementations/racket/preserves/preserves-schema/parser.rkt

@@ -26,7 +26,6 @@
      (maybe-dest dest-pat-stx
                  `(? ,(match atom-kind
                         [(AtomKind-Boolean) 'boolean?]
-                        [(AtomKind-Float) 'float?]
                         [(AtomKind-Double) 'flonum?]
                         [(AtomKind-SignedInteger) 'exact-integer?]
                         [(AtomKind-String) 'string?]

implementations/racket/preserves/preserves-schema/reader.rkt

@@ -80,9 +80,8 @@
 
 (define (namelike v)
   (match v
-    [(? string? s) (string->symbol s)]
     [(? symbol? s) s]
-    [(? number? n) (string->symbol (number->string n))]
+    [(? string? s) (string->symbol s)]
     [(? boolean? b) (if b 'true 'false)]
     [_ #f]))
 
@@ -132,7 +131,6 @@
     (match (peel-annotations item)
       ['any (ks (SimplePattern-any))]
       ['bool (ks (SimplePattern-atom (AtomKind-Boolean)))]
-      ['float (ks (SimplePattern-atom (AtomKind-Float)))]
       ['double (ks (SimplePattern-atom (AtomKind-Double)))]
       ['int (ks (SimplePattern-atom (AtomKind-SignedInteger)))]
       ['string (ks (SimplePattern-atom (AtomKind-String)))]
@@ -194,9 +192,9 @@
                       values
                       (lambda () (error 'parse-simple-dsl "Compound patterns not accepted here: ~a" (input->string stx)))))
 
-  (define ((maybe-named* knamed kanonymous recur [literal-name #f]) b)
+  (define ((maybe-named* knamed kanonymous recur [literal-name '#:none]) b)
     (define n (or (find-name b)
-                  (and (valid-id? literal-name) literal-name)))
+                  (let ((n (namelike literal-name))) (and (valid-id? n) n))))
     (if n
         (let ((p (parse-simple-dsl
                   b

implementations/racket/preserves/preserves-schema/schema.prs

@@ -38,10 +38,10 @@ SimplePattern =
   # any
   / =any
 
-  # special builtins: bool, float, double, int, string, bytes, symbol
+  # special builtins: bool, double, int, string, bytes, symbol
   / <atom @atomKind AtomKind>
 
-  # matches an embedded value in the input: #!p
+  # matches an embedded value in the input: #:p
   / <embedded @interface SimplePattern>
 
   # =symbol, <<lit> any>, or plain non-symbol atom
@@ -79,7 +79,7 @@ CompoundPattern =
 
 DictionaryEntries = { any: NamedSimplePattern ...:... }.
 
-AtomKind = =Boolean / =Float / =Double / =SignedInteger / =String / =ByteString / =Symbol .
+AtomKind = =Boolean / =Double / =SignedInteger / =String / =ByteString / =Symbol .
 
 NamedAlternative = [@variantLabel string @pattern Pattern].
 

implementations/racket/preserves/preserves-schema/unparser.rkt

@@ -14,7 +14,6 @@
   (match (unwrap pattern)
     [(Binding n p) (pattern->unparser p (escape n))]
     [(SimplePattern-any) `(*->preserve ,src-stx)]
-    [(SimplePattern-atom (AtomKind-Float)) `(->float (*->preserve ,src-stx))]
     [(SimplePattern-atom (AtomKind-Double)) `(exact->inexact (*->preserve ,src-stx))]
     [(SimplePattern-atom _) `(*->preserve ,src-stx)]
     [(SimplePattern-embedded _interface) `(embedded ,src-stx)]

implementations/racket/preserves/preserves/block-write.rkt

@@ -0,0 +1,88 @@
+#lang racket/base
+
+(provide emit
+         (struct-out vertical-mode)
+         (struct-out separated)
+         (struct-out grouped))
+
+(require racket/match)
+(require (only-in racket/string string-trim))
+(require (only-in racket/pretty pretty-print-columns))
+(require (only-in racket/port call-with-output-string))
+
+;; mode ∈ {'never, 'variable, 'always, 'extra-newline}
+;;
+;; Things that don't know how to write themselves vertically always write horizontally.
+;; Otherwise, mode is relevant:
+;;
+;;  - 'never: always horizontally
+;;  - 'variable: horizontally, unless it doesn't fit, in which case vertically
+;;  - 'always: always vertically
+;;  - 'extra-newline: as 'normal, but with extra newlines separating items
+;;
+(struct vertical-mode (mode item) #:transparent)
+
+(struct separated (items separator terminator) #:transparent)
+(struct grouped (items separator terminator opener closer) #:transparent)
+
+(struct emitter (port indentation) #:transparent)
+
+(define (emit-seq op vs sep ter cl e indent mode)
+  (define p (emitter-port e))
+  (when op (display op p))
+  (let loop ((vs vs) (need-sep #f))
+    (match vs
+      ['()
+       (when need-sep (display ter p))
+       (when cl (display cl p))]
+      [(cons v vs)
+       (when need-sep (display sep p))
+       (emit* v e indent mode)
+       (loop vs #t)])))
+
+(define (emit-vertical op vs sep ter cl e indent mode)
+  (define inner-indent (if op (string-append indent (emitter-indentation e)) indent))
+  (let ((sep (string-append (trim-delimiter-right sep) inner-indent))
+        (ter (trim-delimiter-right ter))
+        (op (and op (string-append (trim-delimiter-right op) inner-indent)))
+        (cl (and cl (string-append indent (trim-delimiter-left cl)))))
+    (emit-seq op vs sep ter cl e inner-indent mode)))
+
+(define (trim-delimiter-right s)
+  (string-trim #:left? #f s #px"[ \t\f\v]+"))
+
+(define (trim-delimiter-left s)
+  (string-trim #:right? #f s #px"[ \t\f\v]+"))
+
+(define (try-horizontal op vs sep ter cl e indent mode)
+  (call-with-output-string
+   (lambda (p)
+     (emit-seq op vs sep ter cl (emitter p (emitter-indentation e)) indent 'never))))
+
+(define (emit-separated op vs sep ter cl e indent mode)
+  (match mode
+    ['never (emit-seq op vs sep ter cl e indent mode)]
+    ['variable (let ((s (try-horizontal op vs sep ter cl e indent mode)))
+                 (if (> (string-length s) (pretty-print-columns))
+                     (emit-vertical op vs sep ter cl e indent mode)
+                     (display s (emitter-port e))))]
+    ['always (emit-vertical op vs sep ter cl e indent mode)]
+    ['extra-newline (let ((sep (string-append sep "\n")))
+                      (emit-vertical op vs sep ter cl e indent mode))]))
+
+(define (emit* v e indent mode)
+  (define (w v) (emit* v e indent mode))
+  (match v
+    [(? string? s) (display s (emitter-port e))]
+    [(? list? vs) (map w vs)]
+    [(separated vs sep ter) (emit-separated #f vs sep ter #f e indent mode)]
+    [(grouped vs sep ter op cl) (emit-separated op vs sep ter cl e indent mode)]
+    [(vertical-mode new-mode v) (emit* v e indent new-mode)]))
+
+(define (emit v [o (current-output-port)]
+              #:mode [initial-mode 'variable]
+              #:indentation [indentation0 "    "]
+              #:initial-indent [initial-indent "\n"])
+  ;; (local-require racket/pretty) (pretty-write v)
+  (define indentation (if (number? indentation0) (make-string indentation0 #\space) indentation0))
+  (emit* v (emitter o indentation) initial-indent initial-mode))

implementations/racket/preserves/preserves/float-bytes.rkt

@@ -9,7 +9,6 @@
          bytes->double
          double->bytes)
 
-(require "float.rkt")
 (require (only-in racket/math nan? infinite?))
 
 (module binary racket/base
@@ -37,21 +36,20 @@
                               #x0070000000000000
                               (arithmetic-shift payload 29)))
              (dbs (integer->integer-bytes vd 8 #f #t)))
-        (float (floating-point-bytes->real dbs #t 0 8)))
-      (float (floating-point-bytes->real bs #t 0 4))))
+        (floating-point-bytes->real dbs #t 0 8))
+      (floating-point-bytes->real bs #t 0 4)))
 
 (define (float->bytes v)
-  (let ((v (float-value v)))
-    (if (or (nan? v) (infinite? v))
-        (let* ((dbs (real->floating-point-bytes v 8 #t))
-               (vd (integer-bytes->integer dbs #f #t))
-               (signexp (bitwise-bit-field vd 55 64))
-               (payload (bitwise-bit-field vd 29 52))
-               (vf (bitwise-ior (arithmetic-shift signexp 23)
-                                payload))
-               (bs (integer->integer-bytes vf 4 #f #t)))
-          bs)
-        (real->floating-point-bytes v 4 #t))))
+  (if (or (nan? v) (infinite? v))
+      (let* ((dbs (real->floating-point-bytes v 8 #t))
+             (vd (integer-bytes->integer dbs #f #t))
+             (signexp (bitwise-bit-field vd 55 64))
+             (payload (bitwise-bit-field vd 29 52))
+             (vf (bitwise-ior (arithmetic-shift signexp 23)
+                              payload))
+             (bs (integer->integer-bytes vf 4 #f #t)))
+        bs)
+      (real->floating-point-bytes v 4 #t)))
 
 (define (bytes->double bs)
   (floating-point-bytes->real bs #t 0 8))

implementations/racket/preserves/preserves/float.rkt

@@ -1,14 +0,0 @@
-#lang racket/base
-;; Wrapper struct to mark a need for 32-bit IEEE floating-point
-;; precision (de)serialization. In many circumstances, Racket lacks
-;; 32-bit floating point support, and single-flonum? always yields #f.
-
-(provide (struct-out float)
-         ->float)
-
-(struct float (value) #:transparent)
-
-(define (->float v)
-  (if (float? v)
-      v
-      (float (exact->inexact v))))

implementations/racket/preserves/preserves/jelly.rkt

@@ -8,7 +8,7 @@
 ;;---------------------------------------------------------------------------
 ;; Representing values
 
-(require "float.rkt" "float-bytes.rkt")
+(require "float-bytes.rkt")
 (struct record (label fields) #:transparent)
 (struct annotated (annotation item) #:transparent)
 (struct embedded (value) #:transparent)
@@ -26,9 +26,7 @@
         [#x84 '#:end]
         [#x85 (let ((a (next))) (annotated a (next)))]
         [#x86 (embedded (next))]
-        [#x87 (match (next-byte)
-                [4 (bytes->float (next-bytes 4))]
-                [8 (bytes->double (next-bytes 8))])]
+        [#x87 (match (next-byte) [8 (bytes->double (next-bytes 8))])]
         [#xB0 (next-integer (next-varint))]
         [#xB1 (bytes->string/utf-8 (next-bytes (next-varint)))]
         [#xB2 (next-bytes (next-varint))]
@@ -74,7 +72,6 @@
     (match v
       [#f (write-byte #x80 out-port)]
       [#t (write-byte #x81 out-port)]
-      [(float _) (write-byte #x87 out-port) (write-byte 4 out-port) (output-bytes (float->bytes v))]
       [(? flonum?) (write-byte #x87 out-port) (write-byte 8 out-port) (output-bytes (double->bytes v))]
 
       [(annotated a v) (write-byte #x85 out-port) (output a) (output v)]

implementations/racket/preserves/preserves/main.rkt

@@ -2,11 +2,11 @@
 ;; Preserve, as in Fruit Preserve, as in a remarkably weak pun on pickling/dehydration etc
 
 (provide (all-from-out "record.rkt")
-         (all-from-out "float.rkt")
          (all-from-out "annotation.rkt")
          (all-from-out "order.rkt")
          (all-from-out "embedded.rkt")
          (all-from-out "merge.rkt")
+         (all-from-out "pexprs.rkt")
 
          (all-from-out "read-binary.rkt")
          (all-from-out "read-text.rkt")
@@ -18,7 +18,10 @@
          detect-preserve-syntax
          read-preserve
          port->preserves
-         file->preserves)
+         file->preserves
+         port->pexprs
+         file->pexprs
+         string->pexprs)
 
 (require racket/dict)
 (require racket/match)
@@ -27,11 +30,11 @@
 (require (only-in racket/port port->list))
 
 (require "record.rkt")
-(require "float.rkt")
 (require "annotation.rkt")
 (require "order.rkt")
 (require "embedded.rkt")
 (require "merge.rkt")
+(require "pexprs.rkt")
 
 (require "read-binary.rkt")
 (require "read-text.rkt")
@@ -79,7 +82,7 @@
                                     #:decode-embedded decode-embedded
                                     #:source path))))
 
-(define (port->preserves in-port
+(define (port->preserves [in-port (current-input-port)]
                          #:read-syntax? [read-syntax? #f]
                          #:decode-embedded [decode-embedded #f]
                          #:source [source (object-name in-port)])
@@ -89,3 +92,34 @@
                                #:decode-embedded decode-embedded
                                #:source source))
               in-port))
+
+(define (file->pexprs path
+                      #:read-syntax? [read-syntax? #f]
+                      #:decode-embedded [decode-embedded #f])
+  (call-with-input-file path (lambda (p) (port->pexprs p
+                                                       #:read-syntax? read-syntax?
+                                                       #:decode-embedded decode-embedded))))
+
+(define (port->pexprs [in-port (current-input-port)]
+                      #:read-syntax? [read-syntax? #f]
+                      #:decode-embedded [decode-embedded #f]
+                      #:source [source (object-name in-port)])
+  (define vs
+    (port->list (lambda (in-port)
+                  (read-pexpr in-port
+                              #:read-syntax? read-syntax?
+                              #:decode-embedded decode-embedded
+                              #:source source))
+                in-port))
+  (if read-syntax?
+      vs
+      (remove-trailer vs)))
+
+(define (string->pexprs s
+                        #:read-syntax? [read-syntax? #f]
+                        #:decode-embedded [decode-embedded #f]
+                        #:source [source "<string>"])
+  (port->pexprs (open-input-string s)
+                #:read-syntax? read-syntax?
+                #:decode-embedded decode-embedded
+                #:source source))