preserves/preserves-expressions.md

title
P-expressions

Tony Garnock-Jones tonyg@leastfixedpoint.com
October 2023. Version 0.1.2.

This document defines a grammar called Preserves Expressions (P-expressions, pexprs) that includes ordinary Preserves text syntax but offers extensions sufficient to support a Lisp- or Haskell-like programming notation.

Motivation. The text syntax for Preserves works well for writing Values, i.e. data. However, in some contexts, Preserves applications need a broader grammar that allows interleaving of expressions with data. Two examples are the Preserves Schema language and the Synit configuration scripting language, both of which (ab)use Preserves text syntax as a kind of programming notation.

Preliminaries

The P-expression grammar takes the text syntax grammar as its base and modifies it.

**Whitespace.** Whitespace is redefined as any number of spaces, tabs, carriage returns, or line feeds. Commas are *not* considered whitespace in P-expressions.

            ws = *(%x20 / %x09 / CR / LF)

Delimiters. Because commas are no longer included in class ws, class delimiter is widened to include them explicitly.

     delimiter = ws / ","
               / "<" / ">" / "[" / "]" / "{" / "}"
               / "#" / ":" / DQUOTE / "|" / "@" / ";"

Grammar

P-expressions add comma, semicolon, and sequences of one or more colons to the syntax class Value.

        Value =/ Comma / Semicolon / Colons
         Comma = ","
     Semicolon = ";"
        Colons = 1*":"

Now that colon is in Value, the syntax for Dictionary is replaced with Block everywhere it is mentioned.

         Block = "{" *Value ws "}"

Syntax for Record is loosened to allow empty angle brackets.

        Record = "<" *Value ws ">"

New syntax for explicit uninterpreted grouping of sequences of values is introduced, and added to class Value.

        Value =/ ws Group
         Group = "(" *Value ws ")"

Finally, class Document is replaced in order to allow standalone documents to directly comprise a sequence of multiple values.

      Document = *Value ws

No changes to the Preserves semantic model are made. Every Preserves text-syntax term is a valid P-expression, but in general P-expressions must be rewritten or otherwise interpreted before a meaningful Preserves value can be arrived at (see below).

Annotations and Comments

Annotations and comments attach to the term following them, just as in the ordinary text syntax. However, it is common in programming notations to allow comments at the end of a file or other sequential construct:

{
    key: value
    # example of a comment at the end of a dictionary
}
# example of a comment at the end of the input file

While the ordinary text syntax forbids comments in these positions, P-expressions allow them:

     Document =/ *Value Trailer ws
       Record =/  "<" *Value Trailer ws ">"
     Sequence =/  "[" *Value Trailer ws "]"
          Set =/ "#{" *Value Trailer ws "}"
        Block =/  "{" *Value Trailer ws "}"

       Trailer = 1*Annotation

Encoding P-expressions as Preserves

We write ⌜p⌝ for the encoding into Preserves of P-expression p.

{:.pseudocode.equations} | ⌜·⌝ : P-expression | ⟶ | Preserves |

Aside from the special classes Group, Block, Comma, Semicolon, Colons, Trailer, or empty Record, P-expressions are encoded directly as Preserves data.

{:.pseudocode.equations} | ⌜[p ...]⌝ | = | [⌜p⌝ ...] | | ⌜<p ...>⌝ | = | <⌜p⌝ ...> | | ⌜#{p ...}⌝ | = | #{⌜p⌝ ...} | | ⌜#!p⌝ | = | #!⌜p⌝ | | ⌜@p q⌝ | = | @⌜p⌝ ⌜q⌝ | | ⌜p⌝ | = | p when p ∈ Atom |

Everything else is encoded as Preserves dictionaries¹.

{:.pseudocode.equations} | ⌜<>⌝ | = | {r:[]} | | ⌜(p ...)⌝ | = | {g:[⌜p⌝ ...]} | | ⌜{p ...}⌝ | = | {b:[⌜p⌝ ...]} | | ⌜,⌝ | = | {s:|,|} | | ⌜;⌝ | = | {s:|;|} | | ⌜: ...⌝ | = | {s:|: ...|} | | ⌜t⌝ | = | ⌜a⌝ ... {}, where a ... are the annotations in t and t ∈ Trailer |

The empty dictionary {} acts as an anchor for the annotations in a Trailer.

We overload the ⌜·⌝ notation for encoding whole Documents into sequences of Preserves values.

{:.pseudocode.equations} | ⌜·⌝ : P-expression Document | ⟶ | Preserves Sequence | | ⌜p ...⌝ | = | [⌜p⌝ ...] |

Interpreting P-expressions as Preserves

The previous section discussed ways of representing P-expressions using Preserves. Here, we discuss interpreting P-expressions as Preserves, so that (1) a Preserves datum (2) written using Preserves text syntax and then (3) read as a P-expression can be (4) interpreted from that P-expression to yield the original datum.

A reader for P-expressions can be adapted to yield a reader for Preserves terms by processing (subterms of) each P-expression that the reader produces. The only subterms that need processing are the special classes mentioned above.

1. Every Group or Semicolon that appears is an error.
2. Every Colons with two or more colons in it is an error.
3. Every Comma that appears is discarded.
4. Every Trailer that appears is an error.²
5. Every Record with no values in it is an error.
6. Every Block must contain triplets of Value, Colons (with a single colon), Value. Any Block not following this pattern is an error. Each Block following the pattern is translated to a Dictionary containing a key/value pair for each triplet.

Appendix: Examples

Examples are given as pairs of P-expressions and their Preserves text-syntax encodings.

Individual P-expression Values

 ⌜<date 1821 (lookup-month "February") 3>⌝
= <date 1821 {g:[lookup-month "February"]} 3>

 ⌜<>⌝
= {r:[]}

 ⌜(begin (println! (+ 1 2)) (+ 3 4))⌝
= {g:[begin {g:[println! {g:[+ 1 2]}]} {g:[+ 3 4]}]}

 ⌜()⌝
= {g:[]}

 ⌜[() () ()]⌝
= [{g:[]}, {g:[]}, {g:[]}]

 ⌜{
      setUp();
      # Now enter the loop
      loop: {
          greet("World");
      }
      tearDown();
  }⌝
= {b:[
      setUp {g:[]} {s:|;|}
      # Now enter the loop
      loop {s:|:|} {b:[
          greet {g:["World"]} {s:|;|}
      ]}
      tearDown {g:[]} {s:|;|}
  ]}

 ⌜[1 + 2.0, print "Hello", predicate: #t, foo, #!remote, bar]⌝
= [1 + 2.0 {s:|,|} print "Hello" {s:|,|} predicate {s:|:|} #t {s:|,|}
   foo {s:|,|} #!remote {s:|,|} bar]

 ⌜{
      optional name: string,
      address: Address,
  }⌝
= {b:[
      optional name {s:|:|} string {s:|,|}
      address {s:|:|} Address {s:|,|}
  ]}

Whole Documents

 ⌜{
      key: value
      # example of a comment at the end of a dictionary
  }
  # example of a comment at the end of the input file⌝
= [ {b:[
        key {s:|:|} value
        @"example of a comment at the end of a dictionary" {}
    ]}
    @"example of a comment at the end of the input file"
    {}
  ]

Appendix: Reading vs. Parsing

Lisp systems first read streams of bytes into S-expressions and then parse those S-expressions into more abstract structures denoting various kinds of program syntax. Separation of reading from parsing is what gives Lisp its syntactic flexibility.

Similarly, the Apple programming language Dylan included a reader-parser split, with the Dylan reader producing D-expressions that are somewhat similar to P-expressions.

Finally, the Racket dialects Honu and Something use a reader-parser-macro setup, where the reader produces Racket data, the parser produces "syntax" and is user-extensible, and Racket's own modular macro system rewrites this "syntax" down to core forms to be compiled to machine code.

Similarly, when using P-expressions as the foundation for a language, a generic P-expression reader can then feed into special-purpose parsers. The reader captures the coarse syntactic structure of a program, and the parser refines this.

Often, a parser will wish to extract structure from sequences of P-expression Values.

• A simple technique is repeated splitting of sequences; first by Semicolon, then by Comma, then by increasingly high binding-power operators.

• More refined is to use a Pratt parser or similar (1, 2, 3) to build a parse tree using an extensible specification of the pre-, in-, and postfix operators involved.

• Finally, if you treat sequences of Values as pre-lexed token streams, almost any parsing formalism (such as PEG parsing, Ometa, etc.) can be used to extract further syntactic structure.

Notes

---

1. In principle, it would be nice to use records for this purpose, but if we did so we would have to also encode usages of records! ↩︎

2. Implementation note. When implementing parsing of P-expressions into Preserves, consider offering an optional mode where trailing annotations Trailer are discarded instead of causing an error to be signalled. ↩︎