Compare commits

...

8 Commits

Author SHA1 Message Date
Tony Garnock-Jones 73c6593f84 Python 0.994 docs 2024-02-05 23:01:01 +01:00
Tony Garnock-Jones a9e226f759 Bump python version to 0.994.0 2024-02-05 22:58:16 +01:00
Tony Garnock-Jones 33db0b8718 Publish
- @preserves/core@0.994.0
 - @preserves/schema-cli@0.994.0
 - @preserves/schema@0.994.0
2024-02-05 22:55:15 +01:00
Tony Garnock-Jones e923d87fa5 Switch from `#!` to `#:` for embedded values 2024-02-05 22:38:49 +01:00
Tony Garnock-Jones 83697b0e56 Update docs 2024-01-28 18:49:18 +01:00
Tony Garnock-Jones 1798e64615 Remove split-out Rust implementation 2024-01-28 15:18:08 +01:00
Tony Garnock-Jones be32f9b7c8 Remove single-precision floats from the implementations 2024-01-27 14:40:37 +01:00
Tony Garnock-Jones dc1b0ac54d Remove single-precision floats from the specs 2024-01-27 11:34:51 +01:00
298 changed files with 44807 additions and 19431 deletions

View File

@ -22,4 +22,3 @@ test-all:
(cd implementations/javascript; npm test)
(cd implementations/python; make test)
(cd implementations/racket/preserves; make testonly)
(cd implementations/rust; cargo test)

View File

@ -38,14 +38,14 @@ automatic, perfect-fidelity conversion between syntaxes.
#### Implementations of the data model, plus Preserves textual and binary transfer syntax
| Language[^pre-alpha-implementations] | Code | Package | Docs |
|-----------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------|-------------------------------------------|
| Nim | [git.syndicate-lang.org](https://git.syndicate-lang.org/ehmry/preserves-nim) | | |
| Python | [preserves.dev]({{page.projecttree}}/implementations/python/) | [`pip install preserves`](https://pypi.org/project/preserves/) | [docs](python/latest/) |
| Racket | [preserves.dev]({{page.projecttree}}/implementations/racket/preserves/) | [`raco pkg install preserves`](https://pkgs.racket-lang.org/package/preserves) | |
| Rust | [preserves.dev]({{page.projecttree}}/implementations/rust/) | [`cargo add preserves`](https://crates.io/crates/preserves) | [docs](https://docs.rs/preserves/latest/) |
| Squeak Smalltalk | [SqueakSource](https://squeaksource.com/Preserves.html) | `Installer ss project: 'Preserves';`<br>`  install: 'Preserves'` | |
| TypeScript/JavaScript | [preserves.dev]({{page.projecttree}}/implementations/javascript/) | [`yarn add @preserves/core`](https://www.npmjs.com/package/@preserves/core) | |
| Language[^pre-alpha-implementations] | Code | Package | Docs |
|--------------------------------------|------------------------------------------------------------------------------|--------------------------------------------------------------------------------|-------------------------------------------|
| Nim | [git.syndicate-lang.org](https://git.syndicate-lang.org/ehmry/preserves-nim) | | |
| Python | [preserves.dev]({{page.projecttree}}/implementations/python/) | [`pip install preserves`](https://pypi.org/project/preserves/) | [docs](python/latest/) |
| Racket | [preserves.dev]({{page.projecttree}}/implementations/racket/preserves/) | [`raco pkg install preserves`](https://pkgs.racket-lang.org/package/preserves) | |
| Rust | [preserves.dev](https://gitlab.com/preserves/preserves-rs/) | [`cargo add preserves`](https://crates.io/crates/preserves) | [docs](https://docs.rs/preserves/latest/) |
| Squeak Smalltalk | [SqueakSource](https://squeaksource.com/Preserves.html) | `Installer ss project: 'Preserves';`<br>`  install: 'Preserves'` | |
| TypeScript/JavaScript | [preserves.dev]({{page.projecttree}}/implementations/javascript/) | [`yarn add @preserves/core`](https://www.npmjs.com/package/@preserves/core) | |
[^pre-alpha-implementations]: Pre-alpha implementations also exist for
[C]({{page.projecttree}}/implementations/c/) and

View File

@ -105,7 +105,7 @@ A few more interesting differences:
{"dictionaries": "as keys???"}: "well, why not?"}
```
Preserves technically provides a few types of numbers:
Preserves technically provides various types of numbers:
```
# Signed Integers
@ -114,9 +114,6 @@ Preserves technically provides a few types of numbers:
5907212309572059846509324862304968273468909473609826340
-5907212309572059846509324862304968273468909473609826340
# Floats (Single-precision IEEE floats) (notice the trailing f)
3.1415927f
# Doubles (Double-precision IEEE floats)
3.141592653589793
```

View File

@ -13,5 +13,5 @@ defaults:
layout: page
title: "Preserves"
version_date: "October 2023"
version: "0.992.0"
version_date: "February 2024"
version: "0.994.0"

View File

@ -1,5 +1,7 @@
[
{"version":"0.992.2","title":"0.992.2","aliases":["latest"]},
{"version":"0.994.0","title":"0.994.0","aliases":["latest"]},
{"version":"0.993.0","title":"0.993.0","aliases":[]},
{"version":"0.992.2","title":"0.992.2","aliases":[]},
{"version":"0.992.1","title":"0.992.1","aliases":[]},
{"version":"0.992.0","title":"0.992.0","aliases":[]},
{"version":"0.991.0","title":"0.991.0","aliases":[]},

View File

@ -5,9 +5,8 @@ For a value `V`, we write `«V»` for the binary encoding of `V`.
«#t» = [0x81]
«@W V» = [0x85] ++ «W» ++ «V»
«#!V» = [0x86] ++ «V»
«#:V» = [0x86] ++ «V»
«V» if V ∈ Float = [0x87, 0x04] ++ binary32(V)
«V» if V ∈ Double = [0x87, 0x08] ++ binary64(V)
«V» if V ∈ SignedInteger = [0xB0] ++ varint(|intbytes(V)|) ++ intbytes(V)
@ -29,5 +28,4 @@ For a value `V`, we write `«V»` for the binary encoding of `V`.
signedBigEndian(n >> 8) ++ [n & 255] otherwise
```
The functions `binary32(F)` and `binary64(D)` yield big-endian 4- and
8-byte IEEE 754 binary representations of `F` and `D`, respectively.
The function `binary64(D)` yields the big-endian 8-byte IEEE 754 binary representation of `D`.

View File

@ -8,10 +8,9 @@ class="postcard-grammar binarysyntax">*V*</span>.
{:.postcard-grammar.binarysyntax}
«`@`*W* *V*» | = | `85` «*W*» «*V*»
«`#!`*V*» | = | `86` «*V*»
«`#:`*V*» | = | `86` «*V*»
{:.postcard-grammar.binarysyntax}
«*V*» | = | `87``04` **binary32**(*V*) | if *V* ∈ Float
«*V*» | = | `87``08` **binary64**(*V*) | if *V* ∈ Double
{:.postcard-grammar.binarysyntax}
@ -37,10 +36,9 @@ class="postcard-grammar binarysyntax">*V*</span>.
**signedBigEndian**(*n*) | = | <span class="outputish">*n* &amp; 255</span> | if 128 ≤ *n* ≤ 127
| | **signedBigEndian**(*n* &gt;&gt; 8) <span class="outputish">*n* &amp; 255</span> | otherwise
The functions <span class="postcard-grammar binarysyntax">**binary32**(*F*)</span> and <span
class="postcard-grammar binarysyntax">**binary64**(*D*)</span> yield big-endian 4- and 8-byte
IEEE 754 binary representations of <span class="postcard-grammar binarysyntax">*F*</span> and
<span class="postcard-grammar binarysyntax">*D*</span>, respectively.
The function <span class="postcard-grammar binarysyntax">**binary64**(*D*)</span> yields the
big-endian 8-byte IEEE 754 binary representation of <span class="postcard-grammar
binarysyntax">*D*</span>.
<!--
Together, <span class="postcard-grammar binarysyntax">**div**</span> and <span

View File

@ -15,7 +15,7 @@ Set := `#{` Expr* Trailer ws `}`
Trailer := (ws Annotation)*
Embedded := `#!` SimpleExpr
Embedded := `#:` SimpleExpr
Annotated := Annotation SimpleExpr
Annotation := `@` SimpleExpr | `#` ((space | tab) linecomment) (cr | lf)
```

View File

@ -18,6 +18,6 @@ The definitions of `Atom`, `ws`, and `linecomment` are as given in the Preserves
| *Trailer* | := | (**ws** *Annotation*)<sup></sup>
{:.postcard-grammar.textsyntax}
| *Embedded* | := | `#!` *SimpleExpr*
| *Embedded* | := | `#:` *SimpleExpr*
| *Annotated* | := | *Annotation* *SimpleExpr*
| *Annotation* | := | `@` *SimpleExpr* &#124; `#` ((**space** &#124; **tab**) *linecomment*) (**cr** &#124; **lf**)

View File

@ -9,7 +9,7 @@ Set := `#{` (commas Value)* commas `}`
Dictionary := `{` (commas Value ws `:` Value)* commas `}`
commas := (ws `,`)* ws
Embedded := `#!` Value
Embedded := `#:` Value
Annotated := Annotation Value
Annotation := `@` Value | `#` ((space | tab) linecomment) (cr | lf)
@ -21,8 +21,7 @@ ByteString := `#"` binchar* `"`
String := `"` («any unicode scalar except `\` or `"`» | escaped | `\"`)* `"`
QuotedSymbol := `|` («any unicode scalar except `\` or `|`» | escaped | `\|`)* `|`
Symbol := (`A`..`Z` | `a`..`z` | `0`..`9` | sympunct | symuchar)+
Number := Float | Double | SignedInteger
Float := flt (`f`|`F`) | `#xf"` (ws hex hex)4 ws `"`
Number := Double | SignedInteger
Double := flt | `#xd"` (ws hex hex)8 ws `"`
SignedInteger := int

View File

@ -11,7 +11,7 @@
| **commas** | := | (**ws** `,`)<sup></sup> **ws** |
{:.postcard-grammar.textsyntax}
| *Embedded* | := | `#!`*Value* |
| *Embedded* | := | `#:`*Value* |
| *Annotated* | := | *Annotation* *Value* |
| *Annotation* | := | `@`*Value* &#124;`#` ((**space** &#124; **tab**) *linecomment*) (**cr** &#124; **lf**) |
@ -22,8 +22,7 @@
| *String* | := | `"` (« any unicode scalar value except `\` or `"` » &#124; *escaped* &#124;`\"`)<sup>⋆</sup> `"` |
| *QuotedSymbol* | := | `|` (« any unicode scalar value except `\` or `|` » &#124; *escaped* &#124;`\|`)<sup>⋆</sup> `|` |
| *Symbol* | := | (`A`..`Z`&#124;`a`..`z`&#124;`0`..`9`&#124; *sympunct* &#124; *symuchar*)<sup>+</sup> |
| *Number* | := | *Float* &#124; *Double* &#124; *SignedInteger* |
| *Float* | := | *flt* (`f`&#124;`F`) &#124;`#xf"` (**ws** *hex* *hex*)<sup>4</sup> **ws**`"` |
| *Number* | := | *Double* &#124; *SignedInteger* |
| *Double* | := | *flt* &#124;`#xd"` (**ws** *hex* *hex*)<sup>8</sup> **ws**`"` |
| *SignedInteger* | := | *int* |

View File

@ -1,5 +1,5 @@
Python's strings, byte strings, integers, booleans, and double-precision floats stand directly
for their Preserves counterparts. Wrapper objects for [Float][preserves.values.Float] and
for their Preserves counterparts. Wrapper objects for
[Symbol][preserves.values.Symbol] complete the suite of atomic types.
Python's lists and tuples correspond to Preserves `Sequence`s, and dicts and sets to

View File

@ -2,7 +2,6 @@ Here are a few example values, written using the [text
syntax](https://preserves.dev/preserves-text.html):
Boolean : #t #f
Float : 1.0f 10.4e3f -100.6f
Double : 1.0 10.4e3 -100.6
Integer : 1 0 -100
String : "Hello, world!\n"
@ -12,6 +11,6 @@ syntax](https://preserves.dev/preserves-text.html):
Sequence : [value1 value2 ...]
Set : #{value1 value2 ...}
Dictionary : {key1: value1 key2: value2 ...: ...}
Embedded : #!value
Embedded : #:value
Commas are optional in sequences, sets, and dictionaries.

View File

@ -4,7 +4,6 @@
| Embedded
Atom = Boolean
| Float
| Double
| SignedInteger
| String

View File

@ -38,7 +38,7 @@ representations of their keys.[^no-need-for-by-value]
**Other kinds of `Value`.**
There are no special canonicalization restrictions on
`SignedInteger`s, `String`s, `ByteString`s, `Symbol`s, `Boolean`s,
`Float`s, `Double`s, `Record`s, `Sequence`s, or `Embedded`s. The
`Double`s, `Record`s, `Sequence`s, or `Embedded`s. The
constraints given for these `Value`s in the [specification][spec]
suffice to ensure canonicity.

View File

@ -23,10 +23,10 @@ Appropriately-labelled `Record`s denote these domain-specific data
types.[^why-dictionaries]
[^why-dictionaries]: Given `Record`'s existence, it may seem odd
that `Dictionary`, `Set`, `Float`, etc. are given special
that `Dictionary`, `Set`, `Double`, etc. are given special
treatment. Preserves aims to offer a useful basic equivalence
predicate to programmers, and so if a data type demands a special
equivalence predicate, as `Dictionary`, `Set` and `Float` all do,
equivalence predicate, as `Dictionary`, `Set` and `Double` all do,
then the type should be included in the base language. Otherwise,
it can be represented as a `Record` and treated separately.
`Boolean`, `String` and `Symbol` are seeming exceptions. The first

View File

@ -31,7 +31,6 @@ fi
# Ensure that various copies of schema.prs, schema.bin, path.bin,
# samples.pr and samples.bin are in fact identical.
${COMMAND} path/path.bin implementations/python/preserves/path.prb
${COMMAND} path/path.bin implementations/rust/preserves-path/path.bin
${COMMAND} schema/schema.bin implementations/python/preserves/schema.prb
${COMMAND} schema/schema.prs implementations/racket/preserves/preserves-schema/schema.prs
@ -40,11 +39,4 @@ ${COMMAND} tests/samples.bin implementations/python/tests/samples.bin
${COMMAND} tests/samples.pr implementations/python/tests/samples.pr
${COMMAND} tests/samples.pr implementations/racket/preserves/preserves/tests/samples.pr
${COMMAND} _includes/what-is-preserves.md implementations/rust/preserves/doc/what-is-preserves.md
${COMMAND} _includes/cheatsheet-binary-plaintext.md implementations/rust/preserves/doc/cheatsheet-binary-plaintext.md
${COMMAND} _includes/cheatsheet-text-plaintext.md implementations/rust/preserves/doc/cheatsheet-text-plaintext.md
${COMMAND} _includes/value-grammar.md implementations/rust/preserves/doc/value-grammar.md
${COMMAND} _includes/what-is-preserves-schema.md implementations/rust/preserves-schema/doc/what-is-preserves-schema.md
[ -z "$failed" ]

View File

@ -13,9 +13,9 @@ Here you may find:
- [racket](racket/), an implementation for Racket 7.x and newer
(though older Rackets may also work with it).
- [rust](rust/), an implementation for Rust that interoperates with
serde.
Other implementations are also available:
- [Preserves for Rust](https://gitlab.com/preserves/preserves-rs/), an implementation for Rust
that interoperates with serde.
- [Preserves for Squeak Smalltalk](https://squeaksource.com/Preserves.html)

View File

@ -261,7 +261,6 @@ PRESERVES_OUTOFLINE
typedef enum preserves_type_tag {
PRESERVES_BOOLEAN = 0,
PRESERVES_FLOAT,
PRESERVES_DOUBLE,
PRESERVES_SIGNED_INTEGER,
@ -283,7 +282,6 @@ typedef enum preserves_type_tag {
PRESERVES_OUTOFLINE(char const *preserves_type_tag_name(preserves_type_tag_t type), {
switch (type) {
case PRESERVES_BOOLEAN: return "BOOLEAN";
case PRESERVES_FLOAT: return "FLOAT";
case PRESERVES_DOUBLE: return "DOUBLE";
case PRESERVES_SIGNED_INTEGER: return "SIGNED_INTEGER";
case PRESERVES_STRING: return "STRING";
@ -366,7 +364,6 @@ PRESERVES_OUTOFLINE
/*
PRESERVES_BOOLEAN: repr==PRESERVES_REPR_NONE, len=0, data._boolean
PRESERVES_FLOAT: repr=PRESERVES_REPR_NONE, len=0, data._float
PRESERVES_DOUBLE: repr=PRESERVES_REPR_NONE, len=0, data._double
PRESERVES_SIGNED_INTEGER:
@ -418,7 +415,6 @@ typedef struct preserves_index_entry {
union {
bool _boolean;
float _float;
double _double;
int64_t _signed;
uint64_t _unsigned;
@ -818,18 +814,6 @@ PRESERVES_OUTOFLINE
uint8_t *bs = _preserves_reader_next_bytes(r, len);
if (bs == NULL) return _preserves_reader_finish(r, PRESERVES_END_INCOMPLETE_INPUT);
switch (len) {
case 4: {
uint32_t i;
memcpy(&i, bs, 4);
i = ntohl(i);
float f;
memcpy(&f, &i, 4);
RETURN_ON_FAIL(_preserves_reader_emit_entry(r, &count, (preserves_index_entry_t) {
.type = PRESERVES_FLOAT, .repr = PRESERVES_REPR_NONE, .len = 0, .data = {
._float = f
}}));
break;
}
case 8: {
uint32_t lo, hi;
memcpy(&hi, bs, 4);
@ -995,10 +979,6 @@ PRESERVES_IMPLEMENTATION_CHUNK
fprintf(f, i->data._boolean ? " #t" : " #f");
break;
case PRESERVES_FLOAT:
fprintf(f, " %f", i->data._float);
break;
case PRESERVES_DOUBLE:
fprintf(f, " %f", i->data._double);
break;

View File

@ -41,15 +41,6 @@ namespace Preserves {
decodeEmbedded(decodeEmbedded)
{}
boost::optional<float> next_float() {
uint8_t buf[4];
if (!next_chunk(buf, sizeof(buf))) return boost::none;
uint32_t n = buf[0] << 24 | buf[1] << 16 | buf[2] << 8 | buf[3];
float f;
memcpy(&f, &n, sizeof(f));
return f;
}
boost::optional<double> next_double() {
uint8_t buf[8];
if (!next_chunk(buf, sizeof(buf))) return boost::none;
@ -113,7 +104,6 @@ namespace Preserves {
return BinaryReader<>(i).next().map(decodeEmbedded).map(Value<T>::from_embedded);
case BinaryTag::Ieee754: return varint(i).flat_map([&](size_t len)-> boost::optional<Value<T>> {
switch (len) {
case 4: return next_float().map(Value<T>::from_float);
case 8: return next_double().map(Value<T>::from_double);
default: return boost::none;
}

View File

@ -151,19 +151,6 @@ namespace Preserves {
return (*this) << (b ? BinaryTag::True : BinaryTag::False);
}
BinaryWriter& operator<<(float f) {
uint32_t n;
memcpy(&n, &f, sizeof(f));
uint8_t buf[4];
buf[0] = (n >> 24) & 0xff;
buf[1] = (n >> 16) & 0xff;
buf[2] = (n >> 8) & 0xff;
buf[3] = (n) & 0xff;
(*this) << BinaryTag::Ieee754;
put(uint8_t(sizeof(buf)));
return write(buf, sizeof(buf));
}
BinaryWriter& operator<<(double d) {
uint64_t n;
memcpy(&n, &d, sizeof(d));

View File

@ -35,13 +35,6 @@ namespace Preserves {
BinaryWriter& write(BinaryWriter& w) const override {
return w << this->_value();
});
PRESERVES_ATOMIC_VALUE_CLASS(Float, float, float, ValueKind::Float, as_float,
BinaryWriter& write(BinaryWriter& w) const override {
return w << this->_value();
}
boost::optional<double> as_double() const override {
return this->value;
});
PRESERVES_ATOMIC_VALUE_CLASS(Double, double, double, ValueKind::Double, as_double,
BinaryWriter& write(BinaryWriter& w) const override {
return w << this->_value();
@ -57,13 +50,6 @@ namespace Preserves {
return boost::none;
}
}
boost::optional<float> as_float() const override {
if (uint64_t(float(this->value)) == this->value) {
return float(this->value);
} else {
return boost::none;
}
}
boost::optional<double> as_double() const override {
if (uint64_t(double(this->value)) == this->value) {
return double(this->value);
@ -82,13 +68,6 @@ namespace Preserves {
return boost::none;
}
}
boost::optional<float> as_float() const override {
if (int64_t(float(this->value)) == this->value) {
return float(this->value);
} else {
return boost::none;
}
}
boost::optional<double> as_double() const override {
if (int64_t(double(this->value)) == this->value) {
return double(this->value);
@ -295,7 +274,6 @@ namespace Preserves {
bool is_mutable() const override { return underlying.is_mutable(); }
boost::optional<bool> as_bool() const override { return underlying.as_bool(); }
boost::optional<float> as_float() const override { return underlying.as_float(); }
boost::optional<double> as_double() const override { return underlying.as_double(); }
boost::optional<uint64_t> as_unsigned() const override { return underlying.as_unsigned(); }
boost::optional<int64_t> as_signed() const override { return underlying.as_signed(); }
@ -355,12 +333,6 @@ namespace Preserves {
return Value<T>(new Boolean<T>(b));
}
template <typename T>
Value<T> Value<T>::from_float(float f)
{
return Value<T>(new Float<T>(f));
}
template <typename T>
Value<T> Value<T>::from_double(double d)
{

View File

@ -14,7 +14,6 @@ namespace Preserves {
enum class ValueKind {
Boolean,
Float,
Double,
SignedInteger,
String,
@ -43,7 +42,6 @@ namespace Preserves {
std::shared_ptr<ValueImpl<T>> _impl() const { return p; }
static Value from_bool(bool b);
static Value from_float(float f);
static Value from_double(double d);
static Value from_int(uint64_t i);
static Value from_int(int64_t i);
@ -67,11 +65,9 @@ namespace Preserves {
static Value from_number(uint64_t i) { return from_int(i); }
static Value from_number(int64_t i) { return from_int(i); }
static Value from_number(float f) { return from_float(f); }
static Value from_number(double d) { return from_double(d); }
static Value from(bool b) { return from_bool(b); }
static Value from(float f) { return from_float(f); }
static Value from(double d) { return from_double(d); }
static Value from(uint64_t i) { return from_int(i); }
static Value from(unsigned i) { return from_int(uint64_t(i)); }
@ -95,7 +91,6 @@ namespace Preserves {
bool is_mutable() const;
bool is_bool() const { return value_kind() == ValueKind::Boolean; }
bool is_float() const { return value_kind() == ValueKind::Float; }
bool is_double() const { return value_kind() == ValueKind::Double; }
bool is_int() const { return value_kind() == ValueKind::SignedInteger; }
bool is_string() const { return value_kind() == ValueKind::String; }
@ -109,9 +104,6 @@ namespace Preserves {
boost::optional<bool> as_bool() const;
bool to_bool() const { return as_bool().value(); }
boost::optional<float> as_float() const;
float to_float() const { return as_float().value(); }
boost::optional<double> as_double() const;
double to_double() const { return as_double().value(); }
@ -175,7 +167,6 @@ namespace Preserves {
virtual bool is_mutable() const { return false; }
virtual boost::optional<bool> as_bool() const { return boost::none; }
virtual boost::optional<float> as_float() const { return boost::none; }
virtual boost::optional<double> as_double() const { return boost::none; }
virtual boost::optional<uint64_t> as_unsigned() const { return boost::none; }
virtual boost::optional<int64_t> as_signed() const { return boost::none; }
@ -219,7 +210,6 @@ namespace Preserves {
#define PRESERVES_DELEGATE_CAST(t, name) \
template <typename T> boost::optional<t> Value<T>::name() const { return p->name(); }
PRESERVES_DELEGATE_CAST(bool, as_bool);
PRESERVES_DELEGATE_CAST(float, as_float);
PRESERVES_DELEGATE_CAST(double, as_double);
PRESERVES_DELEGATE_CAST(uint64_t, as_unsigned);
PRESERVES_DELEGATE_CAST(int64_t, as_signed);
@ -265,7 +255,6 @@ namespace Preserves {
if (bKind < aKind) return false;
switch (aKind) {
case ValueKind::Boolean: return a.to_bool() < b.to_bool();
case ValueKind::Float: return a.to_float() < b.to_float();
case ValueKind::Double: return a.to_double() < b.to_double();
case ValueKind::SignedInteger: {
if (auto av = a.as_signed()) {

View File

@ -41,7 +41,7 @@ let render
)
m
++ " }"
, embedded = λ(value : Text) → "#!${value}"
, embedded = λ(value : Text) → "#:${value}"
}
let Preserves/boolean = ./boolean.dhall
@ -94,7 +94,7 @@ let example0 =
)}
''
≡ ''
{ a: 1 b: [ 2 3 ] c: { d: 1.0 e: -1.0 } d: #!#t e: <capture <_>> }
{ a: 1 b: [ 2 3 ] c: { d: 1.0 e: -1.0 } d: #:#t e: <capture <_>> }
''
in render

View File

@ -1,6 +1,6 @@
{
"name": "@preserves/core",
"version": "0.992.4",
"version": "0.994.0",
"description": "Preserves data serialization format",
"homepage": "https://gitlab.com/preserves/preserves",
"license": "Apache-2.0",

View File

@ -2,7 +2,7 @@ import { Annotated } from "./annotated";
import { DecodeError, ShortPacket } from "./codec";
import { Tag } from "./constants";
import { Set, Dictionary } from "./dictionary";
import { DoubleFloat, SingleFloat } from "./float";
import { DoubleFloat } from "./float";
import { Record } from "./record";
import { Bytes, BytesLike, underlying, hexDigit } from "./bytes";
import { Value } from "./values";
@ -31,7 +31,6 @@ export interface TypedDecoder<T> {
body: (d: TypedDecoder<S>) => R): R;
nextBoolean(): boolean | undefined;
nextFloat(): SingleFloat | undefined;
nextDouble(): DoubleFloat | undefined;
nextEmbedded(): Embedded<T> | undefined;
nextSignedInteger(): number | bigint | undefined;
@ -241,7 +240,6 @@ export class Decoder<T = never> implements TypedDecoder<T> {
case Tag.Embedded: return this.state.wrap<T>(embed(this.embeddedDecode.decode(this.state)));
case Tag.Ieee754:
switch (this.state.varint()) {
case 4: return this.state.wrap<T>(SingleFloat.fromBytes(this.state.nextbytes(4)));
case 8: return this.state.wrap<T>(DoubleFloat.fromBytes(this.state.nextbytes(8)));
default: throw new DecodeError("Invalid IEEE754 size");
}
@ -308,14 +306,6 @@ export class Decoder<T = never> implements TypedDecoder<T> {
});
}
nextFloat(): SingleFloat | undefined {
return this.skipAnnotations((reset) => {
if (this.state.nextbyte() !== Tag.Ieee754) return reset();
if (this.state.nextbyte() !== 4) return reset();
return SingleFloat.fromBytes(this.state.nextbytes(4));
});
}
nextDouble(): DoubleFloat | undefined {
return this.skipAnnotations((reset) => {
if (this.state.nextbyte() !== Tag.Ieee754) return reset();

View File

@ -26,7 +26,7 @@ export class Embedded<T> {
}
toString(): string {
return '#!' + (this.embeddedValue as any).toString();
return '#:' + (this.embeddedValue as any).toString();
}
__as_preserve__<R>(): T extends R ? Value<R> : never {

View File

@ -5,7 +5,8 @@ import type { Encoder, Preservable } from "./encoder";
import type { Writer, PreserveWritable } from "./writer";
import { Bytes, dataview } from "./bytes";
export type FloatType = 'Single' | 'Double';
// v Previously included 'Single'; may again in future. Also, 'Half', 'Quad'?
export type FloatType = 'Double';
export const FloatType = Symbol.for('FloatType');
export abstract class Float {
@ -37,7 +38,6 @@ export abstract class Float {
abstract get [FloatType](): FloatType;
static isFloat = (x: any): x is Float => x?.[FloatType] !== void 0;
static isSingle = (x: any): x is SingleFloat => x?.[FloatType] === 'Single';
static isDouble = (x: any): x is DoubleFloat => x?.[FloatType] === 'Double';
}
@ -58,76 +58,39 @@ export function floatlikeString(f: number): string {
return s + '.0';
}
export class SingleFloat extends Float implements Preservable<any>, PreserveWritable<any> {
__as_preserve__<T = GenericEmbedded>(): Value<T> {
return this;
}
static fromBytes(bs: Bytes | DataView): SingleFloat {
const view = dataview(bs);
const vf = view.getInt32(0, false);
if ((vf & 0x7f800000) === 0x7f800000) {
// NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
const sign = vf >> 31;
const payload = vf & 0x007fffff;
const dbs = new Bytes(8);
const dview = dataview(dbs);
dview.setInt16(0, (sign << 15) | 0x7ff0 | (payload >> 19), false);
dview.setInt32(2, (payload & 0x7ffff) << 13, false);
return new SingleFloat(dview.getFloat64(0, false));
} else {
return new SingleFloat(dataview(bs).getFloat32(0, false));
}
}
static __from_preserve__<T>(v: Value<T>): undefined | SingleFloat {
return Float.isSingle(v) ? v : void 0;
}
__w(v: DataView, offset: number) {
if (Number.isNaN(this.value)) {
const dbs = new Bytes(8);
const dview = dataview(dbs);
dview.setFloat64(0, this.value, false);
const sign = dview.getInt8(0) >> 7;
const payload = (dview.getInt32(1, false) >> 5) & 0x007fffff;
const vf = (sign << 31) | 0x7f800000 | payload;
v.setInt32(offset, vf, false);
} else {
v.setFloat32(offset, this.value, false);
}
}
__preserve_on__(encoder: Encoder<any>) {
encoder.state.emitbyte(Tag.Ieee754);
encoder.state.emitbyte(4);
encoder.state.makeroom(4);
this.__w(encoder.state.view, encoder.state.index);
encoder.state.index += 4;
}
toBytes(): Bytes {
const bs = new Bytes(4);
this.__w(bs.dataview(), 0);
return bs;
}
toString(): string {
if (Number.isFinite(this.value)) {
return floatlikeString(this.value) + 'f';
} else {
return '#xf"' + this.toBytes().toHex() + '"';
}
}
get [FloatType](): 'Single' {
return 'Single';
}
}
export function Single(value: number | Float): SingleFloat {
return new SingleFloat(value);
}
// -- These snippets are useful to keep in mind for promoting 4-byte, single-precision floats
// -- to 8-byte, double-precision floats *while preserving NaN bit-patterns*:
//
// static fromBytes(bs: Bytes | DataView): SingleFloat {
// const view = dataview(bs);
// const vf = view.getInt32(0, false);
// if ((vf & 0x7f800000) === 0x7f800000) {
// // NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
// const sign = vf >> 31;
// const payload = vf & 0x007fffff;
// const dbs = new Bytes(8);
// const dview = dataview(dbs);
// dview.setInt16(0, (sign << 15) | 0x7ff0 | (payload >> 19), false);
// dview.setInt32(2, (payload & 0x7ffff) << 13, false);
// return new SingleFloat(dview.getFloat64(0, false));
// } else {
// return new SingleFloat(dataview(bs).getFloat32(0, false));
// }
// }
//
// __w(v: DataView, offset: number) {
// if (Number.isNaN(this.value)) {
// const dbs = new Bytes(8);
// const dview = dataview(dbs);
// dview.setFloat64(0, this.value, false);
// const sign = dview.getInt8(0) >> 7;
// const payload = (dview.getInt32(1, false) >> 5) & 0x007fffff;
// const vf = (sign << 31) | 0x7f800000 | payload;
// v.setInt32(offset, vf, false);
// } else {
// v.setFloat32(offset, this.value, false);
// }
// }
export class DoubleFloat extends Float implements Preservable<any>, PreserveWritable<any> {
__as_preserve__<T = GenericEmbedded>(): Value<T> {

View File

@ -3,12 +3,11 @@ import { Bytes } from "./bytes";
import { Value } from "./values";
import { Set, Dictionary } from "./dictionary";
import { annotate, Annotated } from "./annotated";
import { Double, Float, Single } from "./float";
import { Double, Float } from "./float";
import { Embedded } from "./embedded";
export enum ValueClass {
Boolean,
Float,
Double,
SignedInteger,
String,
@ -26,7 +25,6 @@ export type Fold<T, R = Value<T>> = (v: Value<T>) => R;
export interface FoldMethods<T, R> {
boolean(b: boolean): R;
single(f: number): R;
double(f: number): R;
integer(i: number | bigint): R;
string(s: string): R;
@ -45,7 +43,6 @@ export interface FoldMethods<T, R> {
export class VoidFold<T> implements FoldMethods<T, void> {
boolean(b: boolean): void {}
single(f: number): void {}
double(f: number): void {}
integer(i: number | bigint): void {}
string(s: string): void {}
@ -73,9 +70,6 @@ export abstract class ValueFold<T, R = T> implements FoldMethods<T, Value<R>> {
boolean(b: boolean): Value<R> {
return b;
}
single(f: number): Value<R> {
return Single(f);
}
double(f: number): Value<R> {
return Double(f);
}
@ -134,7 +128,7 @@ export function valueClass<T>(v: Value<T>): ValueClass {
return ValueClass.Boolean;
case 'number':
if (!Number.isInteger(v)) {
throw new Error("Non-integer number in Preserves valueClass; missing SingleFloat/DoubleFloat wrapper?");
throw new Error("Non-integer number in Preserves valueClass; missing Float wrapper?");
} else {
return ValueClass.SignedInteger;
}
@ -157,8 +151,6 @@ export function valueClass<T>(v: Value<T>): ValueClass {
return ValueClass.Annotated;
} else if (Bytes.isBytes(v)) {
return ValueClass.ByteString;
} else if (Float.isSingle(v)) {
return ValueClass.Float;
} else if (Float.isDouble(v)) {
return ValueClass.Double;
} else {
@ -202,8 +194,6 @@ export function fold<T, R>(v: Value<T>, o: FoldMethods<T, R>): R {
return o.annotated(v, walk);
} else if (Bytes.isBytes(v)) {
return o.bytes(v);
} else if (Float.isSingle(v)) {
return o.single(v.value);
} else if (Float.isDouble(v)) {
return o.double(v.value);
} else {

View File

@ -9,7 +9,7 @@ export function fromJS<T = GenericEmbedded>(x: any): Value<T> {
case 'number':
if (!Number.isInteger(x)) {
// We require that clients be explicit about integer vs. non-integer types.
throw new TypeError("Refusing to autoconvert non-integer number to Single or Double");
throw new TypeError("Refusing to autoconvert non-integer number to Double");
}
// FALL THROUGH
case 'bigint':

View File

@ -32,7 +32,6 @@ export function merge<T>(
}
return fold<T, Value<T>>(a, {
boolean: die,
single(_f: number) { return is(a, b) ? a : die(); },
double(_f: number) { return is(a, b) ? a : die(); },
integer: die,
string: die,

View File

@ -19,9 +19,7 @@ export function typeCode<V>(v: Value<V>): number {
case 'symbol':
return 6;
case 'object':
if (Float.isFloat(v)) {
return Float.isSingle(v) ? 1 : 2;
}
if (Float.isFloat(v)) return 2; // 1 was for single-precision floats
if (Bytes.isBytes(v)) return 5;
if (Array.isArray(v)) {
return ('label' in v) ? 7 : 8;
@ -55,7 +53,7 @@ export function compare<V>(
const vb = b as any;
return va < vb ? -1 : va > vb ? 1 : 0;
}
case 1:
// case 1: // was single-precision
case 2: {
const va = (a as Float).value;
const vb = (b as Float).value;

View File

@ -8,7 +8,7 @@ import { Bytes, unhexDigit } from './bytes';
import { Decoder, DecoderState, neverEmbeddedTypeDecode } from './decoder';
import { Record } from './record';
import { Annotated, newPosition, Position, updatePosition } from './annotated';
import { Double, DoubleFloat, FloatType, Single, SingleFloat } from './float';
import { Double, DoubleFloat, FloatType } from './float';
import { stringify } from './text';
import { embed, GenericEmbedded, EmbeddedTypeDecode } from './embedded';
@ -24,12 +24,10 @@ export interface ReaderOptions<T> extends ReaderStateOptions {
const MAX_SAFE_INTEGERn = BigInt(Number.MAX_SAFE_INTEGER);
const MIN_SAFE_INTEGERn = BigInt(Number.MIN_SAFE_INTEGER);
export const NUMBER_RE: RegExp = /^([-+]?\d+)(((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))([fF]?))?$/;
export const NUMBER_RE: RegExp = /^([-+]?\d+)((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))?$/;
// Groups:
// 1 - integer part and sign
// 2 - decimal part, exponent and Float marker
// 3 - decimal part and exponent
// 7 - Float marker
// 2 - decimal part and exponent
export class ReaderState {
buffer: string;
@ -131,20 +129,14 @@ export class ReaderState {
}
}
readHexFloat(precision: FloatType): SingleFloat | DoubleFloat {
readHexFloat(): DoubleFloat {
const pos = this.copyPos();
if (this.nextchar() !== '"') {
this.error("Missing open-double-quote in hex-encoded floating-point number", pos);
}
const bs = this.readHexBinary();
switch (precision) {
case 'Single':
if (bs.length !== 4) this.error("Incorrect number of bytes in hex-encoded Float", pos);
return SingleFloat.fromBytes(bs);
case 'Double':
if (bs.length !== 8) this.error("Incorrect number of bytes in hex-encoded Double", pos);
return DoubleFloat.fromBytes(bs);
}
if (bs.length !== 8) this.error("Incorrect number of bytes in hex-encoded Double", pos);
return DoubleFloat.fromBytes(bs);
}
readBase64Binary(): Bytes {
@ -180,10 +172,8 @@ export class ReaderState {
} else {
return Number(v);
}
} else if (m[7] === '') {
return Double(parseFloat(m[1] + m[3]));
} else {
return Single(parseFloat(m[1] + m[3]));
return Double(parseFloat(acc));
}
} else {
return Symbol.for(acc);
@ -360,12 +350,11 @@ export class Reader<T> {
case '"': return this.state.readLiteralBinary();
case 'x': switch (this.state.nextchar()) {
case '"': return this.state.readHexBinary();
case 'f': return this.state.readHexFloat('Single');
case 'd': return this.state.readHexFloat('Double');
case 'd': return this.state.readHexFloat();
default: this.state.error('Invalid #x syntax', startPos);
}
case '[': return this.state.readBase64Binary();
case '!': return embed(this.embeddedType.fromValue(
case ':': return embed(this.embeddedType.fromValue(
new Reader<GenericEmbedded>(this.state, genericEmbeddedTypeDecode).next(),
this.state.options));
default:

View File

@ -1,7 +1,7 @@
// Preserves Values.
import type { Bytes } from './bytes';
import type { DoubleFloat, SingleFloat } from './float';
import type { DoubleFloat } from './float';
import type { Annotated } from './annotated';
import type { Set, Dictionary } from './dictionary';
import type { Embedded, GenericEmbedded } from './embedded';
@ -13,7 +13,6 @@ export type Value<T = GenericEmbedded> =
| Annotated<T>;
export type Atom =
| boolean
| SingleFloat
| DoubleFloat
| number | bigint
| string

View File

@ -297,7 +297,7 @@ export class Writer<T> {
}
else {
((v: Embedded<T>) => {
this.state.pieces.push('#!');
this.state.pieces.push('#:');
if ('write' in this.embeddedWrite) {
this.embeddedWrite.write(this.state, v.embeddedValue);
} else {

View File

@ -1,12 +1,6 @@
import { Single, Double, fromJS, Dictionary, IDENTITY_FOLD, fold, mapEmbeddeds, Value, embed, preserves } from '../src/index';
import { Double, fromJS, Dictionary, IDENTITY_FOLD, fold, mapEmbeddeds, Value, embed, preserves } from '../src/index';
import './test-utils';
describe('Single', () => {
it('should print reasonably', () => {
expect(Single(123.45).toString()).toEqual("123.45f");
});
});
describe('Double', () => {
it('should print reasonably', () => {
expect(Double(123.45).toString()).toEqual("123.45");
@ -21,7 +15,7 @@ describe('fold', () => {
new Dictionary([[[3, 4], fromJS([5, 6])],
['a', 1],
['b', true]]),
Single(3.4),
Double(3.4),
t,
]);
}

View File

@ -1,6 +1,6 @@
{
"name": "@preserves/schema-cli",
"version": "0.992.5",
"version": "0.994.0",
"description": "Command-line tools for Preserves Schema",
"homepage": "https://gitlab.com/preserves/preserves",
"license": "Apache-2.0",
@ -26,8 +26,8 @@
"@types/minimatch": "^3.0"
},
"dependencies": {
"@preserves/core": "^0.992.4",
"@preserves/schema": "^0.992.5",
"@preserves/core": "^0.994.0",
"@preserves/schema": "^0.994.0",
"chalk": "^4.1",
"chokidar": "^3.5",
"commander": "^7.2",

View File

@ -1,6 +1,6 @@
{
"name": "@preserves/schema",
"version": "0.992.5",
"version": "0.994.0",
"description": "Schema support for Preserves data serialization format",
"homepage": "https://gitlab.com/preserves/preserves",
"license": "Apache-2.0",
@ -20,12 +20,12 @@
"compile:watch": "yarn compile -w",
"rollup": "rollup -c",
"rollup:watch": "yarn rollup -w",
"copy-schema": "mkdir -p ./dist && cp -a ../../../../schema/*.prs ./dist",
"copy-schema": "mkdir -p ./dist && cp -a ../../../../schema/schema.prs ../../../../schema/host.prs ./dist",
"test": "jest",
"test:watch": "jest --watch",
"veryclean": "yarn run clean && rm -rf node_modules"
},
"dependencies": {
"@preserves/core": "^0.992.4"
"@preserves/core": "^0.994.0"
}
}

View File

@ -154,7 +154,6 @@ export function converterForSimple(
let valexp: Item = `${src}`;
switch (p.atomKind._variant) {
case 'Boolean': test = `typeof ${src} === 'boolean'`; break;
case 'Float': test = `_.Float.isSingle(${src})`; valexp = `${src}.value`; break;
case 'Double': test =`_.Float.isDouble(${src})`; valexp = `${src}.value`; break;
case 'SignedInteger': test = `typeof ${src} === 'number'`; break;
case 'String': test = `typeof ${src} === 'string'`; break;

View File

@ -37,7 +37,6 @@ export function simpleType(resolver: RefResolver, p: M.SimplePattern): FieldType
case 'atom':
switch (p.atomKind._variant) {
case 'Boolean': return Type.ref(`boolean`, null);
case 'Float': return Type.ref(`number`, null);
case 'Double': return Type.ref(`number`, null);
case 'SignedInteger': return Type.ref(`number`, null);
case 'String': return Type.ref(`string`, null);

View File

@ -40,7 +40,6 @@ function unconverterFor(ctx: FunctionContext, p: M.Pattern, src: string): Item {
return `${src}`;
case 'atom':
switch (p.atomKind._variant) {
case 'Float': return `_.Single(${src})`;
case 'Double': return `_.Double(${src})`;
default: return `${src}`;
}

View File

@ -5,7 +5,6 @@ import * as M from '../meta';
export function sourceCodeFor(v: Value<M.InputEmbedded>): Item {
return fold(v, {
boolean(b: boolean): Item { return b.toString(); },
single(f: number): Item { return f.toString(); },
double(f: number): Item { return f.toString(); },
integer(i: number): Item { return i.toString(); },
string(s: string): Item { return JSON.stringify(s); },

File diff suppressed because one or more lines are too long

View File

@ -285,7 +285,6 @@ export class SchemaInterpreter<V> {
case 'any': return input;
case 'atom': switch (p.atomKind._variant) {
case 'Boolean': return inputIf(typeof input === 'boolean');
case 'Float': return inputIf(Float.isFloat(input));
case 'Double': return inputIf(Float.isDouble(input));
case 'SignedInteger': return inputIf(typeof input === 'number' || typeof input === 'bigint');
case 'String': return inputIf(typeof input === 'string');

View File

@ -225,7 +225,6 @@ function parsePattern(name: symbol, body0: Array<Input>): Pattern {
switch (str) {
case 'any': return ks(M.SimplePattern.any());
case 'bool': return ks(M.SimplePattern.atom(M.AtomKind.Boolean()));
case 'float': return ks(M.SimplePattern.atom(M.AtomKind.Float()));
case 'double': return ks(M.SimplePattern.atom(M.AtomKind.Double()));
case 'int': return ks(M.SimplePattern.atom(M.AtomKind.SignedInteger()));
case 'string': return ks(M.SimplePattern.atom(M.AtomKind.String()));

View File

@ -18,7 +18,7 @@ describe('reader schema', () => {
expect(s.embeddedType._variant).toBe('false');
});
it('understands patterns under embed', () => {
const s = readSchema('version 1 . X = #!0 .');
const s = readSchema('version 1 . X = #:0 .');
const def: Meta.Definition = s.definitions.get(Symbol.for('X'))!;
if (def._variant !== 'Pattern') fail('bad definition 1');
if (def.value._variant !== 'SimplePattern') fail ('bad definition 2');

View File

@ -7,7 +7,6 @@ The main package re-exports a subset of the exports of its constituent modules:
- From [preserves.values][]:
- [Annotated][preserves.values.Annotated]
- [Embedded][preserves.values.Embedded]
- [Float][preserves.values.Float]
- [ImmutableDict][preserves.values.ImmutableDict]
- [Record][preserves.values.Record]
- [Symbol][preserves.values.Symbol]
@ -56,7 +55,7 @@ Finally, it provides a few utility aliases for common tasks:
'''
from .values import Float, Symbol, Record, ImmutableDict, Embedded, preserve
from .values import Symbol, Record, ImmutableDict, Embedded, preserve
from .values import Annotated, is_annotated, strip_annotations, annotate

View File

@ -206,7 +206,6 @@ class Decoder(BinaryCodec):
return self.wrap(Embedded(self.decode_embedded(self.next())))
if tag == 0x87:
count = self.nextbyte()
if count == 4: return self.wrap(Float.from_bytes(self.nextbytes(4)))
if count == 8: return self.wrap(struct.unpack('>d', self.nextbytes(8))[0])
raise DecodeError('Invalid IEEE754 size')
if tag == 0xb0: return self.wrap(self.nextint(self.varint()))

View File

@ -33,12 +33,12 @@ import numbers
from enum import Enum
from functools import cmp_to_key
from .values import preserve, Float, Embedded, Record, Symbol, cmp_floats, _unwrap
from .values import preserve, Embedded, Record, Symbol, cmp_floats, _unwrap
from .compat import basestring_
class TypeNumber(Enum):
BOOL = 0
FLOAT = 1
# FLOAT = 1 # single-precision
DOUBLE = 2
SIGNED_INTEGER = 3
STRING = 4
@ -57,7 +57,6 @@ def type_number(v):
raise ValueError('type_number expects Preserves value; use preserve()')
if isinstance(v, bool): return TypeNumber.BOOL
if isinstance(v, Float): return TypeNumber.FLOAT
if isinstance(v, float): return TypeNumber.DOUBLE
if isinstance(v, numbers.Number): return TypeNumber.SIGNED_INTEGER
if isinstance(v, basestring_): return TypeNumber.STRING

View File

@ -9,7 +9,7 @@ def map_embeddeds(f, v):
```python
>>> map_embeddeds(lambda w: Embedded(f'w={w}'), ['a', Embedded(123), {'z': 6.0}])
('a', #!'w=123', {'z': 6.0})
('a', #:'w=123', {'z': 6.0})
```
"""

View File

@ -1,5 +1,5 @@
´³schema·³version°³ definitions·³Axis´³orµµ±values´³rec´³lit³values„´³tupleµ„„„„µ± descendants´³rec´³lit³ descendants„´³tupleµ„„„„µ±at´³rec´³lit³at„´³tupleµ´³named³key³any„„„„„µ±label´³rec´³lit³label„´³tupleµ„„„„µ±keys´³rec´³lit³keys„´³tupleµ„„„„µ±length´³rec´³lit³length„´³tupleµ„„„„µ± annotations´³rec´³lit³ annotations„´³tupleµ„„„„µ±embedded´³rec´³lit³embedded„´³tupleµ„„„„µ±parse´³rec´³lit³parse„´³tupleµ´³named³module´³seqof´³atom³Symbol„„„´³named³name´³atom³Symbol„„„„„„µ±unparse´³rec´³lit³unparse„´³tupleµ´³named³module´³seqof´³atom³Symbol„„„´³named³name´³atom³Symbol„„„„„„„„³Step´³orµµ±Axis´³refµ„³Axis„„µ±Filter´³refµ„³Filter„„µ±Function´³refµ„³Function„„„„³Filter´³orµµ±nop´³rec´³lit³nop„´³tupleµ„„„„µ±compare´³rec´³lit³compare„´³tupleµ´³named³op´³refµ„³
Comparison„„´³named³literal³any„„„„„µ±regex´³rec´³lit³regex„´³tupleµ´³named³regex´³atom³String„„„„„„µ±test´³rec´³lit³test„´³tupleµ´³named³pred´³refµ„³ Predicate„„„„„„µ±real´³rec´³lit³real„´³tupleµ„„„„µ±int´³rec´³lit³int„´³tupleµ„„„„µ±kind´³rec´³lit³kind„´³tupleµ´³named³kind´³refµ„³ ValueKind„„„„„„„„³Function´³rec´³lit³count„´³tupleµ´³named³selector´³refµ„³Selector„„„„„³Selector´³seqof´³refµ„³Step„„³ Predicate´³orµµ±Selector´³refµ„³Selector„„µ±not´³rec´³lit³not„´³tupleµ´³named³pred´³refµ„³ Predicate„„„„„„µ±or´³rec´³lit³or„´³tupleµ´³named³preds´³seqof´³refµ„³ Predicate„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³named³preds´³seqof´³refµ„³ Predicate„„„„„„„„„³ ValueKind´³orµµ±Boolean´³lit³Boolean„„µ±Float´³lit³Float„„µ±Double´³lit³Double„„µ± SignedInteger´³lit³ SignedInteger„„µ±String´³lit³String„„µ±
Comparison„„´³named³literal³any„„„„„µ±regex´³rec´³lit³regex„´³tupleµ´³named³regex´³atom³String„„„„„„µ±test´³rec´³lit³test„´³tupleµ´³named³pred´³refµ„³ Predicate„„„„„„µ±real´³rec´³lit³real„´³tupleµ„„„„µ±int´³rec´³lit³int„´³tupleµ„„„„µ±kind´³rec´³lit³kind„´³tupleµ´³named³kind´³refµ„³ ValueKind„„„„„„„„³Function´³rec´³lit³count„´³tupleµ´³named³selector´³refµ„³Selector„„„„„³Selector´³seqof´³refµ„³Step„„³ Predicate´³orµµ±Selector´³refµ„³Selector„„µ±not´³rec´³lit³not„´³tupleµ´³named³pred´³refµ„³ Predicate„„„„„„µ±or´³rec´³lit³or„´³tupleµ´³named³preds´³seqof´³refµ„³ Predicate„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³named³preds´³seqof´³refµ„³ Predicate„„„„„„„„„³ ValueKind´³orµµ±Boolean´³lit³Boolean„„µ±Double´³lit³Double„„µ± SignedInteger´³lit³ SignedInteger„„µ±String´³lit³String„„µ±
ByteString´³lit³
ByteString„„µ±Symbol´³lit³Symbol„„µ±Record´³lit³Record„„µ±Sequence´³lit³Sequence„„µ±Set´³lit³Set„„µ±
Dictionary´³lit³

View File

@ -170,7 +170,6 @@ FILTER_RE2 = Symbol('=r')
FILTER_LABEL = Symbol('^')
FILTER_BOOL = Symbol('bool')
FILTER_FLOAT = Symbol('float')
FILTER_DOUBLE = Symbol('double')
FILTER_INT = Symbol('int')
FILTER_STRING = Symbol('string')
@ -226,7 +225,6 @@ def parse_step(tokens):
if t == TRANSFORM_REAL: return syntax.Step.Filter(syntax.Filter.real)
if t == TRANSFORM_INT: return syntax.Step.Filter(syntax.Filter.int)
if t == FILTER_BOOL: return kind_filter(syntax.ValueKind.Boolean())
if t == FILTER_FLOAT: return kind_filter(syntax.ValueKind.Float())
if t == FILTER_DOUBLE: return kind_filter(syntax.ValueKind.Double())
if t == FILTER_INT: return kind_filter(syntax.ValueKind.SignedInteger())
if t == FILTER_STRING: return kind_filter(syntax.ValueKind.String())
@ -448,8 +446,6 @@ def exec(self, v):
@extend(syntax.Filter.real)
def exec(self, v):
v = preserve(_unwrap(v))
if isinstance(v, Float):
return (v.value,)
if type(v) == float:
return (v,)
if type(v) == int:
@ -459,8 +455,6 @@ def exec(self, v):
@extend(syntax.Filter.int)
def exec(self, v):
v = preserve(_unwrap(v))
if isinstance(v, Float):
return (int(v.value()),)
if type(v) == float:
return (int(v),)
if type(v) == int:
@ -476,10 +470,6 @@ def exec(self, v):
def exec(self, v):
return (v,) if type(v) == bool else ()
@extend(syntax.ValueKind.Float)
def exec(self, v):
return (v,) if isinstance(v, Float) else ()
@extend(syntax.ValueKind.Double)
def exec(self, v):
return (v,) if type(v) == float else ()

View File

@ -1,6 +1,6 @@
´³schema·³version°³ definitions·³Ref´³rec´³lit³ref„´³tupleµ´³named³module´³refµ„³
ModulePath„„´³named³name´³atom³Symbol„„„„„³Bundle´³rec´³lit³bundle„´³tupleµ´³named³modules´³refµ„³Modules„„„„„³Schema´³rec´³lit³schema„´³tupleµ´³dict·³version´³named³version´³refµ„³Version„„³ definitions´³named³ definitions´³refµ„³ Definitions„„³ embeddedType´³named³ embeddedType´³refµ„³EmbeddedTypeName„„„„„„„³Binding´³rec´³lit³named„´³tupleµ´³named³name´³atom³Symbol„„´³named³pattern´³refµ„³ SimplePattern„„„„„³Modules´³dictof´³refµ„³
ModulePath„´³refµ„³Schema„„³Pattern´³orµµ± SimplePattern´³refµ„³ SimplePattern„„µ±CompoundPattern´³refµ„³CompoundPattern„„„„³Version´³lit°„³AtomKind´³orµµ±Boolean´³lit³Boolean„„µ±Float´³lit³Float„„µ±Double´³lit³Double„„µ± SignedInteger´³lit³ SignedInteger„„µ±String´³lit³String„„µ±
ModulePath„´³refµ„³Schema„„³Pattern´³orµµ± SimplePattern´³refµ„³ SimplePattern„„µ±CompoundPattern´³refµ„³CompoundPattern„„„„³Version´³lit°„³AtomKind´³orµµ±Boolean´³lit³Boolean„„µ±Double´³lit³Double„„µ± SignedInteger´³lit³ SignedInteger„„µ±String´³lit³String„„µ±
ByteString´³lit³
ByteString„„µ±Symbol´³lit³Symbol„„„„³
Definition´³orµµ±or´³rec´³lit³or„´³tupleµ´³ tuplePrefixµ´³named³pattern0´³refµ„³NamedAlternative„„´³named³pattern1´³refµ„³NamedAlternative„„„´³named³patternN´³seqof´³refµ„³NamedAlternative„„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³ tuplePrefixµ´³named³pattern0´³refµ„³ NamedPattern„„´³named³pattern1´³refµ„³ NamedPattern„„„´³named³patternN´³seqof´³refµ„³ NamedPattern„„„„„„„„µ±Pattern´³refµ„³Pattern„„„„³

View File

@ -279,7 +279,6 @@ DICT = Symbol('dict')
DICTOF = Symbol('dictof')
DOUBLE = Symbol('Double')
EMBEDDED = Symbol('embedded')
FLOAT = Symbol('Float')
LIT = Symbol('lit')
NAMED = Symbol('named')
OR = Symbol('or')
@ -469,7 +468,6 @@ class SchemaObject:
if p.key == ATOM:
k = p[0]
if k == BOOLEAN and isinstance(v, bool): return v
if k == FLOAT and isinstance(v, Float): return v
if k == DOUBLE and isinstance(v, float): return v
if k == SIGNED_INTEGER and isinstance(v, int): return v
if k == STRING and isinstance(v, str): return v

View File

@ -28,7 +28,7 @@ from .binary import Decoder
class TextCodec(object):
pass
NUMBER_RE = re.compile(r'^([-+]?\d+)(((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))([fF]?))?$')
NUMBER_RE = re.compile(r'^([-+]?\d+)((\.\d+([eE][-+]?\d+)?)|([eE][-+]?\d+))?$')
class Parser(TextCodec):
"""Parser for the human-readable Preserves text syntax.
@ -251,15 +251,13 @@ class Parser(TextCodec):
if c == '=': continue
acc.append(c)
def read_hex_float(self, bytecount):
def read_hex_float(self):
if self.nextchar() != '"':
raise DecodeError('Missing open-double-quote in hex-encoded floating-point number')
bs = self.read_hex_binary()
if len(bs) != bytecount:
if len(bs) != 8:
raise DecodeError('Incorrect number of bytes in hex-encoded floating-point number')
if bytecount == 4: return Float.from_bytes(bs)
if bytecount == 8: return struct.unpack('>d', bs)[0]
raise DecodeError('Unsupported byte count in hex-encoded floating-point number')
return struct.unpack('>d', bs)[0]
def upto(self, delimiter, skip_commas):
vs = []
@ -308,10 +306,8 @@ class Parser(TextCodec):
if m:
if m[2] is None:
return int(m[1])
elif m[7] == '':
return float(m[1] + m[3])
else:
return Float(float(m[1] + m[3]))
return float(acc)
else:
return Symbol(acc)
@ -357,11 +353,10 @@ class Parser(TextCodec):
if c == 'x':
c = self.nextchar()
if c == '"': return self.wrap(self.read_hex_binary())
if c == 'f': return self.wrap(self.read_hex_float(4))
if c == 'd': return self.wrap(self.read_hex_float(8))
if c == 'd': return self.wrap(self.read_hex_float())
raise DecodeError('Invalid #x syntax')
if c == '[': return self.wrap(self.read_base64_binary())
if c == '!':
if c == ':':
if self.parse_embedded is None:
raise DecodeError('No parse_embedded function supplied')
return self.wrap(Embedded(self.parse_embedded(self.next())))

View File

@ -39,135 +39,6 @@ def cmp_floats(a, b):
if b & 0x8000000000000000: b = b ^ 0x7fffffffffffffff
return a - b
class Float(object):
"""Wrapper for treating a Python double-precision floating-point value as a
single-precision (32-bit) float, from Preserves' perspective. (Python lacks native
single-precision floating point support.)
```python
>>> Float(3.45)
Float(3.45)
>>> import preserves
>>> preserves.stringify(Float(3.45))
'3.45f'
>>> preserves.stringify(3.45)
'3.45'
>>> preserves.parse('3.45f')
Float(3.45)
>>> preserves.parse('3.45')
3.45
>>> preserves.encode(Float(3.45))
b'\\x87\\x04@\\\\\\xcc\\xcd'
>>> preserves.encode(3.45)
b'\\x87\\x08@\\x0b\\x99\\x99\\x99\\x99\\x99\\x9a'
```
Attributes:
value (float): the double-precision representation of intended single-precision value
"""
def __init__(self, value):
self.value = value
def __eq__(self, other):
other = _unwrap(other)
if other.__class__ is self.__class__:
return cmp_floats(self.value, other.value) == 0
def __lt__(self, other):
other = _unwrap(other)
if other.__class__ is self.__class__:
return cmp_floats(self.value, other.value) < 0
def __ne__(self, other):
return not self.__eq__(other)
def __hash__(self):
return hash(self.value)
def __repr__(self):
return 'Float(' + repr(self.value) + ')'
def to_bytes(self):
"""Converts this 32-bit single-precision floating point value to its binary32 format,
taking care to preserve the quiet/signalling bit-pattern of NaN values, unlike its
`struct.pack('>f', ...)` equivalent.
```python
>>> Float.from_bytes(b'\\x7f\\x80\\x00{')
Float(nan)
>>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
b'\\x7f\\x80\\x00{'
>>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
nan
>>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
b'\\x7f\\xc0\\x00{'
>>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
b'\\x7f\\xc0\\x00{'
```
(Note well the difference between `7f80007b` and `7fc0007b`!)
"""
if math.isnan(self.value) or math.isinf(self.value):
dbs = struct.pack('>d', self.value)
vd = struct.unpack('>Q', dbs)[0]
sign = vd >> 63
payload = (vd >> 29) & 0x007fffff
vf = (sign << 31) | 0x7f800000 | payload
return struct.pack('>I', vf)
else:
return struct.pack('>f', self.value)
def __preserve_write_binary__(self, encoder):
encoder.buffer.append(0x87)
encoder.buffer.append(4)
encoder.buffer.extend(self.to_bytes())
def __preserve_write_text__(self, formatter):
if math.isnan(self.value) or math.isinf(self.value):
formatter.chunks.append('#xf"' + self.to_bytes().hex() + '"')
else:
formatter.chunks.append(repr(self.value) + 'f')
@staticmethod
def from_bytes(bs):
"""Converts a 4-byte-long byte string to a 32-bit single-precision floating point value
wrapped in a [Float][preserves.values.Float] instance. Takes care to preserve the
quiet/signalling bit-pattern of NaN values, unlike its `struct.unpack('>f', ...)`
equivalent.
```python
>>> Float.from_bytes(b'\\x7f\\x80\\x00{')
Float(nan)
>>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
b'\\x7f\\x80\\x00{'
>>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
nan
>>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
b'\\x7f\\xc0\\x00{'
>>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
b'\\x7f\\xc0\\x00{'
```
(Note well the difference between `7f80007b` and `7fc0007b`!)
"""
vf = struct.unpack('>I', bs)[0]
if (vf & 0x7f800000) == 0x7f800000:
# NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
sign = vf >> 31
payload = vf & 0x007fffff
dbs = struct.pack('>Q', (sign << 63) | 0x7ff0000000000000 | (payload << 29))
return Float(struct.unpack('>d', dbs)[0])
else:
return Float(struct.unpack('>f', bs)[0])
# FIXME: This regular expression is conservatively correct, but Anglo-chauvinistic.
RAW_SYMBOL_RE = re.compile(r'^[-a-zA-Z0-9~!$%^&*?_=+/.]+$')
@ -704,7 +575,7 @@ class Embedded:
>>> import io
>>> e = Embedded(io.StringIO('some text'))
>>> e # doctest: +ELLIPSIS
#!<_io.StringIO object at ...>
#:<_io.StringIO object at ...>
>>> e.embeddedValue # doctest: +ELLIPSIS
<_io.StringIO object at ...>
@ -717,7 +588,7 @@ class Embedded:
...
TypeError: Cannot preserves-format: None
>>> print(preserves.stringify(Embedded(None), format_embedded=lambda x: 'abcdef'))
#!"abcdef"
#:"abcdef"
```
@ -739,12 +610,12 @@ class Embedded:
return hash(self.embeddedValue)
def __repr__(self):
return '#!%r' % (self.embeddedValue,)
return '#:%r' % (self.embeddedValue,)
def __preserve_write_binary__(self, encoder):
encoder.buffer.append(0x86)
encoder.append(encoder.encode_embedded(self.embeddedValue))
def __preserve_write_text__(self, formatter):
formatter.chunks.append('#!')
formatter.chunks.append('#:')
formatter.append(formatter.format_embedded(self.embeddedValue))

View File

@ -1,6 +1,6 @@
[project]
name = "preserves"
version = "0.992.2"
version = "0.994.0"
description = "Data serialization format"
readme = "README.md"
requires-python = ">=3.6, <4"

View File

@ -0,0 +1,73 @@
The code below deals with expansion of single-precision IEEE 754 floating point values to
double-precision (and vice-versa) without losing detail of NaN bit-patterns.
```python
def to_bytes(self):
"""Converts this 32-bit single-precision floating point value to its binary32 format,
taking care to preserve the quiet/signalling bit-pattern of NaN values, unlike its
`struct.pack('>f', ...)` equivalent.
```python
>>> Float.from_bytes(b'\\x7f\\x80\\x00{')
Float(nan)
>>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
b'\\x7f\\x80\\x00{'
>>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
nan
>>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
b'\\x7f\\xc0\\x00{'
>>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
b'\\x7f\\xc0\\x00{'
```
(Note well the difference between `7f80007b` and `7fc0007b`!)
"""
if math.isnan(self.value) or math.isinf(self.value):
dbs = struct.pack('>d', self.value)
vd = struct.unpack('>Q', dbs)[0]
sign = vd >> 63
payload = (vd >> 29) & 0x007fffff
vf = (sign << 31) | 0x7f800000 | payload
return struct.pack('>I', vf)
else:
return struct.pack('>f', self.value)
@staticmethod
def from_bytes(bs):
"""Converts a 4-byte-long byte string to a 32-bit single-precision floating point value
wrapped in a [Float][preserves.values.Float] instance. Takes care to preserve the
quiet/signalling bit-pattern of NaN values, unlike its `struct.unpack('>f', ...)`
equivalent.
```python
>>> Float.from_bytes(b'\\x7f\\x80\\x00{')
Float(nan)
>>> Float.from_bytes(b'\\x7f\\x80\\x00{').to_bytes()
b'\\x7f\\x80\\x00{'
>>> struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]
nan
>>> Float(struct.unpack('>f', b'\\x7f\\x80\\x00{')[0]).to_bytes()
b'\\x7f\\xc0\\x00{'
>>> struct.pack('>f', struct.unpack('>f', b'\\x7f\\x80\\x00{')[0])
b'\\x7f\\xc0\\x00{'
```
(Note well the difference between `7f80007b` and `7fc0007b`!)
"""
vf = struct.unpack('>I', bs)[0]
if (vf & 0x7f800000) == 0x7f800000:
# NaN or inf. Preserve quiet/signalling bit by manually expanding to double-precision.
sign = vf >> 31
payload = vf & 0x007fffff
dbs = struct.pack('>Q', (sign << 63) | 0x7ff0000000000000 | (payload << 29))
return Float(struct.unpack('>d', dbs)[0])
else:
return Float(struct.unpack('>f', bs)[0])
```

View File

@ -119,32 +119,6 @@
double15: @"-sNaN" <Test #x"8708fff8000000000111" #xd"fff8000000000111">
double16: @"+sNaN" <Test #x"87087ff8000000000001" #xd"7ff8000000000001">
double17: @"+sNaN" <Test #x"87087ff8000000000111" #xd"7ff8000000000111">
float0: <Test #x"870400000000" 0.0f>
float+0: <Test #x"870400000000" +0.0f>
float-0: <Test #x"870480000000" -0.0f>
float1: <Test #x"87043f800000" 1.0f>
float1u: <Test #x"87043f800000" 1.0F>
float1a: <Test #x"87043f800000" 1e0f>
float1b: <Test #x"87043f800000" 1.0e0f>
float1c: <Test #x"87043f800000" 1e-0f>
float1d: <Test #x"87043f800000" 1.0e-0f>
float1e: <Test #x"87043f800000" 1e+0f>
float1f: <Test #x"87043f800000" 1.0e+0f>
float2: <Test #x"870412345678" #xf"12 34 56 78">
float3: @"Fewer than 8 digits" <ParseError "#xf\"123456\"">
float4: @"More than 8 digits" <ParseError "#xf\"123456789a\"">
float5: @"Invalid chars" <ParseError "#xf\"12zz5678\"">
float6: @"Positive infinity" <Test #x"87047f800000" #xf"7f800000">
float7: @"Negative infinity" <Test #x"8704ff800000" #xf"ff800000">
float8: @"+sNaN" <Test #x"87047f800001" #xf"7f800001">
float9: @"+sNaN" <Test #x"87047f800111" #xf"7f800111">
float10: @"-sNaN" <Test #x"8704ff800001" #xf"ff800001">
float11: @"-sNaN" <Test #x"8704ff800111" #xf"ff800111">
float12: @"Bad spacing" <ParseError "#xf\"12345 678\"">
float13: @"+qNaN" <Test #x"87047fc00001" #xf"7fc00001">
float14: @"+qNaN" <Test #x"87047fc00111" #xf"7fc00111">
float15: @"-qNaN" <Test #x"8704ffc00001" #xf"ffc00001">
float16: @"-qNaN" <Test #x"8704ffc00111" #xf"ffc00111">
int-98765432109876543210987654321098765432109: <Test #x"b012feddc125aed4226c770369269596ce3f0ad3" -98765432109876543210987654321098765432109>
int-12345678123456781234567812345678: <Test #x"b00eff642cf6684f11d1dad08c4a10b2" -12345678123456781234567812345678>
int-1234567812345678123456781234567: <Test #x"b00df06ae570d4b4fb62ae746dce79" -1234567812345678123456781234567>
@ -193,9 +167,9 @@
list11: <Test #x"b5b0010184" [01]>
list12: <Test #x"b5b0010c84" [12]>
noinput0: @"No input at all" <DecodeEOF #x"">
embed0: <Test #x"86b000" #!0>
embed1: <Test #x"8686b000" #!#!0>
embed2: <Test #x"b586b00086b10568656c6c6f84" [#!0 #!"hello"]>
embed0: <Test #x"86b000" #:0>
embed1: <Test #x"8686b000" #:#:0>
embed2: <Test #x"b586b00086b10568656c6c6f84" [#:0 #:"hello"]>
record1: <Test #x"b4 b30763617074757265 b4 b30764697363617264 84 84" <capture <discard>>>
record2: <Test #x"b4 b3076f627365727665 b4 b305737065616b b4 b30764697363617264 84 b4 b30763617074757265 b4 b30764697363617264 84 84 84 84" <observe <speak <discard> <capture <discard>>>>>
record2a: @"Commas not allowed in records" <ParseError "<observe <speak <discard>, <capture <discard>>>>">

View File

@ -135,7 +135,6 @@ class BinaryCodecTests(PreservesTestCase):
self._roundtrip(131072, _buf(0xb0, 0x03, 0x02, 0x00, 0x00))
def test_floats(self):
self._roundtrip(Float(1.0), _buf(0x87, 0x04, 0x3f, 0x80, 0, 0))
self._roundtrip(1.0, _buf(0x87, 0x08, 0x3f, 0xf0, 0, 0, 0, 0, 0, 0))
self._roundtrip(-1.202e300, _buf(0x87, 0x08, 0xfe, 0x3c, 0xb7, 0xb7, 0x59, 0xbf, 0x04, 0x26))

View File

@ -14,7 +14,6 @@
(define :encode-embedded values)
(define (AtomKind? p)
(or (AtomKind-Boolean? p)
(AtomKind-Float? p)
(AtomKind-Double? p)
(AtomKind-SignedInteger? p)
(AtomKind-String? p)
@ -29,15 +28,6 @@
((define/generic *->preserve ->preserve)
(define (->preserve preservable)
(match preservable ((AtomKind-Boolean) 'Boolean)))))
(struct
AtomKind-Float
()
#:transparent
#:methods
gen:preservable
((define/generic *->preserve ->preserve)
(define (->preserve preservable)
(match preservable ((AtomKind-Float) 'Float)))))
(struct
AtomKind-Double
()
@ -87,7 +77,6 @@
(match
input
((and dest 'Boolean) (AtomKind-Boolean))
((and dest 'Float) (AtomKind-Float))
((and dest 'Double) (AtomKind-Double))
((and dest 'SignedInteger) (AtomKind-SignedInteger))
((and dest 'String) (AtomKind-String))

View File

@ -27,8 +27,7 @@
(define (->preserve preservable)
(for/hash [((k v) (in-hash preservable))]
(values (*->preserve k) (*->preserve v))))])
#:defaults ([float? (define (->preserve preservable) preservable)]
[embedded? (define (->preserve preservable) preservable)]
#:defaults ([embedded? (define (->preserve preservable) preservable)]
[record?
(define/generic *->preserve ->preserve)
(define (->preserve preservable)

View File

@ -26,7 +26,6 @@
(maybe-dest dest-pat-stx
`(? ,(match atom-kind
[(AtomKind-Boolean) 'boolean?]
[(AtomKind-Float) 'float?]
[(AtomKind-Double) 'flonum?]
[(AtomKind-SignedInteger) 'exact-integer?]
[(AtomKind-String) 'string?]

View File

@ -131,7 +131,6 @@
(match (peel-annotations item)
['any (ks (SimplePattern-any))]
['bool (ks (SimplePattern-atom (AtomKind-Boolean)))]
['float (ks (SimplePattern-atom (AtomKind-Float)))]
['double (ks (SimplePattern-atom (AtomKind-Double)))]
['int (ks (SimplePattern-atom (AtomKind-SignedInteger)))]
['string (ks (SimplePattern-atom (AtomKind-String)))]

View File

@ -38,10 +38,10 @@ SimplePattern =
# any
/ =any
# special builtins: bool, float, double, int, string, bytes, symbol
# special builtins: bool, double, int, string, bytes, symbol
/ <atom @atomKind AtomKind>
# matches an embedded value in the input: #!p
# matches an embedded value in the input: #:p
/ <embedded @interface SimplePattern>
# =symbol, <<lit> any>, or plain non-symbol atom
@ -79,7 +79,7 @@ CompoundPattern =
DictionaryEntries = { any: NamedSimplePattern ...:... }.
AtomKind = =Boolean / =Float / =Double / =SignedInteger / =String / =ByteString / =Symbol .
AtomKind = =Boolean / =Double / =SignedInteger / =String / =ByteString / =Symbol .
NamedAlternative = [@variantLabel string @pattern Pattern].

View File

@ -14,7 +14,6 @@
(match (unwrap pattern)
[(Binding n p) (pattern->unparser p (escape n))]
[(SimplePattern-any) `(*->preserve ,src-stx)]
[(SimplePattern-atom (AtomKind-Float)) `(->float (*->preserve ,src-stx))]
[(SimplePattern-atom (AtomKind-Double)) `(exact->inexact (*->preserve ,src-stx))]
[(SimplePattern-atom _) `(*->preserve ,src-stx)]
[(SimplePattern-embedded _interface) `(embedded ,src-stx)]

View File

@ -9,7 +9,6 @@
bytes->double
double->bytes)
(require "float.rkt")
(require (only-in racket/math nan? infinite?))
(module binary racket/base
@ -37,21 +36,20 @@
#x0070000000000000
(arithmetic-shift payload 29)))
(dbs (integer->integer-bytes vd 8 #f #t)))
(float (floating-point-bytes->real dbs #t 0 8)))
(float (floating-point-bytes->real bs #t 0 4))))
(floating-point-bytes->real dbs #t 0 8))
(floating-point-bytes->real bs #t 0 4)))
(define (float->bytes v)
(let ((v (float-value v)))
(if (or (nan? v) (infinite? v))
(let* ((dbs (real->floating-point-bytes v 8 #t))
(vd (integer-bytes->integer dbs #f #t))
(signexp (bitwise-bit-field vd 55 64))
(payload (bitwise-bit-field vd 29 52))
(vf (bitwise-ior (arithmetic-shift signexp 23)
payload))
(bs (integer->integer-bytes vf 4 #f #t)))
bs)
(real->floating-point-bytes v 4 #t))))
(if (or (nan? v) (infinite? v))
(let* ((dbs (real->floating-point-bytes v 8 #t))
(vd (integer-bytes->integer dbs #f #t))
(signexp (bitwise-bit-field vd 55 64))
(payload (bitwise-bit-field vd 29 52))
(vf (bitwise-ior (arithmetic-shift signexp 23)
payload))
(bs (integer->integer-bytes vf 4 #f #t)))
bs)
(real->floating-point-bytes v 4 #t)))
(define (bytes->double bs)
(floating-point-bytes->real bs #t 0 8))

View File

@ -1,14 +0,0 @@
#lang racket/base
;; Wrapper struct to mark a need for 32-bit IEEE floating-point
;; precision (de)serialization. In many circumstances, Racket lacks
;; 32-bit floating point support, and single-flonum? always yields #f.
(provide (struct-out float)
->float)
(struct float (value) #:transparent)
(define (->float v)
(if (float? v)
v
(float (exact->inexact v))))

View File

@ -8,7 +8,7 @@
;;---------------------------------------------------------------------------
;; Representing values
(require "float.rkt" "float-bytes.rkt")
(require "float-bytes.rkt")
(struct record (label fields) #:transparent)
(struct annotated (annotation item) #:transparent)
(struct embedded (value) #:transparent)
@ -26,9 +26,7 @@
[#x84 '#:end]
[#x85 (let ((a (next))) (annotated a (next)))]
[#x86 (embedded (next))]
[#x87 (match (next-byte)
[4 (bytes->float (next-bytes 4))]
[8 (bytes->double (next-bytes 8))])]
[#x87 (match (next-byte) [8 (bytes->double (next-bytes 8))])]
[#xB0 (next-integer (next-varint))]
[#xB1 (bytes->string/utf-8 (next-bytes (next-varint)))]
[#xB2 (next-bytes (next-varint))]
@ -74,7 +72,6 @@
(match v
[#f (write-byte #x80 out-port)]
[#t (write-byte #x81 out-port)]
[(float _) (write-byte #x87 out-port) (write-byte 4 out-port) (output-bytes (float->bytes v))]
[(? flonum?) (write-byte #x87 out-port) (write-byte 8 out-port) (output-bytes (double->bytes v))]
[(annotated a v) (write-byte #x85 out-port) (output a) (output v)]

View File

@ -2,7 +2,6 @@
;; Preserve, as in Fruit Preserve, as in a remarkably weak pun on pickling/dehydration etc
(provide (all-from-out "record.rkt")
(all-from-out "float.rkt")
(all-from-out "annotation.rkt")
(all-from-out "order.rkt")
(all-from-out "embedded.rkt")
@ -31,7 +30,6 @@
(require (only-in racket/port port->list))
(require "record.rkt")
(require "float.rkt")
(require "annotation.rkt")
(require "order.rkt")
(require "embedded.rkt")

View File

@ -12,7 +12,6 @@
(require (for-syntax racket/base))
(require "record.rkt")
(require "annotation.rkt")
(require "float.rkt")
(require racket/set)
(require racket/dict)
(require data/order)
@ -24,7 +23,7 @@
(define (typecode v)
(match v
[(? boolean?) 0]
[(or (? float?) (? single-flonum?)) 1]
;; [(or (? float?) (? single-flonum?)) 1]
[(? double-flonum?) 2]
[(? integer? x) 3]
[(? string?) 4]

View File

@ -279,7 +279,7 @@
[(BLOCK ps ...) (grouped (convert-inner ps) " " "" "{ " " }")]
[(SET ps ...) (grouped (convert-inner ps) " " "" "#{ " " }")]
[(TRAILER-ANCHOR) ""]
[(embedded v) (list "#!" (convert (encode-embedded v)))]
[(embedded v) (list "#:" (convert (encode-embedded v)))]
[(strip-annotations (record 'p (list s))) (symbol->string s)]
[v (preserve->string v)]))

View File

@ -6,7 +6,6 @@
(require racket/match)
(require "record.rkt")
(require "embedded.rkt")
(require "float.rkt")
(require "float-bytes.rkt")
(require "annotation.rkt")
(require "varint.rkt")
@ -76,7 +75,6 @@
(next)))]
[#x86 (embedded (decode-embedded (next)))]
[#x87 (match (next-varint)
[4 (bytes->float (next-bytes 4))]
[8 (bytes->double (next-bytes 8))]
[n (return (on-fail "Invalid Preserves IEEE754 size: ~v" n))])]
[#xB0 (next-integer (next-varint))]

View File

@ -12,7 +12,6 @@
(require racket/match)
(require "embedded.rkt")
(require "annotation.rkt")
(require "float.rkt")
(require "float-bytes.rkt")
(require syntax/readerr)
(require (only-in file/sha1 hex-string->bytes))
@ -112,11 +111,10 @@
[#\" (read-literal-binary)]
[#\x (match (next-char*)
[#\" (read-hex-binary '())]
[#\f (read-hex-float 'float)]
[#\d (read-hex-float 'double)]
[#\d (read-hex-float)]
[c (parse-error* "Invalid #x syntax: ~v" c)])]
[#\[ (read-base64-binary '())]
[#\! (embedded (decode-embedded (next)))]
[#\: (embedded (decode-embedded (next)))]
[c (on-hash c)])]
[c (on-char c)]))
@ -242,15 +240,13 @@
;;---------------------------------------------------------------------------
;; Hex-encoded floating point numbers
(define (read-hex-float precision)
(define (read-hex-float)
(unless (eqv? (next-char*) #\")
(parse-error* "Missing open-double-quote in hex-encoded floating-point number"))
(define bs (read-hex-binary '()))
(unless (= (bytes-length bs) (match precision ['float 4] ['double 8]))
(unless (= (bytes-length bs) 8)
(parse-error* "Incorrect number of bytes in hex-encoded floating-point number"))
(match precision
['float (bytes->float bs)]
['double (bytes->double bs)]))
(bytes->double bs))
;;---------------------------------------------------------------------------
;; Base64-encoded ByteStrings
@ -296,12 +292,8 @@
(define (analyze-number input)
(match input
[(pregexp #px"^([-+]?\\d+)(((\\.\\d+([eE][-+]?\\d+)?)|([eE][-+]?\\d+))([fF]?))?$"
(list _ whole _ frac _ _ _ f))
(define n (string->number (if frac (string-append whole frac) whole)))
(cond [(not n) #f]
[(and f (positive? (string-length f))) (float n)]
[else n])]
[(pregexp #px"^([-+]?\\d+)((\\.\\d+([eE][-+]?\\d+)?)|([eE][-+]?\\d+))?$" (? list?))
(string->number input)]
[_ #f]))
;;---------------------------------------------------------------------------

View File

@ -119,32 +119,6 @@
double15: @"-sNaN" <Test #x"8708fff8000000000111" #xd"fff8000000000111">
double16: @"+sNaN" <Test #x"87087ff8000000000001" #xd"7ff8000000000001">
double17: @"+sNaN" <Test #x"87087ff8000000000111" #xd"7ff8000000000111">
float0: <Test #x"870400000000" 0.0f>
float+0: <Test #x"870400000000" +0.0f>
float-0: <Test #x"870480000000" -0.0f>
float1: <Test #x"87043f800000" 1.0f>
float1u: <Test #x"87043f800000" 1.0F>
float1a: <Test #x"87043f800000" 1e0f>
float1b: <Test #x"87043f800000" 1.0e0f>
float1c: <Test #x"87043f800000" 1e-0f>
float1d: <Test #x"87043f800000" 1.0e-0f>
float1e: <Test #x"87043f800000" 1e+0f>
float1f: <Test #x"87043f800000" 1.0e+0f>
float2: <Test #x"870412345678" #xf"12 34 56 78">
float3: @"Fewer than 8 digits" <ParseError "#xf\"123456\"">
float4: @"More than 8 digits" <ParseError "#xf\"123456789a\"">
float5: @"Invalid chars" <ParseError "#xf\"12zz5678\"">
float6: @"Positive infinity" <Test #x"87047f800000" #xf"7f800000">
float7: @"Negative infinity" <Test #x"8704ff800000" #xf"ff800000">
float8: @"+sNaN" <Test #x"87047f800001" #xf"7f800001">
float9: @"+sNaN" <Test #x"87047f800111" #xf"7f800111">
float10: @"-sNaN" <Test #x"8704ff800001" #xf"ff800001">
float11: @"-sNaN" <Test #x"8704ff800111" #xf"ff800111">
float12: @"Bad spacing" <ParseError "#xf\"12345 678\"">
float13: @"+qNaN" <Test #x"87047fc00001" #xf"7fc00001">
float14: @"+qNaN" <Test #x"87047fc00111" #xf"7fc00111">
float15: @"-qNaN" <Test #x"8704ffc00001" #xf"ffc00001">
float16: @"-qNaN" <Test #x"8704ffc00111" #xf"ffc00111">
int-98765432109876543210987654321098765432109: <Test #x"b012feddc125aed4226c770369269596ce3f0ad3" -98765432109876543210987654321098765432109>
int-12345678123456781234567812345678: <Test #x"b00eff642cf6684f11d1dad08c4a10b2" -12345678123456781234567812345678>
int-1234567812345678123456781234567: <Test #x"b00df06ae570d4b4fb62ae746dce79" -1234567812345678123456781234567>
@ -193,9 +167,9 @@
list11: <Test #x"b5b0010184" [01]>
list12: <Test #x"b5b0010c84" [12]>
noinput0: @"No input at all" <DecodeEOF #x"">
embed0: <Test #x"86b000" #!0>
embed1: <Test #x"8686b000" #!#!0>
embed2: <Test #x"b586b00086b10568656c6c6f84" [#!0 #!"hello"]>
embed0: <Test #x"86b000" #:0>
embed1: <Test #x"8686b000" #:#:0>
embed2: <Test #x"b586b00086b10568656c6c6f84" [#:0 #:"hello"]>
record1: <Test #x"b4 b30763617074757265 b4 b30764697363617264 84 84" <capture <discard>>>
record2: <Test #x"b4 b3076f627365727665 b4 b305737065616b b4 b30764697363617264 84 b4 b30763617074757265 b4 b30764697363617264 84 84 84 84" <observe <speak <discard> <capture <discard>>>>>
record2a: @"Commas not allowed in records" <ParseError "<observe <speak <discard>, <capture <discard>>>>">

View File

@ -7,7 +7,6 @@
(require (only-in racket/port call-with-output-bytes))
(require "record.rkt")
(require "embedded.rkt")
(require "float.rkt")
(require "float-bytes.rkt")
(require "annotation.rkt")
(require "varint.rkt")
@ -87,10 +86,6 @@
[#f (output-byte #x80)]
[#t (output-byte #x81)]
[(float _)
(output-byte #x87)
(output-byte 4)
(output-bytes (float->bytes v))]
[(? flonum?)
(output-byte #x87)
(output-byte 8)

View File

@ -11,7 +11,6 @@
(require net/base64)
(require "embedded.rkt")
(require "annotation.rkt")
(require "float.rkt")
(require "float-bytes.rkt")
(require "record.rkt")
(require "object-id.rkt")
@ -135,14 +134,10 @@
(write-binary-stringlike v)
(write-binary-base64 outer-distance v)))))
(define (write-float v precision)
(define (write-float v)
(if (or (nan? v) (infinite? v))
(! "#x~a\"~a\""
(match precision ['float "f"] ['double "d"])
(bytes->hex-string (match precision
['float (float->bytes (float v))]
['double (double->bytes v)])))
(! "~v~a" v (match precision ['float "f"] ['double ""]))))
(! "#xd\"~a\"" (bytes->hex-string (double->bytes v)))
(! "~v" v)))
(define (write-value distance v)
(match v
@ -155,8 +150,7 @@
(write-value distance item)]
[#f (! "#f")]
[#t (! "#t")]
[(float v) (write-float v 'float)]
[(? flonum?) (write-float v 'double)]
[(? flonum?) (write-float v)]
[(? integer? x) (! "~v" v)]
[(? string?)
(! "\"")
@ -182,7 +176,7 @@
[(? set?) (write-sequence distance "#{" (if commas? "," "") "}" write-value (set->list v))]
[(? dict?) (write-sequence distance "{" (if commas? "," "") "}" write-key-value (dict->list v))]
[(embedded value)
(! "#!")
(! "#:")
(write-value distance (encode-embedded value))]
[other (error 'write-preserve/text "Attempt to serialize non-preserve: ~v" other)]))

View File

@ -1,3 +0,0 @@
Cargo.lock
scratch/
target/

View File

@ -1,15 +0,0 @@
[workspace]
exclude = [
"examples/schema-no-build",
]
members = [
"preserves",
"preserves-path",
"preserves-schema",
"preserves-schema-macros",
"preserves-tools",
]
[profile.bench]
debug = true

View File

@ -1,41 +0,0 @@
# Use cargo release to manage publication and versions etc.
#
# cargo install cargo-release
all:
cargo build --all-targets
doc:
cargo doc --workspace
x86_64-binary: x86_64-binary-release
x86_64-binary-release:
cross build --target=x86_64-unknown-linux-musl --release --all-targets
armv7-binary: armv7-binary-release
armv7-binary-release:
cross build --target=armv7-unknown-linux-musleabihf --release --all-targets
aarch64-binary: aarch64-binary-release
aarch64-binary-release:
cross build --target=aarch64-unknown-linux-musl --release --all-targets
test:
cargo test
test-all:
cargo test --all-targets
ws-bump:
cargo workspaces version \
--no-global-tag \
--individual-tag-prefix 'rust-%n@' \
--allow-branch 'main' \
--ignore-changes '../*'
ws-publish:
cargo workspaces publish \
--from-git

View File

@ -0,0 +1,13 @@
# Split out to separate repository
The Rust implementation of Preserves has been split out into a separate git repository,
<https://gitlab.com/preserves/preserves-rs>.
The final released versions that were here were
- `preserves` v4.992.2,
- `preserves-schema` v5.992.0,
- `preserves-tools` v4.992.2, and
- `preserves-path` v5.992.0.
Subsequent releases live in the other repository.

View File

@ -1,10 +0,0 @@
[package]
name = "schema-no-build"
version = "0.1.0"
edition = "2021"
[dependencies]
lazy_static = "1"
preserves = { path = "../../preserves" }
preserves-schema = { path = "../../preserves-schema" }
preserves-schema-macros = { path = "../../preserves-schema-macros" }

View File

@ -1,27 +0,0 @@
use preserves::value::IOValue;
use preserves_schema_macros::compile_preserves_schemas;
compile_preserves_schemas!(
crate::schemas,
load("<CARGO_MANIFEST_DIR>/../../../../path/path.bin"),
external_module(EntityRef = crate::demo_entity_ref),
);
pub mod demo_entity_ref {
use preserves::value::IOValue;
pub type Cap = IOValue;
}
preserves_schema::define_language!(language(): Language<IOValue> {
demo: crate::schemas::Language,
});
fn main() {
use crate::schemas::path::*;
use preserves::value::NestedValue;
use preserves_schema::support::Unparse;
println!("Hello, world! {:?}", (Filter::Compare {
op: Box::new(Comparison::Eq),
literal: IOValue::new(123),
}).unparse(language()));
}

View File

@ -1,23 +0,0 @@
[package]
name = "preserves-path"
version = "5.992.0"
authors = ["Tony Garnock-Jones <tonyg@leastfixedpoint.com>"]
edition = "2018"
description = "Implementation of preserves-path, a query language for Preserves documents."
homepage = "https://preserves.dev/"
repository = "https://gitlab.com/preserves/preserves"
license = "Apache-2.0"
[build-dependencies]
preserves-schema = { path = "../preserves-schema", version = "5.992.0"}
[dependencies]
preserves = { path = "../preserves", version = "4.992.1"}
preserves-schema = { path = "../preserves-schema", version = "5.992.0"}
num = "0.4"
regex = "1.5"
thiserror = "1.0"
[package.metadata.workspaces]
independent = true

View File

@ -1,18 +0,0 @@
use preserves_schema::compiler::*;
use std::io::Error;
use std::path::PathBuf;
fn main() -> Result<(), Error> {
let buildroot = PathBuf::from(std::env::var_os("OUT_DIR").unwrap());
let mut gen_dir = buildroot.clone();
gen_dir.push("src/schemas");
let mut c = CompilerConfig::new("crate::schemas".to_owned());
let inputs = expand_inputs(&vec!["path.bin".to_owned()])?;
c.load_schemas_and_bundles(&inputs, &vec![])?;
compile(&c, &mut CodeCollector::files(gen_dir))
}

View File

@ -1,7 +0,0 @@
´³schema·³version°³ definitions·³Axis´³orµµ±values´³rec´³lit³values„´³tupleµ„„„„µ± descendants´³rec´³lit³ descendants„´³tupleµ„„„„µ±at´³rec´³lit³at„´³tupleµ´³named³key³any„„„„„µ±label´³rec´³lit³label„´³tupleµ„„„„µ±keys´³rec´³lit³keys„´³tupleµ„„„„µ±length´³rec´³lit³length„´³tupleµ„„„„µ± annotations´³rec´³lit³ annotations„´³tupleµ„„„„µ±embedded´³rec´³lit³embedded„´³tupleµ„„„„µ±parse´³rec´³lit³parse„´³tupleµ´³named³module´³seqof´³atom³Symbol„„„´³named³name´³atom³Symbol„„„„„„µ±unparse´³rec´³lit³unparse„´³tupleµ´³named³module´³seqof´³atom³Symbol„„„´³named³name´³atom³Symbol„„„„„„„„³Step´³orµµ±Axis´³refµ„³Axis„„µ±Filter´³refµ„³Filter„„µ±Function´³refµ„³Function„„„„³Filter´³orµµ±nop´³rec´³lit³nop„´³tupleµ„„„„µ±compare´³rec´³lit³compare„´³tupleµ´³named³op´³refµ„³
Comparison„„´³named³literal³any„„„„„µ±regex´³rec´³lit³regex„´³tupleµ´³named³regex´³atom³String„„„„„„µ±test´³rec´³lit³test„´³tupleµ´³named³pred´³refµ„³ Predicate„„„„„„µ±real´³rec´³lit³real„´³tupleµ„„„„µ±int´³rec´³lit³int„´³tupleµ„„„„µ±kind´³rec´³lit³kind„´³tupleµ´³named³kind´³refµ„³ ValueKind„„„„„„„„³Function´³rec´³lit³count„´³tupleµ´³named³selector´³refµ„³Selector„„„„„³Selector´³seqof´³refµ„³Step„„³ Predicate´³orµµ±Selector´³refµ„³Selector„„µ±not´³rec´³lit³not„´³tupleµ´³named³pred´³refµ„³ Predicate„„„„„„µ±or´³rec´³lit³or„´³tupleµ´³named³preds´³seqof´³refµ„³ Predicate„„„„„„„µ±and´³rec´³lit³and„´³tupleµ´³named³preds´³seqof´³refµ„³ Predicate„„„„„„„„„³ ValueKind´³orµµ±Boolean´³lit³Boolean„„µ±Float´³lit³Float„„µ±Double´³lit³Double„„µ± SignedInteger´³lit³ SignedInteger„„µ±String´³lit³String„„µ±
ByteString´³lit³
ByteString„„µ±Symbol´³lit³Symbol„„µ±Record´³lit³Record„„µ±Sequence´³lit³Sequence„„µ±Set´³lit³Set„„µ±
Dictionary´³lit³
Dictionary„„µ±Embedded´³lit³Embedded„„„„³
Comparison´³orµµ±eq´³lit³eq„„µ±ne´³lit³ne„„µ±lt´³lit³lt„„µ±ge´³lit³ge„„µ±gt´³lit³gt„„µ±le´³lit³le„„„„„³ embeddedType€„„

View File

@ -1,52 +0,0 @@
use preserves::value::IOValue;
use preserves_schema::compiler::load_schema_or_bundle;
use preserves_schema::gen::schema::Definition;
use std::io;
#[derive(Default)]
pub struct Env(pub preserves_schema::support::interpret::Env<IOValue>);
pub struct Context<'a> {
pub env: &'a Env,
path: Vec<IOValue>,
}
impl<'a> Context<'a> {
pub fn new(env: &'a Env) -> Self {
Context {
env,
path: Vec::new(),
}
}
pub fn with_path_step<R, F: FnOnce(&mut Self) -> R>(&mut self, v: &IOValue, f: F) -> R {
self.path.push(v.clone());
let result = f(self);
self.path.pop();
result
}
}
impl Env {
pub fn new() -> Self {
Default::default()
}
pub fn load_bundle(&mut self, filename: &std::path::PathBuf) -> io::Result<bool> {
load_schema_or_bundle(&mut self.0, filename)
}
pub fn lookup_definition(
&self,
module: &Vec<String>,
name: &str,
) -> Option<&Definition<IOValue>> {
self.0.get(module).and_then(|s| s.definitions.0.get(name))
}
pub fn to_context(&self) -> Context {
Context::new(self)
}
}

View File

@ -1,17 +0,0 @@
use std::io;
use thiserror::Error;
#[derive(Error, Debug)]
pub enum CompilationError {
#[error(transparent)]
IoError(#[from] io::Error),
#[error("Cannot mix binary operators")]
MixedOperators,
#[error("Invalid step")]
InvalidStep,
#[error("Undefined schema definition name: {0}")]
UndefinedSchemaDefinitionName(String),
#[error(transparent)]
RegexError(#[from] regex::Error),
}

View File

@ -1,24 +0,0 @@
pub mod context;
pub mod error;
pub mod parse;
pub mod predicate;
pub mod schemas {
include!(concat!(env!("OUT_DIR"), "/src/schemas/mod.rs"));
}
pub mod step;
pub use context::Context;
pub use context::Env;
pub use error::CompilationError;
pub use parse::parse_predicate;
pub use parse::parse_selector;
pub use schemas::path::Predicate;
pub use schemas::path::Selector;
pub use schemas::path::Step;
pub use step::Node;

View File

@ -1,323 +0,0 @@
use crate::context::Env;
use crate::schemas::path;
use crate::step::Node;
use crate::CompilationError;
use preserves::value::BinarySource;
use preserves::value::BytesBinarySource;
use preserves::value::IOValue;
use preserves::value::NestedValue;
use preserves::value::Reader;
use std::iter::Iterator;
#[derive(Debug)]
enum Binop {
Union,
Intersection,
}
fn split_values_by_symbol<'a>(tokens: &'a [IOValue], separator: &str) -> Vec<&'a [IOValue]> {
tokens
.split(|t| matches!(t.value().as_symbol(), Some(s) if s == separator))
.collect()
}
fn split_binop(tokens: &[IOValue]) -> Result<(Vec<&[IOValue]>, Option<Binop>), CompilationError> {
let union_pieces = split_values_by_symbol(&tokens, "+");
let intersection_pieces = split_values_by_symbol(&tokens, "&");
match (union_pieces.len(), intersection_pieces.len()) {
(1, 1) => Ok((union_pieces, None)),
(_, 1) => Ok((union_pieces, Some(Binop::Union))),
(1, _) => Ok((intersection_pieces, Some(Binop::Intersection))),
_ => Err(CompilationError::MixedOperators),
}
}
pub fn parse_selector(env: &Env, tokens: &[IOValue]) -> Result<path::Selector, CompilationError> {
let mut steps = Vec::new();
let mut tokens = tokens;
while let Some((s, remaining)) = parse_step(env, tokens)? {
steps.push(s);
tokens = remaining;
}
Ok(path::Selector(steps))
}
pub fn parse_predicate(env: &Env, tokens: &[IOValue]) -> Result<path::Predicate, CompilationError> {
let (pieces, binop) = split_binop(tokens)?;
match binop {
None => parse_non_binop(env, &pieces[0]),
Some(o) => {
let preds = pieces
.into_iter()
.map(|ts| parse_non_binop(env, &ts))
.collect::<Result<_, _>>()?;
Ok(match o {
Binop::Union => path::Predicate::Or { preds },
Binop::Intersection => path::Predicate::And { preds },
})
}
}
}
fn parse_non_binop(env: &Env, tokens: &[IOValue]) -> Result<path::Predicate, CompilationError> {
if !tokens.is_empty() {
let t = tokens[0].value();
if let Some("!") = t.as_symbol().map(|s| s.as_str()) {
return Ok(path::Predicate::Not {
pred: Box::new(parse_non_binop(env, &tokens[1..])?),
});
}
}
Ok(path::Predicate::Selector(Box::new(parse_selector(
env, tokens,
)?)))
}
fn parse_schema_definition_name(
env: &Env,
token: &IOValue,
) -> Result<(Vec<String>, String), CompilationError> {
let defpath = token
.value()
.to_symbol()
.map_err(|_| CompilationError::InvalidStep)?;
let mut module: Vec<String> = defpath.split('.').map(|s| s.to_string()).collect();
let name = module
.pop()
.expect("at least one element in the Schema name");
match env.lookup_definition(&module, &name) {
Some(_) => Ok((module, name)),
None => Err(CompilationError::UndefinedSchemaDefinitionName(format!(
"{:?}",
token
))),
}
}
fn parse_step<'a>(
env: &Env,
tokens: &'a [IOValue],
) -> Result<Option<(path::Step, &'a [IOValue])>, CompilationError> {
if tokens.is_empty() {
return Ok(None);
}
let remainder = &tokens[1..];
if tokens[0].value().is_sequence() {
return Ok(Some((
path::Step::Filter(Box::new(path::Filter::Test {
pred: Box::new(parse_predicate(
env,
tokens[0].value().as_sequence().unwrap(),
)?),
})),
remainder,
)));
}
match tokens[0].value().as_record(None) {
None => (),
Some(r) => match r.label().value().as_symbol() {
None => return Err(CompilationError::InvalidStep),
Some(t) => match t.as_str() {
"count" => {
return Ok(Some((
path::Step::Function(Box::new(path::Function {
selector: parse_selector(env, r.fields())?,
})),
remainder,
)))
}
_ => return Err(CompilationError::InvalidStep),
},
},
}
match tokens[0].value().as_symbol() {
None => return Err(CompilationError::InvalidStep),
Some(t) => match t.as_str() {
"/" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Values)),
remainder,
))),
"//" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Descendants)),
remainder,
))),
"." => {
let (key, remainder) = pop_step_arg(remainder)?;
Ok(Some((
path::Step::Axis(Box::new(path::Axis::At { key })),
remainder,
)))
}
".^" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Label)),
remainder,
))),
".keys" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Keys)),
remainder,
))),
".length" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Length)),
remainder,
))),
".annotations" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Annotations)),
remainder,
))),
".embedded" => Ok(Some((
path::Step::Axis(Box::new(path::Axis::Embedded)),
remainder,
))),
"%" => {
let (defpath, remainder) = pop_step_arg(remainder)?;
let (module, name) = parse_schema_definition_name(env, &defpath)?;
Ok(Some((
path::Step::Axis(Box::new(path::Axis::Parse { module, name })),
remainder,
)))
}
"%-" => {
let (defpath, remainder) = pop_step_arg(remainder)?;
let (module, name) = parse_schema_definition_name(env, &defpath)?;
Ok(Some((
path::Step::Axis(Box::new(path::Axis::Unparse { module, name })),
remainder,
)))
}
"*" => Ok(Some((
path::Step::Filter(Box::new(path::Filter::Nop)),
remainder,
))),
"eq" | "=" => parse_comparison(remainder, path::Comparison::Eq),
"ne" | "!=" => parse_comparison(remainder, path::Comparison::Ne),
"lt" => parse_comparison(remainder, path::Comparison::Lt),
"gt" => parse_comparison(remainder, path::Comparison::Gt),
"le" => parse_comparison(remainder, path::Comparison::Le),
"ge" => parse_comparison(remainder, path::Comparison::Ge),
"re" | "=r" => {
let (regex_val, remainder) = pop_step_arg(remainder)?;
let regex = regex_val
.value()
.to_string()
.map_err(|_| CompilationError::InvalidStep)?
.clone();
let _ = regex::Regex::new(&regex)?;
Ok(Some((
path::Step::Filter(Box::new(path::Filter::Regex { regex })),
remainder,
)))
}
"^" => {
let (literal, remainder) = pop_step_arg(remainder)?;
Ok(Some((
path::Step::Filter(Box::new(path::Filter::Test {
pred: Box::new(path::Predicate::Selector(Box::new(path::Selector(vec![
path::Step::Axis(Box::new(path::Axis::Label)),
path::Step::Filter(Box::new(path::Filter::Compare {
op: Box::new(path::Comparison::Eq),
literal,
})),
])))),
})),
remainder,
)))
}
"~real" => Ok(Some((
path::Step::Filter(Box::new(path::Filter::Real)),
remainder,
))),
"~int" => Ok(Some((
path::Step::Filter(Box::new(path::Filter::Int)),
remainder,
))),
"bool" => Ok(Some((
path::Step::from(path::ValueKind::Boolean),
remainder,
))),
"float" => Ok(Some((path::Step::from(path::ValueKind::Float), remainder))),
"double" => Ok(Some((path::Step::from(path::ValueKind::Double), remainder))),
"int" => Ok(Some((
path::Step::from(path::ValueKind::SignedInteger),
remainder,
))),
"string" => Ok(Some((path::Step::from(path::ValueKind::String), remainder))),
"bytes" => Ok(Some((
path::Step::from(path::ValueKind::ByteString),
remainder,
))),
"symbol" => Ok(Some((path::Step::from(path::ValueKind::Symbol), remainder))),
"rec" => Ok(Some((path::Step::from(path::ValueKind::Record), remainder))),
"seq" => Ok(Some((
path::Step::from(path::ValueKind::Sequence),
remainder,
))),
"set" => Ok(Some((path::Step::from(path::ValueKind::Set), remainder))),
"dict" => Ok(Some((
path::Step::from(path::ValueKind::Dictionary),
remainder,
))),
"embedded" => Ok(Some((
path::Step::from(path::ValueKind::Embedded),
remainder,
))),
_ => Err(CompilationError::InvalidStep),
},
}
}
impl From<path::ValueKind> for path::Step {
fn from(k: path::ValueKind) -> Self {
path::Step::Filter(Box::new(path::Filter::Kind { kind: Box::new(k) }))
}
}
fn pop_step_arg(tokens: &[IOValue]) -> Result<(IOValue, &[IOValue]), CompilationError> {
if tokens.is_empty() {
return Err(CompilationError::InvalidStep);
}
Ok((tokens[0].clone(), &tokens[1..]))
}
fn parse_comparison(
tokens: &[IOValue],
op: path::Comparison,
) -> Result<Option<(path::Step, &[IOValue])>, CompilationError> {
let (literal, remainder) = pop_step_arg(tokens)?;
Ok(Some((
path::Step::Filter(Box::new(path::Filter::Compare {
op: Box::new(op),
literal,
})),
remainder,
)))
}
impl path::Selector {
pub fn from_str(env: &Env, s: &str) -> Result<Self, CompilationError> {
parse_selector(
env,
&(BytesBinarySource::new(s.as_bytes())
.text_iovalues()
.configured(false)
.collect::<Result<Vec<_>, _>>()?),
)
}
}
impl Node {
pub fn from_str(env: &Env, s: &str) -> Result<Self, CompilationError> {
let expr = path::Selector::from_str(env, s)?;
expr.compile()
}
}

View File

@ -1,68 +0,0 @@
use crate::context::Context;
use crate::schemas::path;
use crate::step::BoolCollector;
use crate::step::Node;
use crate::step::StepMaker;
use crate::CompilationError;
use preserves::value::IOValue;
pub trait Predicate: std::fmt::Debug {
fn test(&mut self, ctxt: &mut Context, value: &IOValue) -> bool;
}
#[derive(Debug)]
pub enum CompiledPredicate {
Selector(Node),
Not(Box<CompiledPredicate>),
Or(Vec<CompiledPredicate>),
And(Vec<CompiledPredicate>),
}
impl path::Predicate {
pub fn compile(&self) -> Result<CompiledPredicate, CompilationError> {
match self {
path::Predicate::Selector(b) => Ok(CompiledPredicate::Selector(
(&**b).connect(BoolCollector::new())?,
)),
path::Predicate::Not { pred } => {
Ok(CompiledPredicate::Not(Box::new((&**pred).compile()?)))
}
path::Predicate::Or { preds } => Ok(CompiledPredicate::Or(
preds.iter().map(Self::compile).collect::<Result<_, _>>()?,
)),
path::Predicate::And { preds } => Ok(CompiledPredicate::And(
preds.iter().map(Self::compile).collect::<Result<_, _>>()?,
)),
}
}
pub fn exec(&self, ctxt: &mut Context, value: &IOValue) -> Result<bool, CompilationError> {
Ok(self.compile()?.test(ctxt, value))
}
}
impl Predicate for CompiledPredicate {
fn test(&mut self, ctxt: &mut Context, value: &IOValue) -> bool {
match self {
CompiledPredicate::Selector(n) => n.test(ctxt, value),
CompiledPredicate::Not(p) => !p.test(ctxt, value),
CompiledPredicate::Or(ps) => {
for p in ps.iter_mut() {
if p.test(ctxt, value) {
return true;
}
}
return false;
}
CompiledPredicate::And(ps) => {
for p in ps.iter_mut() {
if !p.test(ctxt, value) {
return false;
}
}
return true;
}
}
}
}

View File

@ -1,511 +0,0 @@
// Selectors operate on IOValues because the AST includes keys of IOValue type.
// If we could make Schemas produce generics...
use crate::context::Context;
use crate::predicate::CompiledPredicate;
use crate::predicate::Predicate;
use crate::schemas::path;
use crate::CompilationError;
use num::bigint::BigInt;
use num::traits::cast::FromPrimitive;
use num::traits::cast::ToPrimitive;
use preserves::value::AtomClass;
use preserves::value::CompoundClass;
use preserves::value::IOValue;
use preserves::value::NestedValue;
use preserves::value::Value;
use preserves::value::ValueClass;
use preserves_schema::support::interpret;
use std::cell::RefCell;
use std::iter::Iterator;
use std::rc::Rc;
pub trait StepMaker {
fn connect(&self, step: Node) -> Result<Node, CompilationError>;
}
pub trait Step: std::fmt::Debug {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue);
fn finish(&mut self);
fn reset(&mut self) -> Vec<IOValue>;
}
macro_rules! delegate_finish_and_reset {
($self:ident, $target:expr) => {
fn finish(&mut $self) { $target.finish() }
fn reset(&mut $self) -> Vec<IOValue> { $target.reset() }
}
}
#[derive(Clone, Debug)]
pub struct Node(pub Rc<RefCell<dyn Step>>);
#[derive(Debug)]
struct AxisStep {
step: Node,
axis: path::Axis,
}
#[derive(Debug)]
struct CompareStep {
op: path::Comparison,
literal: IOValue,
step: Node,
}
#[derive(Debug)]
struct RegexStep {
regex: regex::Regex,
step: Node,
}
#[derive(Debug)]
struct TestStep {
pred: CompiledPredicate,
step: Node,
}
#[derive(Debug)]
struct RealStep {
step: Node,
}
#[derive(Debug)]
struct IntStep {
step: Node,
}
#[derive(Debug)]
struct VecCollector {
accumulator: Vec<IOValue>,
}
#[derive(Debug)]
pub struct BoolCollector {
seen_value: bool,
}
#[derive(Debug)]
struct KindStep {
kind: ValueClass,
step: Node,
}
#[derive(Debug)]
pub struct CountCollector {
count: usize,
}
#[derive(Debug)]
struct CountStep {
step: Node,
counter: Node,
}
impl Node {
fn new<S: Step + 'static>(s: S) -> Self {
Node(Rc::new(RefCell::new(s)))
}
pub fn test(&self, ctxt: &mut Context, value: &IOValue) -> bool {
!self.exec(ctxt, value).is_empty()
}
pub fn accept(&self, ctxt: &mut Context, value: &IOValue) {
self.0.borrow_mut().accept(ctxt, value)
}
pub fn finish(&self) {
self.0.borrow_mut().finish()
}
pub fn reset(&self) -> Vec<IOValue> {
self.0.borrow_mut().reset()
}
pub fn exec(&self, ctxt: &mut Context, value: &IOValue) -> Vec<IOValue> {
self.accept(ctxt, value);
self.finish();
self.reset()
}
}
impl StepMaker for path::Selector {
fn connect(&self, step: Node) -> Result<Node, CompilationError> {
self.0.connect(step)
}
}
impl<S: StepMaker> StepMaker for Vec<S> {
fn connect(&self, mut step: Node) -> Result<Node, CompilationError> {
for s in self.iter().rev() {
step = s.connect(step)?;
}
Ok(step)
}
}
impl StepMaker for path::Step {
fn connect(&self, step: Node) -> Result<Node, CompilationError> {
match self {
path::Step::Axis(b) => (&**b).connect(step),
path::Step::Filter(b) => (&**b).connect(step),
path::Step::Function(b) => (&**b).connect(step),
}
}
}
impl StepMaker for path::Axis {
fn connect(&self, step: Node) -> Result<Node, CompilationError> {
Ok(Node::new(AxisStep {
step,
axis: self.clone(),
}))
}
}
fn descendants(ctxt: &mut Context, step: &mut Node, v: &IOValue) {
step.accept(ctxt, v);
for c in v.value().children() {
ctxt.with_path_step(&c, |ctxt| descendants(ctxt, step, &c));
}
}
impl Step for AxisStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
ctxt.with_path_step(value, |ctxt| match &self.axis {
path::Axis::Values => {
for c in value.value().children() {
self.step.accept(ctxt, &c)
}
}
path::Axis::Descendants => descendants(ctxt, &mut self.step, value),
path::Axis::At { key } => match value.value() {
Value::String(s) | Value::Symbol(s) => step_index(
ctxt,
s.chars(),
&key,
|c| IOValue::new(String::from(c)),
&mut self.step,
),
Value::Record(r) => {
step_index(ctxt, r.fields().iter(), &key, |v| v.clone(), &mut self.step)
}
Value::Sequence(vs) => {
step_index(ctxt, vs.iter(), &key, |v| v.clone(), &mut self.step)
}
Value::Dictionary(d) => {
if let Some(v) = d.get(&key) {
self.step.accept(ctxt, v)
}
}
_ => (),
},
path::Axis::Label => {
if let Some(r) = value.value().as_record(None) {
self.step.accept(ctxt, r.label())
}
}
path::Axis::Keys => match value.value() {
Value::String(s) | Value::Symbol(s) => step_keys(ctxt, s.len(), &mut self.step),
Value::ByteString(bs) => step_keys(ctxt, bs.len(), &mut self.step),
Value::Record(r) => step_keys(ctxt, r.arity(), &mut self.step),
Value::Sequence(vs) => step_keys(ctxt, vs.len(), &mut self.step),
Value::Dictionary(d) => {
for k in d.keys() {
self.step.accept(ctxt, k)
}
}
_ => (),
},
path::Axis::Length => match value.value() {
Value::String(s) | Value::Symbol(s) => {
self.step.accept(ctxt, &IOValue::new(s.len()))
}
Value::ByteString(bs) => self.step.accept(ctxt, &IOValue::new(bs.len())),
Value::Record(r) => self.step.accept(ctxt, &IOValue::new(r.arity())),
Value::Sequence(vs) => self.step.accept(ctxt, &IOValue::new(vs.len())),
Value::Dictionary(d) => self.step.accept(ctxt, &IOValue::new(d.len())),
_ => self.step.accept(ctxt, &IOValue::new(0)),
},
path::Axis::Annotations => {
for c in value.annotations().slice() {
self.step.accept(ctxt, &c)
}
}
path::Axis::Embedded => {
if let Some(d) = value.value().as_embedded() {
self.step.accept(ctxt, d)
}
}
path::Axis::Parse { module, name } => {
if let Some(p) =
interpret::Context::new(&ctxt.env.0).dynamic_parse(module, name, value)
{
self.step.accept(ctxt, &p)
}
}
path::Axis::Unparse { module, name } => {
if let Some(p) =
interpret::Context::new(&ctxt.env.0).dynamic_unparse(module, name, value)
{
self.step.accept(ctxt, &p)
}
}
})
}
delegate_finish_and_reset!(self, self.step);
}
fn step_index<T, Ts: Iterator<Item = T>, F: FnOnce(T) -> IOValue>(
ctxt: &mut Context,
mut vs: Ts,
key: &IOValue,
f: F,
step: &mut Node,
) {
if let Some(i) = key.value().as_usize() {
match vs.nth(i) {
None => (),
Some(v) => step.accept(ctxt, &f(v)),
}
}
}
fn step_keys(ctxt: &mut Context, count: usize, step: &mut Node) {
for i in 0..count {
step.accept(ctxt, &IOValue::new(i))
}
}
impl StepMaker for path::Filter {
fn connect(&self, step: Node) -> Result<Node, CompilationError> {
match self {
path::Filter::Nop => Ok(step),
path::Filter::Compare { op, literal } => Ok(Node::new(CompareStep {
op: (**op).clone(),
literal: literal.clone(),
step,
})),
path::Filter::Regex { regex } => Ok(Node::new(RegexStep {
regex: regex::Regex::new(regex)?,
step,
})),
path::Filter::Test { pred } => Ok(Node::new(TestStep {
pred: (&**pred).compile()?,
step,
})),
path::Filter::Real => Ok(Node::new(RealStep { step })),
path::Filter::Int => Ok(Node::new(IntStep { step })),
path::Filter::Kind { kind } => Ok(Node::new(KindStep {
kind: match &**kind {
path::ValueKind::Boolean => ValueClass::Atomic(AtomClass::Boolean),
path::ValueKind::Float => ValueClass::Atomic(AtomClass::Float),
path::ValueKind::Double => ValueClass::Atomic(AtomClass::Double),
path::ValueKind::SignedInteger => ValueClass::Atomic(AtomClass::SignedInteger),
path::ValueKind::String => ValueClass::Atomic(AtomClass::String),
path::ValueKind::ByteString => ValueClass::Atomic(AtomClass::ByteString),
path::ValueKind::Symbol => ValueClass::Atomic(AtomClass::Symbol),
path::ValueKind::Record => ValueClass::Compound(CompoundClass::Record),
path::ValueKind::Sequence => ValueClass::Compound(CompoundClass::Sequence),
path::ValueKind::Set => ValueClass::Compound(CompoundClass::Set),
path::ValueKind::Dictionary => ValueClass::Compound(CompoundClass::Dictionary),
path::ValueKind::Embedded => ValueClass::Embedded,
},
step,
})),
}
}
}
impl Step for CompareStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
if match self.op {
path::Comparison::Eq => value == &self.literal,
path::Comparison::Ne => value != &self.literal,
path::Comparison::Lt => value < &self.literal,
path::Comparison::Ge => value >= &self.literal,
path::Comparison::Gt => value > &self.literal,
path::Comparison::Le => value <= &self.literal,
} {
self.step.accept(ctxt, value)
}
}
delegate_finish_and_reset!(self, self.step);
}
impl Step for RegexStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
match value.value() {
Value::String(s) | Value::Symbol(s) => {
if self.regex.is_match(s) {
self.step.accept(ctxt, value)
}
}
_ => (),
}
}
delegate_finish_and_reset!(self, self.step);
}
impl Step for TestStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
if self.pred.test(ctxt, value) {
self.step.accept(ctxt, value)
}
}
delegate_finish_and_reset!(self, self.step);
}
impl Step for RealStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
match value.value() {
Value::SignedInteger(i) => {
if let Some(r) = BigInt::from(i).to_f64() {
self.step.accept(ctxt, &IOValue::new(r))
}
}
Value::Float(f) => self.step.accept(ctxt, &IOValue::new(f32::from(*f) as f64)),
Value::Double(_) => self.step.accept(ctxt, value),
_ => (),
}
}
delegate_finish_and_reset!(self, self.step);
}
impl Step for IntStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
match value.value() {
Value::SignedInteger(_) => self.step.accept(ctxt, value),
Value::Float(f) => {
if let Some(i) = BigInt::from_f32(f32::from(*f)) {
self.step.accept(ctxt, &IOValue::new(i))
}
}
Value::Double(d) => {
if let Some(i) = BigInt::from_f64(f64::from(*d)) {
self.step.accept(ctxt, &IOValue::new(i))
}
}
_ => (),
}
}
delegate_finish_and_reset!(self, self.step);
}
impl VecCollector {
fn new() -> Node {
Node::new(VecCollector {
accumulator: Vec::new(),
})
}
}
impl Step for VecCollector {
fn accept(&mut self, _ctxt: &mut Context, value: &IOValue) {
self.accumulator.push(value.clone())
}
fn finish(&mut self) {}
fn reset(&mut self) -> Vec<IOValue> {
std::mem::take(&mut self.accumulator)
}
}
impl BoolCollector {
pub fn new() -> Node {
Node::new(BoolCollector { seen_value: false })
}
}
impl Step for BoolCollector {
fn accept(&mut self, _ctxt: &mut Context, _value: &IOValue) {
self.seen_value = true
}
fn finish(&mut self) {}
fn reset(&mut self) -> Vec<IOValue> {
let result = if self.seen_value {
vec![IOValue::new(true)]
} else {
vec![]
};
self.seen_value = false;
result
}
}
impl Step for KindStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
if value.value_class() == self.kind {
self.step.accept(ctxt, value)
}
}
delegate_finish_and_reset!(self, self.step);
}
impl path::Selector {
pub fn compile(&self) -> Result<Node, CompilationError> {
self.connect(VecCollector::new())
}
pub fn exec(
&self,
ctxt: &mut Context,
value: &IOValue,
) -> Result<Vec<IOValue>, CompilationError> {
Ok(self.compile()?.exec(ctxt, value))
}
}
impl StepMaker for path::Function {
fn connect(&self, step: Node) -> Result<Node, CompilationError> {
// For now, there's just one function: `count`.
Ok(Node::new(CountStep {
step,
counter: self.selector.connect(CountCollector::new())?,
}))
}
}
impl CountCollector {
pub fn new() -> Node {
Node::new(CountCollector { count: 0 })
}
}
impl Step for CountCollector {
fn accept(&mut self, _ctxt: &mut Context, _value: &IOValue) {
self.count += 1
}
fn finish(&mut self) {}
fn reset(&mut self) -> Vec<IOValue> {
let result = vec![IOValue::new(self.count)];
self.count = 0;
result
}
}
impl Step for CountStep {
fn accept(&mut self, ctxt: &mut Context, value: &IOValue) {
let count = &self.counter.exec(ctxt, value)[0];
self.step.accept(ctxt, count)
}
delegate_finish_and_reset!(self, self.step);
}

View File

@ -1,23 +0,0 @@
[package]
name = "preserves-schema-macros"
version = "0.992.1"
authors = ["Tony Garnock-Jones <tonyg@leastfixedpoint.com>"]
edition = "2018"
description = "Implementation of Preserves Schema code generation macros for Rust."
homepage = "https://preserves.dev/"
repository = "https://gitlab.com/preserves/preserves"
license = "Apache-2.0"
[lib]
proc-macro = true
[dependencies]
preserves = { path = "../preserves", version = "4.992.1" }
preserves-schema = { path = "../preserves-schema", version = "5.992.0" }
proc-macro2 = { version = "1", features = ["span-locations"] }
quote = "1"
syn = { version = "2", features = ["parsing", "extra-traits", "full"] }
[package.metadata.workspaces]
independent = true

View File

@ -1,227 +0,0 @@
use preserves::value::Map;
use preserves_schema::compiler::*;
use preserves_schema::compiler::types::Purpose;
use preserves_schema::gen::schema::Schema;
use proc_macro2::Span;
use quote::ToTokens;
use quote::quote;
use std::fmt::Display;
use syn::LitStr;
use syn::Token;
use syn::parenthesized;
use syn::parse::Parser;
use syn::punctuated::Punctuated;
mod kw {
use syn::custom_keyword;
custom_keyword!(load);
custom_keyword!(cross_reference);
custom_keyword!(external_module);
}
#[derive(Debug)]
enum Instruction {
Namespace(String),
Load(LitStr),
CrossReference {
namespace: String,
bundle_path: LitStr,
},
ExternalModule {
module_path: ModulePath,
namespace: String,
}
}
fn syn_path_string(p: syn::Path) -> String {
p.to_token_stream().to_string().replace(&[' ', '\t', '\n', '\r'], "")
}
fn syn_litstr_resolve(s: &syn::LitStr) -> String {
let s: String = s.value();
match s.chars().nth(0) {
Some('/') => s.into(),
Some('<') => match &s[1..].split_once('>') {
Some((envvar, remainder)) => match std::env::var(envvar) {
Ok(p) => p + "/" + remainder,
Err(_) => panic!("No such environment variable: {:?}", s),
}
None => panic!("Invalid relative path syntax: {:?}", s),
},
_ => panic!("Invalid path syntax: {:?}", s)
}
}
impl syn::parse::Parse for Instruction {
fn parse(input: syn::parse::ParseStream) -> syn::Result<Self> {
let lookahead = input.lookahead1();
if lookahead.peek(kw::load) {
let _: kw::load = input.parse()?;
let content;
let _ = parenthesized!(content in input);
let bundle_path: syn::LitStr = content.parse()?;
Ok(Instruction::Load(bundle_path))
} else if lookahead.peek(kw::cross_reference) {
let _: kw::cross_reference = input.parse()?;
let content;
let _ = parenthesized!(content in input);
let namespace: syn::Path = content.parse()?;
let _: Token![=] = content.parse()?;
let bundle_path: syn::LitStr = content.parse()?;
Ok(Instruction::CrossReference {
namespace: syn_path_string(namespace),
bundle_path,
})
} else if lookahead.peek(kw::external_module) {
let _: kw::external_module = input.parse()?;
let content;
let _ = parenthesized!(content in input);
let module_path = Punctuated::<syn::Ident, syn::Token![.]>::parse_separated_nonempty(&content)?;
let _: Token![=] = content.parse()?;
let namespace: syn::Path = content.parse()?;
Ok(Instruction::ExternalModule {
module_path: module_path.into_iter().map(|p| p.to_string()).collect(),
namespace: syn_path_string(namespace),
})
} else {
let ns: syn::Path = input.parse()?;
Ok(Instruction::Namespace(syn_path_string(ns)))
}
}
}
struct ModuleTree {
own_body: String,
children: Map<String, ModuleTree>,
}
impl Default for ModuleTree {
fn default() -> Self {
ModuleTree {
own_body: String::new(),
children: Map::default(),
}
}
}
impl ModuleTree {
fn build(outputs: Map<Option<ModulePath>, String>) -> Self {
let mut mt = ModuleTree::default();
for (p, c) in outputs.into_iter() {
match p {
None => mt.own_body = c,
Some(k) => {
let mut r = &mut mt;
for e in k { r = mt.children.entry(names::render_modname(&e)).or_default(); }
r.own_body = c;
}
}
}
mt
}
}
impl Display for ModuleTree {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}", self.own_body)?;
for (label, mt) in self.children.iter() {
write!(f, "\npub mod {} {{ ", label)?;
mt.fmt(f)?;
write!(f, "}}")?;
}
Ok(())
}
}
#[proc_macro]
pub fn compile_preserves_schemas(src: proc_macro::TokenStream) -> proc_macro::TokenStream {
let instructions = Punctuated::<Instruction, syn::Token![,]>::parse_terminated
.parse(src)
.expect("valid sequence of compile_preserves_schemas instructions");
let mut namespace = None::<String>;
let mut bundles_to_load = Vec::<LitStr>::new();
let mut bundles_to_xref = Vec::<LitStr>::new();
let mut external_modules = Vec::<ExternalModule>::new();
for i in instructions.into_iter() {
match i {
Instruction::Namespace(n) => {
if namespace.is_some() {
panic!("Only one namespace is permitted")
}
namespace = Some(n)
}
Instruction::Load(p) => bundles_to_load.push(p),
Instruction::ExternalModule { module_path, namespace } => {
external_modules.push(ExternalModule::new(module_path, &namespace));
}
Instruction::CrossReference { namespace, bundle_path } => {
let mut bundle = Map::<ModulePath, Schema>::new();
let is_schema = load_schema_or_bundle(&mut bundle, &syn_litstr_resolve(&bundle_path).into())
.expect("Invalid schema/bundle binary");
bundles_to_xref.push(bundle_path);
for (k, _v) in bundle.into_iter() {
external_modules.push(if is_schema {
ExternalModule::new(k, &namespace)
} else {
let ns = namespace.clone();
let mut pieces = vec![ns.clone()];
pieces.extend(
k.iter().map(|p| names::render_modname(&p)).collect::<Vec<_>>());
ExternalModule::new(k, &pieces.join("::"))
.set_fallback_language_types(
move |v| vec![format!("{}::Language<{}>", ns, v)].into_iter().collect())
});
}
}
}
}
let namespace = namespace.expect("Missing namespace");
let mut dependency_paths = Vec::<syn::LitStr>::new();
let mut c = CompilerConfig::new(namespace.clone());
for b in bundles_to_load.into_iter() {
dependency_paths.push(syn::LitStr::new(&syn_litstr_resolve(&b), Span::call_site()));
load_schema_or_bundle_with_purpose(&mut c.bundle, &syn_litstr_resolve(&b).into(), Purpose::Codegen)
.expect(&b.value());
}
for b in bundles_to_xref.into_iter() {
dependency_paths.push(syn::LitStr::new(&syn_litstr_resolve(&b), Span::call_site()));
load_schema_or_bundle_with_purpose(&mut c.bundle, &syn_litstr_resolve(&b).into(), Purpose::Xref)
.expect(&b.value());
}
for m in external_modules.into_iter() {
c.add_external_module(m);
}
let mut outputs = Map::<Option<ModulePath>, String>::new();
let mut collector = CodeCollector {
emit_mod_declarations: false,
collect_module: CodeModuleCollector::Custom {
collect_output: &mut |p, c| {
outputs.insert(p.cloned(), c.to_owned());
Ok(())
},
},
};
compile(&c, &mut collector).expect("Compilation failed");
let top_module_source = format!(
"pub mod {} {{ {} }}",
names::render_modname(namespace.split("::").last().unwrap()),
ModuleTree::build(outputs));
let top_module: syn::Item = syn::parse_str(&top_module_source)
.expect("Invalid generated code");
quote!{
// TODO: this is ugly, but makes the code depend on the actual schema bundle:
// See https://doc.rust-lang.org/nightly/proc_macro/tracked_path/fn.path.html
// and https://github.com/rust-lang/rust/issues/99515
#( const _: &'static [u8] = include_bytes!(#dependency_paths); )*
#top_module
}.into()
}

View File

@ -1,22 +0,0 @@
[package]
name = "preserves-schema"
version = "5.992.0"
authors = ["Tony Garnock-Jones <tonyg@leastfixedpoint.com>"]
edition = "2018"
description = "Implementation of Preserves Schema code generation and support for Rust."
homepage = "https://preserves.dev/"
repository = "https://gitlab.com/preserves/preserves"
license = "Apache-2.0"
[dependencies]
preserves = { path = "../preserves", version = "4.992.1"}
convert_case = "0.4.0"
glob = "0.3.0"
lazy_static = "1.4.0"
regex = "1.5"
structopt = "0.3.14"
thiserror = "1.0"
[package.metadata.workspaces]
independent = true

View File

@ -1,10 +0,0 @@
all:
@echo please use cargo
regenerate:
cargo run -- \
--prefix crate::gen \
--support-crate crate \
--rustfmt-skip \
-o $(CURDIR)/src/gen \
../../../schema/schema.bin

View File

@ -1,6 +0,0 @@
```shell
cargo add preserves preserves-schema
```
This crate ([`preserves-schema` on crates.io](https://crates.io/crates/preserves-schema)) is an
implementation of [Preserves Schema](https://preserves.dev/preserves-schema.html) for Rust.

View File

@ -1,112 +0,0 @@
# Example
[preserves-schemac]: https://preserves.dev/doc/preserves-schemac.html
[preserves-schema-rs]: https://preserves.dev/doc/preserves-schema-rs.html
Preserves schemas are written in a syntax that (ab)uses [Preserves text
syntax][preserves::value::text] as a kind of S-expression. Schema source code looks like this:
```preserves-schema
version 1 .
Present = <Present @username string> .
Says = <Says @who string @what string> .
UserStatus = <Status @username string @status Status> .
Status = =here / <away @since TimeStamp> .
TimeStamp = string .
```
Conventionally, schema source code is stored in `*.prs` files. In this example, the source code
above is placed in `simpleChatProtocol.prs`.
The Rust code generator for schemas requires not source code, but instances of the [Preserves
metaschema](https://preserves.dev/preserves-schema.html#appendix-metaschema). To compile schema
source code to metaschema instances, use [preserves-schemac][]:
```shell
yarn global add @preserves/schema
preserves-schemac .:simpleChatProtocol.prs > simpleChatProtocol.prb
```
Binary-syntax metaschema instances are conventionally stored in `*.prb` files. If you have a
whole directory tree of `*.prs` files, you can supply just "`.`" without the "`:`"-prefixed
fileglob part.[^converting-metaschema-to-text] See the [preserves-schemac documentation][preserves-schemac].
[^converting-metaschema-to-text]:
Converting the `simpleChatProtocol.prb` file to Preserves text syntax lets us read the
metaschema instance corresponding to the source code:
```shell
cat simpleChatProtocol.prb | preserves-tool convert
```
The result:
```preserves
<bundle {
[
simpleChatProtocol
]: <schema {
definitions: {
Present: <rec <lit Present> <tuple [
<named username <atom String>>
]>>
Says: <rec <lit Says> <tuple [
<named who <atom String>>
<named what <atom String>>
]>>
Status: <or [
[
"here"
<lit here>
]
[
"away"
<rec <lit away> <tuple [
<named since <ref [] TimeStamp>>
]>>
]
]>
TimeStamp: <atom String>
UserStatus: <rec <lit Status> <tuple [
<named username <atom String>>
<named status <ref [] Status>>
]>>
}
embeddedType: #f
version: 1
}>
}>
```
#### Generating Rust code from a schema
Generate Rust definitions corresponding to a metaschema instance with [preserves-schema-rs][].
The best way to use it is to integrate it into your `build.rs` (see [the
docs][preserves-schema-rs]), but you can also use it as a standalone command-line tool.
The following command generates a directory `./rs/chat` containing rust sources for a module
that expects to be called `chat` in Rust code:
```shell
preserves-schema-rs --output-dir rs/chat --prefix chat simpleChatProtocol.prb
```
Representative excerpts from one of the generated files, `./rs/chat/simple_chat_protocol.rs`:
```rust,noplayground
pub struct Present {
pub username: std::string::String
}
pub struct Says {
pub who: std::string::String,
pub what: std::string::String
}
pub struct UserStatus {
pub username: std::string::String,
pub status: Status
}
pub enum Status {
Here,
Away {
since: std::boxed::Box<TimeStamp>
}
}
pub struct TimeStamp(pub std::string::String);
```

View File

@ -1,16 +0,0 @@
A Preserves schema connects Preserves `Value`s to host-language data
structures. Each definition within a schema can be processed by a
compiler to produce
- a simple host-language *type definition*;
- a partial *parsing* function from `Value`s to instances of the
produced type; and
- a total *serialization* function from instances of the type to
`Value`s.
Every parsed `Value` retains enough information to always be able to
be serialized again, and every instance of a host-language data
structure contains, by construction, enough information to be
successfully serialized.

Some files were not shown because too many files have changed in this diff Show More