Experiment with zero-copy format
This commit is contained in:
parent
54b9bb6f25
commit
9f556fd9e6
|
@ -0,0 +1,211 @@
|
||||||
|
---
|
||||||
|
no_site_title: true
|
||||||
|
title: "Preserves: Zero-copy Binary Syntax"
|
||||||
|
---
|
||||||
|
|
||||||
|
Tony Garnock-Jones <tonyg@leastfixedpoint.com>
|
||||||
|
{{ site.version_date }}. Version {{ site.version }}.
|
||||||
|
|
||||||
|
*Preserves* is a data model, with associated serialization formats. This
|
||||||
|
document defines one of those formats: a binary syntax for `Value`s from
|
||||||
|
the [Preserves data model](preserves.html) that avoids, in many cases, use
|
||||||
|
of intermediate data structures during reading and writing. This makes it
|
||||||
|
suitable for use for representation of very large values whose
|
||||||
|
fully-decoded representations may not fit in working memory.
|
||||||
|
|
||||||
|
## Zero-Copy Binary Syntax
|
||||||
|
|
||||||
|
A `Buf` is a zero-copy syntax encoding, or representation, of a
|
||||||
|
non-immediate `Value`. A `Ref` is either a type-tagged representation of a
|
||||||
|
small immediate `Value` or a type-tagged pointer to a `Buf`.
|
||||||
|
|
||||||
|
Each `Ref` is a 64-bit unsigned value. Its tag appears in the low 4 bits.
|
||||||
|
The remaining 60 bits encode either an unsigned offset pointing to a
|
||||||
|
previously-encoded `Buf`, or an immediate value. Pointers always point
|
||||||
|
backwards to earlier positions.
|
||||||
|
|
||||||
|
Each `Buf` is prefixed with a 64-bit payload length, counted in units of
|
||||||
|
bytes, and is zero-padded to the nearest multiple of 16 bytes. Neither the
|
||||||
|
length of the padding nor the length of the length itself are included in
|
||||||
|
the length.
|
||||||
|
|
||||||
|
Offsets in pointer `Ref`s are counted in 16-byte units, measuring from the
|
||||||
|
beginning of the length indicator of the `Buf` in which the `Ref` appears.
|
||||||
|
A zero offset is special: it denotes an *empty value* of the type
|
||||||
|
associated with the tag in the `Ref`.
|
||||||
|
|
||||||
|
All multi-byte quantities are encoded using little-endian byte order.
|
||||||
|
|
||||||
|
### Header.
|
||||||
|
|
||||||
|
Because `Ref`s are typed, but `Buf`s are not, the outermost `Value` in e.g.
|
||||||
|
a file or network stream is always encoded preceded by a special header.
|
||||||
|
|
||||||
|
Offset Length Description
|
||||||
|
-------- ------ -----------
|
||||||
|
00000000 1 Marker byte 0xFF
|
||||||
|
00000001 1 Version number 0x00
|
||||||
|
00000002 6 Reserved, 0x00
|
||||||
|
00000008 8 Special Ref
|
||||||
|
00000010 8 Length ("n") of encoded data, in bytes
|
||||||
|
00000018 n Encoded data
|
||||||
|
- - Zero-padding to next 16-byte boundary
|
||||||
|
|
||||||
|
The `Ref` in the header at offset 8 is special.
|
||||||
|
|
||||||
|
If it encodes an immediate `Value`, that `Value` is the encoded value, and
|
||||||
|
the length field and encoded data are omitted. The entire encoded value is
|
||||||
|
exactly 16 bytes long in this case.
|
||||||
|
|
||||||
|
However, if the special `Ref` is an encoding of a pointer to a `Buf`, the
|
||||||
|
offset is interpreted as counting back from the very end of the padding at
|
||||||
|
the end of the encoded data. The entire encoded value is the length of the
|
||||||
|
encoded data, plus 24, rounded up to the next multiple of 16.
|
||||||
|
|
||||||
|
Either way, the tag on the special `Ref` is the type of the encoded value.
|
||||||
|
|
||||||
|
### Tags and Refs.
|
||||||
|
|
||||||
|
Tag Type Interpretation of 60-bit payload
|
||||||
|
--- ------------- --------------------------------
|
||||||
|
0 Boolean 0 = False, 1 = True
|
||||||
|
1 IEEE 754 Offset to Buf holding little-endian 32/64-bit float
|
||||||
|
2 SignedInteger Signed 60-bit integer
|
||||||
|
3 SignedInteger Offset to Buf holding little-endian signed integer
|
||||||
|
4 String 0-7 bytes of UTF-8; length in lower 4 bits
|
||||||
|
5 String Offset to Buf holding UTF-8 data
|
||||||
|
6 ByteString 0-7 bytes of raw binary; length in lower 4 bits
|
||||||
|
7 ByteString Offset to Buf holding raw binary data
|
||||||
|
8 Symbol 0-7 bytes of UTF-8; length in lower 4 bits
|
||||||
|
9 Symbol Offset to Buf holding UTF-8 data
|
||||||
|
A Record Offset to Buf holding Refs (label, fields)
|
||||||
|
B Sequence Offset to Buf holding Refs (sequence values)
|
||||||
|
C Set Offset to Buf holding Refs (elements in arbitrary order)
|
||||||
|
D Dictionary Offset to Buf holding Refs (key/value pairs)
|
||||||
|
E Embedded Offset to Buf holding a single Ref
|
||||||
|
F - (reserved)
|
||||||
|
|
||||||
|
### Records, Sequences, Sets and Dictionaries.
|
||||||
|
|
||||||
|
Offset Length Description
|
||||||
|
-------- ------ -----------
|
||||||
|
00000000 8 n*8: length of following sequence of n Refs, in bytes
|
||||||
|
00000008 8 Ref 0
|
||||||
|
... ... ...
|
||||||
|
n*8 8 Ref n-1
|
||||||
|
(n+1)*8 8 Padding, only if n is even
|
||||||
|
|
||||||
|
Each compound datum is represented as a sequence of `Ref`s representing the
|
||||||
|
contained `Value`s. Each `Record`'s sequence represents the label, followed
|
||||||
|
by the fields in order. Each `Sequence`'s representation is just its
|
||||||
|
contained values in order. `Set`s are ordered arbitrarily into a sequence.
|
||||||
|
The key-value pairs in a `Dictionary` are ordered arbitrarily, alternating
|
||||||
|
between keys and their matching values.
|
||||||
|
|
||||||
|
There is *no* ordering requirement on the elements of `Set`s or the
|
||||||
|
key-value pairs in a `Dictionary`. They may appear in any order. However,
|
||||||
|
the elements and keys *MUST* be pairwise distinct according to the
|
||||||
|
[Preserves equivalence relation](preserves.html#equivalence).
|
||||||
|
|
||||||
|
### SignedIntegers.
|
||||||
|
|
||||||
|
Integers between -2<sup>59</sup> and 2<sup>59</sup>-1, inclusive, are
|
||||||
|
represented as immediate values in a `Ref` with tag 2. Integers outside
|
||||||
|
this range are represented with a `Ref` with tag 3 pointing to a `Buf`
|
||||||
|
containing exactly as many 64-bit words as needed to unambiguously identify
|
||||||
|
the value and its sign, in little-endian byte and word ordering. Every
|
||||||
|
`SignedInteger` *MUST* be represented with its shortest possible encoding.
|
||||||
|
|
||||||
|
For example,
|
||||||
|
|
||||||
|
Number (decimal) Ref (64-bit) Buf (hex bytes)
|
||||||
|
----------------------------------------- ---------------- ----------------
|
||||||
|
-576460752303423488 8000000000000002 -
|
||||||
|
-257 FFFFFFFFFFFFEFF2 -
|
||||||
|
-1 FFFFFFFFFFFFFFF2 -
|
||||||
|
0 0000000000000002 -
|
||||||
|
1 0000000000000012 -
|
||||||
|
257 0000000000001012 -
|
||||||
|
576460752303423487 7FFFFFFFFFFFFFF2 -
|
||||||
|
|
||||||
|
1000000000000000000000000000000 ...............3 1000000000000000
|
||||||
|
00000040EAED7446
|
||||||
|
D09C2C9F0C000000
|
||||||
|
0000000000000000
|
||||||
|
|
||||||
|
-1000000000000000000000000000000 ...............3 1000000000000000
|
||||||
|
000000C015128BB9
|
||||||
|
2F63D360F3FFFFFF
|
||||||
|
0000000000000000
|
||||||
|
|
||||||
|
87112285931760246646623899502532662132736 ...............3 1800000000000000
|
||||||
|
0000000000000000
|
||||||
|
0000000000000000
|
||||||
|
0001000000000000
|
||||||
|
|
||||||
|
### Strings, ByteStrings and Symbols.
|
||||||
|
|
||||||
|
Syntax for these three types varies only in the tag used. For `String` and
|
||||||
|
`Symbol`, the encoded data is a UTF-8 encoding of the `Value`'s code
|
||||||
|
points, while for `ByteString` it is the raw data contained within the
|
||||||
|
`Value` unmodified.
|
||||||
|
|
||||||
|
Encoded data of length 7 bytes or shorter is represented as an immediate
|
||||||
|
`Ref` with tag 4 (`String`), 6 (`ByteString`) or 8 (`Symbol`). The lower 4
|
||||||
|
bits of the 60-bit payload are the length of the encoded data; the upper 56
|
||||||
|
bits are 7 bytes of data, with the first data byte in the uppermost byte.
|
||||||
|
|
||||||
|
Data longer than 7 bytes is represented with a `Ref` with tag 5, 7 or 9
|
||||||
|
pointing to a `Buf` containing the bytes of encoded data. Empty values
|
||||||
|
(length 0) *MUST* be encoded using immediate `Ref` form.
|
||||||
|
|
||||||
|
For example,
|
||||||
|
|
||||||
|
Value Ref (64-bit) Buf (hex bytes)
|
||||||
|
----------------------------------------- ---------------- ----------------
|
||||||
|
"" 0000000000000004 -
|
||||||
|
#"" 0000000000000006 -
|
||||||
|
|| 0000000000000008 -
|
||||||
|
"Hello" 48656C6C6F000054 -
|
||||||
|
"a\0a" 6100610000000034 -
|
||||||
|
|
||||||
|
"Hello, world!" ...............5 0D00000000000000
|
||||||
|
48656C6C6F2C2077
|
||||||
|
6F726C6421000000
|
||||||
|
0000000000000000
|
||||||
|
|
||||||
|
### Booleans.
|
||||||
|
|
||||||
|
Value Ref (64-bit) Buf (hex bytes)
|
||||||
|
----------------------------------------- ---------------- ----------------
|
||||||
|
#f 0000000000000000 -
|
||||||
|
#t 0000000000000010 -
|
||||||
|
|
||||||
|
### Floats and Doubles.
|
||||||
|
|
||||||
|
Each IEEE 754 4- and 8-byte binary representation is encoded into a `Buf`,
|
||||||
|
pointed to with a `Ref` with tag 1. The length of the `Buf` disambiguates
|
||||||
|
between 32-bit floats and 64-bit doubles.
|
||||||
|
|
||||||
|
((This is a very sparse encoding! Each float/double takes up 24 bytes split
|
||||||
|
across the `Buf` and `Ref`.))
|
||||||
|
|
||||||
|
### Embeddeds.
|
||||||
|
|
||||||
|
To encode an `Embedded`, first choose a `Value` to represent the denoted
|
||||||
|
object, and encode that, producing a `Ref`. Place that ref in a `Buf` all
|
||||||
|
of its own (with length 8). Finally, point to the `Buf` with a `Ref` with
|
||||||
|
tag 15.
|
||||||
|
|
||||||
|
### Annotations.
|
||||||
|
|
||||||
|
((Not sure: put them as a trailer after a Header?))
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
((TBD))
|
||||||
|
|
||||||
|
## Appendix. Autodetection of textual or binary syntax
|
||||||
|
|
||||||
|
The first byte of a Header is 0xFF, which may not appear in any UTF-8
|
||||||
|
string. ((...))
|
Loading…
Reference in New Issue