Schema Reference

A schema tells EByte what a binary format looks like: the fields in order, how wide each one is, and how they depend on each other. This page covers version 1 of the schema format, the one EByte reads. The repeat type needs EByte 0.2.0 or later, derive forms other than count_of an array need 0.3.0, and checksums need 0.4.0. Worked schemas for real formats are in the schema library. The full contract, with every edge case, is schema-v1.md on GitHub.

The File

A schema is a UTF-8 JSON file with exactly four members:

{
  "schema_version": 1,
  "name": "ebyte_packet",
  "endian": "little",
  "fields": [ ... ]
}

endian is little or big, and integers inherit it unless they set their own. Names are ASCII letters, digits and underscores, up to 64 characters, starting with a letter or underscore. EByte is strict about the JSON: unknown members, duplicate members, null values and invalid Unicode are all rejected before any binary input is opened.

A schema’s identity is the SHA-256 of its exact bytes, whitespace included. Documents carry that hash, so a document decoded with one schema can’t be encoded with another by accident. Reformatting a schema file changes its identity.

Field Types

Every field has a name and a type.

Integers are strings in the document so that 64-bit values never pass through floating point.

An integer can set its own endian. A length or count is either a literal number or a reference to an earlier integer in the same object, written as {"field": "data_size"}. Strings are exact-width: EByte doesn’t trim, pad or look for a terminator. If a format pads a field, declare the padding as its own field and it’ll be kept.

Repeating Blocks

Many formats are a run of blocks with no count anywhere: RIFF chunks, ZIP entries, PNG chunks. A repeat field reads them, with until saying when to stop.

"until": "end_of_input" keeps reading elements while any byte is left. It has to be the last field of the schema and can’t have when.

{"name": "chunks", "type": "repeat", "until": "end_of_input", "fields": [
  {"name": "id", "type": "string", "encoding": "ascii", "length": 4},
  {"name": "size", "type": "u32"},
  {"name": "body", "type": "bytes", "length": {"field": "size"}}
]}

"until": "tag_mismatch" keeps reading while the next bytes equal a tag. The element’s first field is the tag: an unconditional bytes field of 1 to 16 bytes with expect. When the next bytes differ, or the input runs out, the repeat ends and the following fields carry on from there.

{"name": "entries", "type": "repeat", "until": "tag_mismatch", "fields": [
  {"name": "signature", "type": "bytes", "length": 4, "expect": "504b0304"},
  ...
]}

In the document a repeat is an array of objects, like an array, and you can add, remove or reorder elements freely. Every element has to take at least one byte. On encode, EByte refuses output whose bytes right after a tag repeat start with the tag, because reading the file back would take them for another element.

Floats, Bit Fields and Forward Jumps

These arrived in 0.6.0.

Floats. f32 and f64 follow the field’s endian. Decoding writes the shortest decimal that turns back into exactly the same bits, such as "0.1" or "-0", and "nan:7fc00001" for a NaN, with every bit, so even NaN payloads survive. When encoding you can write any ordinary decimal, such as "0.5" or "2.5e-3"; it’s rounded to the nearest float, and a number too big for the type is refused.

Bit fields. Consecutive bits fields share bytes. Their sizes have to add up to whole bytes, at most 8, with nothing else in between. The bytes are read as one big-endian number and the fields take their bits from the most significant end, in order:

{"name": "version", "type": "bits", "bits": 4, "expect": "4"},
{"name": "header_words", "type": "bits", "bits": 4}

A bit field works like an unsigned integer: it can have expect, and other fields can use it as a length, a count or a when test. It can’t have when or derive itself.

Forward jumps. A bytes field can run up to an offset instead of having a length: {"length": {"until_offset": "pixel_offset"}} takes every byte from where the field starts up to the absolute offset in pixel_offset, or a literal number. The bytes are kept, so nothing is lost. If the offset is behind the field, decoding fails, and when encoding the bytes you give must reach the offset exactly.

Checks, Conditions and Derived Counts

expect checks a fixed value. On an integer it’s a decimal string, on a fixed-length bytes field it’s lowercase hex. A magic number is the usual case:

{"name": "magic", "type": "bytes", "length": 2, "expect": "4542"}

A mismatch is an error on decode and on encode. EByte never corrects a value to make it pass.

when makes a field optional. The field is present only if an earlier integer in the same object equals a value:

{"name": "note", "type": "bytes", "length": 2, "when": {"field": "flags", "equals": "1"}}

Equality is the only test. When it’s false, the field takes no bytes and must be absent from the document.

derive lets the encoder work out a count. An unsigned integer marked {"count_of": "records"} is left out of the editable document, and on encode EByte writes the length of the records array into it. The array’s count must point back at that integer. A count field without derive stays in the document, and encode fails if it doesn’t match the array, rather than guessing which one is right.

Since EByte 0.3.0, derive also works out sizes and offsets, and since 0.4.0 checksums. Each one picks exactly one form:

Targets are fields in the same object, before or after the integer.

Checksums come in five kinds. The integer’s type has to match the width:

A checksum covers the bytes as stored, after any sizes or offsets inside its range have been written, and its range can’t include itself or another checksum next to it. A stale checksum fails decoding with checksum_mismatch at the checksum field. EByte doesn’t decompress anything, so a checksum over uncompressed data, like a ZIP CRC of a deflated file, can’t be declared.

Size, offset and repeat-count forms can add a constant, written as a decimal string: {"offset_of": "items", "add": "-4"}. These measured values never decide how a file is read, so nothing may use them as a length, a count or a when test. Decoding checks each one against the file and reports constraint_mismatch if the stored number is stale. Encoding writes it after the rest of its object, and fails with value_out_of_range if the result is negative or too big for the integer.

Rules That Keep It Bounded

• References only point backwards, to an earlier integer in the same object that has no when. The forward links allowed are a derived count or size to the one field it drives, and measured sizes and offsets, which never change how a file is read.
• Arrays and repeats can nest up to 8 levels deep. A repeat looks ahead at most 16 bytes.
• There are no expressions, no arithmetic between fields beyond the derive forms, no seeking to an offset, no recursion and no plugins.
• Decoding must use every byte. Data left over after the last field is reported as trailing_data, not ignored.

Limits

These are the default ceilings. A program or a command-line flag can lower any of them, but never raise one.

Sizes are checked before EByte reads or allocates for a field, so a header that claims a 2^64-byte field is rejected after reading just that header.

Not in Version 1

Compression, reading backwards to an earlier offset, little-end-first bit numbering, and arithmetic beyond the derive forms. If your format needs one of these, say which: it helps decide what comes next.