FEN Canonical Serialization Spec

Status: v1 draft · companion to fen.md (see §3.3, §3.9, §3.10, §4.1) Format version: fmt = 1 (carried in every event envelope). fmt = 1 signs all context fields but does not bind them as AEAD additional data in the current implementation; wiring AAD requires fmt = 2 and is tracked by issue #66.

This spec defines the one and only byte representation of a FEN event that is hashed, signed, verified, deduplicated, and exchanged between implementations. The governing rule:

Equal event data MUST serialize to byte-for-byte identical output on every device and every implementation.

If two encoders disagree by a single byte, signatures fail to verify and valid events are silently rejected. Everything below exists to make that impossible.

1. Layering — what is signed

FEN is encrypt-then-sign. The author signature covers the envelope containing the ciphertext, so a peer can verify authorship of an event it cannot decrypt (wrong key epoch) and re-serve it without being able to forge it.

plaintext payload  ──XChaCha20-Poly1305(content_key, nonce[, AAD])──▶  ciphertext
                                                                        │
envelope { …metadata…, payload:{nonce, ciphertext} }  ──canonicalize──▶ canonical bytes
                                                                        │
                                        author_signature = Ed25519(sk, canonical bytes)

Two layers, two reasons to be disciplined:

Layer	Canonicalized for	Consequence if not
Envelope (this spec, §4–§7)	signature + `event_id` determinism across implementations	signatures fail; dedup breaks
Plaintext payload (§8)	correct/consistent computation and equality (money, text)	wrong balances; false duplicates

2. Wire format

Encoding: UTF-8, no BOM.
On storage: JSON Lines (.jsonl) — exactly one event per line, terminated by a single \n (U+000A). A serialized event MUST NOT contain a raw \n or any unescaped control character, so line boundaries are unambiguous.
Readable by design: the stored line is the canonical form (see §6), so a human can inspect it and any implementation can re-verify it by re-canonicalizing.

3. Canonicalization algorithm

Base algorithm: RFC 8785 JSON Canonicalization Scheme (JCS) — deterministic key ordering, minimal whitespace, fixed string escaping, UTF-8 output. FEN adds the following stricter constraints on top of JCS (they remove whole classes of ambiguity rather than relying on JCS's more permissive rules):

#	Rule	Rationale
C1	No bare JSON numbers anywhere. Every numeric value is a JSON string holding a base-10 integer with no leading zeros, no `+`, optional leading `-`, no exponent, no decimal point. Amounts are integers in minor units.	Eliminates the entire IEEE-754 / 2^53 / `1.0` vs `1` class of bugs. JCS's number rules never fire.
C2	Object keys are sorted per JCS (ascending by UTF-16 code unit). Applied recursively to every object, including maps like `wrappedKeys`.	Deterministic order.
C3	No insignificant whitespace. `{"a":"1","b":"2"}` — no spaces, no newlines.	Byte determinism.
C4	Strings: all human-supplied text is Unicode-normalized to NFC before serialization. JSON string escaping per RFC 8785: escape only `"` `\` and U+0000–U+001F (via the short forms `\b \t \n \f \r` where defined, else lowercase `\u00xx`); `/` is not escaped; all other characters emitted as literal UTF-8.	Prevents the "é one-codepoint vs e+accent" and escape-variant divergences.
C5	Binary blobs (`nonce`, `ciphertext`, `author_signature`, wrapped keys) use base64url without padding (RFC 4648 §5, `-`/`_`, no `=`).	One encoding, compact.
C6	Identifiers / public keys (`author_id`, member pubkeys, `event_id`, `group_id`) use lowercase hex (event/group ids are lowercase UUIDv4 strings incl. hyphens).	Ids stay readable and match the scaffold.
C7	Timestamps use fixed-precision UTC: `YYYY-MM-DDTHH:MM:SS.sssZ` — always millisecond precision, always `Z`, never an offset.	No trailing-zero / precision / timezone ambiguity.
C8	Absent means omitted. Optional fields that have no value are left out entirely. `null` MUST NOT appear anywhere.	`{}` vs `{"x":null}` can't diverge.
C9	No duplicate keys. A parser encountering a duplicate key MUST reject the event.	Removes JSON's worst ambiguity.
C10	Booleans are the literals `true` / `false`.	—
C11	Enums are the exact strings in §5 (case-sensitive).	—

Alternative considered: deterministic CBOR (RFC 8949 §4.2) is a stronger footgun-free base, but it is not human-readable and conflicts with the "inspectable JSONL export" goal in fen.md. JSON+JCS is chosen for alignment; CBOR remains a fallback if perf/robustness ever demands it. See §10.

4. Signing algorithm (normative)

Let E be the envelope object without the author_signature field.

canonical  = JCS(E)                       // UTF-8 bytes, per §3
signature  = Ed25519_sign(member_sk, canonical)   // Ed25519 hashes internally; do NOT pre-hash
author_signature = base64url_nopad(signature)
stored_line = JCS(E ∪ { "author_signature": author_signature })

Note step 4 re-canonicalizes including the signature field, so the stored line is itself fully canonical.

Verification

parse the line as JSON (reject on duplicate keys, raw control chars, or any
   value violating §3 — see §9)
sig      = base64url_nopad_decode(line.author_signature)
E'       = line without the "author_signature" field
canonical= JCS(E')
accept iff Ed25519_verify(author_pubkey, canonical, sig)
   where author_pubkey = hex_decode(E'.author_id) and is a current roster member

event_id is assigned by the author (UUIDv4), not derived from a hash — it must exist before encryption because it is an AAD input (§7). The signature authenticates it because it is inside the signed envelope.

5. The event envelope

All fields are REQUIRED and appear in every event. (Order here is for humans; JCS sorts them on the wire.)

Field	Type / encoding	Notes
`fmt`	string int (C1)	Format version. `"1"` for this spec.
`event_id`	lowercase UUIDv4 string	Assigned by author; unique; the dedup key.
`group_id`	lowercase UUIDv4 string	Stable per group.
`epoch`	string int (C1)	Key epoch this event is encrypted under (§3.10). AAD input.
`author_id`	lowercase hex (32-byte Ed25519 pubkey → 64 chars)	The signer; must be a roster member.
`author_seq`	string int (C1)	Per-author contiguous counter (gap detection, §3.9). Signed.
`lamport`	string int (C1)	Lamport clock; canonical order scalar `(lamport, event_id)`. Signed.
`event_type`	enum string (C11)	One of the values below. AAD input.
`happened_at`	timestamp (C7)	Device clock; informational/audit only.
`payload`	object `{ "ciphertext": <b64url>, "nonce": <b64url> }`	AEAD output of the plaintext payload (§8).
`author_signature`	b64url-nopad (64-byte Ed25519 sig)	Added last; excluded when signing (§4).

event_type enum values (case-sensitive): GroupCreated, GroupSettingsUpdated, MemberInvited, GroupInviteRelayed, MemberJoined, MemberLeft, MemberRemoved, ExpenseLogged, ExpenseUpdated, ExpenseDeleted, SettlementRecorded, GroupClosed, StorageMigrated, KeyEpochUpdate, SettlementSnapshot, KeyRewrapRequest, AttachmentUploaded, StorageKeyRotated, StorageKeyAck.

Reconciliation note. fmt, epoch, author_seq, and lamport are part of the signed envelope but are not yet on the scaffold's GroupEvent (packages/fen_events/lib/src/group_event.dart). Implementing this spec means adding them.

6. Worked example (illustrative)

Envelope before signing (shown pretty for readability):

{
  "fmt": "1",
  "event_id": "0b3c1e5a-0000-4000-8000-000000000001",
  "group_id": "d41d8cd9-0000-4000-8000-00000000abcd",
  "epoch": "1",
  "author_id": "9e1f00aa...<64 hex>...ff",
  "author_seq": "1",
  "lamport": "1",
  "event_type": "ExpenseLogged",
  "happened_at": "2026-07-01T14:32:00.000Z",
  "payload": { "ciphertext": "AAECAw", "nonce": "Zm9vYmFy" }
}

Canonical bytes signed (keys sorted, minified, payload sorted too):

{"author_id":"9e1f00aa...ff","author_seq":"1","epoch":"1","event_id":"0b3c1e5a-0000-4000-8000-000000000001","event_type":"ExpenseLogged","fmt":"1","group_id":"d41d8cd9-0000-4000-8000-00000000abcd","happened_at":"2026-07-01T14:32:00.000Z","lamport":"1","payload":{"ciphertext":"AAECAw","nonce":"Zm9vYmFy"}}

Stored line (adds author_signature, which sorts right after author_seq):

{"author_id":"9e1f00aa...ff","author_seq":"1","author_signature":"<b64url-64B>","epoch":"1","event_id":"0b3c1e5a-0000-4000-8000-000000000001","event_type":"ExpenseLogged","fmt":"1","group_id":"d41d8cd9-0000-4000-8000-00000000abcd","happened_at":"2026-07-01T14:32:00.000Z","lamport":"1","payload":{"ciphertext":"AAECAw","nonce":"Zm9vYmFy"}}

The reference implementation MUST publish a golden vector file (test/golden/serialization/*.json) pairing known events → canonical bytes → signature under a fixed test key, so all implementations can self-check.

7. AEAD and AAD construction (normative)

Payload confidentiality/integrity: XChaCha20-Poly1305.

content_key = HKDF-SHA256(group_key, info="fen-v1-content") (§4.1).
nonce: 24 random bytes from the CSPRNG, per event.
For fmt = 1, additionalData is empty. Context fields are still covered by the Ed25519 signature over the canonical envelope, and receivers verify that signature before decrypting. This documents the shipped wire behavior found in implementation-review issue #66.
For the planned fmt = 2, AAD additionally binds the ciphertext to its context (§3.10). Naive concatenation is ambiguous, so AAD is the JCS of this fixed object (same rules as §3):
```
AAD = JCS({ "a": author_id, "e": event_type, "g": group_id,
            "i": event_id, "p": epoch })
```
(keys sort to a,e,g,i,p.) This is unambiguous and length-safe.
fmt = 1: ciphertext = XChaCha20Poly1305_encrypt(content_key, nonce, plaintext).
fmt = 2: ciphertext = XChaCha20Poly1305_encrypt(content_key, nonce, plaintext, AAD).

Decryption for fmt = 2 recomputes AAD from the (signed, therefore trusted) envelope fields and fails closed if any was tampered with. Decryption for fmt = 1 does not pass AAD; tampering with those fields is still rejected by the signature verification that must happen before decrypt.

8. Plaintext payload rules

The plaintext (before encryption) is a JSON object serialized with the same rules as §3 (NFC text, numbers-as-strings, sorted keys, no null). Although it is encrypted and thus not part of signature verification, canonicalizing it keeps computation and equality stable and makes re-encryption deterministic.

All monetary values are decimal-string integers in minor units ("2160" = €21.60), never JSON numbers, never floats — this is the hard rule from fen.md.
Split invariant (validated on write and on receipt): sum(splits[i].settlement_minor_units) == settlement_amount_minor_units.
User text (groupName, displayName, expense name, notes) is NFC.
Payload shape per event_type follows packages/fen_events/lib/src/payloads.dart.

9. Parser hardening (reject, don't sanitize)

A conforming parser MUST reject (drop + log, never forward) an event that:

has a duplicate object key (C9);
contains any bare JSON number (C1) or a null (C8);
contains a raw control char / embedded newline;
has an unknown or missing fmt, or a missing required field (§5);
fails signature verification (§4) or is authored by a non-roster pubkey;
fails the split invariant or other domain validation (fen.md §4.4).

Never "fix up" a non-canonical event and accept it — that reintroduces ambiguity. Canonicality is validated, not repaired.

10. Versioning & evolution

fmt is bumped only for a breaking change to these rules. Readers reject fmt values they don't understand.
fmt = 2 is reserved for the same JSON canonicalization rules with AEAD AAD binding enabled (§7, issue #66).
New optional payload fields are additive (older readers ignore unknown payload keys after decryption — but the envelope schema is closed and versioned).
If deterministic CBOR is ever adopted, it ships as a later fmt with a documented migration; old lines remain verifiable under their original format.

11. Implementation notes (Dart)

Put this in packages/fen_events/lib/src/serialization.dart, exposing Uint8List canonicalBytes(GroupEvent e) (envelope minus signature), String encodeLine(GroupEvent e), and GroupEvent decodeLine(String line).
Use a vetted JCS implementation or a small hand-rolled canonical encoder — but cover it with the golden vectors first; this is the highest-leverage unit test in the codebase.
Extend GroupEvent with fmt, epoch, authorSeq, lamport (§5 note).
Signing/verifying lives in fen_crypto (sign / verify, Ed25519) and consumes canonicalBytes — never JSON strings directly.

12. Open decisions to confirm

JSON+JCS vs deterministic CBOR — this draft chooses JSON+JCS for readability/alignment. Confirm, or switch (fmt makes it reversible).
Numbers-as-strings everywhere (C1) — chosen for safety. Confirm you're comfortable with author_seq/lamport/epoch as strings too (recommended), vs. allowing bounded JCS integers for those three ordering counters.
event_id = UUIDv4 (assigned) vs. a content hash. RESOLVED (issue #19): keep author-assigned UUIDv4. A content hash would remove the suppression risk below for free, but event_id is an AAD input that must exist before encryption (§7) — a content hash of the canonical (encrypted) bytes is not available at that point, so UUIDv4 stays. The suppression risk an author-chosen event_id creates — a member reusing another member's event_id so peers drop the honest event as a "duplicate" — is instead closed by making event identity/dedup key on (group_id, author_id, author_seq) rather than event_id (see data-model.md §3.9 rule 5 & reducer invariant 5). author_id is signature-verified and author_seq is signed, so that tuple is unforgeable across authors; event_id survives purely as an AAD input and a (non-unique) middle order tiebreak.

1. Layering — what is signed​

2. Wire format​

3. Canonicalization algorithm​

4. Signing algorithm (normative)​

Verification​

5. The event envelope​

6. Worked example (illustrative)​

7. AEAD and AAD construction (normative)​

8. Plaintext payload rules​

9. Parser hardening (reject, don't sanitize)​

10. Versioning & evolution​

11. Implementation notes (Dart)​

12. Open decisions to confirm​