Skip to main content

FEN — Testing Strategy (§6)

Part of the FEN design set. Overview, index, and the legacy-terminology map: fen.md.

6. Testing Strategy​

6.1 Guiding Principles​

The architecture creates a natural testing opportunity: the event reducer (event log → materialised state) is a pure function over a list of value objects. It is cheap to test exhaustively at the unit level. The bulk of the test budget belongs there — not in UI automation.

Test pyramid for FEN:

         /‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾\
        /   E2E / device    \   ← narrow; platform routing, deep links, two-device smoke
       /‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾\
      /  Integration / backend  \  ← backend conformance, sync layer, SQLite reducer
     /‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾\
    /   Unit + Golden (domain)  \ ← wide; money math, reducer, balance, permissions, crypto
    ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾

Three rules that govern every testing decision:

  1. Domain logic must not import Flutter. Pure Dart packages for all business logic; testable without a device or emulator.
  2. Money correctness is the highest-risk area. Float leakage silently diverges across platforms. Every monetary operation must have a unit test.
  3. Assert data state, not UI strings. E2E tests query SQLite directly via an AppDriver abstraction; they are decoupled from screen layout.

6.2 Unit Tests​

Organise by domain invariant, not by file:

test/
  unit/
    money/
      fixed_point_arithmetic_test.dart    ← 100% branch coverage required
      settlement_amount_test.dart         ← 100% branch coverage required
    ledger/
      event_reducer_test.dart             ← 100% branch coverage required
      balance_calculator_test.dart        ← 100% branch coverage required
      debt_simplification_test.dart       ← 100% branch coverage required
      permission_rules_test.dart          ← 100% branch coverage required
    crypto/
      signing_test.dart
      encryption_test.dart
      key_exchange_test.dart
    sync/
      outbox_test.dart
      dedup_test.dart
    events/
      schema_validation_test.dart
      unknown_event_type_test.dart

Money / Fixed-Point Arithmetic — 100% coverage, no exceptions​

Every operation that touches cents must be tested for: zero, negative, overflow guard, rounding direction. A float leak anywhere silently diverges across platforms.

// Use parameterised tables — no magic numbers in assertions
const splitCases = [
  ('100 cents ÷ 3',  100, 3, [34, 33, 33]),   // remainder to first
  ('1 cent ÷ 3',       1, 3, [1,  0,  0]),
  ('21.60 EUR ÷ 3',  2160, 3, [720, 720, 720]),
  ('21.60 EUR ÷ 2',  2160, 2, [1080, 1080]),
];
// Also test: settlement_amount == original_amount when currencies match,
//            settlement_minor_units sum == settlement_amount_minor_units always,
//            serialise/deserialise round-trip, no double intermediates

Event Reducer — 100% coverage​

Every EventType in the schema must have at minimum:

  • A happy-path test (event applies correctly)
  • An idempotency test (same event applied twice = same state as once)
  • An out-of-order test where sequence ordering matters

Structure as (initialEvents, newEvent, expectedMaterialisedState) triples. The reducer is a pure function — test it that way. No shared mutable state between tests.

test('ExpenseDeleted tombstones but does not remove from log', () {
  final state = reduce([expenseLoggedEvent, expenseDeletedEvent]);
  expect(state.expenses.active, isEmpty);
  expect(state.expenses.deleted, hasLength(1));
  expect(state.balances.values, everyElement(equals(0)));
});

Balance Calculator & Debt Simplification — test properties, not just examples​

// Property 1 — conservation of money
expect(balances.values.fold(0, (a, b) => a + b), equals(0));

// Property 2 — debt simplification is bounded
expect(transactions.length, lessThanOrEqualTo(memberCount - 1));

// Property 3 — no member on both sides of any transaction
for (final tx in transactions) {
  expect(tx.from, isNot(equals(tx.to)));
}

// Property 4 — after applying transactions, all balances zero
final settled = applySettlements(balances, transactions);
expect(settled.values, everyElement(equals(0)));

Permission Rules — enumerate every (actor, action, expense) combination​

// creator_only mode
group('creator_only', () {
  test('creator may edit own expense', ...);
  test('creator may delete own expense', ...);
  test('non-creator may NOT edit', () => expect(canEdit, isFalse));
  test('non-creator may NOT delete', () => expect(canDelete, isFalse));
  test('organiser may NOT edit non-own expense in creator_only mode', ...);
});

// any_member mode
group('any_member', () {
  test('any active member may edit', ...);
  test('removed member may NOT edit', ...);
  test('rejected event does not mutate state', ...);  // key: silent rejection, no exception
});

Crypto Unit Tests​

The biggest risk is silent failure — tampered ciphertext returning garbage instead of throwing. Test:

test('tampered ciphertext throws CryptoException', () {
  final ct = encrypt(plaintext, key);
  final tampered = ct..[5] ^= 0xFF;
  expect(() => decrypt(tampered, key), throwsA(isA<CryptoException>()));
});

test('wrong key throws, does not return garbage', ...);
test('signature verification rejects modified payload', ...);
test('ECDH: both sides derive same shared secret', ...);

// Use at least one fixed test vector from libsodium's own test suite
// This tests the FFI binding, not just the wrapper
test('XChaCha20-Poly1305 known test vector', () {
  expect(encrypt(knownPlaintext, knownKey, knownNonce), equals(knownCiphertext));
});

Event Schema & Future-Proofing​

test('unknown event type is skipped, does not crash reducer', () {
  final events = [...knownEvents, unknownFutureEvent];
  expect(() => reduce(events), returnsNormally);
  // State equals what known events would produce
  expect(reduce(events), equals(reduce(knownEvents)));
});

6.3 Integration Tests​

Event Log + SQLite Reducer​

Use drift's NativeDatabase.memory() for an in-memory SQLite database. Runs on CI without a device or emulator.

test/integration/
  reducer_sqlite_test.dart     ← replay scenarios against real SQLite schema
  schema_migration_test.dart   ← DB migration correctness

Scenarios:

  • Fresh replay: 50 events → assert full SQLite table state matches fixture
  • Incremental replay: apply events 1–25, checkpoint; apply 26–50; assert equals full replay from 0
    • Invariant: state(events[0..N]) == state(events[0..K]) + state(events[K+1..N])
  • Tombstone/edit chains: ExpenseLogged → ExpenseDeleted → balance correctly zero; ExpenseLogged → ExpenseUpdated → only new snapshot visible
  • Multi-device writes: two in-process "devices" append to same in-memory DB; final state deterministic
  • Schema migration: v1 schema → run migrations → v2 schema preserves all data

Sync Layer — In-Process Mock Backend​

Stand up a local mock backend using shelf (pure Dart HTTP server) implementing the StorageBackend adapter (list/get/put/delete + conditional write). Runs in the test process — no Docker, no external dependency. The mock backend must enforce the write-capability check (reject writes lacking the group's share token / authenticated session) and honour ETag/If-Match so optimistic-concurrency paths are exercised; do not stub these away.

test/integration/
  sync_outbox_test.dart
  sync_poll_test.dart
  sync_dedup_test.dart
  sync_sse_test.dart          ← add in v2

Scenarios:

  • Outbox flushes N events → backend acknowledges → events marked pushed in SQLite
  • Upload retry on 503 → eventual delivery, exactly one copy in backend
  • Poll returns already-seen events → reducer deduplicates → state unchanged
  • Backend returns events from two different groups → only own-group events processed
  • Partial flush failure → outbox entries with failed status → retry on next sync trigger

Invite Flow Integration​

The invite package is not an ECDH handshake between the two devices — the inviter generates group_key randomly and ships it inside the encrypted package (doc/fen.md §2.4); the joiner never independently derives it. So the scenario worth an integration test is that the #fragment decrypts client-side, with no network access, into the same package the inviter built — using parseInviteLink (apps/fen/lib/features/onboarding):

test('fragment is parsed and decrypted client-side, never sent anywhere', () {
  final secret = randomBytes(32);
  final packageJson = { 'version': 2, 'group_id': 'grp_1', /* … */ 'expiry': futureUnixSeconds };
  final encryptedPkg = encrypt(utf8.encode(jsonEncode(packageJson)), secret);
  final uri = Uri.parse(
    'https://fenapp.net/join#s=${b64url(secret)}&p=${b64url(encryptedPkg)}&v=2',
  );

  expect(isInviteLink(uri), isTrue);
  final result = parseInviteLink(uri); // no HTTP client involved anywhere in this call
  expect(result, isA<InviteLinkParsed>());
  expect((result as InviteLinkParsed).package.groupId, 'grp_1');
});

See apps/fen/test/unit/onboarding/invite_link_parser_test.dart for the full suite (wrong/missing v, missing p, expired package, corrupt ciphertext, wrong invite_secret, and a no-logging regression test).

6.4 Golden / Data Tests​

These are regression anchors for deterministic outputs. When behaviour intentionally changes, update the golden file deliberately — not by accident.

test/golden/
  balance/
    three_way_equal_split.json
    edit_then_balance.json
    deleted_expense_excluded.json
    solo_expense_in_total_cost.json
    multi_currency_trip.json
  debt_simplification/
    five_person_complex.json
    large_group_rounding.json
    already_settled_noop.json
  multi_currency/
    same_currency_noop.json       ← original == settlement when currencies match
    settlement_stored_not_derived.json  ← highest-value golden (see below)

Format: each file has "input" (list of events or parameters in cents) and "expected" (balance map or transaction list in cents). Load JSON, run the pure function, expect(actual, equals(expected)). No screenshots.

settlement_stored_not_derived.json — the highest-value golden test:

test('balance uses settlement_amount_minor_units from event, not any live or re-derived rate', () {
  // 1. Write ExpenseLogged with original:EUR 32000, settlement:CHF 30500
  // 2. Inject a mock FX API that returns a rate (simulating post-write rate change)
  // 3. Calculate balances
  // 4. Assert: calculation used CHF 30500 from the event, not any re-derived amount
  //    If the projector touches the FX API or re-derives from rate, this test fails
  expect(balances['bob'], equals(expectedFromStoredSettlement));
  expect(mockFxApi.callCount, equals(0));
});

Golden tests fail loudly if output ordering changes. That forces deterministic tie-breaking, which is critical for cross-device convergence.

6.5 End-to-End Tests​

Two-Device Strategy​

Flutter's integration test harness runs one app instance. For two-device logic tests use dual in-process app instances sharing a shelf mock backend — fast, no emulator needed. Reserve real emulator pairs for scenarios where OS platform routing matters.

integration_test/
  two_device_sync_test.dart
  offline_then_sync_test.dart
  invite_flow_test.dart
  multi_currency_e2e_test.dart

AppDriver Abstraction​

Assert data state, not UI strings. Decouples E2E tests from screen layout changes:

class AppDriver {
  Future<void> createGroup(String name, String settlementCurrency);
  Future<String> createInviteLink(String groupId);
  Future<void> joinFromInviteLink(String link);
  Future<void> addExpense({required String name, required int amountCents,
                           required String currency, required List<String> memberIds});
  Future<Map<String, int>> getBalances(String groupId);
  Future<List<SettlementTransaction>> getSettlementPlan(String groupId);
  Future<int> getTotalCost(String groupId, String memberId);
}

Scenarios to Automate (in-process dual instances)​

ScenarioAssert
A creates group, B joins via invite linkBoth derive same group_key; B sees GroupCreated in replay
A adds expense offline → comes online → B pollsB's balances match A's without restart
A and B add expenses simultaneously → both syncBoth converge to identical balance state
A edits expense → B pollsB sees updated amount and split; balance recalculates correctly
A soft-deletes expense → B pollsB's balance excludes deleted expense; tombstone visible when toggled
Duplicate poll (backend returns seen events)State unchanged; no double-counting
Upload retry after 503Backend contains exactly one copy; balance correct
Multi-currency: rate from event, not live APIBalance uses captured rate after mock API rate changes
Wrong invite #fragment (bad secret)Group key derivation fails; error surfaced; no crash

Scenarios Reserved for Real Device / Emulator​

ScenarioWhy manual/emulator
Invite link received in WhatsApp → app opensOS intent routing; platform-specific
#fragment survives Universal Links on iOSiOS sometimes strips fragments
App killed during outbox flush → restart → sync completesOS process management
Background fetch wakes app → backend polledBackgroundTasks / WorkManager scheduling

6.6 Storage Backend Conformance Tests​

The storage_conformance package targets the StorageBackend adapter (§2.7) — list/get/put/delete and conditional-write (ETag/If-Match) semantics against Garage (and, from v2, any self-hosted S3-compatible endpoint using the same adapter).

A standalone, implementation-neutral, black-box HTTP test suite configurable via a base URL env var. Run against any storage backend implementation (self-hosted or managed) without code changes.

storage_conformance/
  auth_test.dart           ← token validation
  append_test.dart         ← event storage and round-trip
  fetch_test.dart          ← cursor-based pagination
  ordering_test.dart       ← stable ordering, concurrent writes
  idempotency_test.dart    ← duplicate event_id handling
  isolation_test.dart      ← group isolation by keypair auth
  limits_test.dart         ← payload size, rate limiting

What the suite must verify:

CategoryTest
AuthNo auth response → 401; invalid Ed25519 signature → 401; valid signature → 200
StoragePOST /events → stored; GET /events?after=0 → all events returned
Round-tripBytes returned are byte-for-byte identical to bytes submitted (backend must not deserialise/reserialise)
PaginationGET /events?after=N returns only events after seq N; no skips, no duplicates
OrderingEvents retrieved in documented order; concurrent appends all retrievable, none lost
IdempotencyDuplicate event_id → 200 or 409 (document which); event stored exactly once
Group isolationEvents from group A never returned when querying group B's token
LimitsPayload at max size accepted; payload at max+1 rejected (4xx); 429 on rate limit
No plaintextBackend DB contains no cleartext group data — verify by inspecting storage directly
MigrationObservable behaviour identical across Phase 0 / 1 / 2 backends

This suite is the mandatory safety net before switching backend implementations. Add it in M4 (§5.4).

6.7 Manual & Exploratory Testing​

Some things cannot be reliably automated:

AreaWhy manual
Deep link #fragment survivaliOS Universal Links and Android App Links strip fragments in certain configurations; must test on real devices, multiple OS versions, from: SMS, WhatsApp, Signal, email, browser, QR scanner
Background syncOS scheduling policy (BackgroundTasks / WorkManager) is not testable in CI; behaviour depends on battery state, OS version, and usage patterns
App upgrade with existing encrypted event logMigration from previous app version with real SQLite data; hard to automate reliably across OS versions
Crypto library updateAfter any libsodium version bump: manual smoke test on real iOS + Android to verify FFI binding
Biometric / Keychain / KeystoreSecure storage behaviour varies by device and OS version; automated emulator behaviour does not match real devices
Low storage / low memoryOS behaviour under constraint is not reliably reproducible in automation
Locale and accessibilityCurrency formatting, RTL layouts, dynamic font size, screen reader compatibility
Real network conditionsMobile data latency, airplane mode transitions, hotel captive portals affecting sync

Use scripted charters, not vague exploration. Example: "Join a group from a WhatsApp link while the app is not running. Create an expense while in airplane mode. Reconnect. Verify the other device shows the correct balance."

6.8 CI Strategy​

Every commit (push to any branch)
├── dart analyze + dart format --check
├── Unit tests (pure Dart, no Flutter, no device)
│   ├── Money / fixed-point arithmetic
│   ├── Event reducer (all event types)
│   ├── Balance calculator + debt simplification
│   ├── Permission rules
│   ├── Crypto test vectors (fixed vectors from libsodium)
│   └── Schema validation + unknown event type handling
├── Golden / data tests (balance, debt simplification, rate-embedded)
├── Integration tests (drift NativeDatabase.memory() + shelf mock backend)
│   ├── SQLite reducer (fresh replay, incremental, tombstone, edit)
│   ├── Sync layer (outbox, dedup, retry, pagination)
│   └── Invite flow (ECDH, fragment parsing)
├── Backend conformance suite → local Phase 0 backend (test namespace, wiped after)
└── Coverage gate: domain packages must maintain ≥ 90 % line coverage

Pull request / pre-merge (in addition to commit suite)
├── Flutter integration smoke test on Android emulator
├── DB migration tests (v_prev → v_current schema)
├── One two-device sync scenario (in-process dual instance)
└── Backend conformance suite if backend code changed

Nightly (scheduled)
├── Full Android emulator suite (flutter test integration_test/)
├── Full iOS simulator suite (if available)
├── Two-device E2E matrix (all UC scenarios)
├── Backend conformance suite against all backends (Phase 0, 1, 2)
├── Randomised / property-based tests
│   ├── Arbitrary expense sets → balances always sum to zero
│   ├── Arbitrary event sequences → replay always idempotent
│   └── Arbitrary split configurations → split invariant always holds
├── Flaky-network / retry tests (simulated 503, timeout, partial flush)
└── Upgrade tests: install previous release → run current → verify data integrity

Key discipline: business correctness lives in fast deterministic Dart tests. E2E proves platform behaviour and full-flow confidence. Backend conformance (§6.6) is mandatory before swapping storage backends. Never use E2E to compensate for missing unit test coverage.