FEN — Operations & Debugging (§7)

Part of the FEN design set. Overview, index, and the legacy-terminology map: fen.md.

7. Operations & Debugging

Governing constraint: FEN's E2EE means the support operator — like the storage-backend operator — is untrusted. Logs, crash reports, and remote config responses must be safe to read by an adversary. They must never allow reconstruction of group names, expense amounts, member identities, or key material.

---

7.1 Support Ticket Flow

Entry point: Settings → Help → Report a problem.

Flow:

1. User picks an issue type: Sync / Storage backend / Import or restore / Crash / Other
2. User writes a free-text description
3. App shows an expandable "What will be sent" disclosure — first and last 20 log lines visible in-app
4. Two buttons: Send (with logs) · Send without logs
5. App opens the OS mail composer pre-addressed to support@fenapp.net with the log bundle attached as a JSON file

No account is required. No third-party SDK is introduced. The user can inspect the file before sending.

Support bundle contents

Field	What's included
App metadata	Version, build hash, platform, OS version, device class (phone/tablet — not model or serial)
Install age	Bucket only: new / 1–7d / 7–30d / 30d+ — not the install timestamp
Network state	Offline / Wi-Fi / cellular / VPN at time of report
Backend info	Backend alias (backend_0) — not the full URL
Sync health	Last sync time bucket, pending outbox count, failed event count by error code, DB schema version
Remote config	Config version/hash currently active
Log lines	Last N lines from the rolling log (user chooses 1 h / 24 h / 3 days; default 24 h)

Log format — structured JSONL

{"ts":"2026-06-28T10:42:31Z","level":"WARN","component":"backend_sync","code":"BK_CONN_TIMEOUT","backend_index":0,"attempt":3,"backoff_ms":8000,"network":"wifi"}

Safe to log: timestamps · log levels · component names · machine-readable error codes · HTTP status codes (not full URLs) · sync state machine transitions (IDLE → SYNCING → PARTIAL_FAILURE) · local DB row counts (not values) · retry counts and backoff durations · active feature flag names.

Never log:

• Plaintext event content: group names, expense names, amounts, notes, currency codes, split ratios
• Public or private keys, group keys, seed phrases, auth tokens
• Full backend URLs, blob hashes, full pubkeys, event IDs — all are stable correlation handles
• Member names, device identifiers, IP addresses, clipboard contents

Redaction pass (runs before attachment)

Pattern	Replacement
Hex string ≥ 32 chars	[HEX_REDACTED]
Bech32 string (nsec1…, npub1…, etc.)	[BECH32_REDACTED]
Base64 ≥ 44 chars outside known safe fields	[B64_REDACTED]
URL query string	stripped
Email address or IP address	[REDACTED]

If cross-line pubkey correlation is needed within one ticket, use HMAC(bundle_random_salt, pubkey)[:8] — correlatable within that report, not across reports.

Log bounds

• Ring buffer: 5,000 lines or 2 MB in-memory
• On-disk: 3-day rotation, 2 MB total cap; files older than 72 h deleted on app launch
• Debug logging: off by default; stored as an expiry timestamp (not a boolean) so it auto-expires 24 h after being enabled
• Bundle cap: 5 MB; verbose lines dropped first, then info, then oldest warnings

CI gate: test that fails if any forbidden field name (amount, name, memo, privkey, seed, plaintext) appears in log call sites.

---

7.2 Remote Config (fen-config.json)

The .well-known/fen-config.json endpoint allows backend succession, attachment storage migration, app update notices, and feature rollouts without an app-store release.

Schema

The managed backend URLs below are the intended stable public endpoints. Their NAS deployment is now specified in infra/managed-backend, but a live environment still requires an operator to create or attach the Cloudflare Tunnel, set TUNNEL_TOKEN, and deploy the compose stack. If the cloudflared container runs with only TUNNEL_TOKEN, the tunnel is remotely managed and this repo's local ingress file does not route public hostnames. Add the routes in Cloudflare Zero Trust under Networks -> Tunnels -> [tunnel] -> Public Hostname -> Add a public hostname: api.fenapp.net should point to http://provisioner:8080, alongside the existing s3.fenapp.net -> http://garage:3900 route. cloudflared tunnel route dns can create DNS CNAMEs, but it does not add that ingress service mapping for a remotely-managed tunnel. If the tunnel is migrated to a locally-managed credentials-file setup, the checked-in cloudflared/config.yml becomes the local ingress source of truth.

{
  "version": 12,
  "issued_at": "2026-06-28T00:00:00Z",
  "expires_at": "2026-07-28T00:00:00Z",
  "signature": "<Ed25519 over canonical JSON, this field excluded from signing>",

  "managed_backend": {
    "s3_endpoint": "https://s3.fenapp.net",
    "provisioner": "https://api.fenapp.net"
  },

  "min_version":          { "ios": "1.4.0", "android": "1.4.0" },
  "min_version_deadline": "2026-07-15T00:00:00Z",
  "recommended_version":  { "ios": "1.6.2", "android": "1.6.2" },

  "limits": {
    "max_group_members":    20,
    "max_blob_bytes":  5242880,
    "max_events_per_sync": 500
  },

  "flags": {
    "receipt_ocr":        { "enabled": false, "min_app_version": "1.5.0" },
    "new_sync_scheduler": { "enabled": true,  "rollout_percent": 10 }
  },

  "support_email": "support@fenapp.net",
  "status_url":    "https://status.fenapp.net",

  "banner": {
    "id":      "maint-2026-07-01",
    "text":    "Backend maintenance July 1, 02:00-04:00 UTC.",
    "severity": "info",
    "expires": "2026-07-01T04:00:00Z"
  },

  "next_fetch_after": "PT12H"
}

Behavior on change

Change	When applied	User-visible?
New backend / attachment storage URL added	Next cold launch	Quiet one-time notice in sync status area
All backend URLs replaced	Next cold launch, after confirmation	"FEN is switching to a new backend — Do it now / On Wi-Fi / Keep old backend"
deprecated entry removed	Next cold launch	Silent
Feature flag flipped	Next cold launch	Only if the feature has UI
limits tightened	Next operation that hits the limit	Limits apply to new operations only — never retroactive
min_version tightened	Immediate banner on foreground	Non-dismissable; hard block after min_version_deadline
banner added	Next foreground event	Shown once; app stores banner.id and won't re-show

Fetch mechanics

• Fetch on cold launch if last fetch was > next_fetch_after ago
• Background refresh (WorkManager / BackgroundTasks) throttled by next_fetch_after
• Cache last-known-good; app must function fully offline using it
• Fetch failure → use cache; log WARN remote_config_fetch_failed
• Fetched version < cached version → reject silently; log WARN remote_config_downgrade_rejected
• Cache > 30 days old → log WARN remote_config_stale

Signature verification

• Config signing key is a dedicated Ed25519 keypair, separate from any backend or member identity
• The public key is hardcoded in the app binary
• Signed payload: canonical JSON (keys sorted, no insignificant whitespace, UTF-8); signature field excluded
• On verification failure: reject the entire config, use cache, log ERROR remote_config_sig_invalid

What remote config must never do

• Change cryptographic algorithms or key derivation paths
• Disable E2EE verification
• Add new trust anchors
• Ship or evaluate executable code
• Override a user's manually-set backend (only the default for new groups changes)

Backend succession lifecycle: active → degraded (dual-write) → read-only → removed

Risks

Risk	Mitigation
Config endpoint compromised	Signature + hardcoded key; attacker also needs the signing key
Signing key leaked	Issue app update with new hardcoded public key; overlap window accepting both; treat like PKI root
Bad config strands users	Last-known-good cache always wins on failure; min_version_deadline buffer
Rollout bug breaks sync globally	rollout_percent staged rollout; per-feature kill-switch flag
Limits applied retroactively	Policy: limits only apply to new operations

---

7.3 Crash Reporting

Use a self-hosted crash reporting instance (e.g. Sentry self-hosted) — not a SaaS provider — to keep crash data off third-party infrastructure.

Safe to capture: stack traces (file + line, no argument values), app version, OS version, device class, memory / CPU pressure at crash time, crash type.

Never capture: UI state derived from group or expense content, screen routes if derived from group IDs, raw database rows, decrypted event payloads, any key material.

Configure the crash SDK with explicit field allowlisting before any event leaves the device. Disable automatic breadcrumb capture; add only manually-whitelisted breadcrumbs.

---

7.4 Sync Health Panel

Accessible from Settings → Sync Status.

Shows:

• Backend connection state per backend (connected / reconnecting / failed) — backend alias, not URL
• Last successful sync: time bucket (justnow / < 5 min / < 1 h / > 1 h)
• Outbox: pending count, failed count by error code
• attachment storage: pending upload count, last upload time bucket
• DB schema version, remote config version

"Run diagnostics" button — 7-step backend health check:

1. DNS resolution
2. TLS handshake
3. Storage bucket access
4. Auth token verification
5. Write synthetic test event (not real group data)
6. Read back test event
7. Round-trip latency bucket (< 200 ms / < 1 s / > 1 s / failed)

Results shown as a checklist. Errors shown as code + description — never raw strings that might contain URLs or user data.

attachment storage diagnostics: upload then download a synthetic encrypted test blob. Log HTTP status class and latency bucket only — not real blob hashes.

---

7.5 Backend Operator Guide

Note. The Managed (Garage) tier is operated by FEN. The planned v2 bring-your-own self-hosted S3 tier will be operated by the user against their own S3-compatible endpoint. Full backend operations guidance is in backend-architecture.md.

What operators cannot see: group names, expense amounts, member identities, receipt images — all content is encrypted before transmission.

What operators can see: IP addresses (mitigatable by VPN/Tor), event sizes and timing, pubkeys as stable pseudonyms, connection frequency.

Required NIPs and API contract: §3.5 Backend API Contract.

Recommended practices:

• No long-term IP address logs
• Publish a transparent data retention and deletion policy
• Maintain a public status page
• Rate limit: max events per pubkey per hour, max event size enforced server-side
• TLS with a valid certificate in production
• Regular backups of the event store with tested restore procedure

---

7.6 Security Incident Response

Scenario	Response
Config signing key compromised	App update with new hardcoded public key; overlap window accepting both keys; revoke old key after adoption threshold reached
Default backend compromised	Push new backend list via remote config; dual-write window; notify users via banner; post public incident report
attachment storage server compromised	Push new attachment storage URL via remote config; E2EE means attacker cannot read blob contents; notify users to re-upload receipts if desired
Critical app bug (data loss risk)	Set min_version + min_version_deadline in remote config; publish fix immediately; hard block after deadline
Backend metadata leak (IP logs exposed)	Post transparency report; recommend VPN/Tor for users with high privacy requirements

Draft user-facing notification templates for each scenario in a separate runbook. Templates must describe what happened and what users should do — not technical details that could aid further exploitation.