RFC 0002 — Eventhub v3: eventhub-connect

Design RFC: moving ingest, validation and fan-out onto a three-node NATS/MQTT cluster inside the ARD CN.

• Status: Draft — proposes change
• Proposes: eventhub-connect, a new deployment target inside the ARD CN
• Last reviewed: 2026-08-28
• Tracking: issue #824, discussion #771

This page is written in English, unlike the rest of these docs. It is an engineering-context artifact aimed at maintainers and coding agents, not at publishers integrating with the API. For integration guides, start at Benutzer-Guides.

1. Why this document exists

v3 ships eventhub-connect as a redundant MQTT ingest, validation and plugin-publishing platform inside the ARD CN, deployed as three nodes across three SWR zones, with a UI for operators. The existing eventhub-ingest in GCP keeps serving broadcasters who have not migrated, mirroring their events to a minimal NanoMQ broker that eventhub-bridge relays into the CN. Once every broadcaster is on MQTT-in-CN, the GCP side is decommissioned (expected by mid to end of 2027).

RFC 0001 describes what runs today. This document states what changes, what it costs, and what is still undecided (§19, §20).

What v3 fixes:

• Every broadcaster needs a public HTTPS endpoint to receive push subscriptions, which means firewall negotiation inside each broadcaster’s own network. An outbound MQTT connection removes that.
• There is no path for control bits or radiotext. Now-playing is the only event class, so TA, Regio and RT/DL+ still ride the ZI gateway this is meant to replace.
• An ARD-internal workload depends on GCP for its data plane.

What today’s system gets right and v3 must not lose: it is simple to deploy, the schema is enforced, and it is observable.

2. Terminology

“v3” is one effort in two phases. One version number does not move.

• Eventhub 3.x — the release line, currently 3.0.0-beta.1. Phase one, already shipped: API cleanup, all of it breaking. Radiotext removed, x-ard-eventhub-uid dropped, length mandatory, trace deprecated. Documented for publishers in docs/user/migration-v3.md.
• eventhub-connect — what this RFC proposes: phase two of the same v3 effort, moving the transport and the data plane. A new deployment target, not a new version number, so shipping it does not imply a 4.x.
• de.ard.eventhub.v1.* — the event schema version, which stays at v1 throughout. Moving a publisher from HTTPS to MQTT is a transport change, not a schema change: the payload on inbox/{institutionId} is the same JSON the HTTPS API accepts today, tightened only in that identifiers must arrive as ARD URNs (§12.4), and the two new event types (radio.control, radio.data) are additions to v1. “Eventhub v3” never means de.ard.eventhub.v3.*.

The two phases are coupled: 3.0.0-beta.1 removed the radio.text event type outright, leaving no radiotext path at all, and this RFC reintroduces that capability as part of radio.data (§13.2).

3. North star

• Bidirectional by default. Publishers push to the broker over MQTT; subscribers consume on demand.
• Sovereign in the ARD CN. The data plane lives on three VMs in the CN. GCP is a sync peer, not the source of truth.
• Schema at the edge. Zod validation at the broker boundary; bad data never reaches subscribers.
• One source of truth for radio events. Track playing/next, control bits and radiotext share one broker, one topic tree, one set of consumers.
• Self-contained. No external SaaS in the data or observability path. The cluster runs, validates, routes and stays observable with zero internet access.
• Infra as code. Every VM and every workload lives in the repo; CI deploys; no manual prod edits.

4. Architecture

4.1 Three nodes, three zones

Three eventhub-connect nodes across three SWR zones inside the ARD CN, on near-identical NixOS + K3S images, each running the same workloads.

• connect-bad — Baden-Baden. Primary, in the narrow sense below.
• connect-stg — Stuttgart
• connect-mnz — Mainz

Three is a hard requirement, not a scaling choice. The NATS MQTT gateway is backed by JetStream, JetStream replication is RAFT, and RAFT quorum is (R/2)+1. At R=3 quorum is 2, so the cluster survives the clean loss of any single node and stays writable on the majority side of any partition. At R=2 quorum would be 2 of 2 — losing either node would take JetStream, and therefore MQTT, down on the survivor as well. Node count alternatives are in §21.6.

4.2 What “primary” means, and what it does not

BAD is primary in one sense only: it is where single-instance workloads that need a fixed home are placed — currently eventhub-bridge and the Beszel hub. It is a placement convention, not a role.

The ARD feed CronJob is also one instance cluster-wide but is not pinned here: it is stateless and writes to replicated JetStream KV, so K3S can schedule it in any zone.

It is not primary in the data path. NATS and JetStream are symmetric across all three nodes; the RAFT leader is elected, not assigned, and can be any node at any time. Clients connect to whichever endpoint answers, and no event needs BAD to be up. Losing BAD costs exactly two things: legacy publishers stop being relayed (accepted — §12.7) and host-level uptime monitoring goes dark. Native MQTT publishers, all subscribers, validation and plugin dispatch continue on the STG + MNZ quorum.

The Beszel hub on BAD is a known soft spot — the one observability component that becomes unavailable in the failure where it would be most useful. Tolerable because it is host and uptime monitoring rather than the metrics and logs path, but the SOC alerting path must not depend on it.

4.3 What runs on the nodes

• A NATS server with the built-in MQTT gateway — the public protocol publishers and subscribers connect to, with NATS subjects internally. JetStream enabled; the MQTT gateway requires it for sessions, retained messages and QoS 1.

• eventhub-connect — one service and one container image, with the role selected at startup. Each role is its own K3S workload, so they scale and fail independently while sharing one codebase, one dependency graph and one src/schemas/ import. Replica counts are in §10.5.

  • Validation sidecar — zod-validates every event at the broker boundary before fan-out (§10.2). Any zone.
  • Plugin adapters — one Deployment per target, delivering to external targets over outbound HTTPS (§10.4). Any zone.
  • Operator UI — read-only, no login, intranet-scoped (§14). Any zone.
  • ARD core feed loader — hourly refresh of the mapping the ownership check depends on (§8). A CronJob, exactly one instance cluster-wide, not one per node.

  The role is chosen by the container command, so adding a plugin target or a UI replica is a manifest change, not a new image.

• eventhub-bridge — relay from GCP into inbox/{institutionId}. A separate image, exactly one instance cluster-wide on BAD, not HA.

• The cluster-internal observability stack (§15).

During migration, GCP eventhub-ingest keeps accepting HTTPS posts and publishes raw events to inbox/{institutionId} on a NanoMQ container, which eventhub-bridge relays into the CN.

The broker is only reachable from inside the ARD CN, and the broadcasters are already there. A publisher’s playout system sits in the CN but in its own house’s network, not in SWR’s — SWR hosts the platform, it does not host the publishers. The only participants outside the CN are the legacy GCP project and the external plugin targets.

5. Services and code structure

5.1 Built versus deployed

• Built — three services, all TypeScript, all in this repo: eventhub-connect, eventhub-bridge, eventhub-ingest.
• Deployed — NATS, the GCP-side NanoMQ, and the whole observability stack. Upstream container images, configured by file, never forked and never wrapped. We write no broker code. NATS is deployed exactly like Vector or VictoriaMetrics: pull the official image, mount a config, run it.

5.2 One responsibility per service

eventhub-connect — the CN platform. One codebase, one image, several roles (§4.3).

• Owns: the NATS/MQTT connection surface, zod validation, the subject-versus-payload ownership check, the ARD feed loader, plugin eligibility and fan-out, the adapters that call external targets, and the operator UI.
• Does not: accept HTTPS event submissions, know anything about GCP, or talk to Pub/Sub.

eventhub-bridge — the GCP-to-CN relay, and nothing else.

• Owns: one MQTT subscription against the GCP broker, one MQTT publish into the CN’s inbox/{institutionId}.
• Does not: validate, map topics, resolve institutions, or know that plugins exist. If a code change to the bridge is ever needed because a schema changed, something has been put in the wrong place.

eventhub-ingest — the GCP HTTPS shim, shrinking in behaviour but not in surface.

• Owns: TLS termination, token issuance (/auth/login, /auth/refresh, /auth/reset), request auth (Firebase JWT plus a user allow-list), event-schema validation for synchronous publisher feedback (§5.4), identifier normalization — CRID-to-URN and Core-ID-to-URN via process-services.ts, plus filling in institutionId from the publisher it already resolves, while still emitting the deprecated externalId and type that Pub/Sub subscribers read (§12.4) — and an MQTT publish of the event body.
• Stays single-institution: one Firebase user maps to one user.institution.id, and the multi-institution capability in §7.4 is deliberately not backported.
• Does not: dispatch plugins or run its own fan-out. Its validation pass is advisory — the sidecar revalidates and is authoritative.
• Keeps its entire published route surface until each route is formally deprecated (§5.3).

5.3 The ingest API surface stays until it is deprecated

eventhub-ingest exposes a published, documented HTTP API with external consumers, and none of it may be deleted as a side effect of internal refactoring. Every route stays live, served and maintained until it has been through an explicit deprecation cycle — announced, with a date, and with measured zero traffic before removal.

The current surface:

• POST /auth/login, /auth/refresh, /auth/reset — token issuance.
• POST /events/:eventName — event ingest.
• GET/POST /subscriptions, GET/DELETE /subscriptions/:name — subscription management.
• GET /topics, GET /topics/:topicName — topic listing.
• PUT/POST /pubsub — Pub/Sub push handler and manual replay.
• GET /, GET /health — health checks.

“Maintained” means maintained, not frozen. These routes terminate TLS and verify tokens, so they stay on patched dependencies with their tests green for as long as they are reachable.

They retire on two different schedules:

• Short track — /subscriptions, /topics, /pubsub. These only mean anything while Pub/Sub carries events; a subscription to a topic that receives nothing is a trap. Their deprecation is coupled to the Pub/Sub shutdown, not scheduled independently: notice while subscribers migrate, then 410 Gone, then removal. This is why step 17 cannot simply switch Pub/Sub off.
• Long track — /events/:eventName and /auth/*. The legacy publishing path, surviving until the last HTTPS publisher moves to MQTT. /auth/* keeps the Firebase dependency alive for exactly as long as /events does; dropping Firebase is gated on the last token consumer, not on the CN being ready.

The OpenAPI document keeps describing every live route accurately, including deprecation markers.

5.4 Validation runs twice, on purpose

/events/:eventName keeps enforcing the event schema exactly as it does today, so a legacy publisher still gets a synchronous 400 with zod error detail. The sidecar then validates the event again once the bridge has relayed it into inbox.{institutionId}.

The two passes enforce different schemas by design: ingest accepts the legacy shape — externalId required, publisherId in either form — then normalizes the identifiers, while the sidecar enforces the stricter URN-only shape (§12.4). The sidecar therefore validates ingest’s output, not its input, which is what makes the strict rule enforceable without breaking a single legacy publisher.

Each pass exists for a different reason:

• Ingest validates for publisher feedback. A legacy publisher holds no MQTT connection, so feedback/{institutionId} cannot reach it and the HTTP response is the only diagnostic channel it has. Dropping the check would degrade the legacy path with no benefit to the houses still on it.
• The sidecar validates because it is the trust boundary. It validates everything arriving on inbox.> regardless of origin, so there is no bridged-message bypass and no trusted-publisher branch to get wrong. eventhub-bridge stays a dumb relay (§5.2) and the sidecar needs no notion of where a message came from.

The sidecar is authoritative. If the two passes disagree, the sidecar wins and the event does not reach the radio. tree.

The cost is that src/schemas/events.ts runs in two deployments, which can drift if the services ship from different commits, and that the strict shape is only reachable through ingest’s normalization. A sidecar rejection of a bridged event therefore means one of two things — the schemas diverged, or normalization stopped producing URNs — and both are alarms, not tolerable states (§11.2). CI builds both from the same commit, and the strict schema is derived from the legacy one rather than written twice.

5.5 Repo layout

v3 adds service directories rather than restructuring:

src/
  schemas/     zod schemas — the event contract
  openapi/     openapi.json generation from schemas (build-time)
  types/       shared types (#types)
  utils/       genuinely shared helpers only
  connect/     eventhub-connect  — sidecar, adapters, feed loader, UI backend
  ingest/      eventhub-ingest   — HTTPS auth + MQTT publish
  bridge/      eventhub-bridge   — GCP → CN relay
infra/         NATS + observability configs, K3S manifests, Nix flake

src/schemas/ is the single source of truth for the event contract, imported rather than reimplemented, with three consumers: eventhub-connect at runtime for event validation, src/openapi/ at build time for openapi.json, and eventhub-ingest for both its own event validation and its route schemas.

eventhub-ingest keeps importing all of it — events.ts for its validation pass (§5.4) and auth.ts, subscriptions.ts, topics.ts and common.ts for its surviving routes. Nothing in src/schemas/ can be deleted while a live route references it.

src/utils/ needs a deliberate split, not a wholesale move. Most of it is ingest-specific by accident of history and becomes eventhub-connect’s: ard-feed, ard-core, events/, plugins/. The Pub/Sub and Datastore helpers get deleted with the fan-out. Only what more than one service imports stays in utils/.

One build per service, three container images, one shared dependency graph. No monorepo tooling: separate entry points in one package is enough at this size, and package.json#main already points at a service entry point today.

6. Topics

6.1 MQTT to NATS translation

The NATS MQTT gateway does not pass topics through unchanged. It rewrites them, and getting this wrong produces subscriptions that connect successfully and then silently receive nothing. The conversion is implemented in mqttToNATSSubjectConversion (server/mqtt.go).

Practical consequences:

• Clients speak /, #, +. A publisher targeting subject inbox.urn:ard:institution:a3004ff924ece1a2 publishes to MQTT topic inbox/urn:ard:institution:a3004ff924ece1a2. Publishing to inbox.urn:… over MQTT instead lands on subject inbox//urn:… — one token — which nothing is subscribed to.
• ACLs are written in NATS subject syntax. The server config says inbox.urn:…; the wire says inbox/urn:….
• The URNs themselves are inert. Neither institution nor livestream URNs contain a . or a /, so each stays a single token in both notations and needs no escaping.
• NATS is MQTT v3.1.1 only. A client requesting MQTT 5 is rejected at connect with CONNACK return code 1, “unacceptable protocol version” — so no correlation data, response topics, user properties, session expiry or shared subscriptions. QoS 0, 1 and 2 are all supported (2.10+). The missing shared subscriptions shape the internal topology (§10.1).

6.2 Topic tree

NATS subjects are the source of truth. Both notations are given.

{institutionId} is the ARD institution URN, e.g. urn:ard:institution:a3004ff924ece1a2 for SWR. {livestreamId} is the livestream URN, e.g. urn:ard:permanent-livestream:49267f7d67be180d. {target} is a plugin target (radioplayer, dts, …). {class} is the event class as it appears in the radio. tree (track.playing, track.next, control, data).

The whole tree is URNs, not human-readable names. The institution URN is the string the ARD core feed returns as publisher.institution.id, so the ownership check guarding every event is a direct string comparison between subject token and feed (§8.2) — no mapping table, no join key. The cost is topic length: a broadcaster configures inbox/urn:ard:institution:a3004ff924ece1a2 rather than inbox/swr. It is the same urn:ard:… shape publishers already send today in services[].id, and the shape v3 requires for every identifier (§12.4).

The livestream comes before the event class, at a fixed depth. The dominant subscriber pattern is “everything for stream X” (encoders, broadcaster apps), and putting the ID last would make that inexpressible: radio.*.{urn} matches three-token subjects only, so it would catch control and data but never track.playing. ID-first keeps both access patterns available and turns per-stream ACLs into a prefix match.

plugin.> is internal. No MQTT user holds any permission on it, in either direction. It exists because plugin eligibility is a per-event decision (§10.3).

Publishers publish only to inbox/{institutionId} and subscribe only to feedback/{institutionId} — to one such pair, or to several if the credential is permitted for several institutions (§7.4). The radio., feedback. and plugin. trees are written exclusively by the sidecar. eventhub-bridge is just another writer to inbox/{institutionId}.

6.3 Subscription patterns

6.4 Migration note for existing subscribers

Today’s Pub/Sub topic names percent-encode the URN — de.ard.eventhub.prod.urn%3Aard%3Apermanent-livestream%3A49267f7d67be180d. MQTT and NATS both accept a raw colon, so v3 drops the encoding entirely. This is a breaking change for every subscriber, and it removes the pubsubBuildId / convert-id encode-decode pair.

7. Auth and ACL

Decision: static config users with bcrypt-hashed passwords, one user per principal, scoped to MQTT, over mandatory TLS. Not operator/JWT mode.

7.1 Why not JWT

NATS decentralized (operator) auth is the more powerful model, and the wrong fit here for three reasons.

MQTT can’t do the NATS handshake. NATS-native JWT auth verifies an NKEY signature over a server-issued nonce, and MQTT has no nonce exchange. nats-server works around this — auth.go has an explicit branch, “MQTT can carry JWTs in the password field” — but the JWT must then be marked a bearer token, which tells the server to skip signature verification entirely. The client never signs anything and never needs the NKEY seed; the JWT alone is the credential, and anyone who captures it can impersonate that publisher until it expires.

It puts token-refresh logic in every broadcaster’s playout system. Short-lived credentials mean each house implements refresh-before-expiry and reconnect, in systems we do not control and cannot debug. A static credential has no refresh path to get wrong.

We don’t need the scale it buys. Roughly ten publisher orgs and a handful of subscriber roles, all provisioned by one team.

7.2 Username convention and rotation

Publisher and subscriber credentials are separate user classes with separate naming rules.

• Publishers: pub-{label}-{issued}, e.g. pub-swr-2026-06-26. The pub- prefix is mandatory and reserved — anything carrying it publishes only to inbox.{institutionId} subjects, one or several (§7.4), and nothing else in the config may use it.
• Subscribers: a descriptive role name plus issuance date, e.g. sub-radioplayer-2026-06-26, sub-ard-sounds-2026-06-26. Naming is looser, because subscriber scopes vary and new consumers appear more often than new publishers. The sub- prefix is a convention, not a rule the ACLs depend on.
• Services: svc-sidecar, svc-adapter-{target}, svc-bridge. NATS-native, not MQTT, and not rotated on the same cadence.

The username is a label for humans; the institution URN in the subject is the contract. The swr in pub-swr-2026-06-26 is not parsed, matched or resolved by anything — the ACL binds that user to a set of inbox.urn:ard:institution:… subjects, and the sidecar reads the institution from the subject each message arrived on. The label exists because the operator UI groups connections by username, where a 16-hex-character hash would tell an operator nothing.

The date suffix makes rotation additive rather than destructive:

1. Add pub-swr-2027-01-15 alongside pub-swr-2026-06-26, identical permissions, new password.
2. Reload the config. Both credentials now work.
3. The house switches its client whenever it suits them.
4. Once the operator UI shows no connections on the old user, remove it and reload again.

No coordinated cutover, no downtime window, and no window during which a slow house is locked out. The switch is seamless for the client, because MQTT session state is keyed by client ID within an account, not by user: a publisher reconnecting with the same client ID and a new username resumes its existing persistent session, queued QoS 1 messages and all. This is one reason the design uses a single account. The date never appears in a topic, so rotating changes nothing for the publisher’s topic configuration and nothing for any subscriber. It also makes credential age an audit finding — a pub-* user dated 2026 still in the config in 2028 is visible as such.

7.3 The config

Users live in the authorization block of the NATS config, which is hot-reloadable — adding or revoking a user is a config change plus nats-server --signal reload, with no restart and no dropped connections for anyone else.

authorization {
  users: [
    # publisher, common case: one organization, per issuance. publish-only into its own inbox.
    # the subject carries the institution URN; the username is just a readable label.
    {
      user: "pub-swr-2026-06-26"
      password: "$2a$11$4I9tIK1JVbttZYtn.F.Jse5iY5ves4EtYWIpjlwyvgVYHJc8yTvk."
      allowed_connection_types: ["MQTT"]
      permissions: {
        publish:   { allow: ["inbox.urn:ard:institution:a3004ff924ece1a2"] }
        subscribe: { allow: ["feedback.urn:ard:institution:a3004ff924ece1a2"] }
      }
    }

    # publisher acting for several institutions: one entry per permitted inbox.
    # nats enforces the set; eventhub-connect never sees the username.
    {
      user: "pub-shared-playout-2026-06-26"
      password: "$2a$11$..."
      allowed_connection_types: ["MQTT"]
      permissions: {
        publish: {
          allow: [
            "inbox.urn:ard:institution:a3004ff924ece1a2"
            "inbox.urn:ard:institution:b71c0e4d9a25f338"
          ]
        }
        subscribe: {
          allow: [
            "feedback.urn:ard:institution:a3004ff924ece1a2"
            "feedback.urn:ard:institution:b71c0e4d9a25f338"
          ]
        }
      }
    }

    # subscriber: one per external consumer role, scoped to what it needs. never publishes.
    {
      user: "sub-ard-sounds-2026-06-26"
      password: "$2a$11$..."
      allowed_connection_types: ["MQTT"]
      permissions: {
        subscribe: { allow: ["radio.*.track.playing", "radio.*.track.next"] }
        publish:   { deny:  [">"] }
      }
    }

    # service: the validation sidecar. NATS-native, plus an MQTT connection for retained publishes.
    {
      user: "svc-sidecar"
      password: "$2a$11$..."
      permissions: {
        subscribe: { allow: ["inbox.>"] }
        publish:   { allow: ["radio.>", "feedback.>", "plugin.>"] }
      }
    }

    # service: one per plugin adapter. reads only its own work queue, publishes nothing.
    {
      user: "svc-adapter-radioplayer"
      password: "$2a$11$..."
      permissions: {
        subscribe: { allow: ["plugin.radioplayer.>"] }
        publish:   { deny:  [">"] }
      }
    }

    # service: the legacy relay. writes only to inbox, like any other publisher.
    {
      user: "svc-bridge"
      password: "$2a$11$..."
      permissions: {
        publish:   { allow: ["inbox.>"] }
        subscribe: { deny:  [">"] }
      }
    }
  ]
}

• Passwords are bcrypt hashes, generated with nats server passwd. The config in git holds the hash; the client still sends the plaintext, so bcrypt protects the config at rest, not the wire.
• allowed_connection_types: ["MQTT"] pins publisher and subscriber credentials to the MQTT listener, so a leaked publisher password cannot open a NATS-native connection.
• Permissions are NATS subject patterns — dots, *, >. See §6.1.
• Publishers never subscribe to radio.> and subscribers never publish anything. A publisher that also wants to consume gets a second, sub- prefixed credential, so a compromised publisher credential cannot read the whole event stream and a compromised subscriber credential cannot inject anything.
• no_auth_user is not set, so an unauthenticated connect fails. This is the specific line Q2 proposes reversing for read-only subscribers.
• No MQTT credential has any permission on plugin.>. It is reachable only by NATS-native service users, and each adapter sees only its own target’s subtree.

7.4 One user, several institutions

A single credential can be permitted for more than one institution, by listing several inbox.{institutionId} subjects in its publish.allow. A shared playout system serving two houses, or an operator publishing on behalf of several, needs one credential rather than one per house.

eventhub-connect needs no new logic for this, and that is the point. The multi-institution decision lives entirely in the ACL:

• NATS decides which institutions a user may act for. A publish to an institution outside the allowed set is refused at the broker, before the message exists.
• The subject says which institution this particular message is for. One message carries exactly one institution, whatever the credential’s breadth.
• The sidecar verifies the livestream belongs to the institution in the subject (§7.5) — the same single comparison it makes for a single-institution publisher.

Composed, those three give exactly the required property: a livestream can only be published under an institution the user is permitted for. The sidecar gets there without knowing who published, and the reason it must work this way is that it cannot know. A message read from JetStream carries its subject, headers, payload and sequence — not the identity of the NATS user that published it. Any design where the sidecar checks a livestream against the user’s permitted set would need that identity propagated into every message, which is precisely the mapping table §8.2 exists to avoid.

The practical cost falls on the publisher: a multi-institution client must publish each event to the matching inbox/{institutionId}. Sending an SWR livestream to the NDR inbox is rejected by the sidecar even though the credential holds both permissions, because the subject and the livestream disagree. The services[].institutionId field (§12.4) makes that mistake self-evident in the rejection rather than something to infer.

svc-bridge is already this pattern. It holds publish.allow: ["inbox.>"] — every institution at once — and needs no institution logic of its own, because the subject it publishes on carries the institution and the sidecar checks it like any other message (§11.2). A multi-institution publisher is the same arrangement with an explicit list instead of a wildcard.

None of this applies to eventhub-ingest. A Firebase user there has exactly one user.institution.id, checked directly against the resolved publisher’s institution, and it stays that way for the remainder of the legacy path’s life. Multi-institution is a v3-only capability; nothing in the legacy code needs to learn about it.

7.5 The ACL protects the subject, not the payload

pub-swr-2026-06-26 can only write to inbox.urn:ard:institution:a3004ff924ece1a2, but nothing at the broker level stops it publishing a payload whose services[] array claims an NDR livestream.

The sidecar must therefore verify that every service in the payload belongs to the institution taken from the subject — never from the payload, and never from the credential’s permitted set, which it cannot see. This is the same institution-ownership check processEvent does today (blocked: "User unauthorized for service"); it moves into the sidecar and takes its trusted input from the subject token. services[].institutionId is checked as part of it, as a claim that must agree rather than as the source of truth. The data behind it is the ARD core feed (§8).

7.6 Residual risk

Credentials do not expire on their own, so a leaked password is valid until someone rotates it — the accepted trade for removing refresh logic from ten houses’ playout systems. Mitigations: TLS is mandatory, the blast radius of any single leak is one publisher’s inbox or one subscriber’s read scope, the issuance date makes stale credentials visible, and revocation is one config reload away.

8. The ARD core feed

The ownership check needs to resolve a livestream URN to the organization that owns it. That mapping comes from the ARD core livestream feed — one JSON document containing every livestream, each with its publisher, each publisher with its institution. eventhub-ingest already consumes it (src/utils/ard-feed.ts, just feed); the v3 loader keeps the same shape but changes the failure behaviour.

8.1 What it is used for

• Authorization. livestreamId → publisher → institution.id, compared against the institution URN in the subject. The only security-relevant use.
• Validation. Rejecting events for livestream URNs that do not exist at all, and checking that the payload’s publisherId is the publisher the feed actually lists for that livestream.
• Enrichment. Publisher title, image and homepage for the operator UI and for adapters that need station names.

8.2 Ownership is a direct comparison

Because the subject carries the institution URN, the check is string equality between ids:

subject:  inbox.urn:ard:institution:a3004ff924ece1a2   ← trusted, enforced by the ACL
payload:  services[].institutionId                      ← a claim, must agree
feed:     livestreamId → publisher.institution.id       ← the authority

All three must agree. The subject is trusted because the broker enforced it, the feed is the authority on what a livestream belongs to, and the payload is a claim that has to match both. Equal means accept; any disagreement means blocked, and which pair disagrees says what went wrong: subject against payload is a client routing its event to the wrong inbox — the likely mistake for a multi-institution credential (§7.4) — while payload against feed is a stale or wrong id in the publisher’s configuration.

This holds unchanged when a credential is permitted for several institutions, because the comparison is per message and never consults the credential. There is no mapping table and no join key — no slug → institution config to maintain, drift, or get wrong.

This is stricter than today’s check, and the URN-only rule is what allows it. process-services.ts currently derives the livestream URN by hashing the CRID (createHashedId(service.externalId)) and never looks that URN up in the feed; the institution is resolved from the payload’s publisherId instead, with allowed-livestreams.json covering COMMON_IDS topics. So a payload naming another house’s livestream under its own publisher is only caught for allow-listed entries. When the livestream arrives as a URN rather than as a CRID to be hashed, it can be looked up directly, and the feed’s publisher for that livestream becomes the authority — the payload’s publisherId is then a cross-check rather than the join key.

Two properties make the comparison safe:

• The ACL is the pinning. The binding between a credential and an institution is expressed once, in the ACL, and reviewed in the same PR that creates the user.
• Nothing authorization-relevant reads a mutable upstream string. institution.acronym and institution.title are display fields and must never be used as join keys — an acronym change upstream would break a house’s publishing, and a collision or blanked field could widen access. Ids only, on both sides.

The feed still has to be trusted for livestreamId → institution.id, hence the integrity rules below.

8.3 Refresh through JetStream KV

An hourly K3S CronJob fetches the feed, validates it, and on success writes it to a JetStream KV bucket. Every sidecar watches that key. Preferred over each node fetching independently because:

• All three nodes converge on the same version. With independent fetches, node A can accept an event node B would reject. The skew is bounded by the refresh interval and self-corrects, but it is an avoidable class of “works on one node” bug.
• RAFT replication is already there, so the feed inherits the cluster’s replication rather than needing shared storage.
• KV revision history gives the previous-version requirement natively, including rollback to a known-good revision without re-fetching from an upstream that may still be broken.
• A KV watch means no polling.

If KV turns out awkward, the fallback is a per-node fetch to a local file with the same validate-then-swap discipline, with the version skew visible in the UI. One fetcher, many watchers either way — upstream should see one request per hour, not one per pod per hour.

8.4 Staleness is safe, silence is not

The current implementation must not be carried over: getARDFeed calls process.exit(1) on any failure — non-200, malformed JSON, timeout, or a failed integrity rule. On a node restart during an internet outage, the service does not come up at all.

The v3 rules:

1. Validate before swapping. Fetch into a candidate, run every integrity rule, and only then atomically replace the active feed. A failed refresh is a no-op that leaves the previous version serving.
2. Never fail closed on feed age. A stale feed keeps authorizing events indefinitely. Rejecting valid events because an upstream CMS is unreachable would be a self-inflicted outage, and the feed’s contents change on the order of weeks while an outage lasts hours.
3. Persist the last good copy to disk or KV so a cold start with no network still comes up working.
4. Ship a bootstrap copy in the image. A brand-new node with no network and no persisted state must still start; the Nix store gives this for free.
5. Make the age loud. Warn at 3 hours (three missed refreshes), alert at 12, page at 48, and show feed age and active revision on the UI front page.

8.5 Poison-feed protection

The feed is an authorization input, so a truncated or half-populated upstream response could silently revoke a house’s ability to publish. A candidate is rejected — keeping the previous version — if any of these fail:

• Item count outside bounds. Currently minItems: 190, maxItems: 251.
• Pagination present. totalPageCount > 1 means we are seeing a partial feed.
• Required stations missing. The existing hardcoded canary list (WDR 2, 1LIVE, SWR3, hr3, …).
• Institution count dropped relative to the active version.
• Any publisher with a live connection disappeared. Catches an upstream edit that would break a publisher mid-broadcast. Connection state is already known from the broker, so the check is local.
• generated is not newer than the active version. Rejects replays and clock-confused upstream responses.

8.6 Carry-overs

• TEMP_PUBLISHER_MAPPING — eleven hardcoded publisher-id remaps in ard-core.ts, dated by their own name. Either the upstream ids get fixed or this becomes explicit config with an owner and a review date (Q7).
• allowed-livestreams.json — the overlay for COMMON_IDS livestreams absent from the feed. It survives into v3 as an explicit overlay applied after the feed loads, under the same rule: the institution comes from pinned config, not from the overlay’s own claims.
• Index on load. publisherLookup.getById does a linear scan over ~200 items per lookup. Cheap even at cyclic radio.data volume, but build a Map when the feed is swapped in.

9. Delivery semantics and latency

9.1 Per event class

• radio.control — durable. A dropped {"name": "TA", "state": false} leaves the traffic-announcement bit stuck on, which is a broadcast fault. Three nodes with JetStream give at-least-once delivery: QoS 1 with a persistent session survives a subscriber reconnect, and RAFT quorum survives a node loss. validUntil is a backstop, not the mechanism — receivers drop the state when the TTL passes, so a lost off-event self-corrects, but the explicit state: false event remains the primary path. No cyclic re-assertion.
• radio.data — cyclic from the source. The cycle field carries the source’s own repeat interval, so a missed message self-heals within one cycle without any broker-side guarantee. This is how RT/DL+ already behaves.
• radio.track.playing / .next — last value wins. Stale-by-one is cosmetic.

Retained messages are the late-joiner fix. An encoder that reboots mid-song has, with no retention, nothing to show until the next event — minutes for track events, potentially hours for control. Retained messages are stored in JetStream ($MQTT_rmsgs), another reason JetStream is not optional.

9.2 The 220 ms budget

The ZI Gateway functional specification sets 250 ms as the processing limit, of which 30 ms is ARD CN transit. That leaves 220 ms for everything Eventhub does between a publisher’s PUBLISH and a subscriber receiving the validated event.

The budget binds radio.control: a now-playing event arriving 400 ms late is invisible, a TA bit arriving late is a missed announcement. All classes share one path, so meeting it for control means meeting it for everything.

Where the 220 ms goes:

1. JetStream write to the inbox stream at R=3 — a RAFT quorum commit, so roughly one inter-zone round trip plus fsync.
2. Sidecar consumer delivery — see §9.3.
3. Zod validation, feed lookup, plugin eligibility — in-memory, sub-millisecond.
4. MQTT publish with RETAIN to radio.… — a second quorum commit, plus the retained-message write.
5. Fan-out to subscribers — core NATS, negligible.

The two RAFT commits dominate, and the three-zone spread is what makes them cost anything. A single-zone cluster would commit in well under a millisecond; BAD ↔ STG ↔ MNZ makes each commit an inter-zone round trip. The inter-zone RTT is therefore the number that decides whether the design fits, and it is an open question rather than an assumption (§20).

The end-to-end latency metric in §15 becomes a p99 SLO against the 220 ms, instrumented per stage so a regression points at a hop rather than at the system.

9.3 The pull-consumer latency trap

A JetStream pull consumer only delivers while a fetch is outstanding. The naive loop — fetch a batch, process it, fetch again — means a message arriving just after a fetch returns waits for the next fetch, so broker latency silently becomes poll latency. At 220 ms that is fatal.

• Always keep a fetch parked at the server. Issue the next long-poll fetch before processing the current batch, so there is never a window with no outstanding request. The server then delivers the instant a message lands.
• Small batch, long expires. A large batch encourages the server to wait for it to fill; long expiry keeps the parked fetch alive without churn.
• Never sleep between fetches. Any backoff belongs on error paths only.

9.4 Control events can be head-of-line blocked

inbox.{institutionId} does not encode the event class — the class lives in the payload’s type field — so a JetStream consumer on inbox.> cannot filter or prioritise by class. Every event for an organization goes through one durable consumer in arrival order.

radio.data is about to become the dominant volume: cyclic data at cycle: 8 across ~60 livestreams is roughly 7.5 msg/s against today’s ~0.3 msg/s. A TA event arriving behind a burst of cyclic data waits for it. max_ack_pending and multiple sidecar pods reduce the exposure but cannot remove it, because the ordering is in the stream, not the consumer.

The fix is a class token in the inbox subject — inbox/{institutionId}/{class} — allowing a dedicated, separately-tuned consumer for inbox.*.control. It costs a slightly wider client contract and requires validating the payload’s type against the subject token, and it buys per-class publish ACLs, so a house can be granted now-playing without the authority to assert TA. Recommended; see Q4.

10. Internal topology and scaling

10.1 Work that scales consumes NATS-side

MQTT 3.1.1 has no shared subscriptions. Every MQTT subscriber on a topic receives every message, so no MQTT-consuming workload can run more than one instance without doing the work twice. Every consumer that needs to scale therefore consumes on the NATS side, and touches MQTT only where it needs an MQTT-specific feature.

NATS offers two once-only mechanisms:

• Core queue groups — each message goes to exactly one group member. Lowest latency, but fire-and-forget: if the member dies mid-processing the message is gone, with no redelivery.
• JetStream durable consumers — each message goes to exactly one puller, with explicit ack and redelivery on timeout. Costs one extra round trip; survives a pod dying mid-work.

Both internal tiers use JetStream consumers.

10.2 Validation sidecar

JetStream pull consumer, durable sidecar, filter inbox.>, ack_policy: explicit. Ack after the validated event has been republished, not on receipt.

A core queue group would also prevent duplicate republishing and would be slightly faster, but the publisher has already received its QoS 1 PUBACK by then, so a pod dying between receipt and republish loses the event silently. That breaks the durability promise for radio.control and makes every rolling restart lossy. The extra round trip is low single-digit ms inside the CN.

• max_ack_pending sized to pod count × desired concurrency.
• max_deliver: 3 with a short ack_wait. A message that fails validation is not retried — it is termed and reported on feedback/{institutionId}. Redelivery exists for crashes, not bad data.
• A fetch is always outstanding (§9.3). A hard requirement, not a tuning preference.
• Each pod needs a distinct MQTT client ID on its publish connection, derived from the pod name. Two pods sharing a client ID means the broker evicts one on every connect and they fight in a reconnect loop.

Each pod holds two connections: MQTT to publish radio/… with the RETAIN flag, which has no documented NATS-side equivalent, and NATS-native to pull inbox.> and publish the plugin.{target}.… fan-out.

10.3 Plugin routing

Plugin eligibility is a per-event decision, not a property of a subject. A publisher sets plugins[] on the event, and the auto-enable rule only covers music track.playing — track.next never auto-enables. So two radio.{id}.track.playing events on the same subject can have completely different plugin targets, and an adapter cannot decide what to deliver by filtering radio.>.

The sidecar resolves eligibility and fans out to a dedicated per-target subject, which is what processEvent does today when it publishes one Pub/Sub message per enabled plugin. For each validated event the sidecar publishes:

1. radio.{livestreamId}.{class} over MQTT with RETAIN — the public tree, unconditional, every subscriber sees it.
2. plugin.{target}.{livestreamId}.{class} NATS-native, once per enabled target, and only for enabled targets. No plugins on the event means nothing is published here at all.

No plugin metadata leaks into the public tree, and adapters carry no eligibility logic. Plugin subjects are not retained: an adapter coming back after a restart wants its consumer backlog, not the last event replayed at it.

10.4 Plugin adapters

Adapters live inside eventhub-connect. Each takes a message off its own work queue and delivers it to its target over outbound HTTPS.

• One JetStream stream, PLUGINS, capturing plugin.>, with a short max_age.
• One durable pull consumer per target, filtered on plugin.{target}.> — adapter-radioplayer on plugin.radioplayer.>, adapter-dts on plugin.dts.>. Not one shared consumer with dispatch on a field: per-target consumers give independent failure domains, so if Radioplayer is timing out its ack-pending backs up and its lag climbs while DTS is untouched. A shared consumer lets one slow target eat everyone’s capacity.
• All pods for a target pull from that target’s durable consumer. Each message is delivered once; adding a pod adds concurrency with no config change and no coordination.
• max_ack_pending is the concurrency limit and the backpressure knob — it caps in-flight HTTPS requests per target, and it is what keeps a slow target from wedging the pipeline.
• On HTTP failure: nak with a short delay, or term to drop immediately. max_deliver caps total attempts.
• Adding a target is a sidecar config change plus a consumer and a Deployment. No change to the radio. tree, no change for any publisher or subscriber.

Retries have a freshness ceiling, because redelivering a now-playing event after the song has ended is worse than not delivering it: the PLUGINS stream carries a short max_age in minutes, max_deliver stays at 2 or 3, and adapters check the event’s own start / time before dispatch and term anything already stale.

One behavioural change to settle during implementation: today the DTS plugin receives one HTTP call carrying an array of services, because a single Pub/Sub message covers all non-blocked services on the event. Per-livestream plugin subjects make that one call per livestream instead. It only differs for the handful of shared nightly broadcasts that carry multiple services. If batching turns out to matter for a target, the adapter can window on its own side.

10.5 What scales and what does not

The four eventhub-connect roles share one image but scale independently, each as its own workload (§4.3).

• NATS server — fixed at one pod per zone. The one workload that is not horizontally scalable: adding a pod changes cluster and JetStream meta-group membership, and the quorum math wants an odd count. Scale up (CPU, RAM) or add zones in odd numbers.
• Validation sidecar — N pods, any zone. The JetStream pull consumer gives once-only handling regardless of count.
• Plugin adapters — N pods per target. Same mechanism, one durable consumer per target.
• eventhub-bridge — exactly one instance cluster-wide, not one per zone. It holds an MQTT subscription to the GCP broker, and with no shared subscriptions a second instance would relay every legacy event twice.
• Operator UI — N pods. Read-only. The stats endpoints are stateless, and the live tail holds one core NATS subscription per pod plus its browser WebSockets — no durable consumer, so nothing to coordinate. Session affinity is not required.
• ARD feed CronJob — exactly one instance cluster-wide. Hourly.

Capacity. Peak throughput is roughly 10 msg/s — about 7.5 from cyclic radio.data plus a fraction of that for track changes. Core NATS handles millions of messages per second and JetStream at R=3 tens of thousands on modest hardware, so the broker is four orders of magnitude over-provisioned on day one. Run two pods per scalable workload for availability during rolling updates, not for throughput. The only place capacity could bite is adapter concurrency, since each event becomes one outbound HTTPS request with a multi-second timeout — a top-of-the-hour burst across all stations is the shape to size for. Tune max_ack_pending first and measure before adding pods.

10.6 Ordering

With more than one pod in any tier, per-livestream ordering is not guaranteed: two events for the same stream can be processed concurrently and arrive out of order. Already true on Pub/Sub today, and it matters because a track.next overtaking a track.playing sticks the wrong song.

Subscribers apply last-write-wins on the event’s own timestamp (start for track and data events, time for control), never on arrival order. If strict per-stream ordering becomes a hard requirement, the escape hatch is partitioning the inbox by livestream and running one consumer per partition.

11. Event flow

11.1 CN path

1. Publisher opens a persistent MQTT connection to one of the three nodes.
2. Publisher publishes a raw event to inbox/{institutionId}.
3. One sidecar pod pulls the message from the durable sidecar consumer on inbox.> — exactly one, whatever the pod count — runs zod validation, and verifies that every service in the payload resolves to the institution URN taken from the subject.
4. On valid, the sidecar publishes over MQTT with RETAIN to radio/{livestreamId}/{class}. Unconditional; every event lands here.
5. The sidecar resolves plugin eligibility and publishes NATS-native to plugin.{target}.{livestreamId}.{class}, once per enabled target. An event with no plugins skips this step entirely.
6. JetStream replicates across the three nodes at R=3.
7. All three nodes expose the radio/# tree to MQTT subscribers, who consume from whichever node they are connected to.
8. In parallel, each plugin adapter pulls from its own plugin.{target}.> consumer and delivers over outbound HTTPS.

On invalid: zod rejection produces a structured error to the cluster observability stack, no event crosses the trust boundary, and the publisher is notified on feedback/{institutionId}.

11.2 Legacy path

1. Legacy broadcaster posts to eventhub-ingest over HTTPS as today.
2. eventhub-ingest authenticates the request and publishes the raw body to inbox/{institutionId} on the minimal GCP MQTT broker. It already resolves the publisher’s institution today, so it already has the URN the subject needs. During the earlier steps it also still fans out to Pub/Sub; that ends at step 17.
3. eventhub-bridge subscribes to inbox/# on the GCP broker and republishes each message to the same topic in the CN.
4. From there, identical to §11.1 with no special case anywhere.

One validator, one eligibility implementation, one fan-out — all in the sidecar. Two consequences:

• eventhub-ingest runs its own zod pass for the whole life of the legacy path (§5.4), so a sidecar rejection of a bridged event means the two copies of the schema disagree. That is a standing drift alarm for us rather than feedback for the publisher, and it should alert rather than sit in a log.
• Plugin dispatch must not run on both sides. While GCP-side Pub/Sub dispatch is still live, the sidecar’s fan-out to plugin.> would double-deliver every bridged event to Radioplayer and DTS. This is why the bridge ships before the adapters move: during that window plugin.> messages expire unconsumed (short max_age), or the fan-out stays behind a feature flag. The adapter cutover then flips one switch per target on the GCP side, covering native and bridged publishers identically because both already flow through the sidecar.

12. The client contract

Every publisher and subscriber connects over MQTT and authenticates with a per-principal username and password (§7).

12.1 Protocol

• MQTT v3.1.1. A client requesting v5 is rejected at connect. Set the protocol version explicitly if your library defaults to v5.
• TLS required, port 8883. Plaintext 1883 is not exposed, inside the CN or otherwise.
• Topics use /, never a .. Wildcards are # and + (§6.1).

12.2 Publishers

• Provisioning. Request a credential from the Eventhub team. You receive a username of the form pub-{label}-{issued}, a password, the institution URN or URNs your credential is permitted for (§7.4), and the current node endpoints. No expiry, no refresh logic.
• Connection. Open a persistent connection with all three node endpoints configured, so the client reconnects to a surviving node on drop. Keep-alive on, ping every 30–60s.
• Auth. MQTT username is the issued pub-… username; the date suffix is part of it, don’t strip it.
• Publish. Publish to inbox/{institutionId}, e.g. inbox/urn:ard:institution:a3004ff924ece1a2. The topic uses the institution URN, not the username — neither is derived from the other. QoS 1, clean-session false so QoS 1 messages queue across reconnects, and a stable client ID — a fresh one starts a new session and loses the queue, and keeping it stable is what makes credential rotation seamless.
• If your credential covers several institutions, route each event to the inbox of the institution that owns the livestream, and set services[].institutionId to match. The broker accepts any of your permitted inboxes, so a misrouted event is caught by the sidecar rather than at publish time.
• Feedback. Subscribe to feedback/{institutionId}, for each institution you publish for.
• Schema. The payload is the same JSON the HTTPS API accepts today, except inside services[]: identifiers must be pre-encoded ARD URNs, institutionId is now required, and externalId and type are deprecated — omit them (§12.4). Existing schemas otherwise apply to track.playing / track.next; the two new classes are in §13.
• Idempotency. QoS 1 is at-least-once by definition. A reconnecting publisher may re-send.

12.3 Subscribers

• Provisioning. One credential per consumer role, named sub-{role}-{issued}, with subscribe permissions scoped to the subjects you need. A subscriber credential cannot publish; if you also produce events, request a separate pub- credential. May become optional for track and data events (Q2).
• Connection. Same as publishers: all three endpoints, keep-alive, persistent session, stable client ID.
• Subscribe. See §6.3. QoS 1, clean-session false.
• Retained messages. On connect you receive the last retained message for each matching topic, then live events. Treat the retained message as current state, not as a new event.
• Order by timestamp, not arrival (§10.6). Discard any event older than the one you already hold.
• Tolerate duplicates. A redelivery after a pod restart is normal; handling must be idempotent.
• Trust the sidecar. Every event you receive has been zod-validated at the broker boundary. Re-validation is optional.

12.4 Identifiers

New MQTT delivery accepts pre-encoded ARD URNs only. Every identifier in services[] arrives already resolved, and the sidecar performs no ID conversion.

externalId and type are deprecated, not deleted — the deletion is scheduled for step 19. Both are redundant once the publisher supplies the id: externalId is the CRID the id was hashed from, and type carries the same fact as the id prefix, since core-id-prefixes.json maps them one-to-one. Keeping either means keeping a pair that can disagree, and a payload where type: PermanentLivestream meets id: urn:ard:event-livestream:… has no correct interpretation.

They survive the migration because eventhub-ingest emits one body to two places. The normalized event goes to the GCP MQTT broker and to Pub/Sub, and Pub/Sub subscribers read services[] as it stands today. Dropping the fields at step 3 would silently change the shape those consumers receive long before any of them has migrated — a breaking change to the legacy path, delivered by a step whose purpose is to prepare the new one. So through the migration they stay present and populated, and their removal is bound to the Pub/Sub shutdown at step 19, when the last consumer of that shape is gone.

Deprecation here follows the same discipline as the route surface (§5.3): marked in the schema with .meta({ deprecated: true }) so OpenAPI and the generated docs say so, announced with a date, and removed only after measured zero usage (§15).

type also stays required on HTTPS input, where it is load-bearing rather than redundant: legacy ingest has no id to read and uses type to pick the prefix it hashes the CRID onto (coreIdPrefixes[type] + createHashedId(externalId)). Removing it there would break the very conversion that keeps bridged events valid.

One consequence of the eventual removal: the id regex becomes the single declaration of which livestream kinds exist, doing validation and kind-derivation at once. That is the point — one source instead of two that can drift — but it does mean adding a livestream kind becomes a schema change rather than an enum entry.

Why URN-only:

• The sidecar does no ID resolution. Conversion today is a deterministic hash of the CRID plus a type-to-prefix lookup (process-services.ts, core-id-prefixes.json) — it mints an id rather than verifying one. Keeping it out of the CN path leaves the sidecar one feed dependency — livestreamId → institution.id for the ownership check (§8.2) — and no id minting on the write path.
• No ambiguity about which form was sent. Today publisherId accepts either a Core ID or a URN and Eventhub infers which; with one accepted form there is nothing to infer and nothing to get wrong.
• publisherId becomes checkable. As a URN it can be compared directly against the publisher the feed resolves for that livestream, so a payload claiming the wrong publisher is caught rather than normalized into something plausible.

institutionId is new in v3 and carries the owning house explicitly. It exists for two reasons. Downstream, it makes an event self-describing: a subscriber on radio/… or a plugin adapter knows which house produced it without holding the feed, which is the last thing that forced feed knowledge onto consumers. Upstream, it turns the ownership check into a three-way agreement between the subject, the payload and the feed (§8.2), so a mismatch names which of the three is wrong instead of just failing.

It is a claim, not an input. Nothing trusts institutionId — the subject remains the only trusted carrier of institution, and the feed remains the authority on what a livestream belongs to. A payload whose institutionId disagrees with either is rejected. Ingest fills it in during normalization from the publisher it already resolves, so bridged events satisfy the field without a house changing anything.

eventhub-ingest keeps the conversion. Legacy HTTPS publishers keep sending externalId and a numeric publisherId; ingest normalizes both before publishing to MQTT, so what arrives on inbox.{institutionId} through the bridge already satisfies the rule above. No house is forced to change its HTTPS payload, and eventhub-bridge stays a dumb relay.

A deprecated field is ignored, not rejected. id is required and therefore authoritative in both cases, so there is no ambiguity to protect against and no reason to fail an otherwise correct event. The sidecar reads neither. A native MQTT publisher should simply omit them; a bridged event will carry them for as long as ingest exists, and the counter in §15 is what tells us when nobody sends them any more.

This applies to services[] only, and both names exist elsewhere in the payload. The event-level externalId and references[].externalId are content identifiers — the track’s id in the publisher’s system, the CRID of a show or episode — and the event-level type is the event class (de.ard.eventhub.v1.radio.track.playing), which is required and unchanged. Only the two fields inside services[] are affected; nothing here asks houses to stop identifying content or to stop naming their events.

12.5 Validation feedback

Feedback is a diagnostic channel, not a transactional acknowledgement. The publisher does not block on it and does not need to correlate every publish with a response.

• On zod rejection the sidecar publishes to feedback/{institutionId} with the error detail and enough identity to locate the offending event.
• Correlation is by playlistItemId for track events, which publishers already send and which is unique per item. radio.control and radio.data carry no playlistItemId; for those, feedback references the target subject plus the event’s start.
• Feedback messages are retained per institution, so a client that connects after the fact still sees the most recent problem. A multi-institution client subscribes to each of its feedback/ topics and gets the rejection on the one the event was published under.
• The authoritative record is the operator UI and the cluster logs.

12.6 Migration path

• Publishers on HTTPS today. Keep posting during the migration window; no code changes required and no behaviour change. The synchronous 400 with zod error detail stays for as long as the route does (§5.4).
• Publishers ready to migrate. Point the publisher config at eventhub-connect over MQTT; the username and password replace the static API token. The payload changes if you send CRIDs or numeric Core IDs today: inside services[], id and publisherId must be ARD URNs and institutionId is added. externalId and type are deprecated and ignored — stop sending them (§12.4). A publisher already sending URNs — as the quickstart documents — changes configuration only. If you do not hold the livestream URNs, read them off services[].topic.id in your current HTTPS responses, which already carries the resolved value, before switching.
• Plugin adapters on Pub/Sub today. Keep running during the migration window, then point the adapter at eventhub-connect and decommission the GCP-side adapter.
• Subscribers on Pub/Sub today. Move to MQTT around step 17, which stops the event fan-out to Pub/Sub — after that a Pub/Sub subscription still exists but receives nothing. The /subscriptions and /topics routes stay served through a deprecation cycle, but they stop being useful at the same moment. This is the migration item most likely to be forgotten, because these consumers are not publishers and will not notice the rest of the transition.

12.7 Failover modes

• Single-node outage. Clients reconnect to one of the two survivors via their configured endpoint list. Quorum holds at 2 of 3, so JetStream stays writable and QoS 1 sessions, retained messages and control-event delivery all continue.
• Network partition. The majority side keeps quorum and serves normally. The minority side loses JetStream and stops accepting MQTT connections; its clients reconnect across. No split-brain divergence — the minority side refuses to serve rather than serving stale state.
• Two-node loss. No quorum, no MQTT. The accepted limit of a three-node cluster.
• Plugin API timeout. Adapters have a hard per-request timeout, configurable per adapter. Current values in eventhub-ingest are 10s (DTS) and 7s (Radioplayer); keep those unless measured p99s say otherwise. Because each target has its own consumer with its own max_ack_pending, one unreachable target cannot consume capacity belonging to another (§10.4).
• Credential revoked. The user is removed from the config and the server reloaded. The connection drops and the next reconnect fails auth.

12.8 Legacy is not first-class

eventhub-bridge runs as a single instance with no failover, and legacy publishers are not covered by the availability guarantees above. This is intentional. The HTTPS path exists to give houses time to migrate, not to be a permanent second-class transport.

When the bridge is down or partitioned, legacy events do not reach CN subscribers, and this exposure grows over the migration. While eventhub-ingest still fans out to Pub/Sub, legacy events reach existing Pub/Sub subscribers regardless of bridge state; once that fan-out ends at step 17, the bridge becomes the only path out of GCP and a bridge outage means legacy events are lost.

13. New event schemas

Both extend de.ard.eventhub.v1.* and reuse the services[] shape in its URN-only v3 form (§12.4). Derived from the proposals in discussion #771.

13.1 de.ard.eventhub.v1.radio.control

Control bits — TA, Regio, and whatever comes next.

{
	"event": "de.ard.eventhub.v1.radio.control",
	"start": "2026-05-27T16:03:00+01:00",
	"validUntil": "2026-05-27T16:18:00+01:00",
	"name": "TA",
	"state": true,
	"services": [
		{
			"publisherId": "urn:ard:publisher:75dbb3dace15f610",
			"institutionId": "urn:ard:institution:a3004ff924ece1a2",
			"id": "urn:ard:permanent-livestream:49267f7d67be180d"
		}
	]
}

name is unconstrained because the ZI schema’s fixed element list is one of the limitations v3 exists to remove. A typo costs one broken control bit for one publisher; an enum costs a release cycle every time someone needs a new element.

13.2 de.ard.eventhub.v1.radio.data

Radiotext, dynamic label, and the RT+/DL+ plus services, bundled into one cyclic event.

{
	"event": "de.ard.eventhub.v1.radio.data",
	"start": "2020-01-19T06:00:00+01:00",
	"cycle": 8,
	"data": [
		{ "type": "radiotext", "id": 0, "value": "Sie hören die ARD Popnacht" },
		{ "type": "dynlabel", "id": 0, "value": "Sie hören die ARD Popnacht" },
		{ "type": "rtdlplus", "id": 32, "description": "PROGRAM.Stationname long", "value": "SWR 3" },
		{ "type": "rtdlplus", "id": 4, "description": "Interpret", "value": "Coldplay" },
		{ "type": "rtdlplus", "id": 1, "description": "Titel", "value": "Clocks" },
		{ "type": "rtdlplus", "id": 36, "description": "PROGRAM.Moderator", "value": "Ben Streubel" }
	],
	"services": [
		{
			"publisherId": "urn:ard:publisher:75dbb3dace15f610",
			"institutionId": "urn:ard:institution:a3004ff924ece1a2",
			"id": "urn:ard:permanent-livestream:49267f7d67be180d"
		}
	]
}

cycle and id are numbers, not strings — the discussion draft quoted them but described them as integers.

id is validated against an allowlist in the zod schema (§13.3), to reject typos and unmapped content types before they reach receivers that route on the numeric value. The allowlist applies only when type is rtdlplus — for radiotext and dynlabel the id is always 0 and is not an RT+ code, so validation is conditional on type rather than a flat check on the field.

Receivers walk data[] and filter on id, taking what they can render and ignoring the rest — as UECP consumers already behave, minus the fields (CT and similar) that only existed because of source configuration.

13.3 RT+ content types

The rtdlplus allowlist. 64 six-bit codes, so anything outside 0–63 is invalid by construction. Normative source: IEC 62106-6 Table A.2; this reproduction agrees with the RDS Forum class list.

• Dummy — 0 DUMMY_CLASS
• Item — 1 ITEM.TITLE, 2 ITEM.ALBUM, 3 ITEM.TRACKNUMBER, 4 ITEM.ARTIST, 5 ITEM.COMPOSITION, 6 ITEM.MOVEMENT, 7 ITEM.CONDUCTOR, 8 ITEM.COMPOSER, 9 ITEM.BAND, 10 ITEM.COMMENT, 11 ITEM.GENRE
• Info — 12 INFO.NEWS, 13 INFO.NEWS.LOCAL, 14 INFO.STOCKMARKET, 15 INFO.SPORT, 16 INFO.LOTTERY, 17 INFO.HOROSCOPE, 18 INFO.DAILY_DIVERSION, 19 INFO.HEALTH, 20 INFO.EVENT, 21 INFO.SZENE, 22 INFO.CINEMA, 23 INFO.TV, 24 INFO.DATE_TIME, 25 INFO.WEATHER, 26 INFO.TRAFFIC, 27 INFO.ALARM, 28 INFO.ADVERTISEMENT, 29 INFO.URL, 30 INFO.OTHER
• Programme — 31 STATIONNAME.SHORT, 32 STATIONNAME.LONG, 33 PROGRAMME.NOW, 34 PROGRAMME.NEXT, 35 PROGRAMME.PART, 36 PROGRAMME.HOST, 37 PROGRAMME.EDITORIAL_STAFF, 38 PROGRAMME.FREQUENCY, 39 PROGRAMME.HOMEPAGE, 40 PROGRAMME.SUBCHANNEL
• Interactivity — 41 PHONE.HOTLINE, 42 PHONE.STUDIO, 43 PHONE.OTHER, 44 SMS.STUDIO, 45 SMS.OTHER, 46 EMAIL.HOTLINE, 47 EMAIL.STUDIO, 48 EMAIL.OTHER, 49 MMS.OTHER, 50 CHAT, 51 CHAT.CENTER, 52 VOTE.QUESTION, 53 VOTE.CENTER
• Reserved — 54, 55. Rejected. Undefined in the standard, so a value here is a bug.
• Private — 56, 57, 58. Accepted. Provider-defined by design, so we cannot judge them; they carry no portable meaning, and a receiver without a template for the sending house will ignore them.
• Descriptor — 59 PLACE, 60 APPOINTMENT, 61 IDENTIFIER, 62 PURCHASE, 63 GET_DATA

INFO.SZENE and the CENTER spellings are the standard’s own, kept verbatim so the constant names match the source table rather than being quietly Anglicized.

Practically: accept 0–53 and 56–63, reject 54 and 55. The rejected pair is the whole mechanism — it is not much of a gate, but the alternative is that a typo’d code reaches receivers that route on the number and renders as the wrong field.

14. Operator UI

Two jobs that want different transports: stats, always on screen and changing slowly, and a live event tail for debugging a publisher.

The UI is a read-only observer and must never be in the data path. If it dies, or if twenty people open it at once, events keep flowing.

14.1 What it shows

Ten panels. The first four gate step 11 (§18.2); the rest can land incrementally, and two only become useful later in the migration.

1. On-air state — every livestream at a glance. Retained messages (§9.1) make this close to free: the UI subscribes once and has current state for every station without a history store or a replay.

• Current track.playing per livestream — station, title, artist, publisher — and track.next where it is sent.
• Active radio.control flags with a validUntil countdown. Which stations currently have a control bit asserted, and when it lapses. This is the panel the ZI Gateway replacement is judged on.
• Latest radio.data value per station with the age of its last cycle.
• Age of the last event per station, sorted worst-first. “Which station went quiet” is the most common operator question, and staleness-first ordering answers it without anyone typing a filter.

2. Connections and credentials.

• Connected clients grouped by username — how a rotation is confirmed complete before the old user is removed (§7.2).
• Per connection: client id, source address, connect time, subscribed subjects, QoS, persistent-session state, last publish time.
• Every configured user, not only the connected ones, with the issue date parsed out of the username and its current connection count. A credential with zero connections is visibly safe to delete; an old one still in use is visibly overdue.
• The institutions each credential is permitted for (§7.4). With multi-institution users the ACL is the whole authorization model, so it has to be readable without opening the NATS config.
• Auth failures by username and source address.

3. Validation and rejections.

• Recent failures with the full zod message, not the sanitized public string.
• What was published to feedback/{institutionId} (§12.5).
• Ownership-check failures, showing which of the three ids disagreed — subject, payload institutionId, or the feed (§8.2). For a multi-institution credential this is the difference between “misrouted to the wrong inbox” and “not permitted at all”.
• Sidecar rejections of bridged events, split by cause — schema drift versus a failed normalization (§5.4).
• Native publishers still sending the deprecated services[].externalId or services[].type (§12.4). Bridged events carry them legitimately, so the panel has to separate the two or it reads as noise.
• Filterable to a single institution, so a house diagnoses its own integration instead of opening a ticket. Publishers currently get Bad request with no detail, which is why that support load exists.

4. Cluster and broker health.

• Three nodes — BAD, STG, MNZ: reachable, version, uptime, and which one answered this request.
• JetStream RAFT group state and leader per group, replica lag, stream and consumer counts, storage headroom.
• Per-consumer lag, ack-pending, redelivery and term counts, for the sidecar and each plugin target separately (§10.5). A climbing ack-pending on one target is the earliest degradation signal there is.
• Slow-consumer events, connect/disconnect churn.

5. ARD feed health.

• Age, active revision, and per-node revision if per-node fetching is used (§8.4).
• Last refresh outcome, and which integrity rule fired when a candidate was rejected (§8.5).
• Item count against its bounds, canary stations present or missing.
• Diff between the active revision and a rejected candidate. A poison feed revokes a house’s ability to publish silently, so seeing which publishers would have disappeared is the difference between catching it and hearing about it from the house.

6. Plugin adapters.

• Per target: success rate, latency, timeout counts, staleness drops.
• Outbound calls per target per minute — the step 16 cutover check. Flat across the switch means one path is live; doubled means both are.
• Fan-out ratio, events on radio.> versus messages on plugin.>, per target.

7. Latency against the budget.

• p99 inbox/-to-radio/ split per stage, against the 220 ms (§9.2), so a breach points at a hop rather than at the system.
• Per-zone breakdown. Depends on the metrics store from step 13.

8. Identifier lookup.

• Station name to livestream URN, publisher URN and institution URN — and the reverse, paste a URN and get the station.
• The exact topic strings to publish to and subscribe to for that station, ready to copy.
• Pure feed data with no state of its own, and it answers the most repeated onboarding question without a human in the loop.

9. Migration progress. Useful from step 8 onward, and retired with the GCP side.

• Per house: publishing natively over MQTT, or still on HTTPS through the bridge.
• Bridge liveness, relay lag and GCP connection state. It runs as a single instance (§10.5), so its health has no redundancy to hide behind.
• Traffic per eventhub-ingest route. §5.3 gates removal on measured zero traffic; this is where that measurement lives. Without it, deprecation is an argument instead of an observation.

10. Live tail. Transport and limits in §14.3 and §14.4.

• Subject filter, defaulting to something narrower than radio.>.
• A sampled badge whenever the rate limit is dropping frames.
• Pretty-printed payload, copy-as-JSON.
• The close reason, displayed rather than swallowed.

Deep-linkable state, everywhere. Filter, station and time range belong in the URL so an operator pastes a link into a chat instead of describing which boxes to tick. Cheap to build in from the start, tedious to retrofit.

Explicitly not features:

• No historical event archive and no log search. The UI reads live state and aggregated metrics. An event history turns it into a data store with retention, backup and PII questions; logs and history belong to the observability stack (§15).
• No replay, no re-publish, no send-test-event button, and none of the credential or ACL editing ruled out in §14.5. Every one of them is a mutation.

14.2 Stats over HTTP

Stats are a snapshot, so they are a periodic GET on a 5–10s interval, served from the NATS monitoring endpoints (/varz, /connz, /jsz) and the metrics store. No socket, cacheable, survives a reload, trivially rate-limited. Keeping stats off the socket is what makes the auto-disconnect policy viable: the always-visible part of the UI never needs a persistent connection, so the socket is open only while someone is deliberately watching a tail.

Everything in §14.1 except the live tail is served this way, including the on-air board — retained state is read once per poll rather than streamed, so a station’s current track survives a page reload without a socket.

One addition to the counters already listed: per-publisher event rates split by event class, which is the cheapest way to see a house ramping up during migration.

14.3 Live tail over WebSocket

The UI backend holds one NATS-native subscription and fans out to browser clients over a WebSocket it serves itself.

Rejected alternatives:

• Browser straight to the NATS WebSocket gateway with nats.ws — less code and one hop shorter, but it needs a broker credential in a JS bundle and leaves nowhere to enforce the disconnect policy below. NATS has connection limits, but not “close this when the human stops looking at it”.
• SSE — the closer call. Strictly one-directional, EventSource reconnects for free, passes through proxies without an upgrade handshake. It loses because EventSource reconnects aggressively and automatically, so a server-side close is immediately undone by the browser. SSE stays the fallback if WebSocket turns out painful through the CN proxy chain.

14.4 The socket must not become a monitoring feed

An always-open tail is not an allowed use. Left unchecked it becomes an unversioned, unmonitored API that someone builds a health check against, and it holds a broker fan-out subscription open indefinitely. So:

• Idle disconnect on human presence, not on traffic. The page heartbeats from real interaction and the Page Visibility API, so a hidden tab stops heartbeating. A timer-driven keepalive would defeat the entire point.
• Absolute cap regardless of activity, around 30 minutes. Reopening is one deliberate click.
• A close frame with a reason the UI displays — “live tail stopped after 30 minutes, resume”.
• A cluster-wide cap on concurrent tails, so the UI cannot amplify fan-out.
• Per-connection rate limit with a visible “sampled” badge. A tab tailing radio.> at cyclic radio.data volume will drop frames or grow without bound; the default filter should be narrower than everything, and sampling must be shown rather than hidden.

14.5 No login, and what that forces

Authorization is reachability from the ARD CN intranet. Consequences that are cheaper to accept now than to retrofit:

• The UI stays strictly read-only. No credential management, no user CRUD, no publishing test events, no ACL editing. The first genuinely mutating feature is the point where login stops being optional.
• Network position is not the only control. Bind the listener to the CN-facing interface and put an explicit source-CIDR allow-list in front of it.
• No credential material on screen, ever. Usernames and connection counts yes; passwords, bcrypt hashes and full tokens never.
• Everyone on the CN can read every house’s events, including control bits and validation errors containing payload fragments (Q5).
• No login means no record of who looked. Log source IP and filter per tail session so an unusual tail can be traced after the fact.

15. Observability

Cluster-internal only. No Datadog, no external SaaS, nothing in the observability path that requires internet access — an observability stack that stops working during an internet outage stops working exactly when it is needed. Nothing in the event flow may depend on a metrics or log write succeeding.

The stack is TBD (Q6). Current candidates: Vector shipping to VictoriaLogs for logs; VictoriaMetrics for metrics, scraping the NATS monitoring endpoints via the Prometheus NATS exporter plus application metrics from the sidecar and adapters; Perses for dashboards; and possibly Beszel for host and uptime, hub on BAD (§4.2).

What to instrument, regardless of which tools win:

• Per-publisher event counters, split by event class.
• Validation error rate, with top failing publishers and error classes.
• Sidecar rejections of bridged events, split by cause — schema drift versus an identifier that failed ingest’s normalization (§5.4). Should be zero; anything else means the two passes disagree.
• Payloads still carrying a deprecated services[] field — externalId or type — split by publisher, by field, and by native versus bridged. Bridged events carry them by design until step 19; a native publisher still sending them has not finished migrating. This counter is what gates the removal, since deprecation ends on measured zero usage rather than on a date alone.
• MQTT session counts, connect/disconnect churn, auth failures.
• Active connections per username, so an unused credential is visibly safe to remove and a stale one visibly overdue.
• JetStream health: RAFT group state, replica lag, stream and consumer counts, storage headroom.
• NATS slow-consumer events.
• Per-consumer JetStream lag, ack-pending, redelivery count and term count — for the sidecar and for each plugin target separately. A climbing ack-pending on one target is the earliest signal that it is degrading.
• Plugin adapter success rate, latency and timeout counts per target.
• Outbound calls per target per minute — proves the step 16 adapter cutover worked. Flat across the cutover means one path is live; doubled means both are.
• Events dropped for staleness, split by target.
• ARD feed age, active revision and refresh outcome — success, network failure, or integrity-rule rejection with the rule that fired. Age is the alerting signal; the rejection counter tells you why the age is climbing.
• Operator UI tail sessions — concurrent count, duration distribution, and disconnect reason split by idle / absolute-cap / client-initiated. A distribution piling up at the 30-minute cap means someone is treating the tail as a monitoring feed.
• Fan-out ratio: events published to radio.> versus messages published to plugin.>, per target. A sudden jump means the eligibility logic changed behaviour.
• End-to-end latency from inbox/ publish to radio/ availability, as a p99 SLO against the 220 ms (§9.2) — split per stage so a breach points at a hop. Alert on the p99, not the mean.

One carry-over to fix: processEvent currently logs the full event payload and the raw request body on every event. Affordable at today’s volume but not once radio.data is cycling — a 20x step change in messages and a larger one in log bytes. Estimate the volume before the pilot.

16. Hosting and operations

• Three nodes across three SWR zones in the ARD CN, each on a near-identical NixOS configuration, with one source of truth for the OS image (Nix flake in this repo). Rebuilds are reproducible; rollbacks are atomic. NixOS carries a learning curve and the team must be able to operate it at 03 under pressure, which is a training commitment on top of the build.
• K3S as the workload orchestrator. NATS, the observability stack and one workload per eventhub-connect role — sidecar, adapters, operator UI, feed CronJob — all live as manifests in infra/. The eventhub-connect manifests differ only in the container command and replica count, since they share one image. Scaling is a PR. eventhub-bridge runs as a single instance on the primary node behind a feature flag.
• NATS is a standard container deployment, exactly like the observability tools — official upstream image, a config file, a persistent volume. One instance per node, with a stable identity and its own volume because JetStream needs both. It is configuration, not code.
• We build three images and pull the rest. CI builds eventhub-connect, eventhub-bridge and eventhub-ingest, and pushes them to the local registry. NATS and the observability stack are mirrored upstream images, pinned by digest.
• Local container registry (ZOT) on each VM. K3S pulls from there: no Docker Hub traffic, no rate limits, no runtime supply-chain exposure.
• Offline runtime. Every package, config and image lives in the Nix store or the local registry, so the cluster keeps validating, routing, serving and observing events with zero internet access. Plugin adapters are the one internet-dependent workload — their targets are on the public internet, so their failure must never affect broker availability. Per-target max_ack_pending enforces that (§10.4).
• Persistent storage for JetStream. Each node needs a durable volume for the store_dir, sized for MQTT session state, retained messages and QoS 1 in-flight tracking. Not large, not optional, and it needs a backup and restore story.
• Infra as code. Every change to a node or workload goes through a PR; CI deploys; no manual prod edits.
• Rolling updates. Drain one node (cordon, evict pods), update NixOS + K3S, restart, wait for JetStream to report the RAFT group healthy, then move on. Quorum holds throughout at 2 of 3, so no maintenance window is needed. Never drain two nodes at once.
• Staged config reloads. An authorization change is hot-reloaded one node at a time, so a mistaken ACL costs one node while clients fail over instead of locking every publisher out at once. The config is parsed and validated in CI before merge.
• 24/7 monitoring handed off to the SOC, with runbooks and alerting wired into the existing escalation path.
• Client failover. Clients are configured with all three endpoints and reconnect to a survivor on drop.

17. Reliability and broadcast-criticality

radio.control carries TA and Regio bits, which makes this senderelevant (broadcast-relevant) infrastructure and raises the availability bar above what now-playing metadata alone would justify.

The bound on the blast radius: no Eventhub outage takes anyone off air. This system carries metadata and control signalling, not audio. A total failure degrades what listeners see on a display and stops TA from being signalled; it does not interrupt a broadcast. The ZI gateway this replaces has the same property.

What the design does is described in place: three nodes with quorum at 2 of 3 (§4.1), no privileged node in the data path (§4.2), durable control events and retained messages (§9.1), rolling updates that never drain two nodes and staged config reloads (§16), an offline-capable data path, and cluster-internal observability (§15).

17.1 Fail-safe by construction

The validUntil TTL plays two roles. In normal operation it is a backstop, with the explicit state: false event as the primary path (§9.1). In a total outage it is the only thing still working.

If the cluster stops delivering, control events stop arriving, TTLs expire, and receivers drop the state they hold. A traffic-announcement bit fails to off, not stuck on. The worst outcome of a total outage is that a traffic announcement cannot be signalled; what it prevents is a TA bit stuck on indefinitely, with receivers switching to a station that has nothing to announce.

The schema marks validUntil optional for states with no natural expiry, but for anything safety-relevant it should be treated as required, and the pilot should verify that receivers actually honour it rather than holding last-known state forever.

17.2 Correlated risk: losing all three nodes

Three zones protect against zone failure, not against anything common to all three. Roughly in order of expected impact:

1. ARD CN outage. All three nodes sit inside the ARD CN at SWR. If the CN itself fails, or the SWR portion of it, all three go unreachable simultaneously and quorum is irrelevant. There is no in-design mitigation, and adding zones does not help — they are all in the same network domain. Accepted rather than solved (§17.4).
2. TLS certificate expiry. All three nodes share one certificate lifecycle, so an expiry drops every client connection everywhere at the same instant — the most common cause of a simultaneous total outage in systems of this shape. Automated renewal plus expiry alerting at 30 / 14 / 7 days, and the alert goes to the SOC, not to an inbox.
3. Bad deploy or correlated software failure. Identical images on all three nodes means a NATS bug or a broken sidecar release can take all three in sequence. Canary one node, roll with an automatic abort on health regression rather than a human watching a dashboard, and rely on atomic NixOS rollback. A valid-but-wrong config is the residual case, which is why reloads are staged.
4. JetStream storage exhaustion or a wedged meta-group. A full store_dir fails stream writes, which fails MQTT sessions. A stuck RAFT meta-group needs manual intervention. Both need runbooks and a disk-headroom alert with real headroom — not 90%.
5. DNS. Three hostnames are no help if none of them resolve. Clients should tolerate a resolution failure by retrying against a cached address, and the CN DNS path needs to be understood rather than assumed.
6. Clock skew. Ordering is last-write-wins on the event’s own timestamp and control state expires on a TTL, so both correctness properties depend on time. NTP on all three nodes is mandatory; a publisher with a badly skewed clock can have its events discarded as stale or its TA expire early. Validate at the pilot and alert on node-to-node skew.

17.3 Deliberately not redundant

Three things are single-instance on purpose, none in the critical path: eventhub-bridge (legacy only), the observability stack (single-writer stores; losing it makes the system blind, not broken), and the plugin adapter targets themselves (on the public internet, unreachable for reasons outside the CN).

17.4 The diversity question

The last failure-independent path disappears in two stages. Step 17 removes the Pub/Sub fan-out, at which point every event — native or legacy — must traverse the ARD CN to reach any subscriber. Step 19 then removes GCP entirely. After step 17 the CN is a single point of failure for the whole system; step 19 only removes the HTTPS front door.

The argument for keeping a minimal GCP-hosted emergency path indefinitely is that it is the only mitigation for risk 1. Against it: sovereignty is a stated goal, two paths means two code paths forever, and the encoders are inside the CN. The central encoding docks directly into the CN, so a CN outage takes the encoders out regardless of where events are published from. A GCP fallback would serve internet-side consumers such as broadcaster apps while failing the broadcast-relevant ones.

Recommendation: decommission as planned, and treat the CN as the availability floor. If the SOC’s Senderelevanz classification demands better, that has to be solved at the network layer, not by keeping a GCP deployment alive.

18. Integration steps

Each step is a discrete shippable. The GCP-side work comes first, because it depends on nothing we have to wait for — while the CN work is gated by two external requests with lead times we do not control (§18.1).

18.1 External requests to file first

Two prerequisites are filed with other teams rather than built by us, so their lead time, not their effort, is the schedule risk. Raise both as soon as step 5 has assigned addresses and hostnames — they gate everything from step 6 onward.

• Firewall rules for ARD CN clients. Inbound :8883 to all three nodes from every publisher and subscriber range in the CN. Each house connects outbound to the broker, so nothing is needed in the other direction. Two further rules are easy to forget and both block earlier than the client work does: inter-node :6222 across BAD ↔ STG ↔ MNZ, without which the cluster cannot form a RAFT group at all in step 6, and outbound HTTPS egress for the plugin adapters, whose targets are on the public internet (§10.4). The bridge additionally needs egress to the GCP broker at step 8.
• ARD CN certificates. The MQTT gateway will not serve TLS without them, so they gate step 6, not step 14. Request one certificate covering all three node hostnames, matching the single shared lifecycle assumed in §17.2, and get renewal automated in the same conversation.

The firewall scope depends on Q1. A CN-internal-only request is the smaller ask and the right one if every subscriber sits on the CN; if any do not, it has to cover internet-facing exposure as well. Answering Q1 first avoids filing twice.

18.2 Steps

Steps 1–4 need no CN infrastructure at all, so they run while the firewall and certificate requests in §18.1 are still in a queue. Everything from step 5 is ordered by dependency.

1. New event schemas. radio.control and radio.data in zod, shipped into the existing HTTPS API with OpenAPI and docs. Gets real payloads from real encoders in front of the schema before any infrastructure exists, and unblocks the ZI replacement conversation. Derive the URN-only variant (§12.4) from the same definitions here.
2. MQTT in GCP. NanoMQ in a Docker container, with eventhub-ingest publishing every event to inbox/{institutionId} alongside its existing Pub/Sub fan-out. Nothing consumes it yet; it becomes the CN’s source of real test data at step 8. No persistence and no retained messages — it is a hop, not a store, so events published while the bridge is down are lost by design and Pub/Sub remains the path of record until step 17.
3. Identifier normalization in ingest. Add institutionId and guarantee id and publisherId are URNs, so bridged events are valid on arrival. Keep emitting externalId and type — the same normalized body goes to Pub/Sub, whose subscribers still read them, so removing them here would break the legacy path. Mark both deprecated in the schema and regenerate OpenAPI (§12.4).
4. Plugin publishing feature flag in ingest. A per-target switch held in Datastore, read at runtime with a short cache — step 16 flips a target mid-migration and must not need a deploy to do it, or to undo it. Distinct from the existing per-event plugins[].isDeactivated, which is a publisher’s choice about one event; this one decides whether GCP dispatches that target at all.
5. VM infrastructure. Three VMs — BAD, STG, MNZ — on NixOS with K3S, a local ZOT registry and storage sized for JetStream. Measure inter-zone RTT while the nodes are fresh; it is the input the latency budget depends on. File both requests in §18.1 as soon as hostnames are assigned — they block step 6.
6. NATS cluster. JetStream at R=3, MQTT gateway on :8883 with TLS, cluster sync on :6222, plus the PLUGINS stream (plugin.>, short max_age) for step 16. Blocked on both requests in §18.1: no certificates means no TLS listener, no inter-zone :6222 means three standalone nodes.
7. Auth and ACLs. Config users, bcrypt passwords, allowed_connection_types, permissions bound to the institution URN, hot-reload tested. Include a multi-institution publisher (§7.4) and confirm a misrouted livestream is still rejected.
8. eventhub-bridge. Single instance on BAD behind a feature flag, relaying GCP MQTT into inbox/{institutionId}. This is the step that puts real traffic in the CN, which everything after it is built and verified against.
9. ARD core feed loader. Hourly CronJob writing to a JetStream KV bucket; watcher with validate-then-swap, on-disk persistence and a bootstrap copy in the image. Before the sidecar, which depends on it.
10. Validation sidecar. Durable pull consumer on inbox.>, zod validation against the URN-only schema, ownership check, RETAIN publish to radio/…, eligibility fan-out to plugin.{target}.…, rejections to feedback/{institutionId}. In src/connect/, importing src/schemas/; eligibility ports from event-helpers.ts. Multi-pod from day one so the once-only path is exercised early.
11. Operator UI. Stats over HTTP, live tail over an auto-disconnecting WebSocket, no login. Panels 1–4 of §14.1 are the gate. Lands here because steps 8–10 give it live traffic to display, making it the primary tool for verifying every later step.
12. Docs and changelog. Publisher migration guide, topic and ACL reference, changelog. Before the pilot, so the pilot house reads documentation instead of asking, and its questions correct the docs for everyone behind it.
13. Cluster-internal observability (§15). Before the pilot.
14. Client connection layer. Three DNS names, one per node, all three configured in every client. Wire up certificate renewal and expiry alerting, and confirm the client firewall rules work from a real house network.
15. Pilot. One broadcaster, one livestream, MQTT end to end alongside its existing HTTPS path. Diff the outputs before opening it to anyone else.
16. Plugin adapters. Move Radioplayer and Xperi DTS to eventhub-connect, one target at a time, each a paired switch — CN adapter on, GCP dispatch off via the step 4 flag — verified by the target’s outbound call volume staying flat. Native and bridged publishers move together, since both already flow through the sidecar (§11.2).
17. Reduce eventhub-ingest to auth, validate and publish — behaviour only, no route removals. Drop plugin dispatch and the Pub/Sub fan-out. Keep the event-schema validation pass so legacy publishers keep their synchronous 400 (§5.4). Every route stays served; those three then enter their deprecation cycle with the Pub/Sub shutdown, while /events and /auth/* run until the last HTTPS publisher migrates (§5.3).
18. Handoff to SOC. 24/7 monitoring, runbooks and alerting in the SOC escalation path. Confirm the Senderelevanz classification and the availability floor it implies (§17).
19. Decommission GCP. Remove the GCP broker and eventhub-bridge, then eventhub-ingest and the last project resources. Prerequisite: every route has completed its deprecation cycle with measured zero traffic — this is where the HTTPS API and the Firebase tenant go away. Also drop the deprecated services[].externalId and services[].type from the schema (§12.4); the last consumer of that shape leaves with Pub/Sub.

19. Open decisions

These need someone to choose, not something to be measured. Q1 and Q2 are coupled and should be answered in that order — the second is cheap if the answer to the first is “none”, and expensive if it is not.

19.1 Q1 — are there potential subscribers who cannot connect within the ARD CN?

The design places the broker inside the CN and has publishers and subscribers connect to it there. Plugin targets are already handled — adapters reach Radioplayer and DTS with outbound HTTPS, so those partners never connect inward. The open part is subscribers, and it is currently assumed rather than answered.

Candidates that may not sit on the CN: ARD Sounds / Nucleus / POC, if any run in a public cloud; broadcaster app backends, frequently cloud-hosted even when the broadcaster is on the CN; and third-party metadata consumers, if they are ever to be served by subscription rather than by us pushing to them.

• If “none”: the MQTT gateway never needs internet exposure. It binds to CN-facing interfaces only, the attack surface collapses to the CN population, and Q2 becomes easy.
• If “some”: you need internet-facing MQTT with TLS and auth, or an outbound relay mirroring selected subjects to where those consumers are — the bridge pattern in reverse. That component does not exist in this document, and it would have to be built before step 19 removes GCP, because today those consumers are served from GCP.

This partly invalidates an assumption elsewhere. Q3 argues that eventhub-ingest could publish straight into the CN because the gateway is “internet-reachable by definition”. That only holds if the gateway is in fact internet-facing. If the CN stays closed, ingest cannot publish directly and the bridge is load-bearing rather than optional.

Next step: an inventory of every current Pub/Sub subscriber and where it runs. That list exists in Datastore today.

19.2 Q2 — can subscriber paths be made freely available, with no user auth?

Read-only, unauthenticated subscribe. Today’s design says the opposite — no_auth_user is not set and an unauthenticated connect fails — so this is a deliberate reversal, not a gap.

Now-playing metadata is already public by construction: it goes out over RDS and DAB to anyone with a receiver. Charging a provisioning and rotation process for access to data we broadcast in the clear is friction without a security benefit, and subscriber credentials are the ones that churn most. Same reasoning as the operator UI (§14.4).

What it costs:

• Per-subscriber visibility disappears. The design leans on connections grouped by username for rotation confirmation and per-consumer metrics. Anonymous subscribers are one undifferentiated blob: you cannot attribute load, cannot tell who is affected by a breaking change, and cannot warn anyone before making one.
• Revocation becomes all-or-nothing. With no identity, the only lever against one abusive consumer is an IP block or switching anonymous access off for everybody.
• Abandoned sessions accumulate broker state. MQTT sessions are keyed by client ID, an anonymous client picks its own, and a persistent session with QoS 1 makes the broker retain a queue in JetStream for a client that may never return. Anonymous plus persistent plus QoS 1 is unbounded state growth with nobody to attribute it to.

Recommendation: yes, but split by event class and forbid persistent sessions.

• Open radio.*.track.* and radio.*.data to an unauthenticated, subscribe-only user — the high-demand, low-sensitivity, already-public traffic.
• Keep radio.*.control authenticated. The audience is small and known — encoders — and it is the class where knowing every consumer matters.
• Force clean sessions and cap subscriptions and connections for the anonymous user, so no JetStream session state accrues. Retained messages still work on subscribe, so a late joiner is unaffected.
• Keep named credentials available for consumers that want them, since anything wanting an SLA or advance warning of changes should be identifiable by choice.

Implementable directly in the NATS config: no_auth_user pointing at a subscribe-only user whose permissions allow the two open subtrees and deny everything else. The ACL config then becomes the only thing standing between anonymous users and control, so that user’s permission block needs an explicit test rather than a review.

The answer depends on Q1. CN-only makes this a low-risk change to a trusted population. Internet-facing makes it a public data feed with public-feed consequences — abuse handling, capacity planning for an unbounded audience, and a stronger case for keeping radio.*.data closed as well.

19.3 Q3 — does eventhub-bridge need to exist?

Now that eventhub-ingest is an MQTT publisher, ingest could publish straight into inbox/{institutionId} in the CN, deleting both the GCP broker and the bridge. What the two-hop design buys is store-and-forward: ingest publishes locally and always succeeds, and the bridge catches up after a CN outage instead of failing the broadcaster’s HTTP request. Credential scope is identical either way, since both svc-bridge and a direct-publishing ingest need inbox.>.

Conditional on Q1 — direct publishing requires the CN gateway to accept connections from GCP. Decide before step 8 builds the bridge.

19.4 Q4 — class token in the inbox subject

inbox/{institutionId}/{class} instead of inbox/{institutionId}, to stop control events being head-of-line blocked behind cyclic radio.data and to enable per-class publish ACLs (§9.4). Recommended: adopt it. Cheap now and a breaking change later — decide before the pilot.

19.5 Q5 — cross-house visibility

With no login on the UI, anyone on the CN can tail every house’s events, control bits included. Defensible for RDS-adjacent metadata, but it is the other houses’ call as much as ours. The same question in a different place as Q2 — decide the principle once and apply it to both the UI tail and the subscribe path.

19.6 Q6 — observability stack

Vector / VictoriaLogs / VictoriaMetrics / Perses / Beszel are all provisional. Blocks step 13.

19.7 Q7 — TEMP_PUBLISHER_MAPPING

Eleven hardcoded publisher-id remaps carried in ard-core.ts. Fix upstream or make it owned, dated config — it must not migrate into v3 unexamined.

20. Open questions

Genuine unknowns, to be measured or inventoried rather than decided.

• Inter-zone RTT. BAD ↔ STG ↔ MNZ, measured rather than assumed. It sets the RAFT commit cost and therefore how much of the latency budget the three-zone spread consumes. Blocks nothing, but the pilot cannot validate the budget without it.
• Subscriber inventory. Every current Pub/Sub subscriber, what it is, who owns it, and which network it runs on. Feeds Q1, and it is the list that has to be worked through before step 17 stops the Pub/Sub fan-out — and the same list that decides when /subscriptions and /topics can be retired.
• Route traffic per endpoint. Which eventhub-ingest routes are still called, by whom, and how often. No route is removed without this, and it is not instrumented today.
• Latency of the path we are replacing. The 250 ms budget is the specified limit; what the ZI gateway actually delivers today for a TA bit is unmeasured.

21. Alternatives considered

21.1 Broker: NATS over EMQX / HiveMQ / Mosquitto

The decisive property is that one binary serves both the MQTT edge and the internal fabric. The design needs an MQTT front door for external clients and durable work queues for the validation sidecar and per-target plugin adapters. Every dedicated MQTT broker is only the front door, so it would have to be paired with Kafka, Redis or NATS anyway — two distributed systems, two failure models, two sets of runbooks. Secondary reasons: subject-based ACLs, cluster replication and the queue substrate are all one hot-reloadable config, and the team already runs NATS elsewhere.

The costs: MQTT 3.1.1 only, no MQTT 5 — no shared subscriptions, no per-message expiry, no reason codes on failure, which is what pushes plugin fan-out onto NATS-native subjects. JetStream becomes mandatory rather than optional. And the MQTT gateway is a smaller feature surface than a purpose-built broker’s implementation.

• EMQX — the strongest counter-candidate: MQTT 5, shared subscriptions, a good dashboard, clustering without RAFT quorum arithmetic. Rejected because it solves only the edge, and because it adds Erlang/OTP operational knowledge plus open-source-versus-enterprise feature gating to a three-person operation.
• HiveMQ — excellent MQTT and clustering, but the features that matter are commercial, and it brings JVM operations.
• Mosquitto — smallest and simplest, fine as a single node. Rejected on redundancy: bridging is not clustering, and there is no work-queue story at all.
• Keep Pub/Sub, reach it over a VPN — retains the GCP dependency sovereignty is meant to remove, still requires every broadcaster to run a push endpoint or hold pull credentials, leaves broadcast-relevant control bits on a SaaS, and does not fix the firewall problem as well as an outbound MQTT connection.

21.2 Transport: MQTT at the edge, NATS-native internally

NATS is the better protocol — no gateway translation, no MQTT 5 gap, and proper nonce-signed JWT authentication rather than bearer tokens. The reason external clients speak MQTT is the client population: playout systems, encoders and vendor appliances speak MQTT, and in the embedded corner of that market it is the only realistic option. “MQTT is an open standard, pick any client library” is also an easier conversation across nine broadcasters than “install our broker’s client”. NATS-native access stays available for anyone who wants it, and every internal component uses it.

21.3 Validation: zod, not valibot

Valibot was raised on #824 and is substantially smaller with better tree-shaking. Standard Schema means the Hono validation middleware accepts either, so this is reversible rather than a lock-in.

Zod stays for two reasons. Richer history and solid operational experience with it — the edge cases and failure modes are known, which is worth more on broadcast-relevant infrastructure than a dependency-size win. And zod 4 is already load-bearing: z.toJSONSchema generates openapi.json, which is what the published docs and the client contract are built from. Bundle size, valibot’s main advantage, is irrelevant for a server-side workload in a container. Revisit only on a concrete zod limitation, not on size grounds.

21.4 Auth: static credentials, not the JWT/operator model

Covered in §7.1.

21.5 Hosting: NixOS + K3S

Alternatives were plain Debian with docker compose, or full Kubernetes. NixOS gives reproducible images and atomic rollback, which is what makes offline operation and “no manual prod edits” enforceable. K3S is the smallest thing that still provides declarative workloads, rolling updates with drain semantics, and manifests reviewed in the repo. Full Kubernetes is overhead on three nodes; compose loses the drain and rolling-update semantics that make maintenance window-free.

Talos Linux with Pulumi, provided by SWR, is the strongest alternative and the one most likely to be revisited. Talos would enforce the same goals harder — no shell, no SSH, no package manager and A/B image upgrades turn “no manual prod edits” from a policy into a property — and Pulumi in TypeScript fits this repo better than a second language almost nobody here writes. The real draw is organizational rather than technical: if SWR operates the platform, OS patching and control-plane upgrades leave this team, and the SOC handoff at step 18 inherits infrastructure the organization already supports, which is an easier case to make for a broadcast-relevant system than a flake maintained in this repo.

Decision: NixOS + K3S, because the team has more hands-on experience with it, and operator familiarity is the specific risk §16 names — the stack has to be debuggable at 03, and the one we already know beats the one that is theoretically stricter. The learning-curve caveat in §16 is NixOS measured against a conventional distribution, not against Talos; against Talos, NixOS is the known quantity.

Three questions would need answers before switching, in order of severity: whether SWR’s Talos offering can place nodes in three genuine zones, since single-site placement would break the premise of this RFC rather than its implementation; whether JetStream can get node-local disk instead of network-attached volumes, given the RAFT commits inside the latency budget and the correlated-storage risk that shared backing would add to §17.2; and where a host-level metrics agent runs on a container-only OS (§15, still open under Q6). If “provided” turns out to mean a handed-over template rather than an operated platform, the organizational argument disappears and this stays settled.

21.6 Node count: three

Two was the original proposal and is rejected outright — JetStream quorum is (R/2)+1, so a two-node cluster tolerates zero failures for anything JetStream-backed, making it strictly less available than a single node while costing twice as much. Five survives two simultaneous losses, but needs two more genuine failure domains and doubles the operational surface for a workload running four orders of magnitude below capacity. Three is the smallest count that tolerates one failure; the constraint is quorum, not throughput.

21.7 Decided in place

Three more comparisons live next to the designs they belong to: the UI live-tail transport (§14.2), feed distribution through JetStream KV (§8.3), and the institution URN in the subject instead of a short slug (§6.2).

22. Non-goals

• No time scheduling. Eventhub does not run a traffic plan; the broadcaster owns on-air switching.
• No EPG duplication. Static programme data stays in the POC.
• No UECP emulation. We do not rebuild the ZI byte stream; we replace the use cases it solves with native events.
• No per-house integration work. We deliver a generic interface; broadcasters, encoders and apps do their own integration.