helm

streaming-hub Helm chart

Deploys streaming-hub — the Lerian service that consumes lib-streaming CloudEvents from Kafka/Redpanda and fans them out to per-tenant SaaS/BYOC subscribers.

This is a multi-component chart built on one image and one binary, selected at runtime into a role via STREAMING_HUB_ROLE. There are no database, broker, or OTEL subcharts — Kafka/Redpanda and PostgreSQL are shared external infra, and OTEL is env-wired (see External dependencies).

Chart Contract

Installing the Chart

helm install streaming-hub oci://registry-1.docker.io/lerianstudio/streaming-hub-helm --version <version> -n streaming-hub --create-namespace

With a custom values file:

helm install streaming-hub oci://registry-1.docker.io/lerianstudio/streaming-hub-helm --version <version> -n streaming-hub -f my-values.yaml

The chart is mirrored to oci://ghcr.io/lerianstudio/streaming-hub-helm as well.

Uninstalling the Chart

helm uninstall streaming-hub -n streaming-hub

The mode switch (read this first)

streamingHub.mode decides the topology. It is an either/or:

mode Renders STREAMING_HUB_ROLE When
all (default) ONE Deployment + Service (+ HPA/PDB) all Single co-resident deployment. The dev-st target. Byte-equivalent to the historical single binary.
split TWO Deployments + Services (ingest + delivery), each with its own HPA/PDB ingest / delivery Ingest and delivery scaled independently as N + M replicas.

> DOUBLE-CONSUME HAZARD — the load-bearing rule

NEVER run a mode: all release AND a mode: split release against the same Kafka/Redpanda cluster. All three roles (all, ingest) join the one ingest consumer group. An all pod and an ingest pod consuming together means every event is double-consumed and double-delivered. The mode switch enforces either/or within a single release — do not defeat it by deploying two releases that overlap.

Worked example — mode: all (dev-st)

streamingHub:
  mode: all
  all:
    replicaCount: 1
    poolMaxOpenConns: 25   # role=all holds both planes' connections
    poolMaxIdleConns: 12

Renders: streaming-hub-all Deployment + streaming-hub-all Service, plus the shared streaming-hub ConfigMap + Secret + ServiceAccount. One consumer-group member set — no double-consume possible.

Worked example — mode: split (independent scaling)

streamingHub:
  mode: split
  ingest:
    replicaCount: 3
    poolMaxOpenConns: 8    # consume-poll + inbox tx + partition-cron
    poolMaxIdleConns: 4
    autoscaling: { enabled: true, minReplicas: 3, maxReplicas: 4 }
  delivery:
    replicaCount: 2
    poolMaxOpenConns: 16   # 8 dispatch workers + reclaim + dlq + delivery /readyz
    poolMaxIdleConns: 10
    autoscaling: { enabled: true, minReplicas: 2, maxReplicas: 4 }

Renders streaming-hub-ingest and streaming-hub-delivery Deployments + Services + HPAs, sharing the one ConfigMap + Secret. Each Deployment gets its own STREAMING_HUB_ROLE and pool sizing as explicit per-Deployment env (which wins over the shared envFrom).

Connection-budget invariant

streaming-hub owns ONE shared PostgreSQL database. Every open connection on every pod of every role draws from that single max_connections budget.

Σ over all running pods of (replicas × poolMaxOpenConns) + headroom ≤ PostgreSQL max_connections. Under HPA, use maxReplicas (not replicaCount) in the sum.

Worked example (max_connections = 100, mode: split):

Role replicas poolMaxOpenConns connections
ingest 3 8 24
delivery 4 16 64
total     88 (leaves 12 for admin/migrations) ✅

Adding an all pod (25) → 88 + 25 = 113 > 100 ❌. The mode either/or already forbids that combination; the budget math is why it also matters operationally. See streaming-hub/.env.reference (STREAMING_HUB_POSTGRES_MAX_OPEN_CONNS) for the per-role rationale (all 25/12, ingest 8/4, delivery 16/10).

Health, drain, and termination

Secrets sourcing

Two mutually-exclusive paths:

useExistingSecret Behavior
false (default) The chart renders templates/secrets.yaml from streamingHub.secrets (base64-encoded). STREAMING_HUB_POSTGRES_DSN is ALWAYS emitted (even empty) so a bare helm template succeeds; an empty DSN fails the app fast at boot. Empty optional keys are skipped.
true No Secret is rendered. Deployments reference existingSecretName. This is the gitops/Vault path (an external secret is projected into the named Secret) and is the production default. Key contract: the named Secret MUST carry STREAMING_HUB_POSTGRES_DSN — each Deployment pins that key via an explicit secretKeyRef, so a Secret missing it fails container creation (CreateContainerConfigError, “couldn’t find key”) instead of CrashLooping the app.

Sensitive keys (all in streamingHub.secrets, all default ""): STREAMING_HUB_POSTGRES_DSN, STREAMING_HUB_KAFKA_CA_CERT, STREAMING_HUB_DEV_KEK (dev only) — these three are ALWAYS emitted (even empty) so a bare render succeeds and their non-credential-shaped names are still counted by the coverage check; plus STREAMING_HUB_KAFKA_SCRAM_PASSWORD and POSTGRES_PASSWORD (migration-toolchain parity; no in-cluster consumer), both emit-when-set. Not secrets (live in the ConfigMap): STREAMING_HUB_KAFKA_SCRAM_USERNAME (username, not a credential) and STREAMING_HUB_KEK_REF (a POINTER — the NAME of the env var holding the KEK material).

External dependencies

This chart provisions none of the following — they live outside it:

Top-level values

Key Default Description
streamingHub.mode all Topology switch: all | split. Schema-enforced enum.
streamingHub.image.repository ghcr.io/lerianstudio/streaming-hub Image repo.
streamingHub.image.tag "" Empty falls back to Chart.appVersion.
streamingHub.image.pullPolicy IfNotPresent  
streamingHub.imagePullSecrets [{name: ghcr-credential}] Private registry pull secrets.
streamingHub.service.type ClusterIP Lerian convention (Ingress fronts external).
streamingHub.service.port 8080 Control-plane HTTP port.
streamingHub.ingress.enabled false Control-plane API ingress (opt-in per env).
streamingHub.serviceAccount.create true  
streamingHub.terminationGracePeriodSeconds 80 Defaults to the ~80s drain ceiling; keep at or above it.
streamingHub.securityContext nonroot 65532, drop ALL, RO rootfs, RuntimeDefault Distroless:nonroot.
streamingHub.useExistingSecret false true = Vault/gitops path.
streamingHub.existingSecretName "" Required when useExistingSecret.
streamingHub.configmap {} Non-sensitive env override hatch (defaults live in templates/configmap.yaml). configmap.<KEY> overrides an enumerated (allowlisted) key — the schema propertyNames.enum rejects a key outside the allowlist; use extraEnvVars to inject one that is not enumerated. See README.params.md.
streamingHub.datastores {} Dedicated PostgreSQL mask (host/port/user/name/ssl); global.datastores.postgres is the shared tier.
streamingHub.extraEnvVars {} Per-Deployment env hatch (key → value map). Never credentials.
streamingHub.secrets (all "") Shared sensitive env. DSN always emitted; other empty values skipped.
streamingHub.<role>.replicaCount 1 Per role: all / ingest / delivery.
streamingHub.<role>.poolMaxOpenConns all 25 / ingest 8 / delivery 16 Postgres pool (connection-budget invariant).
streamingHub.<role>.poolMaxIdleConns all 12 / ingest 4 / delivery 10  
streamingHub.<role>.autoscaling.enabled false HPA per role (autoscaling/v2, CPU+memory).
streamingHub.<role>.pdb.enabled false PodDisruptionBudget per role (policy/v1).
global.externalPostgresDefinitions.enabled false Bootstrap Job for the hub’s one DB/role (PreSync, sync-wave -10).
streamingHub.migrations.enabled false Out-of-band schema migration Job (PreSync, sync-wave -1).
streamingHub.migrations.image.repository ghcr.io/lerianstudio/streaming-hub-migrations Migrations image (FROM migrate/migrate + the hub’s migrations/).
streamingHub.migrations.image.tag "" Empty falls back to streamingHub.image.tag, then Chart.appVersion.
streamingHub.migrations.image.digest "" Pin by digest; wins over tag.
streamingHub.migrations.useExistingSecret false true = Job reads STREAMING_HUB_POSTGRES_DSN from existingSecretName.
streamingHub.migrations.existingSecretName "" Required when migrations.useExistingSecret. Must already exist before the PreSync phase — not the main-Sync app Secret.
streamingHub.migrations.backoffLimit 3 Job retry cap.
streamingHub.migrations.activeDeadlineSeconds 600 Job wall-clock cap.
streamingHub.migrations.ttlSecondsAfterFinished 600 Finished-Job GC TTL.

For the full env contract (defaults, required-in-SaaS markers, the F4 tenant caution, KEK source vars), see streaming-hub/.env.reference.

See docs/TOPOLOGY.md for the role model in depth.

This site is open source. Improve this page.