multi-componentrabbitmq.enabled=true fails at render, unless manager.secrets sets RABBITMQ_DEFAULT_USER: plugin and a non-default RABBITMQ_DEFAULT_PASS). The application still needs APP_ENC_KEY, MONGO_USER/MONGO_PASSWORD and RABBITMQ_DEFAULT_USER/RABBITMQ_DEFAULT_PASS at runtime (plus LICENSE_KEY for licensed deployments), see Secrets. Production credentials must not be committed in values files.github.com/LerianStudio/helm; license is Elastic License 2.0.Fetcher is a data extraction service for Lerian Studio that enables automated data collection from external sources. This plugin consists of two main components:
+----------------+ +----------------+ +----------------+
| | | | | |
| Manager API +----->+ RabbitMQ +----->+ Worker |
| (4006) | | | | |
+-------+--------+ +----------------+ +-------+--------+
| |
v v
+-------+--------+ +-------+--------+
| | | |
| MongoDB | | SeaweedFS |
| | | |
+----------------+ +----------------+
^
|
+-------+--------+
| |
| Valkey/Redis |
| |
+----------------+
ScaledObject (the chart installs the KEDA operator with keda.enabled but ships no ScaledObject)helm install fetcher oci://registry-1.docker.io/lerianstudio/fetcher-helm --version <version> -n midaz-plugins --create-namespace
To install the chart with a custom values file:
helm install fetcher oci://registry-1.docker.io/lerianstudio/fetcher-helm --version <version> -n midaz-plugins -f my-values.yaml
helm uninstall fetcher -n midaz-plugins
The following table lists the configurable parameters and their default values.
global.cloud)Point this chart at a managed-cloud environment (AWS/GCP/Azure) instead of the bundled in-cluster MongoDB/Valkey/RabbitMQ/SeaweedFS with one knob, via the lerian-common dependency:
global:
cloud: "aws" # aws | gcp | azure — leave unset for the bundled dev topology
datastores:
mongo: { host: "my-documentdb.example.com", port: "27017" }
redis: { host: "my-elasticache.example.com", port: "6379" }
broker: { host: "my-amazonmq.example.com" }
objectStorage:
fetcher: { endpoint: "https://s3.us-east-1.amazonaws.com", region: "us-east-1", bucket: "my-bucket" }
observability:
enabled: true
auth:
enabled: true
host: "http://plugin-access-manager-auth:4000"
streaming:
enabled: true
brokers: "redpanda.<namespace>:9092" # mandatory: the worker will not boot without it
multiTenant:
enabled: true
url: "http://tenant-manager:8080"
global.cloud sets the connection TOPOLOGY (TLS, AMQP scheme/ports, S3
path-style) for the masks above; only the ENDPOINTS (host/port) still come
from global.datastores/global.objectStorage — a cloud preset can’t know
your DocumentDB/ElastiCache/AmazonMQ host. A native common.configmap.<KEY>
(or manager.configmap.<KEY> / worker.configmap.<KEY> for their own masked
fields) always overrides any mask.
fetcher does not adopt lerian-common.serviceDiscovery — this chart has
no SD_* contract at all today (no Consul/lib-service-discovery integration),
so there is nothing to migrate onto that mask.
| Parameter | Description | Default |
|---|---|---|
nameOverride |
Override chart name | "" |
fullnameOverride |
Override full name | "" |
namespaceOverride |
Namespace of the MongoDB bootstrap Job only (global.externalMongoDefinitions.enabled); every other object, including the RabbitMQ bootstrap Job, uses the release namespace. See Object names and namespaces |
"fetcher" |
global.scheduling.spread |
Pod spreading preset (enabled / hostname / zone / maxSkew / minDomains / nodeTaintsPolicy), see Pod Spreading | {enabled: true, hostname: ScheduleAnyway, zone: "", maxSkew: 1, minDomains: 0, nodeTaintsPolicy: ""} |
manager.spread, worker.spread |
Field-level override of global.scheduling.spread for that Deployment |
{} |
manager.topologySpreadConstraints, worker.topologySpreadConstraints |
Raw list; non-empty replaces the preset entirely | [] |
| Parameter | Description | Default |
|---|---|---|
manager.name |
Manager component name | fetcher-manager |
manager.replicaCount |
Number of manager replicas | 2 |
manager.image.repository |
Manager image repository | lerianstudio/fetcher-manager |
manager.image.tag |
Manager image tag (not below fetcher.minImage) |
4.1.0-beta.1 |
manager.image.pullPolicy |
Manager image pull policy | IfNotPresent |
manager.service.type |
Kubernetes Service type | ClusterIP |
manager.service.port |
Service HTTP port | 4006 |
manager.resources.requests.cpu |
CPU request | 100m |
manager.resources.requests.memory |
Memory request | 256Mi |
manager.resources.limits.cpu |
CPU limit | 200m |
manager.resources.limits.memory |
Memory limit | 512Mi |
manager.autoscaling.enabled |
Enable HPA | false |
manager.ingress.enabled |
Enable ingress | false |
manager.health.stack |
Health preset stack, go |
go |
manager.health.drainSeconds, manager.health.shutdownSeconds |
SHUTDOWN_DRAIN_DELAY and SHUTDOWN_TIMEOUT of the pod |
15, 30 |
| Parameter | Description | Default |
|---|---|---|
worker.name |
Worker component name | fetcher-worker |
worker.replicaCount |
Number of worker replicas | 2 |
worker.image.repository |
Worker image repository | lerianstudio/fetcher-worker |
worker.image.tag |
Worker image tag (not below fetcher.minImage) |
4.1.0-beta.1 |
worker.health.stack |
Health preset stack, go-worker (no Service, no main port) |
go-worker |
worker.health.shutdownSeconds |
SHUTDOWN_TIMEOUT of the pod (the worker drain is 0) |
30 |
worker.image.pullPolicy |
Worker image pull policy | IfNotPresent |
worker.resources.requests.cpu |
CPU request | 100m |
worker.resources.requests.memory |
Memory request | 256Mi |
worker.resources.limits.cpu |
CPU limit | 200m |
worker.resources.limits.memory |
Memory limit | 512Mi |
Both deployables render probes, terminationGracePeriodSeconds, the drain env (HEALTH_PORT=8099, SHUTDOWN_DRAIN_DELAY, SHUTDOWN_TIMEOUT), rolling-update strategy (maxUnavailable: 0), PodDisruptionBudget and the HPA floor from the lerian-common.health preset. Readiness fails only on the pod’s own state; shared dependencies only report.
/health and /readyz are served on port 8099. On the manager it is a ClusterIP-only Service port named health; the Ingress never routes /readyz or /metrics. The worker has no Service.*.livenessProbe, *.readinessProbe, *.startupProbe, any *.terminationGracePeriodSeconds and SHUTDOWN_* or HEALTH_PORT under a component configmap. It also refuses manager.pdb.enabled, manager.pdb.minAvailable, manager.pdb.maxUnavailable and a non-empty manager.deploymentStrategy, and the same keys under worker; *.terminationGracePeriodSeconds is refused even when 0, and pdb.enabled even when false: remove them from values before upgrading (the preset always renders a PDB with maxUnavailable: 1 and a rolling update with maxUnavailable: 0). Tune only *.health.drainSeconds and *.health.shutdownSeconds.fetcher.minImage (templates/_helpers.tpl), the first image that serves 8099. An image-only rollback below it fails the render; roll back the chart instead.| Parameter | Description | Default |
|---|---|---|
common.configmap.MONGO_HOST |
MongoDB host | mongodb |
common.configmap.MONGO_PORT |
MongoDB port | 27017 |
common.configmap.RABBITMQ_HOST |
RabbitMQ host | rabbitmq |
common.configmap.RABBITMQ_PORT_AMQP |
RabbitMQ AMQP port | 5672 |
common.configmap.SEAWEEDFS_HOST |
SeaweedFS host | seaweedfs-filer |
common.configmap.REDIS_HOST |
Redis/Valkey host | valkey |
common.configmap.REDIS_PORT |
Redis/Valkey port | 6379 |
common.configmap.REDIS_TLS |
Connect to Redis/Valkey over TLS. The app reads it, but no datastore mask (global.datastores.redis.tls, global.cloud) emits it: set it here. Passes through as-is when set. |
unset (app default false) |
common.configmap.RABBITMQ_TLS |
Connect to RabbitMQ over TLS. The app reads it, but no datastore mask (global.datastores.broker.scheme, global.cloud) emits it: set it here. Passes through as-is when set. |
unset (app default false) |
common.configmap.ALLOW_INSECURE_TLS |
Skip TLS certificate validation for MongoDB/RabbitMQ/Redis connections. Bundled dev-mode dependencies have no TLS, so the app refuses to connect unless this is "true". Set to "false" in production with TLS-terminated backends. |
"true" |
global.streaming.enabled |
Enables lib-streaming (CloudEvents) job notifications for the worker. When unset, the template defaults STREAMING_ENABLED to "true". worker.configmap.STREAMING_ENABLED overrides it. With streaming on, the worker needs a reachable broker. |
unset (template default "true") |
worker.configmap.STREAMING_BROKERS |
REQUIRED when the worker is deployed - Kafka/Redpanda bootstrap address(es), e.g. redpanda.<namespace>:9092. No working default; not bundled by this chart. |
"" |
The manager publishes its permission manifest to the Access Manager at boot (lib-auth
declaration.WireFromEnv) when IDP_DECLARATION_ENABLED="true"; with the flag on it refuses
to start without IDP_HOST and the manager.secrets.IDP_M2M_* pair.
| Key | Source (highest first) | Rendered |
|---|---|---|
IDP_DECLARATION_ENABLED |
manager.configmap > manager.extraEnvVars (legacy) > global.idp.declarationEnabled |
always, default "false" |
IDP_HOST |
same, with global.idp.host |
only when set |
“REQUIRED” below means the application needs the value at runtime. The chart does not enforce it at render time (the default render works with every one of them empty); the only render-time checks are the bundled-RabbitMQ ones (rabbitmq.enabled=true, see Full Installation). LICENSE_KEY is enforced by the application only outside local deployment mode, and the chart sets no DEPLOYMENT_MODE.
| Parameter | Description | Default |
|---|---|---|
manager.secrets.APP_ENC_KEY |
REQUIRED - Base64 encoded 32-byte encryption key | "" |
worker.secrets.APP_ENC_KEY |
REQUIRED - Base64 encoded 32-byte encryption key | "" |
secrets.MONGO_USER |
REQUIRED - MongoDB username | fetcher |
secrets.MONGO_PASSWORD |
REQUIRED - MongoDB password | "" |
manager.secrets.RABBITMQ_DEFAULT_USER, worker.secrets.RABBITMQ_DEFAULT_USER |
REQUIRED - RabbitMQ username (plugin with the bundled broker) |
unset |
manager.secrets.RABBITMQ_DEFAULT_PASS, worker.secrets.RABBITMQ_DEFAULT_PASS |
REQUIRED - RabbitMQ password | unset |
secrets.LICENSE_KEY |
REQUIRED - Lerian license key | "" |
manager.secrets.IDP_M2M_CLIENT_ID / IDP_M2M_CLIENT_SECRET |
Access Manager client pair the manager publishes its permission manifest with. Required by the app only when IDP_DECLARATION_ENABLED="true". Each is rendered only when set. Keep it in manager.secrets: the top-level secrets Secret also reaches the worker |
unset |
The top-level secrets.* block renders a shared <release>-common Secret with its
non-empty keys only. Both Deployments load it before their own Secret
(manager.secrets / worker.secrets, or existingSecretName when
useExistingSecret=true), so the component Secret wins on any key both define.
When using an external RabbitMQ instance, you can enable the bootstrap job to automatically apply the required queue, exchange, and binding definitions.
| Parameter | Description | Default |
|---|---|---|
externalRabbitmqDefinitions.enabled |
Enable RabbitMQ bootstrap job | false |
externalRabbitmqDefinitions.connection.protocol |
API protocol (http/https) | http |
externalRabbitmqDefinitions.connection.host |
RabbitMQ management API host | "" |
externalRabbitmqDefinitions.connection.port |
Management API port | 15672 |
externalRabbitmqDefinitions.connection.portAmqp |
AMQP port for health check | 5672 |
externalRabbitmqDefinitions.rabbitmqAdminLogin.username |
Admin username | "" |
externalRabbitmqDefinitions.rabbitmqAdminLogin.password |
Admin password | "" |
externalRabbitmqDefinitions.rabbitmqAdminLogin.useExistingSecret.name |
Use existing secret for admin credentials | "" |
externalRabbitmqDefinitions.appCredentials.pluginPassword |
Password for plugin user | "" |
externalRabbitmqDefinitions.appCredentials.useExistingSecret.name |
Use existing secret for plugin password | "" |
The chart includes optional dependencies that can be enabled for local development or testing:
| Dependency | Parameter | Description |
|---|---|---|
| SeaweedFS | seaweedfs.enabled |
Distributed file system for storing files |
| MongoDB | mongodb.enabled |
Document database for metadata |
| RabbitMQ | rabbitmq.enabled |
Message broker for async processing |
| Valkey | valkey.enabled |
In-memory data store (Redis alternative) |
| KEDA | keda.enabled |
Installs the KEDA operator only. This chart has no ScaledObject template, so enabling it does not autoscale the manager or worker. Use manager.autoscaling (HPA) or create your own ScaledObject |
global.scheduling.spread)The manager and worker Deployments render topologySpreadConstraints from the
lerian-common spread preset. The default is soft (ScheduleAnyway) across
nodes, so replicas are spread across nodes (e.g. spot nodes) whenever possible,
but a pod is never left Pending because of it.
Zone spreading is off by default (zone: ""). When scoring a soft spread, the
scheduler skips every node that lacks the topology.kubernetes.io/zone label, so on
clusters without zone labels (bare-metal, k3s) a zone constraint silently cancels the
node spread. Turn it on (zone: ScheduleAnyway) where every node carries zone labels
(EKS, GKE, AKS).
global:
scheduling:
spread:
enabled: true # master switch
hostname: ScheduleAnyway # kubernetes.io/hostname: ScheduleAnyway | DoNotSchedule | "" (off)
zone: "" # topology.kubernetes.io/zone: same values (off by default)
maxSkew: 1 # integer >= 1
minDomains: 0 # DoNotSchedule constraints only; 0 = off
nodeTaintsPolicy: "" # Honor | Ignore | "" (Kubernetes default Ignore)
worker:
spread: { hostname: DoNotSchedule } # field-level override of the global preset
topologySpreadConstraints: [] # raw list; non-empty replaces the preset entirely
Each constraint counts only the component’s own pods of the same ReplicaSet
(matchLabelKeys: [pod-template-hash]), so rolling updates never deadlock.
Requires Kubernetes >= 1.27.
On EKS + Karpenter use a hard node spread:
global:
scheduling:
spread:
hostname: DoNotSchedule
zone: ScheduleAnyway
maxSkew: 1
minDomains: 2
nodeTaintsPolicy: Honor
minDomains: 2 is required with DoNotSchedule. Skew is measured only against nodes
that already exist, so on a single eligible node both replicas would otherwise share it.
With fewer than 2 nodes the second replica stays Pending until Karpenter launches
another node. nodeTaintsPolicy: Honor stops tainted nodes (for example a node being
drained) from counting as domains.
# values-minimal.yaml
manager:
image:
tag: "1.0.0"
secrets:
APP_ENC_KEY: "<your-base64-32byte-key>" # Generate with: openssl rand -base64 32
ingress:
enabled: true
hosts:
- host: fetcher.example.com
paths:
- path: /
pathType: Prefix
worker:
image:
tag: "1.0.0"
secrets:
APP_ENC_KEY: "<your-base64-32byte-key>" # Same key as manager
common:
configmap:
MONGO_HOST: "my-mongodb.database.svc"
RABBITMQ_HOST: "my-rabbitmq.messaging.svc"
RABBITMQ_HEALTH_CHECK_URL: "http://my-rabbitmq.messaging.svc:15672"
SEAWEEDFS_HOST: "my-seaweedfs.storage.svc"
REDIS_HOST: "my-redis.cache.svc"
secrets:
MONGO_USER: "myuser"
MONGO_PASSWORD: "mypassword"
RABBITMQ_DEFAULT_USER: "plugin"
RABBITMQ_DEFAULT_PASS: "mypassword"
LICENSE_KEY: "<your-license-key>"
# values-full.yaml
# When using embedded dependencies, service names include the release name prefix
# Assuming release name is "fetcher":
common:
configmap:
MONGO_HOST: "fetcher-mongodb"
RABBITMQ_HOST: "fetcher-rabbitmq"
RABBITMQ_HEALTH_CHECK_URL: "http://fetcher-rabbitmq:15672"
REDIS_HOST: "fetcher-valkey"
SEAWEEDFS_HOST: "seaweedfs-filer"
manager:
secrets:
APP_ENC_KEY: "<your-base64-32byte-key>" # Generate with: openssl rand -base64 32
RABBITMQ_DEFAULT_USER: "plugin"
RABBITMQ_DEFAULT_PASS: "<your-rabbitmq-password>"
worker:
secrets:
APP_ENC_KEY: "<your-base64-32byte-key>" # Same key as manager
RABBITMQ_DEFAULT_USER: "plugin"
RABBITMQ_DEFAULT_PASS: "<your-rabbitmq-password>" # Same password as manager
mongodb:
enabled: true
auth:
rootUser: fetcher
rootPassword: securepassword
rabbitmq:
enabled: true
seaweedfs:
enabled: true
valkey:
enabled: true
keda:
enabled: true
The bundled RabbitMQ’s only login is plugin with the manager’s RABBITMQ_DEFAULT_PASS, taken from manager.secrets or, with manager.useExistingSecret, read from that Secret at render (helm template and Argo CD cannot read it, so that render is refused). An empty password and Lerian@123, the public password earlier versions shipped, are refused. The broker keeps no volume and takes the password only when its pod starts: after changing it, or after upgrading from a version that shipped Lerian@123, run kubectl -n <namespace> rollout restart statefulset/<release>-rabbitmq (queued messages are lost).
Use this approach when connecting to an external RabbitMQ instance that doesn’t have fetcher definitions pre-configured.
Important: The bootstrap job only runs when rabbitmq.enabled=false (external RabbitMQ). When using the embedded RabbitMQ subchart (rabbitmq.enabled=true), definitions are automatically loaded via customConfig.
# values-external-rabbitmq.yaml
manager:
image:
tag: "1.0.0"
worker:
image:
tag: "1.0.0"
common:
configmap:
RABBITMQ_HOST: "my-rabbitmq.messaging.svc"
secrets:
RABBITMQ_DEFAULT_USER: "plugin"
RABBITMQ_DEFAULT_PASS: "securepassword"
# Disable embedded RabbitMQ
rabbitmq:
enabled: false
# Enable bootstrap job to apply RabbitMQ definitions
externalRabbitmqDefinitions:
enabled: true
connection:
protocol: "http"
host: "my-rabbitmq.messaging.svc"
port: "15672" # Management API port
portAmqp: "5672" # AMQP port (for health check)
rabbitmqAdminLogin:
username: "admin" # Existing admin user
password: "adminpassword" # Admin password
appCredentials:
pluginPassword: "securepassword" # Password for 'plugin' user (will be created)
The bootstrap job will, on every install and upgrade:
plugin user with the specified password (a password containing a control character is refused)load_definitions.jsonSome objects do not follow the Helm release name or namespace:
| Object | Name | Namespace |
|---|---|---|
| Shared ConfigMap and Secret | <fullname>-common, where fullname is fullnameOverride, else nameOverride, else fetcher (not the release name) |
release |
| Manager / worker objects | manager.name / worker.name (fetcher-manager, fetcher-worker) |
release |
Secret fetcher-manager-load-definitions (bundled RabbitMQ definitions) |
fixed | release |
ConfigMap fetcher-bootstrap-rabbitmq-definitions |
fixed | release |
Job fetcher-bootstrap-rabbitmq-<revision> |
fixed | release |
Job <fullname>-bootstrap-mongodb-<revision> |
<fullname> as above |
namespaceOverride (default fetcher), else release |
Consequences: two releases of this chart in the same namespace collide on these names unless fullnameOverride, manager.name and worker.name differ (the fixed-name RabbitMQ objects collide regardless), and the MongoDB bootstrap Job lands in fetcher unless namespaceOverride is set to the release namespace. Subchart Services (<release>-mongodb, <release>-rabbitmq, <release>-valkey) do use the release name, see below.
When enabling the bundled dependencies (mongodb, rabbitmq, valkey), the service names include the Helm release name as prefix. Update the common.configmap values accordingly:
# If your release name is "fetcher":
common:
configmap:
MONGO_HOST: "fetcher-mongodb" # Not just "mongodb"
RABBITMQ_HOST: "fetcher-rabbitmq" # Not just "rabbitmq"
REDIS_HOST: "fetcher-valkey" # Not just "valkey"
SEAWEEDFS_HOST: "seaweedfs-filer" # SeaweedFS doesn't use release prefix
The APP_ENC_KEY must be a valid base64-encoded 32-byte key. Generate one using:
openssl rand -base64 32
SeaweedFS is a distributed file system used for storing extracted files. When enabling SeaweedFS:
Prerequisites:
Architecture Support:
The default SeaweedFS chart uses kubernetes.io/arch=amd64 node selector. For ARM64 clusters (e.g., Apple Silicon Macs, AWS Graviton), the fetcher chart overrides this with empty nodeSelector: {}.
Storage Requirements: | Component | Default Size | Purpose | |———–|————-|———| | Master | 5Gi | Cluster metadata | | Volume | 10Gi | Blob storage | | Filer | 25Gi | File system layer |
Custom Storage Class:
seaweedfs:
enabled: true
master:
data:
storageClass: "my-storage-class"
volume:
dataDirs:
- name: data
storageClass: "my-storage-class"
filer:
data:
storageClass: "my-storage-class"
Verify SeaweedFS is Running:
# Check all pods are ready
kubectl get pods -l app.kubernetes.io/name=seaweedfs
# Test filer connectivity
kubectl port-forward svc/seaweedfs-filer 8888:8888
curl http://localhost:8888/
APP_ENC_KEY must be a valid base64-encoded 32-byte keyfetcher-mongodb instead of mongodb)nodeSelector: {} is set in the SeaweedFS configuration# Check pod status
kubectl get pods -n midaz-plugins -l app.kubernetes.io/name=fetcher-manager
# View manager logs
kubectl logs -n midaz-plugins -l app.kubernetes.io/name=fetcher-manager
# View worker logs
kubectl logs -n midaz-plugins -l app.kubernetes.io/name=fetcher-worker
Elastic License 2.0 (chart). The Fetcher application itself is proprietary to Lerian Studio Ltd. Versions released under the Elastic License 2.0 keep that license.