# Kubernetes

Deploy SteelEngine with Helm

For application-based deployment access from Azure DevOps, follow [Entra workload identities](/self-hosting/entra-workload-identities). Configure installation trust under `app.env`; the worker inherits it unless explicitly overridden. Register each release application and its workspace permissions in the organization dashboard.

## Prerequisites

For content promotion between installations, see [Organization Releases](/self-hosting/releases). On a target installation, enable the [organization release sync CronJob](/self-hosting/releases/kubernetes-setup) to follow the selected Git branch and apply the configured deployment policy. Kubernetes traffic cutover is separate.

* Kubernetes 1.19+
* Helm 3.16.3 (baseline tested in CI)
* PV provisioner support
* A Temporal server reachable from the cluster (see [Temporal](/self-hosting/temporal))

## Installation

The chart enables the Temporal Worker by default and renders it for production. `helm install` fails until you set the Worker's Temporal settings and one Worker credential source. `postgresql.auth.password` is required when the bundled PostgreSQL is enabled. Keep the non-secret values in a file. Pass secrets with `--set`.

```bash
# Clone repo
git clone https://github.com/steelengineai/steelengine.git && cd steelengine

# Generate secrets
BETTER_AUTH_SECRET=$(openssl rand -hex 32)
ENCRYPTION_KEY=$(openssl rand -hex 32)
INTERNAL_API_SECRET=$(openssl rand -hex 32)
POSTGRES_PASSWORD=$(openssl rand -base64 16 | tr -d '/+=')

# Registry credentials from your subscription
kubectl create namespace steelengine
kubectl create secret docker-registry steelengine-ghcr \
  --namespace steelengine \
  --docker-server=ghcr.io \
  --docker-username=<github-user> \
  --docker-password=<token>

# Broker secret shared only by the app and the Worker
kubectl create secret generic steelengine-temporal-worker-broker \
  --namespace steelengine \
  --from-literal=TEMPORAL_WORKER_BROKER_SECRET="$(openssl rand -hex 32)"
```

### Verify images before mirroring

Each image published to `ghcr.io/steelengineai` is signed with [Sigstore](https://www.sigstore.dev) and carries an SPDX software bill of materials (SBOM). Signatures are keyless: they are issued to the SteelEngine publishing workflow's GitHub identity, so there is no public key to distribute. Verify by digest with [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) after `docker login ghcr.io`:

```bash
set -euo pipefail
IMAGE=ghcr.io/steelengineai/steelengine@sha256:<digest>
IDENTITY='^https://github.com/steelengineai/steelengine/\.github/workflows/ghcr-publish\.yml@'
ISSUER=https://token.actions.githubusercontent.com

# Signature
cosign verify "$IMAGE" \
  --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER"

# SBOM (SPDX JSON)
cosign verify-attestation "$IMAGE" --type spdxjson \
  --certificate-identity-regexp "$IDENTITY" --certificate-oidc-issuer "$ISSUER" \
  | jq -r '.payload | @base64d | fromjson | .predicate' > sbom.spdx.json
```

Signatures and SBOMs are stored in the registry next to the image. They are attached to the image as OCI referrers, which `docker pull` and `docker push` leave behind. To keep them when you mirror into your own registry, copy the image together with its referrers, for example with [`oras cp --recursive`](https://oras.land/docs/commands/oras_cp/). Images published before this was introduced are unsigned.

Create `steelengine-values.yaml`:

```yaml
global:
  imagePullSecrets:
    - name: steelengine-ghcr

app:
  env:
    NEXT_PUBLIC_APP_URL: "https://steelengine.yourdomain.com"
    BETTER_AUTH_URL: "https://steelengine.yourdomain.com"
    # HTTPS URL the Worker uses to reach the app; required with the production broker
    INTERNAL_API_BASE_URL: "https://steelengine.yourdomain.com"
    NEXT_PUBLIC_SOCKET_URL: "https://steelengine-ws.yourdomain.com"

realtime:
  env:
    NEXT_PUBLIC_APP_URL: "https://steelengine.yourdomain.com"
    BETTER_AUTH_URL: "https://steelengine.yourdomain.com"
    NEXT_PUBLIC_SOCKET_URL: "https://steelengine-ws.yourdomain.com"
    ALLOWED_ORIGINS: "https://steelengine.yourdomain.com"

worker:
  env:
    TEMPORAL_ADDRESS: "temporal.yourdomain.com:7233"
    TEMPORAL_NAMESPACE: "steelengine-production"
    TEMPORAL_SERVER_NAME: "temporal.yourdomain.com"
    TEMPORAL_AUTH_ISSUER: "https://steelengine.yourdomain.com"
    TEMPORAL_AUTH_JWKS_URL: "https://steelengine.yourdomain.com/api/auth/jwks"
    TEMPORAL_BUILD_ID: "<image digest or git sha>"
  auth:
    credentialBroker:
      enabled: true
      secret:
        name: steelengine-temporal-worker-broker
        key: TEMPORAL_WORKER_BROKER_SECRET
```

```bash
# Install
helm install steelengine ./helm/steelengine \
  --namespace steelengine \
  --values steelengine-values.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
  --set realtime.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set postgresql.auth.password="$POSTGRES_PASSWORD"
```

`realtime.env.BETTER_AUTH_SECRET` must equal the app's value. The realtime server uses it to verify socket tokens. If you leave it empty, realtime reads `BETTER_AUTH_SECRET` from the app Secret instead. The PostgreSQL password may only contain letters, digits, `.`, `_`, and `-`. The chart copies the Worker's `TEMPORAL_*` values into the app pod, so the app and the Worker always agree on the Temporal settings.

## Cloud-Specific Values

The example files set storage classes, ingress annotations, and node selectors for each cloud. They ship a placeholder `postgresql.auth.password` and no `worker:` block. Create the namespace and Secrets from Installation first. Then layer your own values file after the cloud file and keep the `--set` flags.

**AWS EKS**

```bash
helm install steelengine ./helm/steelengine \
  --namespace steelengine \
  --values ./helm/steelengine/examples/values-aws.yaml \
  --values steelengine-values.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
  --set realtime.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set postgresql.auth.password="$POSTGRES_PASSWORD"
```

**Azure AKS**

```bash
helm install steelengine ./helm/steelengine \
  --namespace steelengine \
  --values ./helm/steelengine/examples/values-azure.yaml \
  --values steelengine-values.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
  --set realtime.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set postgresql.auth.password="$POSTGRES_PASSWORD"
```

**GCP GKE**

```bash
helm install steelengine ./helm/steelengine \
  --namespace steelengine \
  --values ./helm/steelengine/examples/values-gcp.yaml \
  --values steelengine-values.yaml \
  --set app.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set app.env.ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --set app.env.INTERNAL_API_SECRET="$INTERNAL_API_SECRET" \
  --set realtime.env.BETTER_AUTH_SECRET="$BETTER_AUTH_SECRET" \
  --set postgresql.auth.password="$POSTGRES_PASSWORD"
```

`helm/steelengine/examples/values-production.yaml` shows the complete production shape, including the Worker values, ingress, autoscaling, and telemetry.

After the install, open `/signup` on the app URL and create the first account. Only an empty database accepts a sign-up without an invitation, so that account is the first admin, and everyone else joins by invitation.

## Key Configuration

```yaml
# Custom values.yaml
app:
  replicaCount: 2
  env:
    NEXT_PUBLIC_APP_URL: "https://steelengine.yourdomain.com"
    OPENAI_API_KEY: "sk-..."

postgresql:
  persistence:
    size: 50Gi

ingress:
  enabled: true
  className: nginx
  tls:
    enabled: true
  app:
    host: steelengine.yourdomain.com
```

See `helm/steelengine/values.yaml` for all options.

### App shutdown

When a rollout or scale-down stops an app pod, the pod can first sleep for `app.preStopSleepSeconds`, a native `preStop` sleep that needs Kubernetes 1.30 or later. It is 0 (off) by default; on Kubernetes 1.30 or later, set it to about 10 so the Service and ingress stop sending the pod new requests before it shuts down. The server then gets SIGTERM and stops accepting connections. It closes idle keep-alive connections, and every response after that carries `Connection: close`. It waits up to `app.shutdownDrainSeconds` for in-flight requests and `after()` work, gives its own shutdown handlers up to 5 seconds and exits. Requests still open when the drain window ends are cut off. Unset, the drain window is `app.terminationGracePeriodSeconds` (30 by default) minus the preStop sleep minus 5, so 25 seconds. The chart refuses values where the preStop sleep plus the drain window does not fit inside the grace period, because Kubernetes kills the pod when the grace period ends.

Synchronous workflow executions (`POST /api/workflows/<id>/execute`) keep their request open until the run finishes. If callers wait for long runs, raise all three values. For example, to cover a 300-second execution timeout:

```yaml
app:
  preStopSleepSeconds: 10
  shutdownDrainSeconds: 300
  terminationGracePeriodSeconds: 330
```

A stopping pod can then take up to five and a half minutes to go away. Outside Kubernetes, the server reads the drain window from `APP_SHUTDOWN_DRAIN_MS` (25000 by default).

## Temporal Worker

`worker.enabled` defaults to `true`. For a production Worker, the chart requires:

* `worker.env.TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_SERVER_NAME`, `TEMPORAL_AUTH_ISSUER`, and `TEMPORAL_BUILD_ID`
* exactly one of `worker.env.TEMPORAL_AUTH_JWKS_URL` or `worker.env.TEMPORAL_AUTH_JWKS_FILE`
* exactly one credential source: `worker.auth.credentialBroker.enabled: true` or `worker.auth.apiKeySecret.name`

Each missing value stops rendering with a message that names it. For example: `worker.env.TEMPORAL_ADDRESS is required for a production Temporal Worker`.

**Credential broker** (used above). The Worker refreshes a short-lived Temporal token from the app. It needs `app.enabled: true`, `worker.env.TEMPORAL_IDENTITY` (default `steelengine-temporal-worker`), an `INTERNAL_API_SECRET` of at least 32 bytes, and an `https://` `app.env.INTERNAL_API_BASE_URL`. Provide the broker secret as `secret.name` (an existing Secret) or `secret.value` (the chart creates one), never both. The chart mounts it into the app and Worker pods only.

**Projected Secret.** You issue a worker-scoped Temporal token yourself and store it in a Secret. The Worker reads it through `TEMPORAL_API_KEY_FILE` and picks up rotations automatically.

```bash
kubectl create secret generic steelengine-temporal-worker-auth \
  --namespace steelengine \
  --from-literal=TEMPORAL_API_KEY="$TEMPORAL_WORKER_JWT"
```

```yaml
worker:
  auth:
    credentialBroker:
      enabled: false
    apiKeySecret:
      name: steelengine-temporal-worker-auth
      key: TEMPORAL_API_KEY
```

The Worker pod defaults to 4 Gi memory and 1000m CPU limits, with `worker.env.NODE_OPTIONS` capping the Node heap at 3072 MiB. If you lower the memory limit, lower the heap cap to about 75% of it, or the Worker can crash with a heap out-of-memory error. It gets a 90-second termination grace period so pollers can drain. Its health server listens on `worker.healthPort` (3003). Per-queue concurrency and database pool settings live under `worker.env`. See [Docker → Temporal Worker](/self-hosting/docker#temporal-worker) for the variable names. Set `worker.enabled: false` only for an installation that will never run workflows. `worker.autoscaling.enabled` adds a CPU/memory autoscaler in place of `worker.replicaCount`. The autoscaler measures usage against `worker.resources.requests`, so keep the CPU request near real usage. Prefer CPU alone (`worker.autoscaling.targetMemoryUtilizationPercentage: null`): Node keeps its heap after a burst, so a memory target can scale out and never scale back in. Set `worker.env.TEMPORAL_ACTIVITY_MEMORY_TARGET` (for example `"0.85"`) alongside it. A busy Worker then leaves extra Activities queued for new replicas instead of running out of memory. CPU must stay above target for `worker.autoscaling.scaleUpStabilizationWindowSeconds` (300 by default) before replicas are added, so the CPU burst of a starting Worker during a rollout does not scale it out.

The Workers and the tool search Job also receive the non-secret storage and model settings you set in `app.env`: keys starting with `AZURE_STORAGE_`, `AZURE_OPENAI_`, `AZURE_ANTHROPIC_`, `S3_` or `VLLM_`, plus `AZURE_ACCOUNT_NAME`, `AWS_REGION`, `KB_OPENAI_MODEL_NAME`, `BLACKLISTED_PROVIDERS`, `BLACKLISTED_MODELS` and `NEXT_PUBLIC_AZURE_CONFIGURED`. A value in `worker.env` takes precedence.

The Worker and document worker have a startup probe on `/health/live` (`worker.startupProbe`, `documentWorker.startupProbe`; 180 and 240 seconds). A Worker loads and bundles its Workflows before its health server starts, which takes about a minute, and the probe keeps liveness from restarting it during a rollout. Keys, secrets and connection strings are never copied this way; keep them in the app Secret.

### Autoscaling on running executions (KEDA)

Most Activities wait on HTTP calls and model responses, so a Worker can fill all its Activity slots while its CPU stays low, and a CPU autoscaler adds Workers too late. With [KEDA](https://keda.sh) 2.x installed, the chart can also scale on the number of running workflow executions. KEDA is off by default. Without it, installs keep the CPU/memory HPAs.

KEDA must run in the cluster before you turn this on, or the install fails on the unknown `ScaledObject` kind. On AKS, enable the managed add-on (`az aks update --enable-keda`; Kubernetes 1.35 ships KEDA 2.17). Elsewhere, install the `kedacore/keda` Helm chart.

```yaml
worker:
  autoscaling:
    enabled: true          # required: KEDA reuses minReplicas, maxReplicas and the CPU target
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 70
    keda:
      enabled: true
      targetQueryValue: 15             # running executions per Worker
      activationTargetQueryValue: 0
      pollingInterval: 15              # seconds between queries
      cooldownPeriod: 300              # only used when scaling to zero

app:
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 80
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 600
    keda:
      enabled: true
      targetQueryValue: 40             # running executions per app pod
```

With `keda.enabled`, the chart renders a KEDA `ScaledObject` instead of the HPA, never both. It has two triggers: the CPU target (and the memory target, if set) and a `postgresql` trigger. KEDA adds replicas once the query's count divided by `targetQueryValue` is more than the current replica count. The app counts running executions (`keda.postgresql.query`): paused runs are `pending`, so they don't count, and runs started more than six hours ago are skipped. Workers count block Activities running now (`worker.autoscaling.keda.query`), which includes child workflows that run inside a parent, such as a Parallel Window calling a worker workflow; a running-executions count would see only the parents. Finished blocks waiting on a fold and rows a crashed Worker abandoned are left out, and preparation, planning and window Activities aren't counted, so keep the CPU target as the second trigger. Set `app.autoscaling.keda.query` or `worker.autoscaling.keda.query` to use another query. The Worker keeps its HPA behavior: scale-up waits `worker.autoscaling.scaleUpStabilizationWindowSeconds`, and scale-in removes one Worker every five minutes after a 600-second window. The app uses `app.autoscaling.behavior`.

The trigger connects to the database the Workers use (`postgresql` or `externalDatabase`) through a `TriggerAuthentication` named `<release>-keda-postgresql`. That reads the password from the same Secret: the chart-managed one, or `postgresql.auth.existingSecret` / `externalDatabase.existingSecret` with its `passwordKey`. With the in-cluster PostgreSQL and `networkPolicy.enabled`, the chart lets the KEDA operator's namespace (`keda.operatorNamespace`, `kube-system` for the AKS add-on, `keda` for the upstream chart) reach the database.

#### Scaling Workers on Task Queue backlog

The running-executions query counts work that has already started, so it only grows once Workers are busy. `worker.autoscaling.keda.backlog` adds Workers as soon as tasks wait. Temporal hands a task straight to an open poll, so a task only waits in the backlog when every Worker's slots are full.

```yaml
worker:
  autoscaling:
    scaleUpStabilizationWindowSeconds: 15
    scaleUpPercent: 100     # one 15-second step may double the fleet...
    scaleUpPods: 10         # ...or add 10 Workers, whichever is more
    keda:
      enabled: true
      backlog:
        enabled: true
        targetPerReplica: 10   # waiting tasks per Worker
        targetAgeSeconds: 15   # keep adding Workers while the oldest task waits longer
```

Each Worker serves the backlog of its eight Task Queues, Workflow and Activity tasks together, as JSON at `/metrics/task-queue-backlog` on `worker.healthPort`. Temporal reports a queue's backlog across all its pollers, so any one Worker answers for the fleet. The chart adds a `<release>-worker` Service for that port and one or two KEDA `metrics-api` triggers:

* `task-queue-backlog` (`AverageValue`): KEDA asks for the backlog divided by `targetPerReplica` Workers. A backlog of 200 at 10 asks for 20.
* `task-queue-backlog-age` (`Value`, only when `targetAgeSeconds` is above 0): while the oldest task has waited longer than `targetAgeSeconds`, each step multiplies the replica count by age divided by target. At twice the target the fleet doubles. Unlike the count, it keeps growing a fleet that is already large.

The Worker reads the backlog with Temporal's `DescribeTaskQueue` and caches it for five seconds. The [SteelEngine Temporal server](/self-hosting/temporal#scaling-workers) must let Worker tokens call `DescribeTaskQueue`, which it does only for the Task Queues the token names. Roll out that server image and a Worker image that serves the endpoint before you turn the backlog on. Until then the endpoint answers 503. The HPA keeps scaling up on the CPU and query triggers, but it won't scale in while a trigger fails.

`worker.autoscaling.scaleUpPercent` and `scaleUpPods` set how many Workers one 15-second step may add, the larger of the two. The defaults, 100% and 4, are the Kubernetes defaults. With backlog scaling, keep `scaleUpStabilizationWindowSeconds` short. A waiting backlog already means every Worker is full, and the HPA ignores the CPU of Workers that are still starting.

KEDA's own `temporal` scaler, added in KEDA 2.17, isn't used. It authenticates with a static API key or a client certificate. The SteelEngine frontend accepts only short-lived transport JWTs, so KEDA would need a long-lived token or a rotation job of its own.

The ScaledObject keeps the chart's HPA name (`<release>-worker`, `<release>-app`) and takes over the existing HPA, so you can turn KEDA on with a normal upgrade. To turn it off again, delete the ScaledObject first (`kubectl delete scaledobject <release>-worker`) so the chart can recreate its own HPA.

#### What scales the app

The app scales on CPU and running executions, not on in-flight requests. Each app pod is one Node.js process, so its event loop saturates as its CPU climbs, and CPU already tracks request load. Workflows that start child runs through the execute API also raise the running-executions count. The chart can't count in-flight requests across pods by itself. That needs a metrics server such as Prometheus, scraping the ingress controller. With one in the cluster, add a request-rate trigger through `app.autoscaling.keda.extraTriggers`. The chart appends each entry to the ScaledObject as written, and `worker.autoscaling.keda.extraTriggers` does the same for Workers:

```yaml
app:
  autoscaling:
    keda:
      extraTriggers:
        - type: prometheus
          metadata:
            serverAddress: http://prometheus.monitoring.svc:9090
            query: sum(rate(nginx_ingress_controller_requests{ingress="steelengine-app"}[1m]))
            threshold: "50"   # requests per second per pod
```

When app pods run hot, check whether the autoscaler is already at `app.autoscaling.maxReplicas` before you add a trigger. A pod at 150% CPU already asks for about twice as many pods. To add pods faster, set a scale-up policy in `app.autoscaling.behavior`:

```yaml
app:
  autoscaling:
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 0
        selectPolicy: Max
        policies:
          - {type: Percent, value: 100, periodSeconds: 15}
          - {type: Pods, value: 8, periodSeconds: 15}
```

Each app pod opens up to `DATABASE_POOL_MAX` database connections, so `maxReplicas` × `DATABASE_POOL_MAX` must stay below the database's `max_connections` unless [PgBouncer](#connection-pooling-pgbouncer) sits in front of it.

`app.autoscaling` scales the app on its own. It works with or without KEDA. When it is on, the top-level `autoscaling` block scales only realtime, so realtime can stay at `realtime.replicaCount`. When it is off, the top-level block scales the app and realtime together, as before. While `app.autoscaling.enabled` is on, `helm upgrade` sets the app Deployment's `replicas` to its current live count (or `minReplicas` on a new install), so an upgrade never resets the app to one pod. Tools that render offline with `helm template`, such as Argo CD or Flux, cannot read the live count and render `minReplicas` on every sync; the autoscaler then adds pods back. For those, set `minReplicas` to the count you want to keep through a sync.

### Document Worker

**Generate document** blocks run on their own Task Queue, which only the document worker polls. Turn it on with `documentWorker.enabled: true`; without it those blocks wait until they time out. It uses the Worker image and `worker.env`, minus the Worker's Temporal identity and port, `OTEL_SERVICE_NAME`, `E2B_ENABLED`, `EGRESS_ALLOWLIST`, `EGRESS_ALLOWLIST_HOSTED_DATABASES` and `NODE_OPTIONS` (its heap cap is `documentWorker.env.NODE_OPTIONS`, sized to its own memory limit), and runs a Gotenberg sidecar that listens only inside the pod for PDF and Office rendering.

The document worker requires Kubernetes 1.29 or later, because Gotenberg runs as a native sidecar container. On older clusters the chart refuses to render with `documentWorker.enabled: true`.

In production the document worker uses the credential broker with its own identity (`documentWorker.identity`) and its own broker secret, so it can only poll the document queue:

```yaml
documentWorker:
  enabled: true
  auth:
    brokerSecret:
      name: steelengine-document-worker-broker   # or leave empty and set value
      key: DOCUMENT_GENERATION_TEMPORAL_WORKER_BROKER_SECRET
```

Use a dedicated Secret for the broker secret, so only the app and the document worker receive it. If you point it at the app Secret, the Worker, which loads that Secret whole, receives it too; the chart then blanks the key in the Worker's environment, but the value is still in a Secret mounted into the Worker pod.

The chart creates the Gotenberg credentials unless you set `documentWorker.gotenberg.auth.existingSecret`. The chart keeps the generated password across upgrades by reading the live Secret, which GitOps tools that render offline (Argo CD, Flux) cannot do. With those tools, create the Secret yourself and set `existingSecret`.

## External Database

```yaml
postgresql:
  enabled: false

externalDatabase:
  enabled: true
  host: "your-db-host"
  port: 5432
  username: "postgres"
  password: "your-password"
  database: "steelengine"
  sslMode: "require"
```

### Database Secrets

By default, the chart builds `DATABASE_URL` from `postgresql` or `externalDatabase` and the corresponding password secret. Enabling `app.secrets.existingSecret` for your authentication and encryption keys does not require a `DATABASE_URL` key in that Secret.

To use a complete connection URL from your app Secret for the app and its migration init container, explicitly map the key:

```yaml
app:
  secrets:
    existingSecret:
      enabled: true
      name: my-app-secrets
      keys:
        DATABASE_URL: connection-uri
```

Here, `connection-uri` is the name of a key in `my-app-secrets`. Leave `keys.DATABASE_URL` empty to keep the constructed URL. The Worker and realtime service still use `postgresql` or `externalDatabase`, so keep those settings and their password secret pointed at the same database as the app.

### Connection pooling (PgBouncer)

Every app pod and Worker keeps its own pool of up to `DATABASE_POOL_MAX` connections, so without a pooler, app replicas × 20 plus Worker replicas × 25 has to stay below the database's `max_connections`. `pgbouncer.enabled` puts PgBouncer in transaction pooling mode between them and the database. Pods then connect to PgBouncer, which runs each transaction on one of a fixed set of server connections, so replica counts no longer depend on `max_connections`.

```yaml
pgbouncer:
  enabled: true
  replicaCount: 2        # each PgBouncer process is single-threaded
  appPoolSize: 50        # server connections per replica for the app...
  workerPoolSize: 100    # ...and for the Workers
  reservePoolSize: 10    # extra per pool once a client waits reservePoolTimeout seconds
  maxClientConn: 5000    # client connections per replica
  tls:
    enabled: true        # clients connect with sslmode=require
    issuerRef:
      name: my-selfsigned-issuer   # optional: cert-manager issues certificatesSecret
```

The database sees at most `replicaCount` × (`appPoolSize` + `workerPoolSize` + 2 × `reservePoolSize`) connections from PgBouncer, 340 with the defaults. Keep that, plus the direct connections below, under `max_connections`. A transaction holds a server connection only while it runs, so a pool far smaller than the client count is normal. Raise a pool only when clients wait in PgBouncer (`cl_waiting` in `SHOW POOLS`). `replicaCount` is fixed, not autoscaled, because each replica adds its own server connections.

Evicting a PgBouncer pod drops the database connections of every client using it, so a running workflow can fail mid-query. On clusters with the cluster autoscaler, set `pgbouncer.podAnnotations` to `cluster-autoscaler.kubernetes.io/safe-to-evict: "false"` so node scale-down leaves PgBouncer alone. Node drains for upgrades still evict it, one pod at a time when `podDisruptionBudget.enabled` is set. A stopping PgBouncer pod keeps serving its existing clients for `pgbouncer.terminationGracePeriodSeconds` (default 60), but busy connections never sit idle long enough to close on their own. Set `DATABASE_MAX_LIFETIME_SECONDS` in `app.env` and `worker.env` well below that window, so every connection rotates to a running pod before the stopping one is killed. A connection rotates only after its running query, so size the window for the `preStop` delay (10 seconds), the lifetime, `pgbouncer.queryWaitTimeout` and the longest statement timeout: for example 45 seconds with a 300-second window when the queue wait is 120 seconds and the Worker's statement timeout is 60 seconds.

The app and the Workers each connect to their own pool, the `<database>_app` and `<database>_worker` aliases, through `<release>-pgbouncer`. These stay on direct connections:

* the migrations init container, which sets session timeouts and builds indexes concurrently
* the app's session-level advisory locks, which serialize cron jobs and source-control syncs. The chart gives the app `DATABASE_SESSION_URL`, the direct URL. Behind a transaction pooler, a lock and its unlock could run on different server sessions.
* realtime, the document worker, the tool search Job and the KEDA trigger

The app and Worker clients use unnamed statements (`prepare: false`) and keep `SET LOCAL` and transaction-level advisory locks inside transactions, so they work with transaction pooling. They send `DATABASE_LOCK_TIMEOUT_MS` and `DATABASE_STATEMENT_TIMEOUT_MS` as the `options` startup parameter, which PgBouncer can't pass on. PgBouncer ignores it and sets the same values, from `app.env` and `worker.env`, on each server connection of that pool with `connect_query`.

The hop from PgBouncer to the database follows `externalDatabase.sslMode`, or `postgresql.tls.enabled`. Client TLS is off by default. With `pgbouncer.tls.enabled`, PgBouncer reads `tls.crt` and `tls.key` from `pgbouncer.tls.certificatesSecret`, and setting `issuerRef.name` asks cert-manager for that certificate. A self-signed issuer is enough, because clients encrypt without verifying the certificate. Use an RSA key if you supply your own: the app runs on Bun, whose TLS client fails the handshake with PgBouncer on an ECDSA certificate. With `networkPolicy.enabled`, only the app and the Workers reach PgBouncer, and PgBouncer reaches only DNS and the database.

## Secret scoping

Chart 1.4.0 supports `app.extraEnvVars` for Kubernetes `value` and `valueFrom` entries in the app container. For an installation Admin API key, reference a dedicated Secret using `valueFrom.secretKeyRef` and remove any previous shared reference. The chart resolves overlapping names in the app with `app.extraEnvVars` first, followed by shared `extraEnvVars`, `app.env`, and generated values. Check the rendered Temporal CA paths against your mounts during the upgrade. When the installed release already contains duplicate env names, use Helm 4's `--server-side=true` or validate a Helm 3 replacement upgrade in QA. Verify the CA and URL settings on the live Deployments after the upgrade.

The app, the Worker and the tool search Job load the app Secret, and every Secret in `app.secrets.additionalSecretNames`, as environment variables. Realtime and the document worker don't. They reference single keys from the app Secret, listed in `realtime.secrets.appSecretKeys` and `documentWorker.secrets.appSecretKeys`:

* **Realtime**: `BETTER_AUTH_SECRET`, `INTERNAL_API_SECRET`, `REDIS_URL` and the OpenTelemetry auth keys.
* **Document worker**: `ENCRYPTION_KEY`, `API_ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, model provider keys and endpoints, `REDIS_URL`, object storage credentials and container names, and the OpenTelemetry auth keys.

Each reference is optional, so a missing key stays unset and the service reports what it needs at startup. A value set in the component's env or in `extraEnvVars` wins. Add keys to those lists, or whole Secrets to `realtime.secrets.extraSecretNames` and `documentWorker.secrets.extraSecretNames`, when you need more. Neither service receives OAuth client secrets, `CRON_SECRET` or the Worker broker secret.

## Other Chart Features

* **Cron jobs**: `cronjobs.enabled` defaults to `true`. The jobs call the app for schedule reconciliation, subscription renewal, inactivity alerts, connector sync, model retirement notices, and cleanup of stale executions and idempotency keys. Every job sends `Authorization: Bearer $CRON_SECRET`, taken from `app.env.CRON_SECRET` when set and otherwise from the app Secret, so it works with an existing Secret or External Secrets too. If `CRON_SECRET` is missing, the app rejects every job with 401. The stale-execution job also deletes old Temporal journals and finished checkpoint chains. Execution log cleanup (`cronjobs.jobs.cleanupExecutionLogs`) is off by default because it permanently deletes run history older than `LOG_RETENTION_DAYS`; see [Run History Retention](/self-hosting/environment-variables#run-history-retention).
* **Tool search index**: `toolEmbeddings.enabled` runs a Job after each install or upgrade that rebuilds the index the Chat assistant uses to find integrations and blocks. It embeds with Azure OpenAI (`AZURE_OPENAI_API_KEY` from the app Secret, `AZURE_OPENAI_ENDPOINT` from `app.env`) or `OPENAI_API_KEY` from the app Secret. Without it, tool search falls back to name matching. A failed run doesn't fail the deploy.
* **Autoscaling and availability**: `autoscaling` (HPA for the app and realtime), `app.autoscaling` (the app alone), `worker.autoscaling`, optional KEDA scaling on running executions ([above](#autoscaling-on-running-executions-keda)), `podDisruptionBudget` (app, realtime, Worker and document worker), and `monitoring.serviceMonitor` for Prometheus. App probes use `/api/health`.
* **Network policy and storage**: `networkPolicy` restricts pod-to-pod traffic and allows managed Postgres, Redis and Temporal on `networkPolicy.externalServicePorts`, plus DNS, 443 and (with telemetry) OTLP. Realtime, the Workers and the chart's cron jobs reach the app. Set `networkPolicy.ingressControllerNamespaceSelector` (for example `matchLabels: {kubernetes.io/metadata.name: ingress-nginx}`) so only your ingress controller reaches the app and realtime; empty allows any source. App egress is limited to those ports, so add `networkPolicy.egress` rules for anything else. Worker and document worker egress stays open, since integrations and OpenTelemetry use arbitrary ports; set `networkPolicy.restrictWorkerEgress: true` to limit it to DNS, 443, OTLP (4317/4318), the app and `externalServicePorts`. `sharedStorage` provisions ReadWriteMany volumes for workflows that share files between pods.
* **Branding**: `branding.enabled` mounts a ConfigMap of text and base64 binary files at `/app/public/branding`. The `NEXT_PUBLIC_BRAND_*` variables in `app.env` reference them. See [Environment Variables](/self-hosting/environment-variables#branding).
* **Landing pages**: `app.env.NEXT_PUBLIC_LANDING_ENABLED` overrides whether marketing pages are shown or redirect to `/login`. See [Landing Pages](/self-hosting/environment-variables#landing-pages) for hostname-based defaults.
* **Telemetry**: `telemetry.enabled` deploys an OpenTelemetry Collector with Jaeger, Prometheus, and OTLP backends. The chart routes the Worker's traces and metrics to it. To use an external backend such as HyperDX instead, set `otlp.endpoint` and put the auth header in the app Secret as `OTEL_EXPORTER_OTLP_HEADERS` (or `HYPERDX_API_KEY`).
* **Laya Decision**: `laya.enabled` runs the [Laya](/blocks/laya) decision server in-cluster on CPU (about 1 vCPU and 4 GiB; both checkpoints stay resident) and gives the app and Temporal Worker `LAYA_URL` and `LAYA_API_KEY`, which makes the Laya Decision block appear. The chart generates the key once and keeps it across upgrades; set `laya.apiKey.existingSecret` to supply your own. A NetworkPolicy lets only the app and Worker reach it, and it has no egress. The image is `steelengineai/laya`, tagged by laya version and model commit.
* **Code runner**: `codeRunner.enabled` runs Python Function blocks, and JavaScript that imports modules, in your cluster instead of E2B, and gives the Temporal Worker `CODE_RUNNER_URL` and `CODE_RUNNER_TOKEN`. Each call runs in a fresh process as its own Unix user with a private directory and memory, process and time limits. The image (`steelengineai/code-runner`, tagged by a hash of its inputs) has Python's standard library plus `requests` and Node's built-in modules. Run it under a VM-isolating runtime: set `codeRunner.runtimeClassName` (for example `kata-vm-isolation` with AKS Pod Sandboxing) and, on a dedicated tainted pool, `codeRunner.nodeSelector` and `codeRunner.tolerations`. With a runtime class set, `codeRunner.hardLimits: true` enforces each call's memory and process limits through cgroups. The pod then runs privileged inside its own VM, and the chart refuses that setting without a runtime class. The runner also blocks the per-call users from private, loopback, link-local and cloud metadata addresses (`codeRunner.blockedEgressCidrs`) with iptables inside the pod, because a NetworkPolicy alone does not stop a Kata pod reaching the metadata endpoint, and it refuses to start if that block is not in effect. That needs `NET_ADMIN`, which Pod Security `baseline` forbids, so run it in its own namespace (`codeRunner.namespace`) that allows it, holds the image pull Secret, and contains nothing else. A NetworkPolicy lets only the release's Workers connect. If each runner pod fills most of a dedicated node and the pool cannot grow, set `codeRunner.strategy` to `{type: RollingUpdate, rollingUpdate: {maxSurge: 0, maxUnavailable: 1}}` so upgrades replace one pod at a time instead of waiting for room for an extra pod. On shutdown the runner closes each keep-alive connection after its response and waits up to `codeRunner.drainSeconds` for running calls before it exits. Set `codeRunner.preStopSleepSeconds` so the Service stops sending it new calls first, and keep `codeRunner.terminationGracePeriodSeconds` above the two combined. Roll out with `codeRunner.routeWorkers: false`, check the runner pods are Ready, then turn it on to give the Worker `CODE_RUNNER_URL`. Keep E2B enabled if you use Copilot's table files.
* **Ollama**: `ollama.enabled` runs Ollama in-cluster with optional NVIDIA GPU scheduling (`ollama.gpu.strategy` is `time-slicing` or `mig`), and points the app and Workers at it. For an external server, set `app.env.OLLAMA_URL`. With neither, the pods get no `OLLAMA_URL` and Ollama stays off.
* **Secrets**: `app.secrets.existingSecret` and `externalSecrets` (External Secrets Operator) replace inline secret values. See [Secret scoping](#secret-scoping) for which pods receive which keys.
* **Pod security**: containers run as non-root with `allowPrivilegeEscalation: false` and all capabilities dropped, in pods with the `RuntimeDefault` seccomp profile, and no pod mounts a Kubernetes API token (`serviceAccount.automountServiceAccountToken: false`). Override through each component's `securityContext` and `podSecurityContext`.
* **Local rehearsal**: `bun run selfhost:validate` brings up kind, ingress-nginx with TLS, this chart, and a TLS-fronted Temporal server on one machine.

## Commands

```bash
# Port forward for local access
kubectl port-forward deployment/steelengine-app 3000:3000 -n steelengine

# View logs
kubectl logs -l app.kubernetes.io/component=app -n steelengine --tail=100
kubectl logs deployment/steelengine-worker -n steelengine --tail=100

# Upgrade
helm upgrade steelengine ./helm/steelengine --namespace steelengine --values steelengine-values.yaml

# Uninstall
helm uninstall steelengine --namespace steelengine
```

Resource names are `<release>-app`, `<release>-worker`, `<release>-realtime`, and `<release>-postgresql`. Helm drops the chart name when the release name already contains it. A release called `steelengine` yields `steelengine-app`, not `steelengine-steelengine-app`.

## Common Questions

**What are the default resource limits for the SteelEngine app pod?**

The Helm chart defaults to 8 Gi memory limit / 4 Gi request and 2000m CPU limit / 1000m request per app pod. The Temporal Worker gets 4 Gi / 1000m and realtime 1 Gi / 500m. The pods run as non-root (UID 1001) with fsGroup 1001 for security.

**Why does helm install fail before creating anything?**

The chart validates its values before it creates anything. For the default production Worker it requires worker.env.TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_SERVER_NAME, TEMPORAL_AUTH_ISSUER, and TEMPORAL_BUILD_ID. It also requires exactly one of TEMPORAL_AUTH_JWKS_URL or TEMPORAL_AUTH_JWKS_FILE, and one credential source (credentialBroker.enabled or apiKeySecret.name). app.env.BETTER_AUTH_SECRET, app.env.ENCRYPTION_KEY, and postgresql.auth.password are required too. The failure message names the missing value.

**Can I use an external database instead of the bundled PostgreSQL?**

Yes. Set postgresql.enabled to false and configure the externalDatabase section with your host, port, username, password, database name, and sslMode. The external database must have the pgvector extension installed.

**How do I manage secrets securely in Kubernetes?**

The Helm chart supports pre-existing Kubernetes secrets via app.secrets.existingSecret. Set enabled to true and provide the secret name. This integrates with External Secrets Operator, HashiCorp Vault, Azure Key Vault, and similar tools. The app and Worker load the secret as environment variables, so its keys must use the variable names (for example E2B_API_KEY). Realtime and the document worker read only the keys they need. DATABASE_URL, and the keys realtime and the document worker read, can come from a differently named key through app.secrets.existingSecret.keys. The Worker broker secret and the projected Temporal JWT are separate Secrets referenced under worker.auth.

**Can I scale the app to multiple replicas?**

Yes. Set app.replicaCount in your values.yaml, or enable autoscaling for an HPA (app.autoscaling scales the app without realtime). The Worker uses worker.replicaCount, or worker.autoscaling for its own HPA. With KEDA installed, app.autoscaling.keda and worker.autoscaling.keda also scale on running executions. Ensure your database can handle the additional connections from multiple replicas. Each app pod opens up to DATABASE_POOL_MAX (20) connections and each Worker up to 25, unless pgbouncer.enabled pools them.

**Are there cloud-specific example configurations?**

Yes. The Helm chart includes example values files for AWS EKS (values-aws.yaml), Azure AKS (values-azure.yaml), GCP GKE (values-gcp.yaml), as well as files for production, development, local validation, external databases, external secrets, and whitelabeled deployments. The cloud files do not include Worker Temporal settings or a real PostgreSQL password, so combine them with your own values file.

**What ingress options are supported?**

The chart supports ingress with configurable className (e.g., nginx), TLS termination, and separate host configuration for the app and the realtime service. Enable it with ingress.enabled: true and set your hosts under ingress.app.host and ingress.realtime.host.
