Temporal

SteelEngine runs every workflow, schedule, and background job through Temporal. The Compose file and the Helm chart ship the Temporal Worker, not the Temporal server. You run the server and point the app and the Worker at it. SteelEngine publishes its own server image because a stock Temporal server cannot verify SteelEngine's tokens.

The Server Image

SteelEngine workflow checkpoint format upgrades are managed in the application database. For releases containing checkpoint state V4, follow the manual workflow checkpoint upgrade procedure after upgrading all application readers and workers. This activation does not require a Temporal server configuration change.

ghcr.io/steelengineai/temporal-server is Temporal Server 1.31.2 with SteelEngine's token verification built in. It accepts Temporal's standard environment configuration and --config-file. The image also carries the temporal CLI, two dynamic config files, and the namespace bootstrap script. Pull it with the same registry credentials as the app images.

The server keeps Temporal's own system Worker on an internal frontend (port 7236) that skips token verification. Never expose that port or any other internode port through a Service, load balancer, or ingress.

How Authentication Works

The app signs a short-lived token for every request to Temporal. The token names the issuer, the audience (steelengine-temporal-rpc), a subject, a role, and the namespaces and Task Queues it may touch. The signing key is the same one behind the app's JWKS endpoint, <app URL>/api/auth/jwks. The server verifies each token against that endpoint. It never receives the signing key.

There are three roles. The app uses client. The Worker uses worker. Bootstrap tooling uses operator, which is the only administrative role. A Worker token must carry the Worker's TEMPORAL_IDENTITY as its subject and list all eight product Task Queues:

steelengine-workflows
steelengine-knowledge-connectors
steelengine-webhook-ingestion
steelengine-notifications
steelengine-document-processing
steelengine-copy-backfill
steelengine-scheduled-tasks
steelengine-platform

A second signed token travels inside each workflow and identifies the caller. Workers check it before they run Activities. It is not a Temporal Server credential and needs no server configuration.

Server Configuration

Required:

Optional:

Persistence and frontend TLS use Temporal's own variables. For PostgreSQL, set DB=postgres12, POSTGRES_SEEDS, POSTGRES_USER, POSTGRES_PWD, DB_PORT, and DYNAMIC_CONFIG_FILE_PATH. For frontend TLS, set TEMPORAL_TLS_FRONTEND_CERT, TEMPORAL_TLS_FRONTEND_KEY, and TEMPORAL_TLS_FRONTEND_SERVER_NAME. SteelEngine's TEMPORAL_SERVER_NAME must match that certificate name.

The image ships two dynamic config files under /etc/temporal/config/dynamicconfig/. Both record the verified caller on workflow events and allow long IDs. Use development-sql.yaml for a single node and hosted-sql.yaml for a multi-node cluster, which also encrypts cluster membership traffic.

Bootstrap the Schema and Namespace

The server does not create its own tables. Against an empty PostgreSQL it refuses to start.

1. Create the schema. Run docker/temporal/scripts/setup-all.sh from the temporalio/admin-tools:1.31.2 image. Give it the same DB, POSTGRES_*, and SQL_PASSWORD values as the server. It creates the temporal and temporal_visibility databases. No Elasticsearch is needed. Use a PostgreSQL instance dedicated to Temporal in production.
2. Create the namespace. Run /etc/temporal/scripts/bootstrap-product-namespace.sh from the server image as a one-shot task. Set TEMPORAL_PRODUCT_NAMESPACE and TEMPORAL_INTERNAL_ADDRESS. In clustered mode, also pass the internode TLS material. The script creates the namespace and registers eight Keyword search attributes: OrganizationId, WorkspaceId, WorkflowOwnerId, WorkflowTriggerSource, SteelEngineWorkflowId, SteelEngineExecutionId, SteelEngineDeploymentId, and SteelEngineRequestId. SteelEngine writes these attributes when it starts workflows, so register all eight before the first run.
3. Verify. Run bash scripts/temporal-production-preflight.sh from the repository with an operator token. It needs TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_API_KEY, and the TLS flags. It checks cluster health and creates any missing search attribute.

Namespace retention defaults to 7 days. Change it with TEMPORAL_NAMESPACE_RETENTION.

Connect SteelEngine

The app and the Worker read the same connection settings. Environment Variables → Temporal lists every variable. These values must agree across the server, the app, and the Worker:

If the frontend certificate uses a private CA, configure TEMPORAL_TLS_CA_PEM (raw or base64-encoded PEM) or TEMPORAL_TLS_CA_PATH on both the app and the Worker. A file path must be readable inside each container. In Helm, set the connection settings under both app.env and worker.env; setting them on one service does not configure the other. Keep TEMPORAL_TLS_ENABLED=true and set TEMPORAL_SERVER_NAME to the certificate's name. A server-name override alone does not establish trust in a private CA.

The app signs its own tokens. The Worker gets its token from one of two sources:

• Credential broker. This is the Compose default and Helm worker.auth.credentialBroker. The Worker calls the app with INTERNAL_API_SECRET and a proof signed with TEMPORAL_WORKER_BROKER_SECRET. The app returns a short-lived token for the identity in TEMPORAL_WORKER_IDENTITY. The Worker asks for a new token every 30 seconds by default. Production requires an HTTPS APP_URL, which Compose sets from TEMPORAL_WORKER_APP_URL.
• File or static token. This is Helm worker.auth.apiKeySecret, or TEMPORAL_API_KEY_FILE or TEMPORAL_API_KEY in Compose. You issue the token yourself with the Worker's TEMPORAL_IDENTITY as subject and the eight queue names. Rotate it by rewriting the file. The Worker checks the signature, issuer, audience, subject, namespace, role, and queue list before it uses the token.

The two sources are mutually exclusive. If no valid token arrives before the current one expires, the Worker marks itself not ready and stops polling.

Scheduled Sub Agents

A scheduled prompt task runs its complete assistant loop in one Temporal Worker Activity. Model requests, platform tools, and integration tools execute in the Worker process using the shared assistant runtime and platform gateway. The Worker no longer starts these runs by calling /api/workspace-assistant/execute, and platform tool calls do not pass through MCP.

The Activity has a 30-minute start-to-close timeout, a 30-second heartbeat timeout, and one attempt. Heartbeats report liveness; they do not save a checkpoint after each tool. If the Worker stops partway through a run, a new run starts the task again rather than resuming at the next tool. Review external side effects before manually retrying. Persisting the run's final status has a separate retry policy and does not rerun the assistant.

Configure provider access, credential decryption keys, storage, and network access on the Worker as well as the app. Saved tasks use the same model and reasoning validation, BYOK resolution, and current authorization as Chat. See Environment Variables.

The app still provides a credential broker for assistant tools that start Temporal workflows. The Worker calls /api/internal/assistant-temporal-credentials with its internal-service token, a broker proof, and the persisted schedule and execution IDs. The app checks the occurrence and current permissions before signing credentials for the requested operation. The Worker cannot use this broker to request worker or operator authority.

Keep APP_URL, INTERNAL_API_SECRET, and TEMPORAL_WORKER_BROKER_SECRET configured for these operations, including when the Worker's own polling credential comes from a file or static token. Temporal signing private keys remain on the app. Removing the assistant execution HTTP call does not remove the Worker's need to reach the app for credential issuance and other internal callbacks.

Local Development

docker/local/compose.yml runs a single-node server on loopback. gRPC listens on 127.0.0.1:7233, HTTP on 7243, and the Temporal UI on 8233. bun run temporal:up generates keys under .temporal-auth.local/, starts the stack, and writes .env.temporal.local for bun run dev and the host Worker. bun run temporal:smoke runs a real workflow through it.

The local server accepts tokens over plaintext because every port is loopback-only. For a production-shaped rehearsal on one machine, run bun run selfhost:validate. It starts the same server behind TLS, deploys the Helm chart into kind, and checks the Worker end to end.

Scaling Workers

Each Worker process runs one queue Worker per Task Queue: eight for the general Worker, one for the document Worker. Every queue Worker keeps its Activity and Workflow Task polls open as long requests to the frontend. The server limits how many of those a namespace may hold on each frontend host, separately for each kind of poll, with the dynamic config key frontend.namespaceCount. Past the limit, Workers log ResourceExhausted: namespace concurrent poller limit exceeded, and replicas added beyond that point get little or no work.

Total polls of each kind are about:

Worker replicas × Task Queues per Worker × pollers per queue

Keep that below frontend.namespaceCount × frontend hosts, with headroom, because each Worker's connection stays on one frontend host for several minutes and the split between hosts is uneven. Temporal's default is 1200 per host. hosted-sql.yaml raises it to 4000, which fits about 60 general Workers with TEMPORAL_MAX_ACTIVITY_TASK_POLLERS=10 on two frontend hosts. Without that setting, each general Worker holds up to 100 Activity Task polls and 80 Workflow Task polls. For example, 60 Workers with TEMPORAL_MAX_ACTIVITY_TASK_POLLERS=10 and TEMPORAL_MAX_WORKFLOW_TASK_POLLERS=5 hold about 60 × 8 × 10 = 4,800 Activity Task polls and 60 × 8 × 5 = 2,400 Workflow Task polls, about 2,400 and 1,200 per host.

A worker token may also call DescribeTaskQueue on the Task Queues it lists, and on no others. Each Worker uses it to serve its queues' backlog to the Helm chart's KEDA trigger, so the fleet grows as soon as tasks wait. See Kubernetes → Scaling Workers on Task Queue backlog. Server images built before this grant refuse the call, so update the server before you turn that trigger on.

Ten Activity Task polls per queue are usually enough to keep a Worker's Activity slots full: a poll returns as soon as a task is available, so a few polls can start many tasks a second. To hold fewer polls on idle queues, set TEMPORAL_POLLER_AUTOSCALING=true. The maximums still apply. See Environment Variables for the Worker settings. Dynamic config is part of the server image, so a new limit needs a new image and a server restart, before the Worker replica count goes up.

Scaling the Server

Each server pod runs every Temporal service: frontend, history, matching and worker. In clustered mode the pods find each other through ringpop. When a pod joins or leaves, history shards and task queue partitions move to the new membership, as they do in every rolling update, and in-flight requests retry. So you can scale the server by running more identical pods, and a CPU autoscaler with a minimum of two is safe. Scale in slowly: each change moves shards, and a fast autoscaler would move them back and forth with every burst. The Azure stack does this with cluster.temporal.autoscaling, which adds at most two pods a minute and removes one every ten minutes.

• Database connections. Each pod opens up to five services × (SQL_MAX_CONNS + SQL_VIS_MAX_CONNS) connections to the Temporal database. Keep (maximum pods + one rollout surge) × that under its max_connections.
• Polls. frontend.namespaceCount applies to each frontend pod, so more pods hold more Worker polls. The server recycles each Worker connection every few minutes, so a new pod takes its share gradually.
• History shards. NUM_HISTORY_SHARDS is fixed for the life of the database. It caps how far history can spread, not throughput per pod. 512 shards spread over tens of pods. One shard handles a few hundred workflow updates a second, so 512 covers several thousand state transitions a second. Past that, the Temporal database is usually the limit first.

To scale the services separately, start each pod with only some of them (--service frontend, or TEMPORAL_SERVICES). Keep internal-frontend running somewhere, because the system Worker needs it.

Production Checklist

• Frontend TLS on 7233. Set TEMPORAL_TLS_ENABLED=true and a matching TEMPORAL_SERVER_NAME on the app and the Worker.
• An HTTPS JWKS URL served by a highly available app. A fresh server or Worker cannot start while it is down.
• A stable BETTER_AUTH_SECRET across app replicas. Keep old public signing keys while running workflows still reference them.
• TEMPORAL_DEPLOYMENT_MODE=clustered with internode mutual TLS for any multi-node cluster. Keep ports 7234-7239 and 6933-6939 private.
• Schema and namespace bootstrap completed before the Worker starts. Both scripts are safe to run twice.
• bash scripts/temporal-production-preflight.sh passing before the first deployment and after every namespace change.