# Temporal

Run the Temporal server that SteelEngine deployments depend on

SteelEngine runs every workflow, schedule, and background job through Temporal. The Compose file and the Helm chart ship the Temporal Worker, not the Temporal server. You run the server and point the app and the Worker at it. SteelEngine publishes its own server image because a stock Temporal server cannot verify SteelEngine's tokens.

The Temporal server is a separate deployment with its own PostgreSQL databases. Size it in addition to the [Requirements](/self-hosting#requirements) for the SteelEngine containers.

## The Server Image

`ghcr.io/steelengineai/temporal-server` is Temporal Server 1.31.2 with SteelEngine's token verification built in. It accepts Temporal's standard environment configuration and `--config-file`. The image also carries the `temporal` CLI, two dynamic config files, and the namespace bootstrap script. Pull it with the same registry credentials as the app images.

The server keeps Temporal's own system Worker on an internal frontend (port 7236) that skips token verification. Never expose that port or any other internode port through a Service, load balancer, or ingress.

## How Authentication Works

The app signs a short-lived token for every request to Temporal. The token names the issuer, the audience (`steelengine-temporal-rpc`), a subject, a role, and the namespaces and Task Queues it may touch. The signing key is the same one behind the app's JWKS endpoint, `<app URL>/api/auth/jwks`. The server verifies each token against that endpoint. It never receives the signing key.

There are three roles. The app uses `client`. The Worker uses `worker`. Bootstrap tooling uses `operator`, which is the only administrative role. A Worker token must carry the Worker's `TEMPORAL_IDENTITY` as its subject and list all eight product Task Queues:

```
steelengine-workflows
steelengine-knowledge-connectors
steelengine-webhook-ingestion
steelengine-notifications
steelengine-document-processing
steelengine-copy-backfill
steelengine-scheduled-tasks
steelengine-platform
```

A second signed token travels inside each workflow and identifies the caller. Workers check it before they run Activities. It is not a Temporal Server credential and needs no server configuration.

## Server Configuration

Required:

| Variable                  | Description                                                                                              |
| ------------------------- | -------------------------------------------------------------------------------------------------------- |
| `TEMPORAL_AUTH_ISSUER`    | Exact token issuer. Must match `TEMPORAL_AUTH_ISSUER` on the app and the Worker                          |
| `TEMPORAL_AUTH_AUDIENCE`  | Token audience. Use `steelengine-temporal-rpc` unless you changed it everywhere                          |
| `TEMPORAL_AUTH_JWKS_URL`  | HTTPS JWKS endpoint, normally `<app URL>/api/auth/jwks`. Set this or `TEMPORAL_AUTH_JWKS_FILE`, not both |
| `TEMPORAL_AUTH_JWKS_FILE` | Local JWKS file for development or a mounted secret                                                      |

Optional:

| Variable                                                                                                                       | Default      | Description                                                                                  |
| ------------------------------------------------------------------------------------------------------------------------------ | ------------ | -------------------------------------------------------------------------------------------- |
| `TEMPORAL_AUTH_ENVIRONMENT`                                                                                                    | `production` | `development`, `test`, or `production`. Production rejects every insecure exception          |
| `TEMPORAL_AUTH_REQUIRE_TLS`                                                                                                    | `true`       | Reject tokens on plaintext connections                                                       |
| `TEMPORAL_AUTH_ALLOW_INSECURE_JWKS`                                                                                            | `false`      | Permit an HTTP JWKS URL. Development only                                                    |
| `TEMPORAL_AUTH_CLOCK_SKEW`                                                                                                     | `30s`        | Tolerance for token time claims                                                              |
| `TEMPORAL_AUTH_JWKS_REFRESH_INTERVAL`                                                                                          | `5m`         | Maximum age of cached keys                                                                   |
| `TEMPORAL_AUTH_JWKS_HTTP_TIMEOUT`                                                                                              | `5s`         | JWKS HTTP timeout                                                                            |
| `TEMPORAL_DEPLOYMENT_MODE`                                                                                                     | `local`      | `local` permits loopback. `clustered` requires a private broadcast address and internode TLS |
| `TEMPORAL_BROADCAST_ADDRESS`                                                                                                   | detected     | Private IPv4 this node advertises. Detected when the host has one private interface          |
| `TEMPORAL_TLS_INTERNODE_CA`, `TEMPORAL_TLS_INTERNODE_CERT`, `TEMPORAL_TLS_INTERNODE_KEY`, `TEMPORAL_TLS_INTERNODE_SERVER_NAME` | -            | Internode mutual TLS material. Required in `clustered` mode                                  |

Persistence and frontend TLS use Temporal's own variables. For PostgreSQL, set `DB=postgres12`, `POSTGRES_SEEDS`, `POSTGRES_USER`, `POSTGRES_PWD`, `DB_PORT`, and `DYNAMIC_CONFIG_FILE_PATH`. For frontend TLS, set `TEMPORAL_TLS_FRONTEND_CERT`, `TEMPORAL_TLS_FRONTEND_KEY`, and `TEMPORAL_TLS_FRONTEND_SERVER_NAME`. SteelEngine's `TEMPORAL_SERVER_NAME` must match that certificate name.

The image ships two dynamic config files under `/etc/temporal/config/dynamicconfig/`. Both record the verified caller on workflow events and allow long IDs. Use `development-sql.yaml` for a single node and `hosted-sql.yaml` for a multi-node cluster, which also encrypts cluster membership traffic.

## Bootstrap the Schema and Namespace

The server does not create its own tables. Against an empty PostgreSQL it refuses to start.

1. **Create the schema.** Run `docker/temporal/scripts/setup-all.sh` from the `temporalio/admin-tools:1.31.2` image. Give it the same `DB`, `POSTGRES_*`, and `SQL_PASSWORD` values as the server. It creates the `temporal` and `temporal_visibility` databases. No Elasticsearch is needed. Use a PostgreSQL instance dedicated to Temporal in production.
2. **Create the namespace.** Run `/etc/temporal/scripts/bootstrap-product-namespace.sh` from the server image as a one-shot task. Set `TEMPORAL_PRODUCT_NAMESPACE` and `TEMPORAL_INTERNAL_ADDRESS`. In `clustered` mode, also pass the internode TLS material. The script creates the namespace and registers eight `Keyword` search attributes: `OrganizationId`, `WorkspaceId`, `WorkflowOwnerId`, `WorkflowTriggerSource`, `SteelEngineWorkflowId`, `SteelEngineExecutionId`, `SteelEngineDeploymentId`, and `SteelEngineRequestId`. SteelEngine writes these attributes when it starts workflows, so register all eight before the first run.
3. **Verify.** Run `bash scripts/temporal-production-preflight.sh` from the repository with an operator token. It needs `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, `TEMPORAL_API_KEY`, and the TLS flags. It checks cluster health and creates any missing search attribute.

Namespace retention defaults to 7 days. Change it with `TEMPORAL_NAMESPACE_RETENTION`.

## Connect SteelEngine

The app and the Worker read the same connection settings. [Environment Variables → Temporal](/self-hosting/environment-variables#temporal) lists every variable. These values must agree across the server, the app, and the Worker:

| Setting   | Server                              | App and Worker                                                |
| --------- | ----------------------------------- | ------------------------------------------------------------- |
| Endpoint  | frontend on `7233`                  | `TEMPORAL_ADDRESS=host:7233`, no scheme                       |
| Namespace | created in bootstrap                | `TEMPORAL_NAMESPACE`                                          |
| TLS name  | `TEMPORAL_TLS_FRONTEND_SERVER_NAME` | `TEMPORAL_SERVER_NAME` with `TEMPORAL_TLS_ENABLED=true`       |
| Issuer    | `TEMPORAL_AUTH_ISSUER`              | `TEMPORAL_AUTH_ISSUER`                                        |
| Audience  | `TEMPORAL_AUTH_AUDIENCE`            | `TEMPORAL_AUTH_AUDIENCE` (default `steelengine-temporal-rpc`) |
| JWKS      | `TEMPORAL_AUTH_JWKS_URL`            | `TEMPORAL_AUTH_JWKS_URL` on the Worker                        |

The app signs its own tokens. The Worker gets its token from one of two sources:

* **Credential broker.** This is the Compose default and Helm `worker.auth.credentialBroker`. The Worker calls the app with `INTERNAL_API_SECRET` and a proof signed with `TEMPORAL_WORKER_BROKER_SECRET`. The app returns a short-lived token for the identity in `TEMPORAL_WORKER_IDENTITY`. The Worker asks for a new token every 30 seconds by default. Production requires an HTTPS `APP_URL`, which Compose sets from `TEMPORAL_WORKER_APP_URL`.
* **File or static token.** This is Helm `worker.auth.apiKeySecret`, or `TEMPORAL_API_KEY_FILE` or `TEMPORAL_API_KEY` in Compose. You issue the token yourself with the Worker's `TEMPORAL_IDENTITY` as subject and the eight queue names. Rotate it by rewriting the file. The Worker checks the signature, issuer, audience, subject, namespace, role, and queue list before it uses the token.

The two sources are mutually exclusive. If no valid token arrives before the current one expires, the Worker marks itself not ready and stops polling.

## Scheduled Sub Agents

A scheduled prompt task runs its complete assistant loop in one Temporal Worker Activity. Model requests, platform tools, and integration tools execute in the Worker process using the shared assistant runtime and platform gateway. The Worker no longer starts these runs by calling `/api/workspace-assistant/execute`, and platform tool calls do not pass through MCP.

The Activity has a 30-minute start-to-close timeout, a 30-second heartbeat timeout, and one attempt. Heartbeats report liveness; they do not save a checkpoint after each tool. If the Worker stops partway through a run, a new run starts the task again rather than resuming at the next tool. Review external side effects before manually retrying. Persisting the run's final status has a separate retry policy and does not rerun the assistant.

Configure provider access, credential decryption keys, storage, and network access on the Worker as well as the app. Saved tasks use the same model and reasoning validation, BYOK resolution, and current authorization as Chat. See [Environment Variables](/self-hosting/environment-variables#scheduled-assistant-configuration).

The app still provides a credential broker for assistant tools that start Temporal workflows. The Worker calls `/api/internal/assistant-temporal-credentials` with its internal-service token, a broker proof, and the persisted schedule and execution IDs. The app checks the occurrence and current permissions before signing credentials for the requested operation. The Worker cannot use this broker to request worker or operator authority.

Keep `APP_URL`, `INTERNAL_API_SECRET`, and `TEMPORAL_WORKER_BROKER_SECRET` configured for these operations, including when the Worker's own polling credential comes from a file or static token. Temporal signing private keys remain on the app. Removing the assistant execution HTTP call does not remove the Worker's need to reach the app for credential issuance and other internal callbacks.

## Local Development

`docker/local/compose.yml` runs a single-node server on loopback. gRPC listens on `127.0.0.1:7233`, HTTP on `7243`, and the Temporal UI on `8233`. `bun run temporal:up` generates keys under `.temporal-auth.local/`, starts the stack, and writes `.env.temporal.local` for `bun run dev` and the host Worker. `bun run temporal:smoke` runs a real workflow through it.

The local server accepts tokens over plaintext because every port is loopback-only. For a production-shaped rehearsal on one machine, run `bun run selfhost:validate`. It starts the same server behind TLS, deploys the Helm chart into kind, and checks the Worker end to end.

`TEMPORAL_ALLOW_INSECURE_AUTH` on the clients and `TEMPORAL_AUTH_REQUIRE_TLS=false` on the server are for loopback development only. The app and the Worker reject `TEMPORAL_ALLOW_INSECURE_AUTH` together with `TEMPORAL_TLS_ENABLED=true`. The server rejects an HTTP JWKS URL in production.

## Production Checklist

* Frontend TLS on `7233`. Set `TEMPORAL_TLS_ENABLED=true` and a matching `TEMPORAL_SERVER_NAME` on the app and the Worker.
* An HTTPS JWKS URL served by a highly available app. A fresh server or Worker cannot start while it is down.
* A stable `BETTER_AUTH_SECRET` across app replicas. Keep old public signing keys while running workflows still reference them.
* `TEMPORAL_DEPLOYMENT_MODE=clustered` with internode mutual TLS for any multi-node cluster. Keep ports 7234-7239 and 6933-6939 private.
* Schema and namespace bootstrap completed before the Worker starts. Both scripts are safe to run twice.
* `bash scripts/temporal-production-preflight.sh` passing before the first deployment and after every namespace change.

## Common Questions

**Can I use Temporal Cloud or a stock Temporal server instead of the SteelEngine image?**

The app and the Worker send SteelEngine-signed tokens. The server must verify them against the app's JWKS endpoint. The ghcr.io/steelengineai/temporal-server image does this, and it is the only server this guide covers.

**Why does the Temporal server refuse to start against a fresh database?**

The image never creates tables. Run docker/temporal/scripts/setup-all.sh from temporalio/admin-tools:1.31.2 first. It creates the temporal and temporal_visibility databases and their schemas.

**Which value goes in TEMPORAL_AUTH_ISSUER?**

The app writes TEMPORAL_AUTH_ISSUER into every token, and falls back to BETTER_AUTH_URL when the variable is unset. Set it to your public app URL on the app, the Worker, and the server so all three compare the same string. The Worker has no fallback and fails at startup without it.

**How does the Worker get its Temporal credential?**

From the app's credential broker, which is the Compose default. The broker returns a short-lived worker token in exchange for INTERNAL_API_SECRET and a proof signed with TEMPORAL_WORKER_BROKER_SECRET. Or from a token you issue yourself, mounted as a file (TEMPORAL_API_KEY_FILE) or passed as TEMPORAL_API_KEY. The token subject must equal the Worker's TEMPORAL_IDENTITY. The two sources cannot be combined.

**Do I need Elasticsearch for visibility?**

No. The server keeps visibility data in the temporal_visibility PostgreSQL database. The eight Keyword search attributes that SteelEngine registers use its built-in Keyword slots.

**What does the local stack do differently from production?**

docker/local/compose.yml binds every port to 127.0.0.1 and accepts tokens over plaintext. It reads a JWKS file instead of an HTTPS URL and shares one PostgreSQL instance with the app. Production keeps TLS on and verifies against the app's HTTPS JWKS endpoint. It uses clustered mode with internode TLS for multi-node clusters and gives Temporal its own database.