# Troubleshooting

Common issues and solutions

The commands below use the production Compose file. Run them from the repository root.

## Entra Workload Access Fails

Check that web and every worker received the same `WORKLOAD_AUTH_ENABLED=true` and valid `WORKLOAD_AUTH_PROFILE`, then redeploy after environment changes. A successful workspace-read verification does not establish import, deployment, environment-write or execution permission. Check the registered application's enabled state, current revision, target grants and workspace exclusions. See [workload troubleshooting](/self-hosting/entra-workload-identities#audit-and-troubleshooting) for token and worker-admission failures. Keep tokens and secret values out of diagnostic output.

## Database Connection Failed

```bash
# Check database is running
docker compose -f docker/selfhost/compose.yml ps db

# Test connection
docker compose -f docker/selfhost/compose.yml exec db psql -U postgres -c "SELECT 1"
```

Verify `DATABASE_URL` format: `postgresql://user:pass@host:5432/database`

The published images run with `NODE_ENV=production`. The connection pool then requires TLS unless the URL ends in `?sslmode=disable`. A plaintext PostgreSQL outside the Compose file fails with an SSL error until you add that parameter. The bundled `db` service works because the Compose file sets the URL for you.

On Kubernetes with PgBouncer, runs that fail with connection errors right after a PgBouncer pod stops (a node drain, scale-down or rollout) were still using that pod when it was killed. Set `DATABASE_MAX_LIFETIME_SECONDS` below `pgbouncer.terminationGracePeriodSeconds`; see [Connection pooling](/self-hosting/kubernetes#connection-pooling-pgbouncer).

## Ollama Models Not Showing

Inside Docker, `localhost` = the container, not your host machine.

```bash
# For host-machine Ollama, use:
OLLAMA_URL=http://host.docker.internal:11434  # macOS/Windows
OLLAMA_URL=http://192.168.1.x:11434           # Linux (use actual IP)
```

## WebSocket/Realtime Not Working

1. Check `NEXT_PUBLIC_SOCKET_URL` matches your domain
2. Verify realtime service is running: `docker compose -f docker/selfhost/compose.yml ps realtime`
3. Ensure reverse proxy passes WebSocket upgrades (see [Docker guide](/self-hosting/docker))

The realtime container exits with `Invalid environment configuration for the realtime server` in three cases:

* `BETTER_AUTH_SECRET` is shorter than 32 characters
* `INTERNAL_API_SECRET` is shorter than 32 characters
* `DATABASE_URL` is missing

It exits with `set APP_URL (preferred) or NEXT_PUBLIC_APP_URL` when neither URL is set. `BETTER_AUTH_SECRET` must equal the app's value. The health endpoint is `GET /health` on port 3002. Add extra browser origins with `ALLOWED_ORIGINS`.

## 502 Bad Gateway

```bash
# Check app is running
docker compose -f docker/selfhost/compose.yml ps steelengine
docker compose -f docker/selfhost/compose.yml logs steelengine

# Common causes: out of memory, database not ready
```

The `steelengine` container waits for `migrations` to exit successfully and for `realtime` to report healthy. A realtime configuration error therefore keeps the app down too. Check `ps` for the service that is not healthy.

## Migration Errors

```bash
# View migration logs
docker compose -f docker/selfhost/compose.yml logs migrations

# Run manually
docker compose -f docker/selfhost/compose.yml run --rm migrations
```

Migrations run from the `ghcr.io/steelengineai/migrations` image, not from the app container. The app image ships only the compiled server, so `bun run db:migrate` is not available inside `steelengine`.

## pgvector Not Found

Use the correct PostgreSQL image:

```yaml
image: pgvector/pgvector:pg17  # NOT postgres:17
```

## Temporal Worker Not Ready

Workflows stay queued and schedules never fire when the Temporal Worker is not polling.

```bash
# Ready only when every queue Worker is polling
docker compose -f docker/selfhost/compose.yml exec steelengine-worker curl -fsS http://127.0.0.1:3003/health/ready

docker compose -f docker/selfhost/compose.yml logs steelengine-worker
```

Match the log message to its cause:

* `Missing required environment variable TEMPORAL_ADDRESS` (or `TEMPORAL_NAMESPACE`, `TEMPORAL_BUILD_ID`): the variable is empty in `.env`. The Compose file passes an empty `TEMPORAL_BUILD_ID` through, which overrides the image default.
* `Temporal authentication requires TEMPORAL_AUTH_ISSUER and exactly one of TEMPORAL_AUTH_JWKS_URL or TEMPORAL_AUTH_JWKS_FILE`: set `TEMPORAL_AUTH_ISSUER` and `TEMPORAL_AUTH_JWKS_URL`. The Worker has no issuer default. The app defaults to `http://localhost:3000`. Set the same value on both.
* `Temporal Worker credential broker request failed for <url>`: the Worker cannot reach `TEMPORAL_WORKER_APP_URL`. It must be the app's HTTPS URL as seen from inside the Worker container. A `503` from that endpoint means the app lacks `TEMPORAL_WORKER_IDENTITY` or `TEMPORAL_WORKER_BROKER_SECRET`.
* `TEMPORAL_IDENTITY must match the Temporal Worker JWT subject`: the Worker's identity and the subject in its token differ. With the broker, keep `TEMPORAL_WORKER_IDENTITY` (app) equal to `TEMPORAL_IDENTITY` (Worker). With a static token, issue it for the Worker's identity.
* `Production Temporal Workers require bearer authentication`: no credential source is configured. The Compose file enables the broker. Helm needs `worker.auth.credentialBroker` or `worker.auth.apiKeySecret`.
* `TEMPORAL_ALLOW_INSECURE_AUTH is incompatible with TEMPORAL_TLS_ENABLED=true`: the plaintext escape hatch is on while TLS is on. Remove it outside local development.

`TEMPORAL_ADDRESS` is `host:port` with no `http://` or `grpc://` prefix. If the Worker connects but Temporal rejects workflows for missing search attributes, run `bash scripts/temporal-production-preflight.sh`. It needs `TEMPORAL_ADDRESS`, `TEMPORAL_NAMESPACE`, and an operator `TEMPORAL_API_KEY`. See [Temporal](/self-hosting/temporal#bootstrap-the-schema-and-namespace).

### Added Worker Replicas Get No Work

If Worker logs repeat `ResourceExhausted: namespace concurrent poller limit exceeded`, the Worker fleet holds more polls than the Temporal server allows for the namespace. Replicas added past that point get little or no work. Lower `TEMPORAL_MAX_ACTIVITY_TASK_POLLERS` and `TEMPORAL_MAX_WORKFLOW_TASK_POLLERS` on the Worker, or raise `frontend.namespaceCount` on the server. See [Temporal → Scaling Workers](/self-hosting/temporal#scaling-workers) for the arithmetic.

### Runs Fail After the Worker Restarts

If scheduled or API runs fail with only `Workflow execution failed`, check whether the Worker is restarting. On Kubernetes, `kubectl describe pod` shows `Last State: Terminated, Reason: OOMKilled`. The container hit its memory limit mid-run, and the runs it held were marked failed afterwards.

Give the Worker more memory and keep `NODE_OPTIONS=--max-old-space-size` at about 75% of the limit. The rest covers memory outside the JavaScript heap. To stop one Worker taking on more than it can hold, set `TEMPORAL_ACTIVITY_MEMORY_TARGET` and run more than one replica. See [Kubernetes](/self-hosting/kubernetes).

## Blank Page After Login

1. Check browser console for errors
2. Verify `NEXT_PUBLIC_APP_URL` matches your actual domain
3. Clear browser cookies and local storage
4. Check that all services are running: `docker compose -f docker/selfhost/compose.yml ps`

## Windows-Specific Issues

**Running from source on Windows:**

```bash
# Use WSL2 for better compatibility
wsl --install
```

**Line ending issues:**

```bash
# Configure git to use LF
git config --global core.autocrlf input
```

## Workflow Cannot Reach an Internal Service

An API block or a function-block `fetch` fails with `cannot point to private IP addresses`, `resolves to a blocked IP address`, or `must use https:// protocol` when the target is on your own network.

Workflow egress is deny-by-default for private addresses. Add the destination to `EGRESS_ALLOWLIST` on both the app and the Temporal Worker. With production Compose, set it in `.env` and recreate both containers so they receive the changed value. See [Private Network Configuration](/self-hosting/docker#private-network-configuration) and [Workflow Egress Allowlist](/self-hosting/environment-variables#workflow-egress-allowlist).

If the error is `resolves outside the allowlisted CIDR pin`, the host now resolves to an address outside the range you pinned. Confirm the DNS change is expected before widening the pin.

### MCP Cannot Connect to an Internal Server

MCP uses `ALLOWED_MCP_DOMAINS`, independently of `EGRESS_ALLOWLIST`. Set the approved hostname on both the app and Worker, then recreate their containers. The containers still need network access to that server.

If the server redirects, use its final HTTP or HTTPS endpoint URL. Put authentication in headers, without credentials embedded in the URL. Every request and reconnect validates and pins DNS results. Metadata and other absolute-deny addresses stay blocked even for an approved hostname; only an explicit `localhost` or loopback IP URL may reach loopback. See [Allowed Server Domains](/mcp#allowed-server-domains).

## View Logs

```bash
# All services
docker compose -f docker/selfhost/compose.yml logs -f

# Specific service
docker compose -f docker/selfhost/compose.yml logs -f steelengine
docker compose -f docker/selfhost/compose.yml logs -f steelengine-worker
```

## Getting Help

* [GitHub Issues](https://github.com/steelengineai/steelengine/issues)
* [Discord](https://discord.gg/Hr4UWYEcTT)
