# Azure Key Vault secret storage

Configure optional Key Vault storage for workspace secrets and workspace or organization BYOK credentials.

Azure Key Vault can store workspace Secrets/environment variables, workspace BYOK keys and organization BYOK keys. Existing encrypted PostgreSQL storage remains available. A storage default applies to **new entries only**; existing entries move through an explicit migration.

For PostgreSQL-only installations, leave `keyVault.enabled: false` (the default) or leave `AZURE_KEY_VAULT_ENABLED` unset. No Azure credentials or additional environment variables are required. Keep the existing `ENCRYPTION_KEY` and apply the additive database migration with this application revision. PostgreSQL environment batches save encrypted values and synchronize their credential index in one database transaction. Caller-supplied transactions remain supported for PostgreSQL-only writes, including when Key Vault is enabled for other entries.

This integration uses Azure **secrets**. It does not store personal environment variables, OAuth/connector credentials, installation bootstrap secrets, Azure cryptographic keys or certificates. BYOK means the provider API credential, not an Azure key object. The initial deployment profile supports commercial Azure, AKS workload identity and vaults/managed identities in the same Entra resource tenant. Cross-subscription access in that tenant is supported. Cross-tenant, sovereign-cloud and non-AKS deployments need separate qualification.

## What runs where

The operator approval pool binds each vault URL, network mode and permitted exact names/prefixes to an organization. Organization administrators can narrow these approvals, not expand them. A shared Azure identity cannot establish organization ownership by itself. Never approve installation bootstrap/encryption secrets or another organization’s names in this pool.

The installation has a get-only reader managed identity and a separate management writer. Trusted application and worker code selects the approved vault from PostgreSQL metadata and resolves values at use. Tokens never enter workflow/user code. The management UI shows names, references, versions and operation status; it does not load vault values into a form.

The reader is shared within one installation/environment. SteelEngine checks organization, workspace, credential family, name assignments and current caller/workload authority before retrieval and cache hits. Azure identity itself does not isolate workspaces. Keep separate identities and vault assignments between development, staging and production installations.

Deploy this application revision with Key Vault disabled first. Organization settings → Key Vault shows the organization id needed by the operator approval pool.

The steps below configure the repository's Helm chart. Use a pinned application/chart revision that includes this feature. Keep the existing PostgreSQL, Temporal, internal broker, TLS and application secrets configuration from the [Kubernetes guide](/self-hosting/kubernetes) and [Temporal guide](/self-hosting/temporal).

## 1. Prepare Azure resources

Run Azure CLI as an operator authorized to create identities, custom roles, role assignments and the chosen network resources. Set these identifiers for your environment; they are metadata, not credentials:

```bash
export RESOURCE_GROUP='customer-runtime'
export LOCATION='eastus'
export AKS_NAME='customer-aks'
export VAULT_NAME='customer-runtime-vault'
export NAMESPACE='steelengine'
export RELEASE='steelengine'
export APP_SERVICE_ACCOUNT='steelengine-app'
export WORKER_SERVICE_ACCOUNT='steelengine-worker'
export DOCUMENT_SERVICE_ACCOUNT='steelengine-document-worker'
export READER_IDENTITY='steelengine-vault-reader'
export WRITER_IDENTITY='steelengine-vault-writer'
export TENANT_ID="$(az account show --query tenantId -o tsv)"
export SUBSCRIPTION_ID="$(az account show --query id -o tsv)"
```

For a new vault:

```bash
az keyvault create --resource-group "$RESOURCE_GROUP" --name "$VAULT_NAME" \
  --location "$LOCATION" --enable-rbac-authorization true \
  --enable-purge-protection true --retention-days 90 --default-action Deny \
  --query id -o tsv
```

For an existing/pre-seeded vault, inspect its tenant, RBAC mode, network configuration, soft-delete retention and purge protection. Coordinate changes with its owner; do not convert its access model as part of an application upgrade. No import of values is required. SteelEngine can link its existing secret names/versions.

```bash
az keyvault show --resource-group "$RESOURCE_GROUP" --name "$VAULT_NAME" \
  --query '{id:id,tenant:properties.tenantId,rbac:properties.enableRbacAuthorization,publicNetwork:properties.publicNetworkAccess,purgeProtection:properties.enablePurgeProtection,retentionDays:properties.softDeleteRetentionInDays}'
export VAULT_RESOURCE_ID="$(az keyvault show --name "$VAULT_NAME" --query id -o tsv)"
az identity create --resource-group "$RESOURCE_GROUP" --name "$READER_IDENTITY" --location "$LOCATION" --query id -o tsv
az identity create --resource-group "$RESOURCE_GROUP" --name "$WRITER_IDENTITY" --location "$LOCATION" --query id -o tsv
export READER_CLIENT_ID="$(az identity show -g "$RESOURCE_GROUP" -n "$READER_IDENTITY" --query clientId -o tsv)"
export WRITER_CLIENT_ID="$(az identity show -g "$RESOURCE_GROUP" -n "$WRITER_IDENTITY" --query clientId -o tsv)"
export READER_PRINCIPAL_ID="$(az identity show -g "$RESOURCE_GROUP" -n "$READER_IDENTITY" --query principalId -o tsv)"
export WRITER_PRINCIPAL_ID="$(az identity show -g "$RESOURCE_GROUP" -n "$WRITER_IDENTITY" --query principalId -o tsv)"
```

### Least-privilege data-plane roles

The runtime reader needs only `getSecret`. Discovery requires `readMetadata`, and managed writes require `setSecret`. Use custom roles so the runtime reader cannot list names. The writer needs get/set and metadata discovery, with no delete, purge, backup, restore or recovery operations. Microsoft documents the exact [Key Vault data actions](https://learn.microsoft.com/en-us/azure/role-based-access-control/permissions/security#microsoftkeyvault).

```bash
cat > /tmp/steelengine-vault-reader-role.json <<EOF
{"Name":"SteelEngine Vault Reader $RELEASE","IsCustom":true,"Description":"Read secret values for the trusted runtime","Actions":[],"NotActions":[],"DataActions":["Microsoft.KeyVault/vaults/secrets/getSecret/action"],"NotDataActions":[],"AssignableScopes":["/subscriptions/$SUBSCRIPTION_ID"]}
EOF
cat > /tmp/steelengine-vault-writer-role.json <<EOF
{"Name":"SteelEngine Vault Writer $RELEASE","IsCustom":true,"Description":"Manage assigned application secret values and discover metadata","Actions":[],"NotActions":[],"DataActions":["Microsoft.KeyVault/vaults/secrets/getSecret/action","Microsoft.KeyVault/vaults/secrets/setSecret/action","Microsoft.KeyVault/vaults/secrets/readMetadata/action"],"NotDataActions":[],"AssignableScopes":["/subscriptions/$SUBSCRIPTION_ID"]}
EOF
az role definition create --role-definition /tmp/steelengine-vault-reader-role.json --query name -o tsv
az role definition create --role-definition /tmp/steelengine-vault-writer-role.json --query name -o tsv
az role assignment create --assignee-object-id "$READER_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal --role "SteelEngine Vault Reader $RELEASE" --scope "$VAULT_RESOURCE_ID" --query id -o tsv
az role assignment create --assignee-object-id "$WRITER_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal --role "SteelEngine Vault Writer $RELEASE" --scope "$VAULT_RESOURCE_ID" --query id -o tsv
```

Role definitions are subscription-scoped; role assignments above are vault-scoped. Adapt names/scope through your organization's IaC and change process, and allow Azure role propagation before testing. Neither identity needs permission to manage the vault resource itself.

## 2. Choose and validate the network path

The application accepts canonical `https://<name>.vault.azure.net` origins only. Every resolved address must match the selected network mode. Redirects and cross-vault continuation URLs are rejected. Private access never falls back to a public endpoint.

For a restricted public endpoint, allow only the AKS installation's known outbound public IPs and keep the default deny policy. Do not enable a blanket trusted-services bypass to compensate for a missing egress rule.

```bash
export AKS_EGRESS_IP='203.0.113.10' # Replace with the actual installation egress address.
az keyvault update --name "$VAULT_NAME" --public-network-access Enabled \
  --default-action Deny --bypass None --query id -o tsv
az keyvault network-rule add --name "$VAULT_NAME" --ip-address "$AKS_EGRESS_IP" --query id -o tsv
```

For a private endpoint, use an approved endpoint subnet and link the private DNS zone to the AKS VNet:

```bash
export VNET_ID='/subscriptions/.../resourceGroups/.../providers/Microsoft.Network/virtualNetworks/runtime-vnet'
export ENDPOINT_SUBNET_ID="$VNET_ID/subnets/private-endpoints"
az network private-endpoint create -g "$RESOURCE_GROUP" -n "$VAULT_NAME-pe" \
  --subnet "$ENDPOINT_SUBNET_ID" --private-connection-resource-id "$VAULT_RESOURCE_ID" \
  --group-id vault --connection-name "$VAULT_NAME-connection" --query id -o tsv
az network private-dns zone create -g "$RESOURCE_GROUP" -n privatelink.vaultcore.azure.net --query id -o tsv
az network private-dns link vnet create -g "$RESOURCE_GROUP" -n "$VAULT_NAME-aks" \
  --zone-name privatelink.vaultcore.azure.net --virtual-network "$VNET_ID" --registration-enabled false --query id -o tsv
export PRIVATE_DNS_ZONE_ID="$(az network private-dns zone show -g "$RESOURCE_GROUP" -n privatelink.vaultcore.azure.net --query id -o tsv)"
az network private-endpoint dns-zone-group create -g "$RESOURCE_GROUP" \
  --endpoint-name "$VAULT_NAME-pe" -n vault --private-dns-zone "$PRIVATE_DNS_ZONE_ID" --zone-name vault --query id -o tsv
az keyvault update --name "$VAULT_NAME" --public-network-access Disabled --query id -o tsv
```

Reuse an existing private DNS zone/link instead of recreating it. Record the endpoint's actual private IP(s) as narrow CIDRs in both Helm `keyVault.privateCidrs` and the UI connection. Verify DNS and TCP/TLS reachability from app, Temporal worker and document worker pods. Entra token exchange also needs outbound HTTPS to the commercial Entra endpoints; vault private DNS does not provide Entra connectivity.

## 3. Federate separate service accounts

```bash
az aks update -g "$RESOURCE_GROUP" -n "$AKS_NAME" --enable-oidc-issuer --enable-workload-identity --query id -o tsv
export AKS_ISSUER="$(az aks show -g "$RESOURCE_GROUP" -n "$AKS_NAME" --query oidcIssuerProfile.issuerUrl -o tsv)"
az identity federated-credential create -g "$RESOURCE_GROUP" --identity-name "$WRITER_IDENTITY" \
  --name "$RELEASE-app-writer" --issuer "$AKS_ISSUER" \
  --subject "system:serviceaccount:$NAMESPACE:$APP_SERVICE_ACCOUNT" --audiences api://AzureADTokenExchange --query id -o tsv
for ACCOUNT in "$APP_SERVICE_ACCOUNT" "$WORKER_SERVICE_ACCOUNT" "$DOCUMENT_SERVICE_ACCOUNT"; do
  az identity federated-credential create -g "$RESOURCE_GROUP" --identity-name "$READER_IDENTITY" \
    --name "$RELEASE-$ACCOUNT-reader" --issuer "$AKS_ISSUER" \
    --subject "system:serviceaccount:$NAMESPACE:$ACCOUNT" --audiences api://AzureADTokenExchange --query id -o tsv
done
```

The app needs the reader for its own trusted consumers and the writer for management. Worker service accounts receive reader federation only. Never federate the writer to a runtime worker service account. Exact issuer, namespace, service account subject and audience must match. The chart adds the required pod label and identity annotations; the SDK uses `WorkloadIdentityCredential` and the projected `AZURE_FEDERATED_TOKEN_FILE`, with no developer-credential fallback. See Microsoft's [AKS workload identity setup](https://learn.microsoft.com/en-us/azure/aks/workload-identity-deploy-cluster).

## 4. Deploy

Secret values are masked or omitted from logs, trace displays, diagnostics, management responses and audit records. The native adapter disables Azure HTTP body/token logging and returns fixed error categories. Function stdout and errors use known-value redaction, including values without recognizable credential prefixes. Actual authorized workflow inputs and results retain their existing behavior.

This integration adds no execution-data encryption, encrypted checkpoint format or new keyring requirement. PostgreSQL credential encryption continues for entries stored in PostgreSQL. Vault-backed configuration and durable version selections contain references and metadata only; the value cache stays in process memory. Previously returned values, or values deliberately copied or transformed by authorized code, follow the destination's controls. Revoking a vault reference cannot recall those copies.

Overlay these settings on the installation's existing values:

```yaml
componentServiceAccounts:
  enabled: true
  app:
    name: steelengine-app
  worker:
    name: steelengine-worker
  documentWorker:
    name: steelengine-document-worker
keyVault:
  enabled: true
  tenantId: "<resource-tenant-id>"
  readerClientId: "<reader-managed-identity-client-id>"
  writerClientId: "<writer-managed-identity-client-id>"
  allowedVaults:
    - organizationId: "<SteelEngine-organization-id>"
      vaultUrl: "https://customer-runtime-vault.vault.azure.net"
      networkMode: private
      names: ["existing-api-token", "existing-openai-key"]
      prefixes: ["se-", "customer-workload-"]
  privateCidrs: ["10.20.30.4/32"] # Actual endpoint IPs; [] for restricted public access.
  cacheTtlSeconds: 60
```

```bash
helm template "$RELEASE" ./helm/steelengine -n "$NAMESPACE" \
  -f installation-values.yaml -f key-vault-values.yaml > /tmp/steelengine-rendered.yaml
helm upgrade --install "$RELEASE" ./helm/steelengine -n "$NAMESPACE" \
  -f installation-values.yaml -f key-vault-values.yaml --wait --timeout 10m
kubectl -n "$NAMESPACE" rollout status deployment/steelengine-app
kubectl -n "$NAMESPACE" rollout status deployment/steelengine-worker
```

Use actual rendered deployment names if `fullnameOverride` or the release differs. Review the rendered service account names, per-component identity annotations, workload identity pod labels and reader/writer environment variables before applying. The writer client id is blank in reader pods. The document-rendering sidecar and migration init container are excluded from workload identity token injection; native Key Vault settings are injected only into the application and worker containers. A successfully rendered chart does not prove Azure federation or vault permissions.

For non-Helm configuration, the corresponding server-only variables are `AZURE_KEY_VAULT_ENABLED`, `AZURE_KEY_VAULT_TENANT_ID`, `AZURE_KEY_VAULT_ALLOWED_VAULTS` (the JSON approval pool), `AZURE_KEY_VAULT_READER_CLIENT_ID`, management-only `AZURE_KEY_VAULT_WRITER_CLIENT_ID`, `AZURE_KEY_VAULT_PRIVATE_CIDRS` (comma-separated), `AZURE_KEY_VAULT_CACHE_TTL_SECONDS`. AKS injects the token-file variable. Other identity/deployment profiles need separate qualification.

### Automate the installation approval list

Keep `keyVault.allowedVaults` in the installation's version-controlled Helm/GitOps values. This is the central list of approved organization/vault pairs and permitted secret names or prefixes. Organization administrators select from it in the UI; they cannot expand it.

Prefer a dedicated application prefix for existing secrets, plus the managed `se-` namespace when SteelEngine may create new entries:

```yaml
keyVault:
  allowedVaults:
    - organizationId: "<SteelEngine-organization-id>"
      vaultUrl: "https://customer-runtime-vault.vault.azure.net"
      networkMode: private
      names: ["legacy-openai-key"]
      prefixes: ["customer-app-prod-", "se-"]
```

An administrator can then create a **Vault access rule** using `customer-app-prod-`. New secrets with that prefix become permitted automatically; they do not need a separate `names` entry. With **Show allowed secret names when linking an entry** enabled, **Discover available names** lists matching metadata from Azure, including new matching names. It never fetches values for the list. Discovery may require paging, and the picker searches the current page.

For exceptions that need exact names, generate the `names` array from the same infrastructure manifest or pipeline that owns those Azure secrets, review it, and deploy the resulting Helm values. The **Allowed existing secrets** picker consumes the operator's explicit list. A known name covered by an approved prefix can also be selected without permitting the whole prefix in that access rule. Do not automatically approve every name found by an unrestricted vault scan: discovery should obey the approval policy, rather than expanding it.

The installation approval pool accepts at most 200 organization/vault pairs across all organizations and 256 KiB of JSON, with up to 500 exact names and 100 prefixes per approval profile. Each connection supports up to 200 access rules, each with up to 200 exact names. These bounds are application limits. There is no separate per-organization connection-count quota. Updating the Helm approval list requires deploying the new configuration to all application and worker hosts; creating another secret under an already approved and assigned prefix does not.

This uses the existing configuration. It adds no environment variables, database migration or Azure credentials to the browser.

## 5. Configure organization and workspace storage

1. An organization administrator with `organization:manage_secret_storage` opens **Organization settings → Key Vault** and chooses **Connect approved vault**. Select the approved vault and give the connection a name. The resource tenant and network mode are shown under &#x2A;*Connection details (operator setup)**; private endpoint CIDRs must be supplied by the installation operator.
2. Add **Vault access rules** for each workspace's secrets/environment variables, workspace provider API keys or organization provider API keys as needed. Select **Allowed existing secrets** from the operator list, or choose an approved prefix under **Automatically allowed names**. Use **Advanced name matching** only to narrow a prefix. Enable permission to create/update values and to show names separately. SteelEngine's generated names are confined to their original organization/workspace/family even with a broad `se-` approval.
3. Open **Workspace settings → Secrets** or **BYOK**, or **Organization settings → Models** for organization BYOK. Select the storage default for new entries. The existing editors and supported automation writes follow that default for new entries and retain existing entries' backends.
4. To use a pre-seeded value, choose **Link a pre-seeded secret**, discover permitted names or enter an approved name, and optionally pin an exact version under **Advanced: version and write access**. Links are read-only by default; a pin is read-only. Discovery returns metadata only and may require paging.
5. To move an existing entry, open **Advanced: migration and recovery** and use **Migrate an existing entry**. Export/source-use authority is separate from write management. The server copies the value without returning it to the caller. Local-to-vault completion removes the active PostgreSQL value; old WAL/backups still follow their retention policies.

Keep BYOK keys referenced by active or rollback-eligible promoted releases in PostgreSQL. Promotion activation checks the effective provider key at the workspace and organization scopes and requires PostgreSQL storage for BYOK assignments. Native Vault storage remains available for ordinary workspace and organization BYOK use.

Removing a vault-backed workspace BYOK override can expose the already-configured organization fallback, just as removing a local override does. Vault failures, invalid/expired values, denied access and pending writes never select another credential source. Only actual entry absence permits fallback. Cross-backend batches can partially complete; refresh entry/operation status and reconcile pending writes before resubmitting.

### API example

The configuration endpoint is session-authenticated and returns `Cache-Control: no-store`. Use the current browser session or a protected cookie jar. Existing workload automation for workspace environment writes uses its current workload grant and the configured storage default.

```bash
# request.json: {"action":"default","connectionId":"<approved-connection-id>"}
curl --fail-with-body --cookie "$SESSION_COOKIE_JAR" \
  --json @request.json \
  "$APP_URL/api/organizations/$ORGANIZATION_ID/secret-storage?family=environment&workspaceId=$WORKSPACE_ID"
```

Use `family=workspace_byok` for workspace provider keys, or `family=organization_byok` without `workspaceId` for organization keys. Other explicit actions are `link` (key/connectionId/secretName/optional version), `write` (key/value/optional backend target), `migrate` (key/target), `detach` (key), `discover` (connectionId/opaque continuation) and privileged `recover` (bindingId/exact version). Do not send submitted values through command-line arguments or print request bodies in diagnostics.

## Provision a workspace from CI/CD

A self-hosted installation can create a fresh workspace and configure its Key Vault storage in one request after the migration and Helm rollout finish. The target organization and intended workspace owner's active membership must already exist. If needed, provision those records using the existing installation Admin API.

### Allow an existing workload identity

Use the same federated pipeline sign-in and enabled SteelEngine organization binding described in [Entra workload identities](/self-hosting/entra-workload-identities). In the identity details, copy **Workload identity ID** and **Organization ID**. Add that SteelEngine binding ID to `workspaceProvisionerIds` on the target vault's existing Helm approval.

```yaml
keyVault:
  allowedVaults:
    - organizationId: "<organization-id>"
      vaultUrl: "https://customer-runtime-vault.vault.azure.net"
      networkMode: private
      names: ["existing-openai-key"]
      prefixes: ["customer-workload-", "se-"]
      workspaceProvisionerIds:
        - "11111111-1111-4111-8111-111111111111" # Workload identity ID from SteelEngine
```

Use the matching workload identity and vault approval for each installation and organization. Deploy compatible web and worker images before adding the approval to an installation already using Key Vault. DevOps can then provision each workspace entirely from CI/CD. This approval can create workspaces with this vault and select an existing organization member as owner. Give the approval to your installation pipeline identity. To revoke it, remove the binding ID from the vault approval and roll out the values, or disable the existing binding to revoke all its SteelEngine access.

### Apply the workspace manifest

Use the installation's approved vault and a stable workspace UUID in your deployment state. Generate the UUID once and reuse it on every retry. Use that same target organization and workspace ID when deploying workflows. The request creates a workspace and a connection dedicated to it. Multiple workspaces can point to the same approved Azure vault while keeping their access rules separate.

For example, save this as `workspace-vault.json`. Use the owner’s SteelEngine user ID, then fill in the vault URL and secret names for the target installation.

```json
{
  "name": "Claims staging",
  "ownerId": "<owner-user-id>",
  "vault": {
    "name": "Claims staging vault",
    "vaultUrl": "https://customer-runtime-vault.vault.azure.net",
    "privateCidrs": ["10.20.30.4/32"],
    "assignments": [
      { "family": "environment", "prefix": "customer-workload-" },
      { "family": "environment", "prefix": "se-", "writable": true },
      { "family": "workspace_byok", "names": ["existing-openai-key"] }
    ],
    "defaults": ["environment"],
    "links": [
      { "family": "environment", "key": "API_TOKEN", "secretName": "customer-workload-api-token" },
      { "family": "workspace_byok", "key": "openai", "secretName": "existing-openai-key" }
    ]
  }
}
```

The installation approval supplies the network mode and the server configuration supplies the tenant. Include the actual private endpoint CIDRs for a private vault. Omit `privateCidrs` for a restricted public vault. User-supplied names and prefixes must fall within the installation's approval. Use separate rules for an existing secret prefix and the writable `se-` namespace. Add `workspace_byok` to `defaults` only when that family has a writable rule covering SteelEngine-generated names. Links are read-only and can optionally include an Azure `version`.

After the existing federated sign-in, request an access token for the same SteelEngine API scope used by your workload identity pipeline. In Azure DevOps, run this within the existing `AzureCLI@2` task. Set `STEELENGINE_API_SCOPE` to your resource API's identifier URI plus `/.default`. Keep shell tracing off.

```bash
set +x
set -euo pipefail
token=$(az account get-access-token --scope "$STEELENGINE_API_SCOPE" --query accessToken -o tsv)
printf 'Authorization: Bearer %s\n' "$token" |
  curl --fail-with-body --request PUT --header @- --json @workspace-vault.json \
    "$APP_URL/api/v1/admin/organizations/$ORGANIZATION_ID/workspaces/$WORKSPACE_ID/key-vault"
unset token
```

The installation Admin API key is also supported. To use it, supply `STEELENGINE_ADMIN_HEADER_FILE` through the pipeline's protected secret mechanism with the single `x-admin-key` header. Use one credential per request. The manifest and response contain references and metadata.

```bash
curl --fail-with-body --request PUT \
  --header @"$STEELENGINE_ADMIN_HEADER_FILE" \
  --json @workspace-vault.json \
  "$APP_URL/api/v1/admin/organizations/$ORGANIZATION_ID/workspaces/$WORKSPACE_ID/key-vault"
```

A successful first apply returns HTTP 201 with the workspace ID, slug and connection ID. SteelEngine validates linked Azure secrets before committing the workspace, access rules, defaults and references together. An identical replay returns HTTP 200 with `created: false` and skips the Azure lookup. A conflicting workspace or changed declared configuration returns HTTP 409. Provisioning does not overwrite existing entries, migrate PostgreSQL values or create Azure secret versions.

Workspace provisioning uses the application's existing organization binding and records that identity in the audit log. Subsequent workflow deployment and secret writes use their normal workload grants. Automatic workspace access covers newly created workspaces in the organization. Explicit grants require adding the new workspace. The application and workers continue to use their AKS managed identities for Azure Key Vault access. Use a canary workflow to verify runtime access before deploying workload content.

## Version, cache and outage behavior

Each new unpinned run/invocation performs a live latest lookup on first use and durably selects an exact version. Later steps and retries retain that version. An explicitly pinned reference always selects its stated version. Binding/connection revision changes fail the affected run rather than silently selecting another value.

Only exact versions enter a bounded per-process RAM cache, with a default absolute TTL of 60 seconds and an operator maximum of 300 seconds. It is not a shared Redis/database cache and does not cache a latest pointer. Replicas, cold starts and new unpinned runs still incur vault calls. Cached entries never extend their TTL on a hit; enabled/not-before/expiry and current application authority are checked at use. A valid cache hit may bridge an outage until expiry. Azure-side revocation also depends on cache TTL and Azure propagation. No stale-after-expiry fallback is used.

The native SDK uses bounded read retries and transport deadlines; SET has no blind retry because another version might already have been created. Capacity planning must measure run-start rate, bindings per run, replica count, throttles and cold-start traffic. These choices apply Microsoft's [caching and retry guidance](https://learn.microsoft.com/en-us/azure/key-vault/secrets/secure-secrets) while retaining the installation's explicit rotation and revocation bounds.

## Pending writes and recovery

A managed write reserves a durable operation before calling Azure. Until completion or explicit reconciliation, new unpinned consumers of that binding are gated. Already selected versions and explicit pins may continue with current authorization and validity. Timeouts, restarts and lease expiry do not clear the gate. Azure and PostgreSQL do not share a transaction; a timeout can leave an Azure version/orphan even when the application reports failure.

Inspect the binding's operation id and Azure version metadata/tags with an operator identity. Avoid displaying secret values. Confirm the intended version, then use the privileged UI recovery control or `recover` action to verify and pin that exact version. Recovery increments the binding revision and makes the entry read-only. It does not reopen the old latest-version name, because a delayed writer may still complete. Use the explicit migration action to return a pinned entry to writable storage through a new independent managed name after reconciliation; do not repeat the original SET request as a recovery mechanism.

Backend migrations keep the working source active until SteelEngine verifies and publishes the destination. In **Advanced: migration and recovery**, select the pending migration to verify and pin its destination or retire that attempt. Retiring keeps the original source and reserves the old destination name against reuse. Publication checks that the source and destination configuration still match the staged operation.

For an uncertain write to an active entry, **Retire pending entry** (`retire_pending`) removes its binding and retains its operation and name ownership record. Retiring an active workspace BYOK entry can expose the organization fallback.

Detachment never deletes/purges the Azure secret. Operators own orphan inventories, version retention, soft-delete/purge protection and cleanup. Keep ownership records and references for the entire retry/resume and backup recovery window. Cleanup of Azure objects is an independent, approved operator action.

## Verify before customer activation

Run this matrix from the actual deployed worker roles, first in a restricted public profile and then the approved private profile. Record installation/chart revision, pod/service account, identity client id, binding/revision, selected version and outcome; never include tokens or values in the evidence.

* Create a nonproduction canary environment value through SteelEngine. Confirm it is in the configured vault and that PostgreSQL contains only the reference/operation/version metadata for that entry. Link a separate pre-seeded name without importing it.
* Execute an actual workflow using the variable in a Function/custom-code block and a following consumer. Confirm its real authorized output is correct. Retry/resume the same execution after rotation and verify it retains the selected version. A new unpinned run must observe the new version.
* Configure workspace and organization BYOK for existing provider adapters. Exercise model/tool calls and embeddings/OCR where used. Remove the workspace override and verify the organization fallback; deny a configured source and verify failure rather than platform fallback.
* Run a worker/pod longer than the identity token lifetime and execute new uncached reads after renewal. Restart a pod and repeat cold-cache reads. Token exchange alone is insufficient; prove get and workload use.
* With the reader identity, prove get succeeds and list/set/delete/purge fail. With the management writer, prove the approved get/list/set operations work and delete/purge fail. Prove the runtime service account cannot exchange for the writer identity.
* Exercise an unassigned workspace/family/name, a broad-prefix cross-workspace managed name, a disabled binding/connection, revoked caller/workload authority on a warm cache, and expired/not-yet-valid/disabled Azure versions.
* Interrupt a managed SET after Azure accepts it but before local completion. Restart the writer. Verify new unpinned consumers stay gated and prior exact versions remain stable; reconcile by explicit version pin.
* Block Key Vault egress, test valid cached use and absolute expiry, then recover. Validate private DNS never falls back to public addresses. Test Entra connectivity failures separately.
* Use a unique canary to scan logs, trace displays, diagnostic errors, audit records, management responses and the reference-only binding/selection tables. None may contain its value. Check Function stdout and errors, including JSON-escaped and URL-encoded forms. Actual authorized Function results and provider requests must remain correct. A masked UI alone is insufficient evidence.

## Upgrade, disable, rollback and backup recovery

Apply the additive database migrations and deploy backend-aware code to **all** web and worker hosts before creating vault bindings. Verify normal runs/retries/resumes, then enable Key Vault and approve connections. Drain older hosts that would interpret a vault reference as a literal value before activation.

To disable the feature, stop new vault-backed writes/admissions, inventory live bindings and durable selections, and drain or explicitly migrate affected entries. Setting `keyVault.enabled: false` makes remaining vault references unavailable; it never switches them to PostgreSQL. Do not roll back to binaries that lack vault bindings while affected entries or resumable executions still depend on them.

Coordinate PostgreSQL PITR and Azure secret/version retention. Preserve the installation’s existing PostgreSQL credential encryption key through its normal backup process. A restored database may reference an Azure version or pending operation from another point in time. Restore those exact versions before resuming; validate binding revisions and reconcile pending operations explicitly. Never repair a missing retained version by selecting arbitrary latest. Keep the required Azure versions and soft-delete recovery window long enough for the agreed recovery point objectives.

Diagnose in this order: installation enablement, current application scope/permission, service account subject and projected token-file presence, Entra exchange, exact vault DNS/TLS/network policy, Azure data-plane roles, selected version validity, and pending operation state. Log identifiers and fixed error categories only. Never enable HTTP body/token logging or copy mounted tokens into a support ticket.
