Release Operations and Acceptance
Prepare monitoring, recovery, and a client acceptance rehearsal for organization releases
Operate releases using durable IDs and observable outcomes. A browser timeout, successful Git merge, or green Kubernetes health probe does not independently establish that the intended organization generation is serving.
The supported organization Git path is described in Git Approval and Release Status. The UI supports coordinated rollback of an eligible retained organization release, with fresh checks and a durable receipt. The broader blue/green and automatic reconciliation procedures below remain acceptance requirements. For standalone signed-file transfers, use Current CLI Rehearsal. Do not describe planned acceptance cases as tests already passed.
Recover a Supported Organization Operation
Refresh status after a timeout before issuing another action. Commit and PR creation recover the retained workspace snapshot; organization deployment and rollback return the saved generation when the same operation already committed. If preparation or activation fails inside the transaction, all workspaces keep their prior active pointers.
To restore a previous workflow release, use Earlier releases → Check rollback readiness, review the complete plan, and select Restore N workspaces. Expired plans, revoked access or trust, unavailable target resources, and changed organization inventory block the operation. A rollback restores retained executable versions and assignments; it does not restore current drafts, resource values, or runtime data. Whole-workspace removal and unsupported content types remain outside this release path.
Execution recovery is separate from release recovery. New runs can recover an interrupted Slack → Get User Presence operation automatically after Temporal detects worker loss. Preparation retains the reviewed read-only tool version; retries use the saved graph and checkpoints. Completed steps reuse their recorded results, and a replaced worker cannot overwrite the successor's result. An interrupted read may observe newer provider data when retried. The default heartbeat timeout is five minutes, so recovery is not immediate.
This policy applies only to explicitly reviewed read operations, beginning with Slack user presence. Older prepared runs keep their original policy. A worker without the retained tool contract stops before dispatch. Other integrations, generic HTTP requests, and external writes keep their existing conservative policy; an in-progress or ambiguous-outcome error requires checking the provider's outcome before starting a fresh run. Do not clear the fence or replay a write blindly. Provider-deduplicated write retries and an operator resolution UI are not implemented by this increment.
A client timeout alone does not establish that the execution failed; refresh its recorded status first. Existing executions retain their selected workflow version even if an organization rollback changes the version used by new runs.
Release Record
Retain the organization identity, complete workspace inventory, candidate digest, source commit, approving PR/commit, target environment, preflight report and plan hash, operation IDs, active generation, binding revisions, validation evidence, and previous rollback candidate. Record the chart revision and image digests separately.
The audit trail should identify capture, review, apply, binding changes, activation, retirement, and rollback actors. Reference provider accounts only to the extent the reader is authorized to see them. Export safe error codes and correlation IDs rather than secret values or raw provider responses.
Monitoring
| Signal | Investigate when | First check |
|---|---|---|
| Desired commit versus active digest | A reviewed release remains unapplied beyond the agreed window | Reconciliation, readiness blockers, operation status |
| Git delivery and reconciliation freshness | Deliveries fail or no successful check is recorded | Signature/installation setup, network access, scheduled recovery |
| Worker compatibility and readiness | Required workers disappear or contracts disagree | Rollout state, queue polling, lease freshness, installed build |
| Integration readiness | Authorization, scopes, account identity or destination changes | Target binding and provider account; avoid fallback to development |
| Trust freshness | Refresh deadline approaches or clock rollback is reported | Independent trust refresh and clock synchronization |
| Notifications | Jobs remain pending/retrying after healthy sweeps | CronJob authentication and Temporal admission |
| Cutover | Routing and content generation disagree | Fence new work and inspect the original cutover operation |
| Storage and retention | Capacity approaches its operational limit | Retained releases, objects, runs and rollback evidence |
Choose alert thresholds from the client's release frequency, recovery objectives, and measured environment behavior. This guide does not promise a fixed recovery time or throughput.
Incident Decisions
| Observation | Safe next step |
|---|---|
| PR merged, production unchanged | Inspect target readiness and operation status; merge alone does not establish activation |
| One workspace cannot prepare | Leave the previous whole-organization generation active; repair the blocker and revalidate |
| Timeout with an operation ID | Read that operation; a disconnected client does not cancel server work |
| Lost admission response | Retry identical input with the same idempotency key against the same installation |
| Stale plan after a binding, policy or permission change | Generate and review a fresh plan; do not edit the previous report |
| Product committed, notification failed | Recover notification delivery; do not repeat the product mutation |
| Green unhealthy before cutover | Keep current production serving; repair or discard the candidate |
| Routing changed but content activation uncertain | Read durable content and routing results; contain admission before retrying or reversing either |
| Previous release cannot roll back | Inspect trust, retention, resources and compatibility; use an approved forward repair if necessary |
| GitHub unavailable | Preserve the last active organization generation and retry verification when GitHub recovers; workspace file activation cannot bypass the organization boundary |
Do not repair an incident by changing database version pointers, editing trust history, disabling secret checks, or reusing an idempotency key with changed input.
Backup and Recovery
Back up and restore the product database, retained release/object storage, encryption keys, signing/trust material, and Temporal state as a coordinated system. A database-only restore can leave references to missing objects or executions that cannot resume. Test recovery on isolated infrastructure and preserve the installation identity when recovering that same installation.
The reviewed foundation has no general historical release-retention cleanup or supported trust-authority replacement procedure. Abandoned staged-object cleanup is separate and does not remove retained release history. Plan capacity and escalate lost or compromised authority material through an agreed recovery process before relying on promotion in production.
Client Acceptance Rehearsal
Use at least three workspaces: one changed, one unchanged, and one with a new required connection. Include safe examples of every object/tool type the client intends to promote. Record unsupported types as blockers, not skipped successes.
- Capture all workspaces and prove the inventory is complete despite pagination and access boundaries.
- Open a PR against the configured development branch, then promote the exact validated payload to the configured production branch.
- Change the PR head and prove prior checks/review evidence cannot authorize the new candidate.
- Leave one production integration unbound and prove no workspace becomes newly active. Bind it locally, revalidate, then activate the complete organization.
- Replay and reorder GitHub deliveries, miss one deliberately, and verify reconciliation converges without duplicate activation.
- Interrupt preparation and activation around the durable commit. Recover the same operation and demonstrate either the old or the new complete generation, never a mixed one.
- Keep an execution running through cutover, exercise a retry/resume, and verify its retained version and compatible worker remain available.
- Exercise scheduled and webhook work through the handoff. Verify fencing, deduplication, and agreed schedule catch-up behavior.
- Roll back an eligible release, then demonstrate a deliberately ineligible rollback is blocked with a useful reason.
- Restore the coordinated backup in an isolated environment and verify retained releases, credentials, execution state, and rollback evidence.
The full rehearsal must run against separate live installations and the client's Kubernetes topology. Unit tests, mocked webhooks, and one shared test database are useful earlier checks but cannot establish this acceptance.
Handoff Checklist
Record an accountable owner and evidence link for: deployment topology; build/chart versions; workspace mappings; integration inventory; Git protections; target activation policy; trust refresh; backups and restore results; monitoring and on-call routing; rollback window; recovery objectives; known limitations; and the signed acceptance result.
Keep production activation disabled until the required cases pass on the delivered build. Share the resulting evidence and this versioned guide with the client's platform team, administrators, and on-call engineers.