# Blue/Green Operation

Plan coordinated content activation and Kubernetes traffic changes without duplicating work

Blue/green keeps the current production version available while a candidate is prepared and validated. Development is the authoring environment; blue and green are production slots, not development and production themselves.

This is a proposed operating contract. The reviewed Helm chart uses Kubernetes Deployments and does not provide an organization-aware blue/green controller, slot-specific release admission, or trigger fencing. Do not deploy two copies against live production state and assume an ingress switch supplies those guarantees.

## Keep Two Version Boundaries Visible

| Boundary                     | Identifies                                                                    | Recovery                                                        |
| ---------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------- |
| Organization content release | Exact workspace/object inventory, payload digest, bindings, active generation | Validated content rollback or forward repair                    |
| Infrastructure release       | Chart revision, image digests, migrations, routing and worker contracts       | Customer's compatible infrastructure rollback or forward repair |

A content-only release may use the existing compatible runtime without replacing containers. A container upgrade may leave organization content unchanged. When both change, record their compatibility and deployment order rather than conflating their version numbers.

## Recommended Production Shape

Keep development data isolated. Within production, prefer a single authoritative persistence and execution state with compatible runtime slots and a stable public origin. Green preview needs an explicit candidate execution context; it must not become live merely by connecting to the production database.

The future organization coordinator must choose which generation admits new work. The customer ingress or rollout controller chooses the runtime receiving traffic. These are distinct operations, and no database transaction can atomically commit a Kubernetes routing change.

A fully separate database per slot changes the problem: the client needs a separately designed replication, write ownership, consistency, and disaster-recovery strategy. Copying development data into green would overwrite production state and is outside content promotion.

## Planned Cutover Sequence

1. **Prepare.** Retain the exact approved candidate and previous production generation. Stage every workspace while the current generation continues serving.
2. **Check compatibility.** Verify all bindings, supported objects, workers, database changes, resource dependencies, and rollback eligibility. Keep schemas compatible with both generations during overlap.
3. **Validate green.** Run isolated preview checks against the candidate. Prevent production schedules, webhook subscriptions, queue consumers, and uncontrolled external writes from starting on the inactive slot.
4. **Commit content.** Recheck target revisions and publish one organization generation. All newly admitted work must resolve through that generation. A failed workspace prevents the commit.
5. **Switch routing and ownership.** Coordinate the ingress transition and a fenced handoff for new scheduled and webhook work. Record the content generation and controller result separately. If they disagree, pause affected admissions or recover to the last verified state according to the rehearsed policy.
6. **Drain.** Existing executions finish on their retained versions and compatible workers. Retries, child calls, and resumed work must retain the same release context. A web connection drain is not an execution drain.
7. **Observe and retire.** Check error rates, latency, queue backlog, duplicate events, and business outcomes for the agreed observation window. Remove old workers and retained artifacts only after all dependent runs and rollback obligations are satisfied.

The exact cutover order depends on runtime compatibility. The first supported protocol must require old and new runtimes to understand both relevant content generations, or explicitly pause admission during the transition. “Zero downtime” requires measured evidence for the chosen protocol.

## Prevent Duplicate Effects

Every production trigger needs one authoritative owner. A lease alone is insufficient if an old process can continue acting after its lease expires; admission must reject stale ownership epochs. Preserve webhook delivery deduplication through the handoff. Define schedule catch-up behavior so a paused interval neither silently loses runs nor creates duplicates.

Side-effecting operations need application/provider idempotency where available. Neither a PR nor a traffic switch guarantees exactly-once delivery to an external API. Preview should use safe test accounts or explicit non-mutating checks; production credentials existing in green do not authorize arbitrary smoke-test writes.

## Kubernetes and Helm Boundaries

The current chart starts migration init containers when `migrations.enabled` is true. It also provisions workers and optional CronJobs. A second release can repeat these activities. A blue/green chart/controller design must assign ownership of migrations, singleton jobs, subscriptions, and Temporal queues before it is supported.

Use an independently reviewed rollout controller or customer pipeline for traffic operations. Argo Rollouts, for example, provides active and preview Services for blue/green routing; that does not by itself implement SteelEngine release activation or worker/trigger ownership. See [Argo Rollouts blue/green](https://argoproj.github.io/argo-rollouts/features/bluegreen/) and [Kubernetes Deployment behavior](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/).

## Rollback Boundaries

Content rollback restores a retained, eligible generation. It does not undo sent messages, payments, external writes, production rows, or incompatible database migrations. Old executions may still complete with external effects after new admissions have moved back.

Revalidate current trust, access, credentials, resources, schema compatibility, and worker availability before rollback. Keep a forward-repair path when the previous generation is no longer eligible. Record whether the incident requires a content rollback, infrastructure rollback, data restore, or a combination; each has a different procedure and risk.