Migrating AI workflow state without stopping every active agent

AI workflow state migration becomes difficult at exactly the moment the system becomes useful. A few short-lived runs can be stopped and restarted. Thousands of agents waiting for approvals, retrying tools, sleeping until deadlines or holding partially completed work cannot.

The tempting plan is to deploy a database migration, backfill every row and switch the code. That plan assumes persisted state is passive. In a long-running workflow, it is not. Workers are reading it, timers are waking it, webhooks are changing it and external tools may still be completing actions while the backfill runs.

I treat a live workflow migration as a distributed protocol, not a data-cleaning script. The goal is not zero pauses everywhere. The goal is to avoid a global stop by moving each run through a short, explicit safe point while the rest of the system continues.

Workflow state is more than a database row

A normal schema migration changes stored data. A workflow migration changes the meaning of stored data while code is actively making decisions from it.

Consider an agent that reviews a document, asks for approval and then publishes a result. Version 3 stores one field called review_status. Version 4 splits it into policy_decision, human_decision and publication_status. Converting the JSON is easy. Deciding what an old value of approved means while a delayed webhook arrives is the real problem.

A live run usually spans several consistency surfaces:

This is why updating rows in place is insufficient. The migration has to establish which writer owns a run, which schema it may write and how delayed inputs are interpreted.

The distinction also separates this topic from versioning long-running AI workflows. Versioning decides which contracts a run uses. Migration is the controlled operation that moves a particular run from one compatible contract to another.

Define the migration contract before writing the converter

A converter such as V3ToV4 is only one part of the design. Before it runs, I want a migration contract that answers five questions:

1. Eligibility: which workflow names, versions and states can move?
2. Safe point: what must not be in progress when conversion starts?
3. Transformation: how is old state mapped, and which invariants must hold afterward?
4. Compatibility: which old messages and events can the new version still accept?
5. Recovery: can the run roll back, move forward through a repair, or require review?

Put those answers in code and metadata, not only in a release ticket. A useful migration definition is immutable and named:

final class ReviewWorkflowV3ToV4 implements WorkflowStateMigration
{
    public function fromVersion(): int { return 3; }
    public function toVersion(): int { return 4; }

    public function eligible(WorkflowRun $run): bool
    {
        return $run->workflow_name === 'document-review'
            && in_array($run->status, ['waiting', 'ready'], true)
            && $run->open_operations === 0;
    }

    public function migrate(array $state): array
    {
        return [
            ...$state,
            'policy_decision' => $state['review_status'] ?? 'pending',
            'human_decision' => null,
            'publication_status' => 'not_started',
        ];
    }
}

The converter should be deterministic. The same old snapshot and migration version must produce the same new snapshot. It should not call an LLM, inspect current time, query a mutable policy table or trigger an external tool. Those dependencies make retries impossible to reason about.

Use an explicit state envelope

Do not infer a schema from whichever fields happen to exist. Persist an envelope around the domain state:

{
  "workflow": "document-review",
  "definition_version": 3,
  "state_schema_version": 3,
  "revision": 184,
  "migration": null,
  "data": {
    "review_status": "approved"
  }
}

The definition version and state schema version are separate because control flow and storage do not always change together. The monotonic revision protects against stale writes. The migration field records intent and progress instead of forcing operators to reconstruct them from logs.

A six-phase protocol for migrating active agents

The safest rollout expands compatibility first and removes old behavior last. The pattern resembles a zero-downtime database migration, but it adds per-run coordination.

1. Observe and classify the fleet

Before changing anything, count runs by workflow version, schema version, state and age. Identify open tool operations, future timers, queued payload versions and human tasks. This inventory reveals whether a supposedly simple migration includes runs that have been sleeping for six months.

Define cohorts with stable predicates: tenant range, creation date, workflow type or a hash of the run ID. Never select a fresh random sample on each retry. You need to know exactly which runs were intended to move.

2. Deploy compatibility readers

New workers must understand both the old and new state before any run is converted. Event handlers should accept the supported old payload envelope too. This is the expand phase: add without removing.

Prefer a clear adapter boundary:

$state = match ($run->state_schema_version) {
    3 => StateV3::fromArray($run->state),
    4 => StateV4::fromArray($run->state),
    default => throw new UnsupportedStateVersion($run->id),
};

Do not silently fill every unknown shape with defaults. Compatibility code should be tolerant of an expected previous version and loud about anything else.

3. Request a safe point per run

Set a migration request on an eligible cohort. The next worker that owns a run finishes its current deterministic transition, persists the result and moves the run to migration_pending. New dispatches for that run stop, but unrelated agents continue normally.

A sleeping run can usually be claimed directly. A busy run may need seconds or minutes to reach its boundary. A run waiting on an ambiguous external side effect is not safe and should remain deferred until tool-call reconciliation reaches a known outcome.

4. Convert with compare-and-set protection

The migrator reads the state at revision 184, produces version 4, validates it and writes only if the revision is still 184 and the run is still fenced for this migration. If either condition changed, it aborts and reevaluates rather than overwriting newer work.

UPDATE workflow_runs
SET state = :new_state,
    state_schema_version = 4,
    definition_version = 4,
    revision = revision + 1,
    migration_status = 'migrated'
WHERE id = :id
  AND revision = :expected_revision
  AND migration_token = :token
  AND migration_status = 'converting';

One affected row means the ownership assumptions were still true. Zero means something changed; it is a concurrency signal, not permission to force the write.

5. Resume a canary cohort

Remove the per-run barrier and route the migrated run to workers that understand version 4. Start with a small cohort. Compare transition failures, stuck-run age, tool-call duplication, latency, cost and human escalation rate against version 3.

Increase the cohort only after a full business cycle, not merely after a green deployment. If workflows commonly wait overnight for approval, ten quiet minutes prove very little.

6. Contract only after the old population reaches zero

Once no active, sleeping, retryable or manually paused run needs version 3, remove old writers first. Keep old readers for a defined retention period to support delayed events and forensic replay. Only then remove legacy fields, handlers and indexes.

The deletion condition should be a query or metric, not a date in a calendar. “Two weeks have passed” does not prove that the oldest timer or dead-lettered message has disappeared.

Safe points need leases and fencing, not hope

A status field called paused does not stop a worker that already loaded the row. The classic race looks like this:

1. worker A reads revision 184 and starts a slow transition;
2. the migrator marks the run paused and converts it to revision 185;
3. worker A finishes and writes its version 3 snapshot over version 4.

Database row locks can serialize short critical sections, but a transaction should not stay open around a network or model call. For work that crosses process boundaries, use a lease plus a fencing token.

Every successful claim increments a token. The worker includes that token when committing. A later migration claim receives a larger token, so writes from the earlier worker are rejected even if its lease timeout was misjudged.

UPDATE workflow_runs
SET state = :state,
    revision = revision + 1
WHERE id = :id
  AND owner_token = :worker_token
  AND revision = :expected_revision
  AND migration_status IS NULL;

The safe point is therefore an enforced invariant: no open transition owns a valid token, no external result is unresolved, and no new work can be dispatched until conversion commits.

A practical Laravel migration coordinator

Laravel provides transactions, queue middleware and scheduler locks, but the application still needs domain-level coordination. I separate three jobs:

• RequestWorkflowMigration selects a bounded cohort and marks intent.
• MigrateWorkflowRun claims and converts one run transactionally.
• VerifyWorkflowMigration checks invariants and releases the run.

The conversion job can remain small:

DB::transaction(function () use ($runId, $migrationId) {
    $run = WorkflowRun::query()->lockForUpdate()->findOrFail($runId);
    $migration = $this->registry->get($migrationId);

    if ($run->migration_status === 'migrated') {
        return; // idempotent redelivery
    }

    if (! $migration->eligible($run)) {
        $run->deferMigration('not_at_safe_point');
        return;
    }

    $before = $run->state;
    $after = $migration->migrate($before);
    $migration->assertInvariants($before, $after);

    $run->completeMigration(
        migrationId: $migrationId,
        newState: $after,
        checksumBefore: hash('sha256', canonical_json($before)),
        checksumAfter: hash('sha256', canonical_json($after)),
    );
});

The job should have a unique business key such as run_id + migration_id. Laravel queue attempt counts are not enough because jobs can be recreated. Persist migration attempts, last error and next eligible time with the run.

Process in small chunks with a concurrency cap. A backfill that saturates the primary database can stop agents more effectively than a maintenance window. Measure lock wait, replica lag, query latency and queue delay, then throttle automatically.

For commands emitted after migration, use a transactional outbox. The converted state and the first version 4 command should commit together. Otherwise the run can appear resumed without the work needed to move it forward.

Handle approvals, timers and tool calls as versioned inputs

Human approvals

An approval link may outlive several deployments. Give the task its own immutable ID and payload version. After migration, translate the old approval response through a compatibility adapter or replace the task explicitly and invalidate the old link. Do not accept two live approvals for the same decision.

Timers and delayed jobs

A timer should wake a run by ID and event type, then let current compatible code load the state. Avoid serializing an entire workflow object into a delayed job. If old payloads already exist, keep their decoder until the maximum delay plus dead-letter retention has passed.

In-flight tool calls

Never migrate across an operation whose outcome is unknown. First reconcile it using the original idempotency key and correlation ID. Then map the confirmed result into the new schema. Creating a new version 4 operation because the old record is inconvenient risks repeating the real-world action.

Prompt and model artifacts

A state migration should not casually change the evidence under which an earlier AI decision was accepted. Preserve prompt, model profile, policy and output-schema identifiers in the audit history. Version 4 may use new artifacts for future decisions, but it should not pretend they produced the past.

Verification and rollback are separate phases

A successful SQL update proves only that bytes changed. Verification should assert business invariants:

• every completed external operation still has exactly one durable identity;
• the new current step is reachable under the version 4 graph;
• required fields are populated from evidence rather than invented defaults;
• pending approvals and timers still point to the same logical decisions;
• no command already acknowledged by a provider is emitted again;
• the audit record connects old checksum, new checksum and converter version.

Run the converter offline against sanitized historical snapshots before production. In production, shadow-convert a sample and compare invariant results without persisting them. This finds old shapes that fixtures missed.

Prefer roll-forward after new work begins

A byte-for-byte reverse converter is useful before a migrated run resumes. After version 4 has executed transitions or called tools, reversing state may erase facts that version 3 cannot represent. At that point, a tested repair migration or manual review is safer than a blind rollback.

Keep the pre-migration snapshot immutable for audit and diagnosis, but do not automatically restore it over newer state. A snapshot is evidence, not necessarily a valid recovery point.

Metrics that decide whether to continue

I want one dashboard grouped by migration ID and cohort:

• requested, waiting-for-safe-point, converting, verified, resumed and failed counts;
• age of the oldest run in each phase;
• compare-and-set conflicts and stale fencing-token rejections;
• invariant failures by reason;
• database lock time, replica lag and queue latency;
• duplicate-operation alarms and manual-review volume;
• version 3 versus version 4 completion rate, cost and transition latency.

Define pause thresholds before rollout. If invariant failures exceed zero for a destructive mapping, stop the next cohort automatically. The current cohort can remain isolated while unaffected agents keep running.

Production checklist

• Inventory active runs, timers, human tasks and external operations.
• Persist definition, schema and payload versions explicitly.
• Deploy old-and-new readers before producing new state.
• Use deterministic, side-effect-free converters.
• Move each run through an enforced safe point.
• Protect commits with revisions, leases and fencing tokens.
• Defer runs with ambiguous external outcomes.
• Migrate stable cohorts and canary through a full business cycle.
• Validate business invariants, not only row counts.
• Throttle on database and queue health.
• Prefer a forward repair once the new version has performed work.
• Remove old writers before old readers, and retire both with evidence.

Do not stop the fleet; create a safe boundary per run

There is no magic switch that makes a breaking state change safe. There is a protocol: expand compatibility, fence one run, reach a safe point, convert deterministically, verify, resume and observe.

That protocol turns a global maintenance event into many small controlled transitions. Most agents keep working. The difficult runs wait for evidence instead of being forced through. And every migration leaves enough history to explain what changed.

If you are building a Laravel or PHP agent platform with workflows that survive deployments, I can help review its state model, migration protocol and failure boundaries. See my consulting work or contact me.

---

Sources and methodology: the migration protocol was checked against the official Temporal safe deployment guidance, Temporal Worker Versioning documentation, PostgreSQL explicit locking documentation, and GitLab's zero-downtime migration guidance. Sources checked 27 October 2026. The per-run migration protocol, Laravel examples and AI-tool boundary are original synthesis based on production-oriented PHP and agent workflow design. This article was prepared with AI assistance and reviewed and edited for technical accuracy and practical usefulness. Cover image: Pixabay.