Partial failure is the default in AI automation

The Most Dangerous Result Is Almost Successful

A completely failed automation is usually obvious. The request returns an error, nothing changes, and someone investigates. The more difficult incident is the workflow that succeeds in three places and fails in the fourth.

Imagine a document-processing flow. A file is uploaded, text is extracted, a model produces a structured answer, the answer is written to the database, and a notification is sent. The model call succeeds. The database update succeeds. The notification provider times out. What is the state of the operation?

From the backend's perspective, most of the work is complete. From the user's perspective, nothing happened. If a worker retries the entire job, the model may run again, the database may receive another version, and the notification may eventually be sent twice. If the worker does not retry, a valid result remains invisible.

This is partial failure, and it is not an unusual corner case in AI automation. It is the normal consequence of connecting several independent systems. Every model provider, queue, database, storage service, email API, and internal application can succeed or fail independently. The more useful the workflow becomes, the more boundaries it tends to cross.

I used to think about an automation as a line of code moving from input to output. In production, it is closer to a sequence of commitments. Each step changes what the system knows or what the outside world has seen. Reliability depends on making those commitments explicit.

A Green HTTP Response Proves Very Little

A successful response from a model provider proves that one request returned a response. It does not prove that the response was valid, persisted, delivered, or safe to use.

The provider can return syntactically correct JSON with a missing business field. The application can parse it and then lose its database connection. The database can commit while the worker loses its acknowledgement to the queue. An external system can accept an update and time out before confirming it. The browser can disconnect while the backend continues processing.

These are not necessarily bugs in the individual components. Distributed work creates moments when one side knows something the other side does not. A timeout means uncertainty, not always failure. Retrying an uncertain action without checking can be just as dangerous as abandoning it.

This changes how I read success metrics. A dashboard showing a 99% model-call success rate may look healthy while users are missing results because the next stage fails. A job marked completed may have produced an output that validation rejected later. A notification metric may be green even though it contains a link to a record that was never committed.

The useful unit of success is the business outcome. Did the requested document become available to the right person in an acceptable state? Everything below that is a stage metric. Stage metrics help diagnose the system, but they should not be confused with the outcome.

Start With A Durable Operation

Before doing expensive or external work, I create a durable record for the operation. It gets a stable identifier and an initial state such as pending. That record becomes the place where the workflow reports what it has done.

The record does not need to contain every implementation detail. It should contain enough to answer practical questions: who requested the work, what input version it used, what state it is in, which stage is active, when it last progressed, what output it produced, and whether a person needs to intervene.

This is more useful than relying only on a queue payload. Queue messages are excellent for delivering work, but they are not the product's source of truth. They can be retried, delayed, moved to a failed-jobs table, or replaced during deployment. The operation record survives those infrastructure events.

A stable operation ID also connects the layers. It belongs in structured logs, provider metadata where supported, generated artifacts, notifications, and support tools. When someone says that a result did not arrive, the investigation begins with one identifier instead of a timestamp and a guess.

The operation is not merely a log entry. It is part of the domain. It represents a promise the application made to complete some work or explain why it could not.

Model The Workflow As States, Not A Boolean

A single processed flag cannot describe partial failure. It forces every intermediate condition into false and makes recovery ambiguous.

I prefer explicit states that reflect meaningful progress. A document flow might move through pending, extracting, generating, validating, ready, and notified. It may also enter retry_wait, needs_review, or failed. The exact names matter less than having legal transitions and clear ownership.

States make impossible combinations visible. If the operation is notified, a result should exist. If it is ready, validation must have passed. If it is needs_review, the interface should show what decision is required. Those invariants can be tested.

I also separate technical progress from the final business status when necessary. A model request may be technically completed while the output is unsuitable. Calling that operation successful hides the most important fact. The system should record that generation finished and that business validation did not pass.

Explicit states improve the frontend too. Instead of displaying an endless spinner, the application can say that the file was read and the result is being checked. If recovery requires a person, the interface can offer a specific action rather than a generic error.

Checkpoints Make Retries Smaller

A long job that repeats from the beginning after every error is easy to write and expensive to operate.

Suppose text extraction took two minutes and the model call cost real money. If the final notification fails, neither step should run again. Each significant stage should leave a checkpoint: extracted text, model response, validated data, generated artifact, or delivery receipt.

On retry, the worker loads the operation and asks which durable checkpoint already exists. It resumes from the first incomplete stage. This makes recovery faster and reduces the number of opportunities for a different answer to replace a valid one.

Checkpoints need version information. Reusing an old model response after the input changed is not recovery; it is a data integrity bug. I associate intermediate results with an input fingerprint and, when relevant, a prompt or workflow version. A stage can be skipped only when its checkpoint belongs to the same logical operation version.

There is a tradeoff. Persisting every tiny step creates noise and coupling. I checkpoint across expensive calls, irreversible side effects, and boundaries that are likely to fail. A local data transformation that takes milliseconds can usually be repeated. A paid model request or an email sent to a customer deserves durable evidence.

Idempotency Is A Business Rule

Retrying safely requires more than making database inserts unique.

For every side effect, I ask what makes two attempts the same logical action. It might be one generated report for a specific document version, one status update for an operation, or one notification that a result became ready. That identity becomes an idempotency key.

At the database layer, a unique constraint can prevent duplicate records. At a provider boundary, an API idempotency key can prevent duplicate requests when supported. For a service that offers no such mechanism, the application may need an outbox record or a delivery ledger that is checked before sending.

The difficult case is a timeout after an external service accepted the action. The application does not know whether retrying will duplicate it. A good integration provides a way to query by the stable key or reconcile later. If it does not, the workflow should acknowledge that uncertainty and route high-impact actions for review instead of pretending that another attempt is harmless.

Idempotency is therefore defined by the product, not by the queue library. The infrastructure can redeliver a job. Only the application knows whether two deliveries represent one intended email, one intended payment, or two separate actions.

Keep Database Changes And Messages In Agreement

One common partial failure happens between a database commit and publishing the next job or event.

The application saves a generated result and then dispatches a notification job. If the process crashes between those actions, the result exists but no notification will ever be sent. Reversing the order is not better: the notification worker may start before the result commits.

The transactional outbox pattern solves this boundary. In the same database transaction that saves the result, the application adds an outbox entry describing the event to publish. A separate process reads unsent outbox entries, dispatches them, and records delivery. The database guarantees that the result and the intention to notify either both exist or neither exists.

This does not make the external notification exactly once. The dispatcher can still crash after sending and before marking the outbox entry delivered. The notification operation must remain idempotent. The outbox solves atomic intent; the idempotency key solves repeated execution.

I do not add an outbox to every small application on day one. I use it when losing the event would leave the product in a silent, inconsistent state. The pattern earns its complexity when a database change and a later action must stay logically connected.

Validation Is Its Own Failure Boundary

AI output introduces a category that traditional integrations have less often: a request can succeed technically and fail semantically.

A model may return the required JSON shape while inventing a value, ignoring a constraint, selecting an unsupported category, or producing text that should not be sent automatically. Treating parsing as validation is not enough.

I put deterministic checks after generation whenever the domain allows them. Required fields, allowed values, date ranges, cross-field relationships, permission rules, and references to source material can all be checked without another model call. For riskier decisions, a confidence signal may determine whether the workflow continues or asks for review.

A validation failure should not automatically retry the same prompt five times. If the input and instructions are unchanged, repetition can create cost without changing the condition. The policy might use one constrained repair attempt, choose a fallback model, or move the operation to needs_review.

Separating generation from validation makes metrics honest. We can see that the provider was available while the outputs failed product rules. That distinction guides the correct fix: prompt changes, schema changes, better source data, or a different automation boundary.

Compensation Is Not A Time Machine

Some workflows need to undo earlier work after a later stage fails. In distributed systems this is often described as compensation.

Compensation is a new action, not a rollback across the entire world. Deleting a provisional record can reverse a database effect. Revoking a generated link can remove access. Sending a correction can address an earlier notification. None of these makes the original event disappear.

This matters when designing side effects. I delay irreversible actions until validation and approvals are complete. When possible, I create drafts before publishing, reservations before confirmations, and disabled artifacts before exposing them. The system keeps an opportunity to stop safely.

Compensating actions need their own idempotency and observability. They can fail too. A workflow that says "if anything breaks, undo everything" has merely moved the difficult problem into an untested error branch.

I reserve compensation for cases where the business truly needs it. Sometimes the honest response is to preserve the completed result, mark delivery as failed, and let the user retry delivery. Reversing correct work because a later convenience step failed can create more harm than the original partial failure.

Every Failure Needs An Owner And A Next Action

A status called failed is incomplete unless someone knows what should happen next.

Transient infrastructure failures belong to automated retries with bounded backoff. Invalid input belongs to the user who can correct it. A suspicious or low-confidence result may belong to a reviewer. An authentication failure belongs to an operator. A new programming error belongs to the engineering team.

I want the operation to record a safe failure category, not only an exception message. The category drives the next action: retry automatically, edit input, reconnect an integration, approve a result, or contact support with the operation ID.

Retries need an end. Infinite retries can hide a permanent failure while consuming capacity and money. After the automated policy is exhausted, the operation should become visible in a queue that a person actually owns. A dead-letter table nobody checks is storage, not recovery.

This ownership also changes alerting. A single transient provider timeout may need no alert. A growing number of operations stuck in the same stage does. The useful signal is often the age of the oldest unfinished operation and the number of users waiting, not the raw count of exceptions.

Test The Gaps Between The Happy Steps

Happy-path tests prove that the workflow can work. Reliability tests prove that it can stop and continue.

I test failures after each durable boundary: after the provider responded but before persistence, after persistence but before event publication, after an external side effect but before acknowledgement, and during a retry of every stage. The assertions are about outcomes: no duplicate notification, no second model charge when a checkpoint exists, no completed state without a validated result.

These tests are easier when dependencies are behind explicit interfaces and operation state is visible. A fake provider can return a response and then simulate a network exception. A notification adapter can record that it accepted a key but withhold the acknowledgement. A worker can be run twice against the same operation.

Production observability should answer the same questions. Which stage is stuck? How many attempts occurred? Which checkpoints exist? Was the external action accepted? What can safely happen next?

Chaos does not need to mean randomly killing an entire cluster. A focused test that terminates a worker between two lines of important code can reveal more about the workflow's promises.

Design For The In-Between State

Reliable AI automation is not a perfect chain where every component succeeds together. It is a system that can describe incomplete work, avoid repeating completed work, and guide the remaining work to a safe conclusion.

The practical tools are not exotic: durable operation records, explicit states, checkpoints, idempotency keys, transactional outboxes, deterministic validation, bounded retries, and human recovery paths. Their value comes from applying them at the boundaries where the application can lose certainty.

This design may feel slower than putting five API calls in one job and catching exceptions at the end. It becomes faster the first time a provider times out after accepting a request, a worker restarts during deployment, or a customer asks why they received the same message twice.

The goal is not to eliminate partial failure. No architecture can force independent systems to fail as one. The goal is to make partial failure observable, recoverable, and unsurprising.

An automation earns trust when it can answer three questions at any moment: what has definitely happened, what has not happened yet, and what is safe to do next.

In production, an AI workflow is not one transaction. Reliability comes from knowing which promises have already been kept.