Idempotency is the hidden contract of reliable AI automation

The Timeout That Does Not Tell The Truth

An AI agent asks a tool to create a support ticket. The request reaches the provider, the ticket is created, and the network connection disappears before the response returns. From the agent's perspective, the call failed. From the customer's perspective, the action succeeded.

The obvious recovery is to retry. It is also how one request becomes two tickets.

This is one of the most important differences between a demo and a production automation system. In a demo, success and failure are clean states. In production, a timeout means only that the caller did not receive an answer in time. It does not prove that nothing happened.

AI makes this problem more visible because an agent can initiate many different side effects: send a message, update a record, publish a post, book an appointment, start a deployment, or trigger another workflow. The model may choose the correct action and still create damage if the surrounding system treats every retry as a new intention.

The engineering concept that prevents this is idempotency. In plain language, the same intended operation can be submitted more than once without producing the side effect more than once. It is not a small API detail. It is a contract that lets automation recover safely from uncertainty.

At-Least-Once Delivery Is The Normal Case

Queues, webhooks, schedulers, and distributed workers commonly provide at-least-once delivery. They try hard not to lose work, which means they may deliver the same work again. A worker can finish a job and crash before acknowledging it. A queue can decide that a slow job is abandoned while it is still running. A webhook sender can retry because the receiver's response was delayed.

This behavior is not necessarily a bug. Guaranteeing that an event arrives exactly once across unreliable networks is much harder than it sounds. Most infrastructure chooses the practical guarantee: the message will arrive, but the consumer must tolerate duplicates.

An AI agent adds more retry layers. The model may repeat a tool call after an error. The application may retry the model. A queue may retry the entire run. A user may click again because the interface appears stuck. An operator may replay a failed job. Each layer can be reasonable in isolation while the combined system performs the action several times.

I therefore assume that every command can arrive more than once. If a workflow writes to the outside world, duplicate delivery is part of its normal operating environment, not an exotic edge case.

One User Intention Needs One Stable Key

Idempotency starts with identity. The system needs a stable key that represents the intended business operation, not an individual HTTP request or worker attempt.

Suppose a user asks an agent to publish an approved article. The first worker receives the operation, calls the publishing API, and times out. A second worker retries. Both attempts must carry the same operation key because they represent one intention: publish this approved version once.

A random identifier generated inside each attempt does not help. It proves only that two attempts exist. The key must be created before the retry boundary and then preserved through model calls, queue messages, tool calls, and recovery jobs.

I distinguish an operation ID from an attempt ID. The operation stays constant for the user's job. Every execution gets a new attempt ID for observability. This lets me say, “operation 42 had three attempts, but produced one confirmed publication.”

The key should also include the scope that defines uniqueness. A monthly report for May is different from the report for June. An email draft can be regenerated many times, while sending the approved draft should happen once. The business meaning determines the boundary.

Do Not Ask The Model To Detect Duplicates

A language model can reason about a conversation, but it should not be the source of truth for whether a side effect already happened. Context can be summarized, truncated, reordered, or missing. Two attempts may run concurrently before either can tell the other what it did.

Duplicate protection belongs in deterministic infrastructure: a database uniqueness constraint, an idempotency table, a transactional outbox, or a provider's idempotency mechanism. The model may decide that a ticket should be created. Ordinary code must guarantee that one operation key maps to at most one ticket creation.

This separation makes the system easier to test. I do not need to hope that the model notices a previous tool result in a long transcript. I can submit the same command ten times and assert that the database contains one operation and the provider contains one object.

It also keeps prompts focused. Instructions such as “never send twice” express policy, but they cannot enforce concurrency or survive a process crash. Safety properties need a boundary stronger than generated text.

Claim The Operation Before Performing The Work

A common implementation uses a table keyed by the operation ID. The first attempt inserts a record with a status such as processing. A unique constraint prevents another attempt from claiming the same operation at the same time.

If a second request arrives, it reads the existing record. When the operation is complete, it returns the stored result. When it is still processing, it waits, returns an accepted response, or schedules reconciliation. When it failed before creating a side effect, it may be eligible for another controlled attempt.

The record should store more than a Boolean. I want the operation type, safe fingerprint of important inputs, current state, attempt count, timestamps, provider reference, and final result. A key reused with different inputs must be rejected. Otherwise the same identifier could silently refer to two different intentions.

The state transition needs to be atomic. “Check whether the key exists, then insert” can race when two workers check simultaneously. A unique database constraint or atomic insert makes one winner explicit.

This is a small state machine, but it becomes the durable memory of the action. Conversation history can describe what the agent wanted. The operation record says what the system actually committed to doing.

Local Transactions Do Not Cover Remote APIs

When all changes happen in one database, a transaction can update the business record and the idempotency record together. External APIs break that boundary. I cannot wrap my database and an email provider in one ordinary transaction.

The difficult window appears after the provider accepts the action but before my application records success. A crash there leaves the local operation in processing even though the email, payment, ticket, or publication already exists.

If the provider supports idempotency keys, I send my stable operation key with every attempt. The provider can then return the original result instead of creating another object. This is the strongest and simplest arrangement because duplicate protection exists at the side-effect boundary.

If the provider does not support keys, I look for a stable external reference or a way to query the result. A ticket might include my operation ID in a custom field. A publication can be checked by slug and content version. A message provider may accept a client reference. Recovery can search for that reference before attempting another write.

When neither option exists, the operation is not safely retryable. That limitation should be visible in the workflow. It may require manual review instead of pretending an automatic retry is harmless.

Unknown Is A Real Production State

Many applications model an action as pending, successful, or failed. Distributed side effects require another state: unknown.

A timeout after a write request is unknown. The provider may have rejected the request, accepted it, or completed it while losing the response. Marking the operation failed is an unsupported claim. Retrying immediately is a bet with the user's data.

I represent this explicitly with a state such as outcome_unknown. A reconciliation job then checks the provider using the idempotency key or external reference. If it finds the result, it records success. If it confirms that no action exists, it can release a retry. If certainty is impossible, the workflow escalates to a person with the evidence collected so far.

This may feel less elegant than a green or red status, but it is more honest. Reliable systems do not eliminate uncertainty; they keep uncertainty from becoming an unbounded side effect.

Idempotency And Retries Must Be Designed Together

Retry policy is often added as a generic middleware setting: try three times with exponential backoff. That is useful for read operations and dangerous for writes unless the operation is idempotent.

I classify tool calls by consequence. Pure reads can usually be repeated, although cost and rate limits still matter. Local calculations are safe when they do not mutate shared state. Writes with a provider idempotency key are retryable under the provider's retention rules. Writes without duplicate protection require reconciliation or approval.

The error also matters. Validation errors should not be retried. Authentication failures usually need configuration or renewed credentials. Rate limits benefit from backoff. Connection loss before any bytes were sent differs from a timeout after the full request left the process, although libraries do not always expose that distinction clearly.

An agent should not improvise this policy from an error string. Tools can return structured fields such as retryable, side_effect_possible, and reconciliation_required. The orchestration layer then follows a deterministic recovery path.

The Transactional Outbox Protects The Handoff

Another common failure occurs between updating local state and publishing a job or event. The application saves an approved order, then crashes before sending the queue message that starts fulfillment. Or it sends the message first and crashes before marking the order approved.

The transactional outbox pattern solves this handoff. The application writes the business change and an outbox event in the same database transaction. A separate publisher reads unsent outbox rows and delivers them to the queue. Delivery can happen more than once, so the consumer still uses an idempotency key.

This pattern is particularly useful in agent workflows because approval, scheduling, and execution often happen in different processes. The approved action should not depend on a fragile moment where one process updates two systems.

An inbox table on the consumer side provides the matching protection. It records processed message IDs and prevents the same event from applying twice. Outbox plus inbox does not create magical exactly-once networking. It creates an effectively-once business outcome from at-least-once delivery.

Compensation Is Not The Same As Idempotency

Some actions cannot be made perfectly idempotent, and some workflows contain several side effects. In those cases, teams often discuss compensation: if step three fails, reverse steps one and two.

Compensation is useful, but it solves a different problem. Idempotency prevents duplicate execution of one intended action. Compensation performs a new action intended to reduce or reverse the effect of an earlier one.

Sending a second email saying “please ignore the previous email” does not unsend the first. Deleting a duplicated ticket may remove audit history. Refunding a charge creates a separate financial event and may involve fees. Compensation has its own failure modes and needs its own operation key.

I prefer prevention at irreversible boundaries. For multi-step workflows, I define which steps can be retried, which can be compensated, and which require confirmation before the next step begins. A saga can coordinate those transitions, but it still depends on idempotent commands.

Expiration Windows Need Business Meaning

Idempotency records cannot always live forever. Providers may remember keys for 24 hours. Local tables may need retention policies. The correct window depends on how long a duplicate request can realistically return.

A browser double-click may repeat within seconds. A queue can redeliver after hours. An operator may replay a job days later. A delayed webhook can appear after a deployment. Deleting keys too early turns an old retry into a new action.

I choose retention from the workflow's recovery horizon, not from a convenient cache default. High-consequence operations may keep a compact permanent record containing the operation key, input fingerprint, result reference, and timestamps. Less sensitive actions can expire after every producer and retry path is guaranteed to have stopped.

Provider retention must be documented as part of the tool contract. If my queue can retry for seven days but the remote API remembers keys for one day, the system is protected for only one day unless I add local reconciliation.

Test The Crashes, Not Only The Responses

A happy-path test that calls the tool twice is a start. The important tests interrupt the workflow at every boundary.

I test two workers claiming the same operation concurrently. I crash after the local claim, after the remote request is sent, after the provider returns, and before local success is committed. I delay the first attempt until the second starts. I replay a queue message after completion. I reuse the key with different inputs and expect rejection.

The assertion is always about business state: one published article, one ticket, one message, one charge. I also assert that every attempt is visible, the stored result is returned consistently, and uncertain operations enter reconciliation rather than blind retry.

Fault injection is valuable here because the most dangerous window may last only milliseconds. Waiting for production to hit it is not a test strategy. A controlled failure directly after the provider call can prove whether the architecture survives the exact moment we fear.

Observe The Invariant, Not Just The Error Rate

Idempotency needs production signals. I record the operation ID, attempt ID, idempotency decision, state transition, provider reference, and reconciliation outcome. Sensitive payloads are not required to understand whether duplicate protection worked.

Useful metrics include operations with multiple attempts, duplicate requests served from stored results, operations stuck in processing, unknown outcomes, reconciliation duration, and key conflicts caused by different input fingerprints.

The most important alert is a broken invariant: one operation produced more than one external side effect. A low HTTP error rate does not compensate for that. I also alert when unknown operations exceed their recovery window or when a tool classified as non-idempotent receives an automatic retry.

These events connect reliability to user impact. Retries are not inherently bad; they are evidence that recovery is working when the outcome remains singular.

Reliable Agents Need Boring Guarantees

AI agents introduce flexible decisions into software, but the boundaries around those decisions should be deliberately boring. Stable identifiers, unique constraints, state machines, structured errors, outbox records, reconciliation jobs, and provider references are not glamorous. They are what make autonomous execution survivable.

The model can decide what to do. The system must decide whether that intention is new, already in progress, complete, safe to retry, or uncertain. Those decisions cannot depend on the model remembering a previous attempt.

I treat every external write as a contract: one business intention, one stable key, one recorded outcome. If the contract cannot be enforced, the automation should narrow its authority and ask for help.

That is the hidden work behind a reliable agent. Not making every call succeed, but making failure and repetition unable to multiply the consequence.

A retry repeats an attempt. Idempotency ensures it does not repeat the user's intention.