AI workflow concurrency limits: rate limits are not enough

AI workflow concurrency limits solve a different problem from rate limits. A rate limit answers how many operations may start during a time window. A concurrency limit answers how many operations may be in progress right now. Production agents need both.

The distinction looks academic until a slow dependency turns it into an incident. A workflow can stay under 60 requests per minute and still hold 60 database connections, 60 browser sessions or 60 long-lived model streams at once. If each request becomes slower under load, work accumulates faster than it completes. The system is below its nominal rate limit while moving toward failure.

I think about this as three clocks. Rate limits govern the clock on the wall. Concurrency limits govern the work currently occupying capacity. Business limits govern the total scope of one run: branches, tool calls, retries and money. Reliable AI automation makes all three explicit.

Rate and concurrency limits are not interchangeable

A rate limiter controls throughput over time. “No more than 100 requests per minute” is a rate policy. A concurrency limiter controls simultaneous occupancy. “No more than eight model calls in flight” is a concurrency policy.

Consider a service allowed to start one request per second. When calls take 200 milliseconds, average concurrency is low. If the provider slows down and calls take 20 seconds, the same arrival rate produces roughly 20 in-flight calls. Nothing changed in the rate policy, but the occupied capacity increased by two orders of magnitude.

The reverse also matters. A concurrency cap of five does not guarantee a provider rate limit of 60 requests per minute. If every call finishes in one second, those five slots can start hundreds of requests in a minute. A semaphore protects simultaneous capacity; a token bucket protects throughput. One cannot substitute for the other.

Rate limits shape arrivals. Concurrency limits bound occupancy. Timeouts release occupancy. Run limits contain the business process.

Why agents amplify concurrency

Traditional request-response code often has a visible fan-out. One HTTP request performs a known set of downstream calls. An agent can decide at runtime to search again, open several pages, delegate verification or retry a tool. That flexibility makes static capacity estimates less reliable.

The dangerous pattern is nested parallelism. Ten workflow runs start together. Each creates four research branches. Every branch makes three tool calls in parallel. The product accepted ten user actions, but the infrastructure sees up to 120 concurrent operations before retries are counted.

Retries add a feedback loop. A dependency slows down, operations time out and workers retry. The retries add load to the dependency that is already slow, which causes more timeouts. AWS describes this overload cycle as offered load exceeding useful throughput until retries and latency reinforce each other. A rate limiter at the public endpoint may never see the internal amplification.

Long-lived operations make the problem easier to miss:

• a streaming model response holds a connection and worker slot until completion or cancellation;
• a browser task occupies memory and a browser context even while waiting on a page;
• a tool call waiting on a human approval may hold a logical workflow slot for hours;
• a queue job that polls an external system repeatedly consumes attempts and worker time;
• a stalled branch can prevent a parent run from releasing its reservations.

This is why “we have 20 queue workers” is not a complete concurrency policy. Workers are an execution mechanism. The protected resources are model capacity, tenant fairness, browser sessions, database connections, external APIs and the workflow's own branching budget.

Place limits at every scarce boundary

I start by drawing the execution path and marking each resource whose performance degrades under simultaneous use. Then I assign a limit at the narrowest layer that can protect it.

Global provider limit

Cap in-flight calls per provider or model pool. This prevents every application worker from independently assuming the full external quota is available. Keep rate and token limits beside it because providers can enforce requests per minute and tokens per minute independently.

Tool and dependency limit

Browser sessions, search APIs, document converters and internal services need separate pools. A model call and a headless browser task have different capacity profiles. One global “AI concurrency” number hides the actual bottleneck.

Tenant limit

A single customer should not occupy every slot. Give each tenant a ceiling and keep a fleet-wide ceiling above it. The tenant limit protects fairness; the fleet limit protects the dependency.

Workflow and run limit

Bound how many branches one run may execute simultaneously. This prevents one complicated request from crowding out many small requests. A run should also carry limits for total branches, steps, retries, elapsed time and cost. Concurrency answers “how many now”; these limits answer “how much altogether.”

Resource key limit

Some work must be serialized by business key. Two jobs editing the same document, publishing to the same account or reconciling the same payment should not overlap even if the system has spare global capacity. This is a concurrency limit of one for a specific key.

GitHub's Agentic Workflows documentation demonstrates the layered approach: workflow-level and engine-level concurrency sit beside user rate limits, timeouts, output limits and manual approvals. The value is not one perfect threshold. It is defense in depth against different growth paths.

Make admission atomic and lease-based

A limit checked with “count active jobs, then start if the count is below ten” is vulnerable to a race. Several workers can observe nine active jobs and all proceed. Correct admission needs one atomic acquire operation.

The simplest abstraction is a semaphore:

permit = concurrency.acquire(pool, owner, ttl)

if permit is unavailable:
    defer work with jitter

try:
    execute bounded operation
finally:
    concurrency.release(permit)

In a distributed system, the permit should behave like a lease. It has an owner, acquisition time, expiry and stable operation key. Expiry prevents a killed worker from reducing capacity forever. The lease must outlive the expected operation, and healthy workers should renew it when long operations are legitimate.

Expiry introduces its own edge case. If a slow operation continues after its lease expires, another worker may acquire the slot and temporarily exceed the limit. The operation should have a timeout shorter than the lease, or renewal should be tied to evidence that the worker is alive. For side effects, a fencing token can reject stale owners after a newer lease is issued.

Acquire and release must be idempotent. A redelivered queue message should recover its existing permit or learn that the previous attempt finished. A duplicated release must not increase capacity twice.

I store admission decisions as events: pool, limit, observed occupancy, owner, wait duration and outcome. Without that evidence, a saturated system looks like a mysteriously slow queue.

Applying the pattern in Laravel queues

Laravel already provides useful building blocks, but they solve distinct problems. Queue worker counts bound how many jobs a worker group can execute. Rate-limiting middleware shapes job starts. WithoutOverlapping prevents jobs sharing a key from running together. Horizon supervisors can allocate worker processes across queues.

For an AI workflow, I usually combine them rather than asking one feature to do everything:

• separate queues for interactive, batch and recovery work;
• Horizon process limits as the broad worker ceiling;
• a Redis-backed semaphore for shared provider and tool pools;
• rate-limiting middleware for provider requests or tokens per time window;
• WithoutOverlapping for business keys that require serialization;
• job timeouts shorter than the queue connection's retry window;
• a run-level budget for branches, retries, tool calls and cost.

The job should acquire capacity immediately before the scarce operation, not when it first enters the queue. A job may spend time loading state or validating input. Holding a model permit during that preparation wastes the pool.

public function handle(
    AgentConcurrency $concurrency,
    ModelGateway $models,
): void {
    $run = AgentRun::findOrFail($this->runId);
    $run->assertStepMayStart($this->stepId);

    $permit = $concurrency->tryAcquire(
        pool: "model:{$this->model}",
        owner: $this->operationKey,
        ttl: now()->addSeconds(90),
    );

    if (! $permit) {
        $this->release(random_int(5, 15));
        return;
    }

    try {
        $result = $models->complete($run->promptFor($this->stepId));
        $run->recordResultIdempotently($this->stepId, $result);
    } finally {
        $concurrency->release($permit);
    }
}

Releasing a job because no permit is available must not consume the workflow's failure budget as if the model had failed. It is an admission delay, not an execution attempt. Track queue attempts, admission deferrals and provider attempts separately.

The workflow state should also remain explicit. I use state machines for production AI agents so a deferred step returns to a known waiting state. That makes it possible to distinguish “queued for capacity” from “running,” “rate limited,” “failed” and “waiting for approval.”

Turn saturation into backpressure

When every permit is occupied, the system has learned something important: the downstream resource cannot safely accept more work. The worst response is to spin, poll aggressively or create more workers. Saturation should travel upstream as backpressure.

There are four reasonable outcomes:

1. Queue: durable, valuable work waits with a visible position or status.
2. Degrade: optional branches are skipped, a smaller model is used or context is reduced under a tested policy.
3. Reject: low-value or expired work fails quickly before consuming more resources.
4. Escalate: high-value work requests approval or capacity from an operator.

Do not let queues grow without a deadline. A queued answer that arrives after the user no longer needs it is not useful throughput. Store an expiry or “latest useful start” time with each operation. Drop stale work before it acquires an expensive slot.

Use jitter when deferring jobs so hundreds of workers do not wake at the same second. Respect provider retry guidance, but remember that a provider's Retry-After timestamp is an earliest retry boundary, not a reservation of future capacity.

This connects directly to retry budgets for AI automation. Retries need permission from both controls: enough remaining retry budget and enough current capacity. Backoff without a budget only delays a retry storm.

Protect fairness and priority

A single FIFO queue is simple, but it can create head-of-line blocking. Ten long document jobs may make an interactive request wait even though the request needs only one short model call.

I prefer explicit service classes:

• interactive: short deadline, small concurrency reservation and high scheduling priority;
• batch: larger queue, lower priority and permission to use spare capacity;
• recovery: constrained separately so incident retries cannot consume the whole fleet;
• operator-approved: visible and prioritized, but still subject to hard dependency safety limits.

Reserve some capacity for interactive work instead of relying only on priority. Priority cannot help when every slot is already occupied by a long batch. A small reserved pool provides predictable headroom.

Tenant fairness also needs weights or quotas. A strict per-tenant cap is a useful baseline, but unused capacity should not remain idle. Weighted fair scheduling can allow a tenant to borrow spare slots while preserving a minimum share for others when demand returns.

Never let the model assign its own priority. The agent may describe urgency, but application policy decides from authenticated tenant, workflow type, deadline and business rules. Otherwise prompt content becomes a capacity-control interface.

Tune concurrency limits with evidence

The right limit is discovered through load behavior, not copied from a provider dashboard. Start conservatively and increase while watching useful throughput, latency and error rate.

Measure at least:

• active, available and expired permits per pool;
• admission wait time and queue age at p50, p95 and maximum;
• operation latency and timeout rate;
• rate-limit responses and provider errors;
• completed useful outcomes per minute, not merely started calls;
• retry amplification and duplicate suppression;
• fairness by tenant and service class;
• cost per accepted outcome.

Increase load beyond the expected peak in a controlled environment. AWS recommends testing past the point where a service begins to fail, because overload behavior determines whether useful throughput plateaus or collapses. A healthy limiter should keep accepted work completing even when offered load continues to rise.

Do not tune only against average latency. Concurrency equals arrival rate multiplied by time in the system, so tail latency occupies disproportionate capacity. A small number of 60-second calls can dominate a pool whose median call finishes in two seconds.

Adaptive concurrency can react to latency and errors, but it needs guardrails: a minimum, a hard maximum, slow increases and fast decreases. It should never expand automatically in response to a growing queue without confirming that the dependency still has headroom. Autoscaling workers against backlog can otherwise turn backpressure into a larger overload.

Production checklist

• Define separate rate, concurrency, timeout and run-scope limits.
• Map every scarce provider, tool and internal dependency to a pool.
• Enforce fleet, tenant, workflow and resource-key boundaries.
• Acquire permits atomically immediately before scarce work.
• Use owned leases with expiry, renewal and idempotent release.
• Keep operation timeouts shorter than lease and redelivery windows.
• Return saturation upstream through queues, degradation or rejection.
• Separate interactive, batch and recovery capacity.
• Expire stale queued work before it consumes a slot.
• Measure useful throughput, tail latency, wait time and fairness.
• Load test beyond expected capacity and verify graceful saturation.

A production agent is a distributed system with probabilistic decisions inside it. Rate limits protect the timeline, but they do not protect the resources occupied between start and finish. Concurrency limits make that hidden work finite. Combined with deadlines, retry budgets and explicit workflow state, they turn an expanding agent graph into an operable service.

If your AI workflow becomes unstable when traffic, latency or parallel branches increase, I can help review its queue boundaries and admission controls. See my AI engineering consulting work or contact me.

Sources and further reading

• GitHub Agentic Workflows: rate limiting and concurrency controls
• Laravel 13.x documentation: queues, rate limiting and preventing overlaps
• OpenAI API documentation: rate limits
• Amazon Builders' Library: using load shedding to avoid overload