how to handle LLM API outages

To handle LLM API outages, set request timeouts, retry only transient failures with exponential backoff and jitter, and use circuit breakers to stop repeated calls to unhealthy services. Add tested fallback models, queue recoverable work, and offer a degraded experience. Monitor user-visible failures, protect sensitive data, and rehearse recovery before production incidents occur.

Key takeaways

• Classify failures before retrying: outages, quota limits, invalid requests, and broken streams need different responses.
• Use one end-to-end deadline, bounded retries, and circuit breakers to prevent cascading failures.
• Test fallback models for output quality, tool compatibility, latency, and data-handling requirements.
• Keep useful features available through saved work, retrieval-only answers, or background processing.
• Measure completed user tasks, not just API availability, and rehearse failover regularly.

How do you tell whether an LLM API is down or your app is failing?

Start with your own telemetry, then check the provider’s status page. A green status page does not rule out a regional or model-specific problem. Equally, a failed request does not prove a provider outage: your API key, quota, network, or deployment may be responsible.

Classify the failure before taking action

1. Inspect the response: capture the HTTP status, provider error code, request ID, model, region if applicable, and elapsed time.
2. Check your infrastructure: review recent deployments, secret rotation, DNS, outbound networking, and account billing.
3. Compare traffic: determine whether failures affect one model, one customer, or all requests.
4. Run a minimal probe: send a short, non-sensitive request through the same production network path.

Authentication and malformed-request errors generally need configuration fixes, not retries. A 429 may indicate temporary throttling or exhausted quota; inspect the error body. Timeouts and 5xx responses are more likely to justify bounded retries, although provider-specific guidance matters.

Use OpenTelemetry with Grafana, Datadog, or Sentry to connect failures to affected workflows. Low-volume teams can often start on free tiers; a small paid monitoring setup might warrant a $25–$100 monthly budget, depending on event volume and retention. Avoid logging complete prompts by default: outage debugging should not create a sensitive-data archive.

What timeout and retry settings should you use during an LLM API outage?

Give every workflow an end-to-end deadline rather than allowing each attempt a fresh, unlimited window. Interactive chat and overnight document processing need different budgets. Measure normal latency before setting production thresholds.

Use a bounded retry policy

• Interactive example: start testing with a 30-second total deadline and a connection timeout around 3–5 seconds. Adjust for your model and product requirements.
• Retry budget: allow no more than two retries initially, and only when enough deadline remains.
• Backoff: use increasing delays with random jitter, such as randomized waits around one and two seconds.
• Rate limits: respect Retry-After when supplied, provided it fits the deadline.
• Streaming: separately monitor time to first token and stalled-stream intervals.

Python’s Tenacity and Node.js retry libraries can implement these rules. Check the provider SDK first: built-in retries combined with gateway and application retries can multiply attempts unexpectedly. Choose one layer to own the policy.

Retries also multiply cost. If each attempt costs $0.02, three fully billed attempts cost $0.06; actual billing depends on tokens processed and provider rules. A timed-out request may still be running upstream, so cancellation does not guarantee zero charges.

Never blindly retry an entire agent workflow that can send email, charge a card, or update records. Use application-level operation IDs, persisted tool results, and idempotency keys where supported. Restarting generation must not repeat a completed business action.

How do circuit breakers and queues prevent cascading failures?

A circuit breaker stops sending normal traffic to a dependency that is repeatedly failing. Without one, thousands of users can each trigger retries against the same unhealthy endpoint, exhausting your connection pool and delaying unrelated features.

Implement three circuit states

1. Closed: send requests normally and record eligible failures.
2. Open: skip the unhealthy route and use a fallback or degraded response.
3. Half-open: permit a few probe requests before restoring normal traffic.

As an initial test configuration, open the circuit when more than half of the last 20 eligible requests fail, wait 30 seconds, then permit limited probes. These are starting values, not universal settings. Exclude invalid user requests and use separate circuits for routes that can fail independently.

Envoy offers routing and resilience controls; Resilience4j is useful for Java applications. For asynchronous jobs, use Amazon SQS, BullMQ with Redis, or Celery. Store job state in your database, cap worker concurrency, and send repeatedly failing jobs to a dead-letter queue.

Queues require their own safety controls: maximum job age, cancellation, deduplication, and a backlog limit. Never tell users a task is queued unless durable storage has accepted it. A queue also does not solve an outage when every job expires before processing resumes.

Light SQS usage may fit within its free allowance, with request-based charges afterward. Managed Redis commonly requires a budget of tens of dollars monthly, depending on capacity and provider. During recovery, drain queues gradually; releasing the entire backlog can immediately trigger new rate limits.

How do you build a safe fallback between LLM providers?

Treat fallback as a product change, not just an endpoint change. A second model can differ in tool calling, structured output, context limits, refusal behavior, and answer quality. Successful HTTP responses do not establish equivalent results.

Validate the fallback before enabling it

1. Define an internal request format and provider-specific adapters.
2. Test representative prompts, long inputs, tool calls, refusals, and required JSON schemas.
3. Validate outputs server-side before executing tools or saving results.
4. Confirm backup quotas and route a small amount of permitted test traffic regularly.
5. Set per-task spending limits and feature flags for disabling unsafe routes.

LiteLLM can simplify routing across supported providers, but it does not replace evaluation. OpenAI, Anthropic, and Google offer different interfaces and capabilities; cloud deployments may share upstream dependencies, so verify actual failure isolation.

For budgeting, calculate tokens separately for each model. At hypothetical rates of $1 per million input tokens and $5 per million output tokens, a 2,000-input-token, 500-output-token request costs $0.0045. Check current published prices rather than assuming the backup is cheaper.

For US healthcare or regulated customer data, confirm contractual and security requirements before routing elsewhere. Where required, the fallback needs appropriate agreements, such as a BAA, plus acceptable retention and access controls.

What should users see when your AI feature is unavailable?

Preserve the user’s work and explain what remains possible. A specific message such as “Draft saved; generation is temporarily unavailable” is more useful than a spinning indicator or an unsupported claim that recovery will happen soon.

Choose degradation by workflow

• Chat: retain conversation history and offer a deliberate retry.
• Document generation: save inputs and offer background processing with completion notification.
• Search: show retrieved passages and source links without presenting them as a generated answer.
• Agents: pause before consequential actions and display the last confirmed completed step.
• Streaming: mark interrupted output as incomplete instead of silently merging another model’s continuation.

Use LaunchDarkly, Unleash, or a database-backed feature flag to switch modes quickly. A database flag can avoid another vendor bill, while commercial flag platforms introduce subscription or usage costs. Restrict access and test propagation; an emergency switch that takes minutes to reach workers may not stop spending fast enough.

Cached answers can reduce API usage, but only reuse them when permissions, source versions, and freshness requirements still match. Never share tenant-specific responses across customers. Label older information and avoid caching personalized financial, medical, or account answers without carefully defined safeguards.

How should you monitor, test, and recover from LLM API outages?

Monitor whether users complete tasks, not just whether the provider returns 200 responses. Track timeout rate, valid-output rate, latency, fallback share, queue age, interrupted streams, and cost per completed task. Break metrics down by model and route without adding sensitive payloads.

Create a practical outage runbook

1. Detect: alert on sustained user-visible failures and growing queue age.
2. Contain: open affected circuits, limit concurrency, and disable unsafe workflows.
3. Communicate: publish affected features and known workarounds without inventing a recovery estimate.
4. Recover: probe the primary route, restore a small traffic share, and increase gradually.
5. Reconcile: inspect partial jobs, duplicate actions, failed callbacks, and unexpected charges.

Use Toxiproxy or mocked HTTP responses to simulate connection failures, 429s, 503s, malformed JSON, and midstream disconnects. Run load tests with k6 to verify that retries stay bounded. Include a scenario where the backup provider is unavailable too.

These tools have open-source options, though CI compute and staging infrastructure still cost money. Set an explicit drill budget—for example, a $20 API-spend cap—and favor mocks for high-volume failure testing.

After an incident, record which customer tasks failed, which controls helped, and which assumptions were wrong. Do not restore full traffic simply because a status page says “resolved”; confirm recovery through your own path and watch for backlog-driven throttling.

Frequently asked questions

Should you retry every failed LLM API request?

No. Retry transient failures within a deadline, but fix authentication, invalid-input, and exhausted-quota problems directly. Check SDK defaults so retries do not stack across layers.

Can you fail over after a streaming response has started?

Yes, but restarting and clearly labeling the new response is usually safer than silently combining outputs. Preserve the partial answer and ensure any completed tool actions are not repeated.

Is a second API key enough protection against an outage?

Usually not: keys can share the same account limits and provider infrastructure. Use approved capacity arrangements or a genuinely separate fallback rather than rotating keys to evade limits.

How often should you test LLM outage handling?

Run automated failure tests whenever routing, SDKs, or agent workflows change. A monthly staging drill is a reasonable starting point, with frequency adjusted to release pace and business risk.

What is the simplest setup for a small startup?

Start with explicit deadlines, bounded retries, durable input storage, and a clear unavailable state. Add a circuit breaker and one evaluated fallback when the operational risk justifies the maintenance.

Muhammad Tariq at Tariq Services builds AI products, agents, and automations with practical reliability controls. Explore our AI development services or book a call to plan outage handling around your product’s workflows, data requirements, and budget.