OpenAI returns 503 approximately once every few weeks—during peak load or incidents. For a mobile app where an AI assistant is part of the core user flow, this means a white screen or crash if fallback isn't prepared in advance. In such scenarios, a request queue builds up, latency increases, and users churn. To avoid this, we design a degradation cascade: retry with exponential backoff, then switch to a backup AI service like Anthropic Claude or Google Gemini. Our solutions have been proven on projects with a million-user audience. If your AI assistant is key functionality, fallback logic is essential. Average downtime drops from 3 hours to 5 minutes, saving the business up to $5,000 monthly. Support costs are reduced by 30% thanks to automatic recovery.
The Problem: AI Service Unavailable
Failures vary: provider overload, network issues, rate limits. Without fallback, the user faces an error or infinite loading. The result is a drop in retention and negative reviews. Retry and circuit breaker patterns are the baseline, but for critical scenarios a cascade is needed.
How the Degradation Cascade Works?
Proper fallback is not a single stub but multiple levels, each activating when the previous fails.
Level 1: Retry with backoff. Transient errors (429 Rate Limit, 503, timeout) are retried with exponential backoff. Three attempts: after 1s, 3s, 9s. If all three fail, proceed to level 2.
Level 2: Provider switch. If the primary provider is OpenAI, fallback to Anthropic Claude API or Google Gemini. Responses differ in style, but quality is comparable for most tasks. Keys to backup providers are stored in server configuration.
Level 3: Local model. For critical flows, a small local model (Phi-3.5-mini via llama.cpp, ~2.2 GB). Quality is lower than GPT-4o but works offline. On iOS, run via MLModel or llama.swift.
Level 4: Static responses. FAQ and common questions from cache or database. The user receives a useful answer without knowing the AI is unavailable.
Degradation Level Comparison Table
| Level | Latency | Quality | Cost per execution |
|---|---|---|---|
| Retry | ~13 s | Full | Free |
| Provider switch | ~1 s | 90-95% | API calls |
| Local model | ~2 s | 70-80% | Device energy |
| Static responses | <100 ms | Exact only for FAQ | Zero |
Why Circuit Breaker is a Must-Have Pattern?
The circuit breaker pattern prevents cascading load on a degrading service. It's faster than simple retry and saves client and server resources.
// Android — Kotlin
class AIServiceCircuitBreaker {
private var failureCount = 0
private var lastFailureTime = 0L
private val failureThreshold = 5
private val resetTimeout = 60_000L // 1 minute
enum class State { CLOSED, OPEN, HALF_OPEN }
var state = State.CLOSED
fun canCall(): Boolean = when (state) {
State.CLOSED -> true
State.OPEN -> {
if (System.currentTimeMillis() - lastFailureTime > resetTimeout) {
state = State.HALF_OPEN
true
} else false
}
State.HALF_OPEN -> true
}
fun recordSuccess() {
failureCount = 0
state = State.CLOSED
}
fun recordFailure() {
failureCount++
lastFailureTime = System.currentTimeMillis()
if (failureCount >= failureThreshold) state = State.OPEN
}
}
Circuit breaker implementation example on iOS (Swift)
enum CircuitBreakerState {
case closed, open, halfOpen
}
class CircuitBreaker {
private var state: CircuitBreakerState = .closed
private var failureCount = 0
private let threshold = 5
private let timeout: TimeInterval = 60
private var lastFailure: Date?
func canCall() -> Bool {
switch state {
case .closed: return true
case .open:
if let lastFailure = lastFailure, Date().timeIntervalSince(lastFailure) > timeout {
state = .halfOpen
return true
}
return false
case .halfOpen: return true
}
}
func recordSuccess() {
failureCount = 0
state = .closed
}
func recordFailure() {
failureCount += 1
lastFailure = Date()
if failureCount >= threshold { state = .open }
}
}
Retry Strategy Comparison
| Strategy | Wait between attempts | Number of attempts | Use case |
|---|---|---|---|
| Fixed | 5 s | 3 | Low load |
| Exponential | 1 s, 3 s, 9 s | 3-5 | Transient errors |
| Jitter | 1-5 s (random) | 3-5 | Rate Limit |
How Static Responses Work?
Static responses are pre‑prepared messages for frequently asked questions. They are stored as key‑value pairs in a local database or cache. When all higher levels are unavailable, the system returns the most relevant response based on query analysis (e.g., TF‑IDF). This guarantees an instant answer without network.
How to Test Fallback Logic?
We use integration tests that simulate a 503 error from the primary provider. We verify the switch to backup, correct logging of degradation levels, and a UX without technical error messages. These tests run on CI with every commit.
Step‑by‑Step Fallback Implementation Plan
- Analyze current stack and integration with AI providers—identify points of failure.
- Design degradation cascade (retry, circuit breaker, provider switch, local model).
- Implement retry with exponential backoff and jitter on the client.
- Implement circuit breaker with failure thresholds.
- Connect backup provider and local model.
- Develop static responses for frequent queries.
- Write unit and integration tests for each level.
- Document fallback logic and monitoring.
- Provide instructions for adding new providers.
- Offer post‑deployment support (1 month).
UX During Degradation
The user should never see technical errors. When falling back to static responses, show normal UI without labeling. When fully unavailable, display "Assistant temporarily unavailable, please try again in a few minutes" instead of a raw Error 503.
A degradation indicator is useful for internal analytics: log each fallback with level and reason. This helps identify problematic providers and improve stability.
What’s Included in the Work
- Analysis of current stack and integration with AI providers
- Design and implementation of degradation cascade (retry, circuit breaker, provider switch, local model)
- Setup of static responses and cache
- Writing unit and integration tests
- Documentation of fallback logic and monitoring
- Instructions for adding new providers
- Post‑deployment support (1 month)
Timeline Estimates
Basic retry with backoff — 1 day. Full cascade with circuit breaker and two providers — 2–3 days. We’ll provide a precise estimate after a free consultation.
Our Metrics
5+ years in mobile development, 50+ projects with AI integration, certified engineers (iOS, Android, Flutter). Warranty on implemented functionality. Order an audit of your current AI stack—we’ll find weak spots. Get a free consultation on fault‑tolerance design. Contact us for a project assessment.