4 Patterns for a Resilient API Proxy

Every API proxy can eventually have issues if the service on the other end doesn’t cooperate. It gets slow, it goes down for a few minutes, or it starts rejecting requests during a traffic spike. None of that is under your control, but your users still expect your product to work.

We ran into this problem with a proxy that stands between our application frontend and Coral, our third-party commenting platform. At roughly 1,000 requests per minute, that proxy is a single chokepoint for all the communication between the systems. If Coral slows down or becomes unreachable, the proxy’s resources get exhausted, and the failure spreads to users.

To handle these issues, we used four patterns, layered so each one covers a failure mode the others don’t, and wired together so that they reinforce each other.

When Your Proxy Becomes the Bottleneck

A proxy that sits between users and a third-party API inherits that API’s problems at scale. A thousand slow requests per minute, all queued behind a dependency that answers in a few seconds instead of milliseconds, will exhaust your resources.

There are three ways a dependency fails, and each needs a different defense:

• Service is slow: users are left waiting while your proxy’s capacity drains along with them.
• Service is down: requests fail, and retrying immediately just adds more failed requests.
• Service is rate limiting you: the dependency is telling you to stop the requests.

A single mechanism cannot cover all three at once. Each failure mode calls for its own pattern, and the patterns must not conflict when more than one triggers simultaneously.

The Four Patterns

Each one of the below is its own distributed systems pattern:

• Stale-while-revalidate (SWR) cache: don’t make users wait for data that’s still good enough.
• Request coalescing: don’t fetch the same thing multiple times.
• Circuit breaker: don’t keep sending requests to a dependency that’s already failing.
• Exponential backoff with jitter: retry in a way that doesn’t make things worse.

Pattern 1: Stale-While-Revalidate (SWR) Caching

Not all data needs to be fresh. Stale-while-revalidate takes advantage of that by giving each cache entry two lifespans: a fresh window where it’s served as-is, and a longer stale window where it’s still served while a background refresh is triggered.

There are three modes for the SWR cache:

• Fresh: serve from cache, no upstream call at all.
• Stale: serve from cache, but kick off a background refresh so the next request has new data.
• Expired: treat it as a cache miss, fetch synchronously, and cache the result.

Code example:

```

async function getOrFetch(key, fetchFn) {

    const entry = cache.get(key)

    if (entry && !entry.isStale) return entry.value	// fresh: serve cached value

    if (entry && entry.isStale) {

        refreshInBackground(key, fetchFn)		// stale: trigger background refresh

        return entry.value				// stale: serve stale cached value

    }

    const value = await fetchFn()			// expired: fetch new value

    cache.set(key, value)

    return value

}

```

The part that takes real judgment is choosing the two windows correctly for each type of data, because getting it wrong in either direction has a cost: too short and you’re barely caching anything, too long and users start seeing data that’s out of date.

In our proxy, different data gets different TTLs based on how often it actually changes. A story’s comment list, which changes every time someone posts, is cached fresh for 15 seconds and still served stale for another 15 seconds after that while a refresh runs behind it. Data that rarely changes can have the cache TTLs set to a few minutes, hours, or even days. The result is that most reads never touch Coral at all, and the ones that do are quiet background refreshes instead of a user waiting on a spinner.

Pattern 2: Request Coalescing

Caching handles repeated reads over time. It doesn’t handle multiple requests for the same thing arriving within a short timeframe of a few milliseconds, before any of them has had a chance to populate the cache. That’s what request coalescing is for.

The mechanism tracks in-flight requests in a map keyed by what they’re fetching. The first request for a given key starts the fetch and stores the promise. Every other request for the same key, while that promise is pending, awaits it instead of starting a second fetch.

Code example:

```

async function getOrFetchCoalesced(key, fetchFn) {

    if (inflight.has(key)) return inflight.get(key)          			// join the in-flight call

    const promise = fetchFn().finally(() => inflight.delete(key))

    inflight.set(key, promise)                                			// first caller owns the fetch

    return promise

}

```

Getting this right also means deciding how it interacts with the cache. A background refresh triggered by a stale cache hit has to check the same in-flight map; otherwise, a stale-triggered refresh and a concurrent cache miss fetch for the same key would race each other.

In our proxy, this matters most during traffic spikes on a live story. Multiple users can load the same story in the same second, and without coalescing, that’s multiple identical calls to Coral landing at once. With this pattern, the first request does the work, and every other concurrent request for that story rides along on the same promise.

Pattern 3: Circuit Breaker

Caching and coalescing both assume the API is healthy, perhaps only a little slow or busy. When the service is genuinely down, neither pattern helps. Retrying a failing dependency adds more failed requests to a system that’s already struggling. A circuit breaker tracks failures and, past a threshold, stops trying for a while.

This pattern has three modes:

• Closed: normal operation, requests go through.
• Open: the breaker has tripped, requests fail immediately without ever reaching the API.
• Half-open: one request is let through: success closes the breaker, and a failure reopens it.

Code example:

```

async function callWithBreaker(fn) {

    if (state == "open" && now() - lastFailure < recoveryTime)

        return reject("circuit open")                      		// open: fail fast, skip fetch

    try {

        const result = await fn()                          		// closed / half-open: try the call

        state = "closed"; failures = 0

        return result

    } catch (error) {

        if (++failures >= threshold) state = "open"         		// too many fails: switch to open

        throw error

    }

}

```

A circuit breaker needs a few parameters to be set: how many failures before it trips, how long it waits before probing again, and how many probes it allows through in half-open mode. Usually we also treat slow requests as failures, because most of the time these are also going to fail or at least make the user experience bad.

In our proxy, a response slower than 12 seconds is recorded as a failure even if it eventually returns a success. Three failures within the tracked window trip the breaker open. How long it then stays open depends on which endpoint it’s guarding. For user-facing requests, the circuit breaker is usually open from 15 to 30 seconds. Once that window passes, a single probe request is allowed through, and the breaker closes again only if that probe succeeds.

Pattern 4: Exponential Backoff With Jitter

Sometimes a failure happens because of a dropped connection or a momentary timeout. For those failures, retrying makes sense, but retrying immediately in a tight loop is not a good decision. Exponential backoff fixes the timing, so that each retry waits longer than the last, doubling up to a cap. A jitter adds randomness to the delay instead of retrying at a fixed time. This avoids multiple independent clients failing and retrying at the same moment.

Code example:

```

async function withRetry(operation) {

    for (let attempt = 1; attempt <= maxAttempts; attempt++) {

        try {

            return await operation()

        } catch (error) {

            if (!isRetryable(error) || attempt == maxAttempts) throw error

            const cap = min(baseDelay  2 * (attempt - 1), maxDelay)

            const delay = random() * cap

            await sleep(delay)

        }

    }

}

```

Our proxy runs two separate versions of this pattern, tuned to different failure modes. Plain network errors, a dropped connection or a timeout, get a fast, unjittered doubling: 150ms, then 300ms, then 600ms, capped at 3 seconds, because those tend to be isolated to a single request and resolve quickly. A rate limiting error from the API gets a slower retry with 30% jitter layered on top of the doubling, starting at 1 second and capping at 10, because in that failure mode synchronised retries make things worse instead of better.

Layering the Patterns Together

Each of these four patterns is a self-contained mechanism: a cache with two TTLs, a map of in-flight promises, a failure-tracking mechanism, and a jittered delay retry. How we wire them together is important, because each layer has to trust that the layers around it will only pass it the work it’s actually meant to handle.

A schema of my layered solution:

```

function proxyRequest(key) {

    return cache.getOrFetch(key, () =>	// 1. SWR cache + coalescing

        withCircuitBreaker(() =>                   	// 2. breaker guards the upstream

            withRetry(() =>                         	// 3. retry transient failures

                fetchFromUpstream(key))))           // 4. the real call

}

```

In our proxy, this ordering is what makes the composition actually work. A fresh cache hit for a story’s comment list never gets past the first layer, so it never touches the breaker or the retry logic at all. A cache miss that lands on a healthy Coral goes through the breaker and straight to the call. A dropped connection on that call gets retried and resolved by the backoff layer before it ever counts as a failure further out. And if Coral is genuinely down, the breaker trips and every subsequent request fails fast, without wasting a retry cycle.

By the time a request reaches Coral, it has already been filtered three times:

• Is this request actually necessary?
• Is the API known to be healthy?
• Is this worth retrying if it fails?

What Breaks and What Catches It

Every failure mode that this proxy needs to survive eventually shows up as one of three concrete situations, and each one is handled by a different combination of the four patterns:

1. API is temporarily down:

• Cached keys keep serving stale data to users.
• Uncached requests trip the circuit breaker after a few failures.

2. Traffic spike:

• Coalescing collapses a burst of concurrent requests into a single call
• The cache absorbs the repeated reads that follow.

3. API rate limiting:

• Jittered backoff spreads retries out instead of hammering the limit again in sync.
• If the limiting persists, the circuit breaker eventually opens and stops the requests.

The point of building these patterns is not to eliminate the failures, but to make the system absorb the failures and hide them from the users if possible.

Takeaway

Every third-party dependency will eventually be slow, unavailable, or rate-limit you. None of that is avoidable. What’s avoidable is whether your users notice.

The patterns to achieve this are:

• SWR Cache: so you don’t wait on data that’s still good enough.
• Request Coalescing: so concurrent requests don’t multiply load for no reason.
• Circuit Breaker: so a struggling service doesn’t pile further requests.
• Exponential Backoff: so retries help recovery instead of working against it.

Each pattern on its own solves one specific way a dependency fails, and each carries real design decisions: TTL windows, failure thresholds, recovery timers, jitter formulas, all of which have to be tuned to the traffic and the dependency they’re protecting. The way they are layered together makes sure that a request only pays the cost of the protection it actually needs. That layering is the difference between an outage your system resolves quietly, and one your users see in your application.