Software Architecture

Appearance

Chapter 4: Reliability Patterns ​

General Track · Foundational · ~35 min read + exercise

Why this chapter exists ​

Everything you've built so far in this course assumed, quietly, that calls succeed. You model your data, choose a style, draw your service boundaries — and in the back of your mind, a function call returns, a query runs, a request gets a response. In a single process on one machine, that assumption mostly holds: a function call doesn't time out, doesn't half-complete, doesn't mysteriously run twice. The moment your system spans more than one process — a service calling another service, an app calling a database across the network, a job calling a third-party API — that assumption breaks, and it breaks in ways that are genuinely new.

This is the hardest mental shift for engineers moving from monoliths to distributed systems, and it's worth stating plainly: the network is not reliable, and you must design as if it isn't. A network call can succeed. It can fail outright. It can hang forever. It can succeed on the server but lose the response on the way back, so the caller thinks it failed when it didn't. It can be slow enough that the caller gives up while the server is still working. These aren't exotic edge cases — they are the normal behavior of networks at any real scale, happening constantly, and a system that doesn't plan for them isn't robust, it's just untested.

Here's the asymmetry that makes this matter. In a monolith, if the billing module calls the ticketing module and ticketing has a bug, you get an exception with a stack trace, and the whole thing fails cleanly in one process you can debug. In a distributed system, if the billing service calls the ticketing service and the network hiccups, billing might wait thirty seconds for a response that never comes, then retry — and now ticketing has processed the same request twice, double-charging a customer, while billing still doesn't know if either attempt worked. Nobody wrote a bug. The code is correct. The failure came entirely from the space between the services, and that space is where reliability lives.

Consider it concretely. Your Laravel app dispatches a job to charge a customer's card through a payment provider's API. The job fires the HTTP request and waits. The request reaches the provider, the charge succeeds — and then the response times out on the way back. Your job sees a timeout, assumes failure, and per its retry logic, charges the card again. The customer is billed twice for one purchase. Every piece of code did what it was told. The failure is structural: a call that partially completed, leaving the caller unable to tell success from failure, combined with a retry that didn't account for it. This chapter is the toolkit for exactly this class of problem — the patterns that let a system stay correct and standing when its parts, and the network between them, misbehave.

These patterns are not optional knowledge if you build anything distributed. They are the difference between a system that degrades gracefully under the failures that will happen and one that turns a single slow dependency into a cascading outage. And they connect directly back to the reliability quality attribute from Chapter 1 — this is the chapter where "reliability" stops being a word on a list and becomes a set of concrete techniques.

What this chapter covers ​

  • Timeouts — why every remote call needs a deadline, and why the default of "wait forever" is a latent outage.
  • Retries — how to recover from transient failures without making things worse, and the backoff that keeps retries from becoming a self-inflicted denial of service.
  • Idempotency — the property that makes retries safe, and without which retries are dangerous. This is the keystone of the whole chapter.
  • Circuit breakers — how to stop calling a dependency that's clearly down, so one failure doesn't cascade into many.

By the end you'll do a hands-on exercise hardening a realistic remote call — finding where it fails, and applying the patterns to make it reliable.

Let's start with the simplest and most neglected of them: the timeout.

Timeouts: every call needs a deadline ​

The most common reliability bug isn't dramatic. It's a missing timeout. A remote call is made with no deadline, the dependency hangs, and the caller waits — forever, or for some framework default measured in minutes that might as well be forever. It sounds trivial. It is responsible for an enormous share of real outages, and understanding why reveals how distributed failures cascade.

Picture the chain. Your Laravel app handles a web request, which calls an internal service, which calls the payment provider. The payment provider has a bad moment and stops responding — not failing, just hanging. Without a timeout, your call to it waits. The request handler holding that call is now stuck, occupying a PHP-FPM worker. More requests arrive; each tries the same call; each gets stuck; each occupies a worker. Within seconds, every worker is blocked waiting on the dead dependency, and your app stops serving all requests — including ones that have nothing to do with payments. A single slow dependency just took down your entire application, not because it failed, but because you waited for it without limit.

A timeout breaks this chain. It says: this call gets N milliseconds, and if there's no answer by then, we give up and handle it as a failure. The stuck workers free up. The app keeps serving the requests that don't depend on the sick service. The failure stays contained to the feature that needs the slow dependency, instead of spreading to the whole system. This is a core principle of reliability: a slow dependency is often worse than a failed one, because a failure is at least immediate. A timeout converts "hangs indefinitely" into "fails quickly," and failing quickly is something you can handle.

The practical guidance is short and worth following without exception:

Set a timeout on every remote call. Database queries, HTTP calls, cache lookups, message-broker operations — anything that crosses a process boundary. The default for most clients is no timeout or an absurdly long one; override it. A call with no timeout is a latent outage waiting for its dependency to have a bad day.

Choose the value from the call's real latency, not a guess. Look at the actual p99 latency of the call and set the timeout somewhat above it — enough that normal slow-but-fine responses succeed, low enough that a truly hung call is abandoned quickly. A timeout of thirty seconds on a call that normally takes fifty milliseconds isn't protection; it's a thirty-second window to pile up stuck workers.

Mind the budget down a chain. If a web request must respond in two seconds, and it calls a service that calls another service, the inner timeouts must fit inside the outer budget. An inner call with a ten-second timeout inside a request expected to answer in two seconds is incoherent — the outer request will have given up long before the inner timeout fires. Timeouts should tighten as you go deeper, not stay flat or grow.

A timeout tells you a call failed (or took too long, which you treat as failure). The natural next question is: should you try again? That's retries — and it's where reliability gets subtle.

Retries: recovering without making it worse ​

Many failures are transient — a brief network blip, a momentary overload, a dependency restarting. The call failed, but the same call a second later would likely succeed. Retrying is the obvious response, and for transient failures it's exactly right: a retry turns a momentary glitch into a non-event the user never noticed. But retries are also one of the easiest ways to turn a small problem into a large one, and the difference is entirely in how you do them.

The naive retry — catch the failure, immediately try again, maybe a few times in a tight loop — is actively dangerous, for two reasons.

The first is that immediate retries kick a system that's already down. If a dependency is failing because it's overloaded, the worst possible response is to immediately send it the same request again, plus all the other callers doing the same thing. You've just multiplied the load on a service that was struggling, turning a brief overload into a sustained one. This is a retry storm, and it's a classic way that a minor blip becomes a major outage: the retries cause the very failure they're responding to.

The fix is exponential backoff with jitter. Backoff means waiting longer between each successive retry — 100ms, then 200ms, then 400ms — so a struggling dependency gets breathing room to recover instead of a barrage. Jitter means adding a small random amount to each wait, so that a thousand callers who all failed at the same instant don't all retry at the same later instant, re-synchronizing into another thundering wave. Backoff spaces the retries out; jitter spreads them apart. Together they let a system recover instead of hammering it while it's down.

The second danger is deeper, and it's the one that connects to the rest of the chapter: retrying a call that already partially succeeded can cause real damage. Recall the double-charge from the opening — the charge succeeded but the response was lost, so the retry charged again. Backoff doesn't help here at all; the retry is perfectly spaced and still wrong, because the operation it's repeating has a side effect that shouldn't happen twice. You cannot safely retry an operation unless retrying it is harmless. And that property — "safe to do more than once" — has a name, and it's the keystone this whole chapter is built around.

So before you can retry safely, you have to make the thing you're retrying safe to repeat. That's idempotency.

Idempotency: the property that makes retries safe ​

Idempotency is the most important concept in this chapter, and one of the most important in all of distributed systems, so it's worth stating carefully. An operation is idempotent if performing it multiple times has the same effect as performing it once. Reading a value is naturally idempotent — reading it twice gives the same answer and changes nothing. Setting a value to a specific number is idempotent — setting status = 'closed' twice leaves it closed, same as once. Incrementing a value is not — adding 1 twice adds 2. Charging a card is not — charging it twice charges it twice.

This property is the hinge the whole reliability toolkit turns on, because of a fact you cannot engineer away: in a distributed system, you often cannot tell the difference between "the operation failed" and "the operation succeeded but the response was lost." Both look identical to the caller — a timeout, no response. If the operation was idempotent, you don't need to tell the difference: just retry, and whether the first attempt happened or not, the end state is correct. If the operation was not idempotent, that same retry is a double-charge, a duplicate order, a doubled inventory adjustment. Idempotency is what converts "I don't know if it worked" from a catastrophe into a non-problem.

The good news is that idempotency is a property you can design in, and there are a few standard ways to do it.

Make the operation naturally idempotent where you can. Prefer "set this field to this value" over "increment this field." Prefer operations that declare a desired end state over operations that apply a relative change, because declaring an end state is repeatable by construction. A surprising amount of accidental non-idempotency comes from reaching for increments and appends when a set would have done.

Use an idempotency key for operations with side effects. When an operation genuinely can't be made naturally idempotent — charging a card, sending an email, creating an order — the standard technique is a unique key the caller generates and attaches to the request. The first time the server sees that key, it performs the operation and records the key with the result. If it sees the same key again (because the caller retried), it recognizes the duplicate and returns the recorded result without performing the operation a second time. The charge happens once no matter how many times the request arrives. This is exactly how real payment APIs like Stripe handle retries safely, and it's the pattern to reach for whenever a side effect must happen at-most-once. In your Laravel app, that's a unique constraint on the key column plus a check-or-create in a transaction — modest code that turns a dangerous retry into a safe one.

Deduplicate on the consumer side in async flows. When messages can be delivered more than once — which most message brokers explicitly allow, guaranteeing "at least once" rather than "exactly once" — the consumer must recognize and skip messages it has already processed, typically by tracking processed message IDs. Same idea as the idempotency key, applied to message consumption.

The connection back to retries is now complete and worth saying directly: retries and idempotency are a single technique in two halves. Retries provide recovery from transient failure; idempotency makes that recovery safe. Retrying without idempotency is how the double-charge happens. Designing for idempotency without retrying leaves transient failures unhandled. You need both, together — and the order to think about them is idempotency first, because it's the property that makes everything else safe.

There's one failure the patterns so far don't address. Timeouts contain a slow dependency per-call; retries plus idempotency recover from transient blips. But what about a dependency that isn't blipping — that's genuinely down, and will be for minutes? Retrying it, even with perfect backoff, just wastes time and resources failing over and over. For that, you stop calling it.

Circuit breakers: stop calling what's clearly broken ​

When a dependency is down — not slow, not blipping, but failing every call for a sustained stretch — continuing to call it is pure waste. Every call burns a timeout's worth of waiting before failing, every retry adds load to a service that can't handle it, and every stuck call ties up a worker. The smart move is the one named after household electrics: trip a circuit breaker that stops the calls until the dependency recovers.

The metaphor is exact. A home circuit breaker detects a dangerous condition and opens the circuit, cutting the flow to prevent a fire, then can be reset once the problem is resolved. A software circuit breaker wraps a remote call and watches its failures. While calls are succeeding, the breaker is closed — current flows, calls go through normally. When failures cross a threshold (say, more than half of recent calls failed), the breaker opens — and now calls fail immediately without even attempting the dead dependency. No timeout wait, no added load, no stuck workers. After a cooling-off period, the breaker goes half-open — it lets a single trial call through to test whether the dependency has recovered. If that call succeeds, the breaker closes and normal flow resumes; if it fails, the breaker opens again and waits longer.

What the circuit breaker buys you is the prevention of cascading failure — the way one dead component takes down everything that depends on it, and everything that depends on those. We saw the mechanism in the timeouts section: a hung dependency ties up workers until the whole app stalls. Timeouts limit the damage per call; a circuit breaker goes further and stops making the calls at all once it's clear they're futile, which frees the system to fail fast and degrade gracefully — showing the user a "payments are temporarily unavailable" message instantly, rather than a thirty-second spinner that ends in an error. A fast, honest failure is a far better experience than a slow one, and it keeps the rest of the system healthy.

The pattern pairs naturally with a fallback: when the breaker is open, instead of just failing, do something sensible if you can — serve cached data, queue the work to do later, return a degraded-but-useful response. A product page whose recommendation service is down can show the page without recommendations rather than not showing the page. The breaker tells you the dependency is unavailable; the fallback decides what to do about it.

You won't usually implement circuit breakers from scratch — they live in resilience libraries and service meshes (the infrastructure layer that manages service-to-service traffic), and Laravel's HTTP client and queue tooling have related capabilities — but you decide where they belong: around the calls to dependencies whose failure shouldn't be allowed to take down the caller. That's a design judgment, and it's yours to make.

A dependency you'll meet more and more makes all of this vivid: a call to a third-party model (an LLM). It's an unreliable dependency like any other — it can be slow, rate-limited, or down — so it wants the same defenses: a timeout, bounded retries, a circuit breaker, and a fallback, where the fallback is often a cheaper deterministic path you keep for when the model is unavailable or too slow. It adds one twist the others don't have: its output is unpredictable even on success, so you treat what it returns as untrusted and validate it before your system acts on it — the same boundary discipline you'd apply to any input. Two AI-specific names attach to that twist: the runtime checks that constrain and sanitize a model's output are guardrails, and because you can't exact-match-assert a probabilistic output, you verify this dependency with evals (graded sets of sample inputs) rather than ordinary unit tests — but both are just this chapter's "validate at the boundary" applied to a component whose output you can't predict. The patterns don't change; the model is just a particularly vivid dependency to apply them to (you'll design exactly this trade-off in the capstone). (Flagged: LLM specifics are current-ecosystem; the reliability patterns are the chapter's and the source books'.)

Put the four together and you have a layered defense. Timeouts bound how long any single call can hurt you. Retries with backoff recover from the transient failures that bounding reveals. Idempotency makes those retries safe. Circuit breakers stop the retries from beating on a dependency that's genuinely down, and fallbacks keep the system useful when it is. None of them is exotic; all of them are the difference between a system that survives the failures it will encounter and one that amplifies them.

A deeper move: define errors out of existence ​

It's worth naming the design philosophy underneath the keystone of this chapter. John Ousterhout calls it defining errors out of existence in A Philosophy of Software Design: often the most robust way to deal with an error condition is to redesign so the condition can't arise in the first place, leaving nothing to handle. Idempotency is the prime example — it doesn't handle the "did this already run?" error, it makes the question meaningless, so duplicate delivery stops being an error at all. You'll see the same move recur in this course: a saga's compensating actions (Communication Patterns chapter) replace "what if the distributed transaction half-commits?" with a defined, ordinary path rather than an exception to recover from. When a failure mode keeps biting, the strongest question often isn't "how do I handle this?" but "how do I design so this can't happen?" — fewer error paths is itself a reliability gain, because every error path is code that can be wrong.

Practice ​

The skill here is seeing where a remote call can fail and applying the right pattern to each failure mode — and especially seeing why retries without idempotency are dangerous. The exercises build toward hardening a real call. Budget about two hours.

Exercise 1: Find the failure modes (20 min) ​

Pick a remote call in a system you work on — a call to a database, a third-party API, another service. Write down every way it can fail: outright failure, hang/slow, succeed-but-lose-the-response, succeed-twice-under-retry. For each, note what currently happens (does the code handle it, or assume it away?). The goal is to practice seeing the failure space that the happy path hides.

Exercise 2: Trace a double-charge (45 min) ​

Take an operation with a real side effect (a charge, an email, an order creation) and trace, step by step, how a lost response plus a retry could cause it to happen twice. Then redesign it to be safe: decide whether to make it naturally idempotent or to add an idempotency key, and sketch the change (in Laravel terms, the key column, the unique constraint, the check-or-create). Write two or three sentences on why your design makes the retry safe.

If you use an AI assistant to sketch the idempotency-key implementation, interrogate it against your real schema and framework — does its approach actually hold under concurrent duplicate requests, or does it have a race between the check and the create? That race is the most common bug in hand-rolled idempotency; make sure the design closes it.

Exercise 3: Place the patterns (45 min) ​

For the same call or flow, decide which of the four patterns it needs and with what settings: what timeout (from the call's real latency), what retry policy (how many, what backoff), whether idempotency is required (does it have a side effect?), and whether a circuit breaker and fallback make sense (what would the fallback do?). Write it up as a short reliability plan for that call.

Deliverable ​

A one-page reliability plan for a real remote call or flow, covering: its failure modes, the timeout and why, the retry policy, the idempotency approach (and why retries would be unsafe without it), and whether a circuit breaker and fallback apply. A reviewer will read it in about fifteen minutes and respond to one question: if this call's dependency had a bad thirty seconds, would this plan keep the rest of the system healthy and the data correct?

Check yourself ​

Before you consider this chapter done, you should be able to:

  • Explain why a slow dependency can be worse than a failed one, and how a timeout helps.
  • Explain what a retry storm is and how exponential backoff with jitter prevents it.
  • Define idempotency, and explain why retries are unsafe without it — with a concrete example of a non-idempotent operation made safe.
  • Describe the closed/open/half-open states of a circuit breaker and what each one does.
  • Explain how timeouts, retries, idempotency, and circuit breakers fit together as a layered defense.

If idempotency feels shaky, redo Exercise 2 — it's the keystone, and it clicks once you've traced a real double-charge and closed it.

What a strong answer demonstrates: a strong reliability plan treats idempotency as the keystone — it can trace exactly how a lost response plus a retry causes a double effect, and shows the design (natural idempotency or a key) that closes the race under concurrent redelivery. The clearest signal you've understood the chapter is that the four patterns appear as a layered defense answering the question "if this dependency had a bad thirty seconds, would the rest of the system stay healthy and the data stay correct?"

Going deeper (optional) ​

  • Building Microservices (Newman), the chapter on resiliency — the direct, practical source for timeouts, retries, circuit breakers, and bulkheads in a services context, with more patterns (like bulkheads and the broader stability patterns) than we covered.
  • Designing Data-Intensive Applications (Kleppmann), the chapter on the trouble with distributed systems — for the deeper "why": unreliable networks, unreliable clocks, and why you genuinely cannot distinguish a slow node from a dead one. This is the theoretical bedrock under this chapter's practical patterns.
  • Michael Nygard's Release It!: Design and Deploy Production-Ready Software (Pragmatic Bookshelf, 2007) — the canonical source that introduced the circuit breaker and a catalog of stability patterns and anti-patterns drawn from real production failures. If this chapter grabbed you, this is the deep dive.

Primary sources (the foundational statements beneath "the network is not reliable," if you want the theory on-ramp):

  • L. Peter Deutsch and colleagues at Sun Microsystems, "The Fallacies of Distributed Computing" (originated ~1994–1997; the canonical write-up is Arnon Rotem-Gal-Oz, "Fallacies of Distributed Computing Explained," ~2006) — the eight false assumptions ("the network is reliable," "latency is zero," …) that this entire chapter is the antidote to.
  • Leslie Lamport, "Time, Clocks, and the Ordering of Events in a Distributed System" (Communications of the ACM, July 1978) — the seminal paper on why "what happened first" is genuinely hard across machines; the deep root of why you can't reliably distinguish "failed" from "succeeded but lost the response."

None of this is required to continue.

Key takeaways ​

  • The network is not reliable. Across a process boundary, calls can fail, hang, or succeed-but-lose-the-response — design for it, don't assume it away.
  • Every remote call needs a timeout. A slow dependency without a timeout ties up resources until it takes down the whole system; failing fast contains the damage.
  • Retries recover from transient failures — with exponential backoff and jitter, so they don't become a retry storm that worsens the outage.
  • Idempotency is the keystone. Because you can't tell "failed" from "succeeded but lost the response," retries are only safe when the operation is safe to repeat — via natural idempotency or an idempotency key.
  • Circuit breakers stop calling a dependency that's clearly down, preventing cascading failure and enabling fast, graceful degradation with fallbacks. The four patterns together are a layered defense.

Next chapter: Communication Patterns — we've treated remote calls as a thing that happens; now we look hard at the choice between synchronous and asynchronous communication, and between orchestration and choreography, as one of the heaviest trade-offs in distributed design.