Operational resilience and failure design
Keep critical services correct when components fail.
Commit the state and the handoff together
Enlarge to read every label and explore the connections
An outbox makes the event handoff durable alongside the business change. Retries also need a consumer that controls duplicate effects.
Commit state and the outbox event atomically.
Expect repeated delivery; make effects idempotent.
Reconcile intended actions with observed results.
The fraud vendor is slow, the queue is growing, and every client retries. More traffic arrives because the system is struggling with the traffic it already has. Resilience means designing the failure path before this loop begins.
Define the critical service
- ServiceName the customer or control outcome
- DependenciesMap what it requires
- ObjectiveMeasure successful delivery of that outcome
Start from the customer service that must continue or recover: receive a report, protect a balance, make an authorized payout, or preserve evidence. Map the systems, people, partners, and data needed to deliver it.
A service-level objective measures a chosen reliability target. It should include correctness and coverage where they matter, not just uptime. A payment API that returns success quickly while dropping ledger events is not delivering a reliable payment service. Define the observable good event and the acceptable failure budget with the responsible owners.
Inside the mechanism. Define the critical service through the customer and financial outcome: an authorized action completes once, required controls run, records reconcile, and unresolved outcomes remain visible. Infrastructure health is supporting evidence. A green API probe cannot detect every failed business effect. Service measures should include the relevant population, success condition, time window, and exclusions.
A concrete example. A critical payment service includes the customer outcome, control evidence, and financial record. A responsive API alone does not establish completion. 620 intended requests generate 651 processing attempts under this retry assumption. Capacity is 790 attempts per interval, and the critical path consumes 150 ms of a 300 ms budget. The request-based SLO view observes 100 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.
When the assumption fails. Health checks pass while the settlement publisher stops delivering effects. Define success through the full service path and reconcile customer and ledger outcomes. The following worked sequence shows the reference condition, a stress condition, and a response condition with explicit synthetic data. These are comparative assumptions, not measured causal effects.
A critical payment service includes the customer outcome, control evidence, and financial record. A responsive API alone does not establish completion.
- Availability
- Endpoint responds
- Correct service
- Required action and records are completed accurately
Service objective
Illustrative data; not a real customer record or a prescribed policy.
- Endpointresponding
Availability evidence
- Ledger eventmissing
Correctness failure
- Customer outcomeunproven
Response speed is insufficient
Correctness and coverage are part of reliability
Measure the critical outcome beyond uptime. Correctness and coverage are part of reliability.
- Failure mode 1avoid
- Use only HTTP success counts. The financial action may still fail.
- Failure mode 2avoid
- Ignore people and partners. They are dependencies too.
- Failure mode 3avoid
- Set a target without an owner. Breaches may receive no decision.
Control retries and overload
- BoundLimit attempts and waiting work
- SpreadUse backoff and jitter
- ProtectPrioritize accepted critical obligations
Retries can help recover transient failures but can also amplify load. Use bounded attempts, backoff, jitter, and idempotent operation keys where appropriate. Coordinate retry behavior across layers so a client, gateway, worker, and vendor do not multiply attempts unexpectedly.
Apply backpressure and admission control to protect critical work. Rejecting or deferring new optional work can be safer than allowing every queue to grow without bound. Preserve accepted financial obligations and give clients an accurate status. Overload handling must not turn unknown outcomes into automatic new payments.
Retries are useful when failure is temporary, but they can amplify an outage. If every client immediately repeats a timed-out request, the recovering service receives extra work before it can clear the original demand. Use bounded retries, appropriate backoff, and a defined response when the outcome is unknown. Financial requests also need the idempotency and reconciliation controls described earlier in the book.
Protect the work that must continue. Admission controls, queue limits, and priority handling can prevent low-value traffic from consuming capacity required for critical operations. A queue needs a size and age policy; otherwise it can preserve requests until they are too old to be safe or useful. Recovery includes deciding which delayed work remains valid.
Inside the mechanism. Retries can turn a partial failure into overload. Each attempt consumes resources even when the financial operation remains idempotent. Bound retries, use backoff and jitter where appropriate, and avoid independent retry multiplication across layers. Admission control and load shedding need business-aware policies. Never infer that a timed-out financial action failed merely because the caller did not receive its response.
A concrete example. A retry consumes capacity even if it produces no new business value. Independent retry policies at several layers can multiply load during a partial failure. 950 intended requests generate 998 processing attempts under this retry assumption. Capacity is 1150 attempts per interval, and the critical path consumes 150 ms of a 240 ms budget. The request-based SLO view observes 100 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.
When the assumption fails. Clients, gateways, and workers retry the same unknown action at once. Bound retries, use stable action identifiers, and apply admission and backoff policies. The following worked sequence shows the reference condition, a stress condition, and a response condition with explicit synthetic data. These are comparative assumptions, not measured causal effects.
A retry consumes capacity even if it produces no new business value. Independent retry policies at several layers can multiply load during a partial failure.
- Retry recovery
- Another attempt after a transient failure
- Retry storm
- Repeated attempts amplify the outage
Retry multiplication
Illustrative data; not a real customer record or a prescribed policy.
- Client attempts3
Outer retry layer
- Worker attempts3
Inner retry layer
- Potential calls9
Before other retrying layers
Independent policies can multiply load
Coordinate bounded retries across layers. Independent policies can multiply load.
- Failure mode 1avoid
- Retry immediately forever. That can sustain the outage.
- Failure mode 2avoid
- Generate a new payment key each time. Retries can create duplicate effects.
- Failure mode 3avoid
- Accept unlimited work without capacity. Backlogs and deadlines can become unmanageable.
Use a durable handoff pattern
- TransactWrite business state and outbox together
- DeliverPublish pending events with retries
- ConsumeApply each business effect idempotently
A database update and a message publish can fail between the two operations. A transactional outbox records the business change and the event to publish in the same database transaction. A worker later delivers the event and tracks progress.
The pattern still requires idempotent consumers because delivery can repeat. Monitor unpublished events and reconcile them with downstream effects. Do not claim universal exactly-once processing from one outbox table. The useful contract is explicit: durable recording, retryable delivery, and a consumer that prevents duplicate business effects at its boundary.
The transactional outbox pattern records a business state change and a pending message in the same local database transaction. A separate publisher delivers the message and records progress. This reduces the gap in which a state change commits but its notification is lost. Delivery can still repeat, so consumers need their own duplicate handling. The pattern provides a durable handoff, not a universal promise of exactly-once effects across every external system. Reconciliation remains necessary at the financial boundary.
Inside the mechanism. A transactional outbox writes business state and a pending publication in one local transaction. A publisher later sends the effect and can retry after failure. Delivery can still repeat, so the consumer needs a stable business key and idempotent handling. This pattern closes a local handoff gap; it does not create a universal exactly-once guarantee across independent external systems. Reconciliation remains necessary.
A concrete example. A local database commit and a remote effect are separate failure boundaries. Recording intent durably allows recovery, but delivery can still repeat. 370 intended requests generate 388 processing attempts under this retry assumption. Capacity is 460 attempts per interval, and the critical path consumes 150 ms of a 280 ms budget. The request-based SLO view observes 100 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.
When the assumption fails. The process crashes between committing a decision and publishing its effect. Commit state and an outbox together and use idempotent consumers plus reconciliation. The following worked sequence shows the reference condition, a stress condition, and a response condition with explicit synthetic data. These are comparative assumptions, not measured causal effects.
A local database commit and a remote effect are separate failure boundaries. Recording intent durably allows recovery, but delivery can still repeat.
- Durable event
- Recorded for later delivery
- Single business effect
- Consumer prevents duplicate application
Outbox failure case
Illustrative data; not a real customer record or a prescribed policy.
- Databasecommitted
Business state and event saved
- Publishercrashed
Delivery incomplete
- Recoveryretry pending event
Consumer handles duplicates
At-least-once delivery can repeat
Combine durable handoff with idempotent consumption. At-least-once delivery can repeat.
- Failure mode 1avoid
- Publish then forget the database update. The two states can diverge.
- Failure mode 2avoid
- Claim exactly once without boundary definitions. The guarantee may not span all systems.
- Failure mode 3avoid
- Ignore outbox age. Undelivered obligations can accumulate.
Test restoration and data integrity
- RestoreRecover data and required dependencies
- ReconcileCheck events balances and pending work
- VerifyConfirm the critical service outcome
A backup is useful only if it can restore the required service and records. Test restoration with the actual dependencies, keys, schemas, and access controls. Define recovery time and recovery point objectives for the critical service.
After restoration, reconcile ledger state, event streams, decisions, and pending work. Restoring a database snapshot can lose later events or replay earlier ones unless the recovery design accounts for them. Use a controlled exercise with explicit expected results. A successful file restore does not prove the customer balances are correct.
Inside the mechanism. A restoration test must verify data integrity, authority, open obligations, and usable service, not only that a process starts. Compare expected counts, amounts, and unresolved actions with the restored state. Check the meaning of the recovery point and the treatment of events after it. A restored backup can be technically valid while missing a hold or external effect needed for safe operation.
A concrete example. A restored service needs the right data, control state, and unresolved actions. A recent backup timestamp does not prove usable recovery. 285 intended requests generate 299 processing attempts under this retry assumption. Capacity is 390 attempts per interval, and the critical path consumes 150 ms of a 350 ms budget. The request-based SLO view observes 100 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.
When the assumption fails. The restored store lacks open holds and unknown external payment outcomes. Test restoration against expected counts, balances, access, and outstanding obligations. The following worked sequence shows the reference condition, a stress condition, and a response condition with explicit synthetic data. These are comparative assumptions, not measured causal effects.
A restored service needs the right data, control state, and unresolved actions. A recent backup timestamp does not prove usable recovery.
- Recovery time objective
- Target time to restore service
- Recovery point objective
- Target tolerance for lost data history
Restore exercise
Illustrative data; not a real customer record or a prescribed policy.
- Snapshot02:00
Recovered base state
- Last valid event02:17
Additional history needed
- Reconciliationrequired
File restore alone is incomplete
Readable files do not prove correct balances
Verify restored business invariants. Readable files do not prove correct balances.
- Failure mode 1avoid
- Test backups without keys. Encrypted data may be unusable.
- Failure mode 2avoid
- Ignore events after the snapshot. Accepted obligations can disappear.
- Failure mode 3avoid
- Declare recovery at server startup. The service and records still need checks.
Exercise realistic failure combinations
- ScenarioCombine plausible dependent failures
- ExerciseObserve containment and recovery safely
- RepairRetest the specific weakness found
Real incidents often combine failures: a vendor outage, rising retries, an unavailable approver, and a backlog. Test combinations that threaten the critical service. Use safe synthetic traffic and controlled fault injection in an appropriate environment.
Define stop conditions, expected containment, and evidence to collect. Include operations and support in the exercise. The result should produce concrete repairs and a retest, not only a meeting note. A resilience exercise is successful when it reveals and closes weaknesses without creating uncontrolled customer impact.
Inside the mechanism. Combined failures expose interactions that isolated tests miss. Increased traffic can arrive with stale risk data, reduced reviewer capacity, and a partner outage. State the scenario, affected capabilities, permitted degraded behavior, and recovery evidence. The useful result is a tested service boundary and a clear account of which customer outcomes remain supported.
A concrete example. Real incidents can combine dependency loss, staff pressure, stale data, and increased demand. A single isolated fault test can miss their interaction. 1280 intended requests generate 1344 processing attempts under this retry assumption. Capacity is 1440 attempts per interval, and the critical path consumes 150 ms of a 260 ms budget. The request-based SLO view observes 100 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.
When the assumption fails. A partner outage arrives during a traffic spike and a review staffing shortfall. Model the combined service path and verify scoped degraded behavior and recovery evidence. The following worked sequence shows the reference condition, a stress condition, and a response condition with explicit synthetic data. These are comparative assumptions, not measured causal effects.
Real incidents can combine dependency loss, staff pressure, stale data, and increased demand. A single isolated fault test can miss their interaction.
- Component test
- One dependency fails alone
- Service exercise
- Several failures affect the customer outcome
Exercise scenario
Illustrative data; not a real customer record or a prescribed policy.
- Vendortimeout
Primary failure
- Queuenear capacity
Compounding condition
- Approverunavailable
Human dependency
Technical and human dependencies interact
Test the full operating response. Technical and human dependencies interact.
- Failure mode 1avoid
- Simulate only easy isolated failures. Combined weaknesses remain hidden.
- Failure mode 2avoid
- Run uncontrolled faults on live customer funds. The exercise needs a bounded safe environment.
- Failure mode 3avoid
- Close with a meeting summary only. Repairs require verification.
Chapter connections
This chapter builds on Treasury, liquidity, and settlement operations. Continue with Risk incidents, containment, and learning to follow the next part of the system. Use the glossary for terminology and risk mathematics for formulas and worked calculations.