Worked case · Controlled response · 12 figures

Durable handoff

Use a durable handoff pattern

Figure 01 / 12

Dependencies form a critical path

Dependencies form a critical path — Durable handoff. Illustrative service dependency graph. Exact values are in the figure data below.
Illustrative service dependency graph

The case examines use a durable handoff pattern. Each edge is a required handoff in this illustrative path. A service can respond quickly while a downstream effect remains incomplete, so health needs both technical and business evidence.

Figure data and text version
FromToHandoff
Durable handoffDecision serviceAction request
Decision serviceEvidence providerRequired evidence
Decision serviceState storeDurable decision
State storeEffect publisherPending effect
Effect publisherExternal railFinancial action

A local database commit and a remote effect are separate failure boundaries. Recording intent durably allows recovery, but delivery can still repeat.

Commit state and an outbox together and use idempotent consumers plus reconciliation.

All amounts, rates, capacity limits, and outcomes in this case are synthetic. The three conditions are separate assumptions for comparison. A better result in the response condition is not measured proof that the proposed control causes that improvement. The figures expose the calculation and its limits; a real deployment needs its own evidence.

Read the result

370 intended requests generate 400 processing attempts under this retry assumption. Capacity is 529 attempts per interval, and the critical path consumes 145 ms of a 280 ms budget. The request-based SLO view observes 50 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.

Model inputs and calculated values

Inputs below are the case-specific values. Each figure states the condition-specific assumptions and units used in its calculation. Calculated values are rounded for display.

InputValue
arrivals370
capacity460
budgetMs280
Calculated valueResult
arrivals370
attempts400
capacity529
latency145
budget280
remaining135
window100,000
bad50
allowed100
Figure 02 / 12

Allocate the latency budget explicitly

Allocate the latency budget explicitly — Durable handoff. Milliseconds on one critical path. Exact values are in the figure data below.
Milliseconds on one critical path

The four stages total 145 ms against a 280 ms budget, leaving 135 ms. Negative remaining time means this modeled path exceeds the target before adding any unmodeled overhead. The values are fixed teaching observations, not a latency guarantee.

Figure data and text version
StageMilliseconds
Ingress18
Feature reads30
Control evaluation75
Commit and response22
Remaining budget135
Figure 03 / 12

A slow minority changes the tail

A slow minority changes the tail — Durable handoff. Milliseconds; explicitly constructed percentile profile. Exact values are in the figure data below.
Milliseconds; explicitly constructed percentile profile

The constructed distribution separates the median from high percentiles. Percentiles are order statistics over the same request population; summing stage p99 values is not generally the service p99. This chart is a teaching profile, not a measured production distribution. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version
PercentileLatency ms
p50102
p75130
p90160
p95203
p99334
Figure 04 / 12

Retries increase attempted work

Retries increase attempted work — Durable handoff. Attempts per interval. Exact values are in the figure data below.
Attempts per interval

370 original requests produce 400 attempts under the stated average retry multiplier. The additional 30 attempts consume capacity even when their financial effect must remain idempotent. This simple model omits recursive retry storms across multiple layers.

Figure data and text version
Attempt typeCount
Original requests370
Additional attempts30
Figure 05 / 12

Attempted demand versus capacity

Attempted demand versus capacity — Durable handoff. Attempts; measures are not all additive. Exact values are in the figure data below.
Attempts; measures are not all additive

The interval has capacity for 529 attempts against 400 attempted requests. The difference is 0 unserved attempts in this simplified window. An unserved attempt is not necessarily an unexecuted business action: reconcile the stable action identifier before retrying a financial effect.

Figure data and text version
MeasureAttempts
Attempted demand400
Processing capacity529
Served this interval400
Unserved this interval0
Figure 06 / 12

Overload leaves a durable backlog

Overload leaves a durable backlog — Durable handoff. Attempts waiting at interval end. Exact values are in the figure data below.
Attempts waiting at interval end

The six intervals use explicitly varied arrival multipliers and constant capacity. The queue carries forward unfinished attempts. Real systems also need a maximum age, admission policy, and expiry semantics so delayed work does not execute after its business authority has ended. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version
IntervalPending attempts
T10
T20
T30
T4150
T5101
T60
Figure 07 / 12

Throughput can stay flat while demand rises

Throughput can stay flat while demand rises — Durable handoff. Attempts per interval. Exact values are in the figure data below.
Attempts per interval

The completion count is limited by available work and capacity. A saturated completion line with a growing queue is evidence of overload, not stable end-to-end service. Inspect waiting time and customer outcomes alongside throughput. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version
IntervalAttemptsCompletions
T1280280
T2360360
T3519519
T4679529
T5480529
T6320421
Figure 08 / 12

Write state and pending effect together

Write state and pending effect together — Durable handoff. Illustrative reliable handoff. Exact values are in the figure data below.
Illustrative reliable handoff

This sequence describes a transactional outbox within one datastore boundary. The publisher can deliver more than once, so the consumer also needs duplicate handling. A durable local handoff does not create a universal exactly-once guarantee across an external payment system.

Figure data and text version
EventCommit boundaryEvidence
Validate actionBefore transactionStable action key and permitted operation
Write state and outboxOne local transactionBoth records commit or neither does
Publish effectAfter commitDelivery may be retried
Consume effectConsumer boundaryIdempotent handling and stored result
ReconcileIndependent checkExpected and observed effects agree
Figure 09 / 12

Repeated delivery and financial effects differ

Repeated delivery and financial effects differ — Durable handoff. Counts at different boundaries. Exact values are in the figure data below.
Counts at different boundaries

The example contains 400 transport attempts for 370 intended business actions. The protected effect count cannot be inferred from transport success alone. The table states the intended contract and the evidence needed to check it.

Figure data and text version
MeasureIllustrative valueRequired evidence
Business actions370Stable unique action identifiers
Transport attempts400Delivery identifiers and retries
Permitted effects370One effect per authorized action
Observed effectsMust reconcileAuthoritative ledger or external record
Figure 10 / 12

An SLO budget is a measured allowance

An SLO budget is a measured allowance — Durable handoff. Request-based illustrative SLO; one measurement window. Exact values are in the figure data below.
Request-based illustrative SLO; one measurement window

For 100,000 eligible requests and a 99.9% illustrative SLO, the budget is 100 bad requests. This case observes 50. Eligibility, success, and measurement windows must be fixed before interpreting the result. An SLO allowance never overrides a legal or financial correctness requirement.

Figure data and text version
MeasureRequests
Eligible population100,000
Bad requests allowed100
Bad requests observed50
Remaining budget50
Figure 11 / 12

Dependency failures require scoped behavior

Dependency failures require scoped behavior — Durable handoff. Dependency-specific policy boundary. Exact values are in the figure data below.
Dependency-specific policy boundary

The dependency is message transport. The response depends on the operation, required control, and evidence available. The table gives illustrative behavior classes, not a universal fail-open rule.

Figure data and text version
Failure stateIllustrative responseEvidence to retain
Optional signal lateUse approved degraded policyMissing signal and selected policy version
Required control unavailableHold the affected actionPending owner and expiry
Unknown external outcomeQuery and reconcileOriginal external action identifier
Publisher retryRepeat delivery safelyStable business key and attempt history
Figure 12 / 12

Recovery ends with a reconciled population

Recovery ends with a reconciled population — Durable handoff. Business recovery sequence. Exact values are in the figure data below.
Business recovery sequence

The last successful health probe does not close the incident. Recovery must classify affected actions, resolve unknown outcomes, restore required controls, and verify money and records. A replay is allowed only under the action’s current authority and idempotency contract.

Figure data and text version
StageCompletion evidence
ContainThe process crashes between committing a decision and publishing its effect.
ClassifyKnown success, known failure, unknown outcome
ResolveRepair or replay by stable action identifier
VerifyCounts, amounts, and control evidence reconcile
CloseOwners accept remaining exceptions explicitly

Connect the result to the system

Commit state and an outbox together and use idempotent consumers plus reconciliation.

Check the population, evidence, permitted action, and actual effect together. A balanced calculation can still use the wrong population; a successful response can still leave an unknown financial outcome. The case’s numerical result applies only to its stated assumptions.

Sources and further reading

The chapter sources support the concepts and scope. They do not prescribe the synthetic model rates.

  1. Google SRE: handling overload
  2. NIST: Cybersecurity Framework
  3. PostgreSQL: transaction isolation