Retry and overload control

Figure 01 / 12

Dependencies form a critical path

The case examines control retries and overload. Each edge is a required handoff in this illustrative path. A service can respond quickly while a downstream effect remains incomplete, so health needs both technical and business evidence.

Figure data and text version

From	To	Handoff
Retry and overload control	Decision service	Action request
Decision service	Evidence provider	Required evidence
Decision service	State store	Durable decision
State store	Effect publisher	Pending effect
Effect publisher	External rail	Financial action

A retry consumes capacity even if it produces no new business value. Independent retry policies at several layers can multiply load during a partial failure.

Clients, gateways, and workers retry the same unknown action at once.

All amounts, rates, capacity limits, and outcomes in this case are synthetic. The three conditions are separate assumptions for comparison. A better result in the response condition is not measured proof that the proposed control causes that improvement. The figures expose the calculation and its limits; a real deployment needs its own evidence.

Read the result

950 intended requests generate 1710 processing attempts under this retry assumption. Capacity is 632 attempts per interval, and the critical path consumes 305 ms of a 240 ms budget. The request-based SLO view observes 3000 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.

Figure 02 / 12

Allocate the latency budget explicitly

The four stages total 305 ms against a 240 ms budget, leaving -65 ms. Negative remaining time means this modeled path exceeds the target before adding any unmodeled overhead. The values are fixed teaching observations, not a latency guarantee.

Figure data and text version

Stage	Milliseconds
Ingress	25
Feature reads	55
Control evaluation	180
Commit and response	45
Remaining budget	-65

Figure 03 / 12

A slow minority changes the tail

The constructed distribution separates the median from high percentiles. Percentiles are order statistics over the same request population; summing stage p99 values is not generally the service p99. This chart is a teaching profile, not a measured production distribution. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Percentile	Latency ms
p50	214
p75	274
p90	336
p95	427
p99	702

Figure 04 / 12

Retries increase attempted work

950 original requests produce 1710 attempts under the stated average retry multiplier. The additional 760 attempts consume capacity even when their financial effect must remain idempotent. This simple model omits recursive retry storms across multiple layers.

Figure data and text version

Attempt type	Count
Original requests	950
Additional attempts	760

Figure 05 / 12

Attempted demand versus capacity

The interval has capacity for 632 attempts against 1710 attempted requests. The difference is 1078 unserved attempts in this simplified window. An unserved attempt is not necessarily an unexecuted business action: reconcile the stable action identifier before retrying a financial effect.

Figure data and text version

Measure	Attempts
Attempted demand	1,710
Processing capacity	632
Served this interval	632
Unserved this interval	1,078

Figure 06 / 12

Overload leaves a durable backlog

The six intervals use explicitly varied arrival multipliers and constant capacity. The queue carries forward unfinished attempts. Real systems also need a maximum age, admission policy, and expiry semantics so delayed work does not execute after its business authority has ended. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Interval	Pending attempts
T1	565
T2	1,472
T3	3,063
T4	5,338
T5	6,758
T6	7,494

Figure 07 / 12

Throughput can stay flat while demand rises

The completion count is limited by available work and capacity. A saturated completion line with a growing queue is evidence of overload, not stable end-to-end service. Inspect waiting time and customer outcomes alongside throughput. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Interval	Attempts	Completions
T1	1,197	632
T2	1,539	632
T3	2,223	632
T4	2,907	632
T5	2,052	632
T6	1,368	632

Figure 08 / 12

Write state and pending effect together

This sequence describes a transactional outbox within one datastore boundary. The publisher can deliver more than once, so the consumer also needs duplicate handling. A durable local handoff does not create a universal exactly-once guarantee across an external payment system.

Figure data and text version

Event	Commit boundary	Evidence
Validate action	Before transaction	Stable action key and permitted operation
Write state and outbox	One local transaction	Both records commit or neither does
Publish effect	After commit	Delivery may be retried
Consume effect	Consumer boundary	Idempotent handling and stored result
Reconcile	Independent check	Expected and observed effects agree

Figure 09 / 12

Repeated delivery and financial effects differ

The example contains 1710 transport attempts for 950 intended business actions. The protected effect count cannot be inferred from transport success alone. The table states the intended contract and the evidence needed to check it.

Figure data and text version

Measure	Illustrative value	Required evidence
Business actions	950	Stable unique action identifiers
Transport attempts	1,710	Delivery identifiers and retries
Permitted effects	950	One effect per authorized action
Observed effects	Must reconcile	Authoritative ledger or external record

Figure 10 / 12

An SLO budget is a measured allowance

For 100,000 eligible requests and a 99.9% illustrative SLO, the budget is 100 bad requests. This case observes 3000. Eligibility, success, and measurement windows must be fixed before interpreting the result. An SLO allowance never overrides a legal or financial correctness requirement.

Figure data and text version

Measure	Requests
Eligible population	100,000
Bad requests allowed	100
Bad requests observed	3,000
Remaining budget	-2,900

Figure 11 / 12

Dependency failures require scoped behavior

The dependency is payment gateway. The response depends on the operation, required control, and evidence available. The table gives illustrative behavior classes, not a universal fail-open rule.

Figure data and text version

Failure state	Illustrative response	Evidence to retain
Optional signal late	Use approved degraded policy	Missing signal and selected policy version
Required control unavailable	Hold the affected action	Pending owner and expiry
Unknown external outcome	Query and reconcile	Original external action identifier
Publisher retry	Repeat delivery safely	Stable business key and attempt history

Figure 12 / 12

Recovery ends with a reconciled population

The last successful health probe does not close the incident. Recovery must classify affected actions, resolve unknown outcomes, restore required controls, and verify money and records. A replay is allowed only under the action’s current authority and idempotency contract.

Figure data and text version

Stage	Completion evidence
Contain	Clients, gateways, and workers retry the same unknown action at once.
Classify	Known success, known failure, unknown outcome
Resolve	Repair or replay by stable action identifier
Verify	Counts, amounts, and control evidence reconcile
Close	Owners accept remaining exceptions explicitly

Connect the result to the system

Bound retries, use stable action identifiers, and apply admission and backoff policies.

Check the population, evidence, permitted action, and actual effect together. A balanced calculation can still use the wrong population; a successful response can still leave an unknown financial outcome. The case’s numerical result applies only to its stated assumptions.