Combined failure exercise record

Figure 01 / 12

Dependencies form a critical path

The case examines exercise realistic failure combinations. Each edge is a required handoff in this illustrative path. A service can respond quickly while a downstream effect remains incomplete, so health needs both technical and business evidence.

Figure data and text version

From	To	Handoff
Combined failure exercise record	Decision service	Action request
Decision service	Evidence provider	Required evidence
Decision service	State store	Durable decision
State store	Effect publisher	Pending effect
Effect publisher	External rail	Financial action

Real incidents can combine dependency loss, staff pressure, stale data, and increased demand. A single isolated fault test can miss their interaction.

A partner outage arrives during a traffic spike and a review staffing shortfall.

All amounts, rates, capacity limits, and outcomes in this case are synthetic. The three conditions are separate assumptions for comparison. A better result in the response condition is not measured proof that the proposed control causes that improvement. The figures expose the calculation and its limits; a real deployment needs its own evidence.

Read the result

1280 intended requests generate 2304 processing attempts under this retry assumption. Capacity is 792 attempts per interval, and the critical path consumes 305 ms of a 260 ms budget. The request-based SLO view observes 3000 bad requests against an illustrative allowance of 100. These measurements must be connected to the financial effect and control evidence before declaring recovery.

Figure 02 / 12

Allocate the latency budget explicitly

The four stages total 305 ms against a 260 ms budget, leaving -45 ms. Negative remaining time means this modeled path exceeds the target before adding any unmodeled overhead. The values are fixed teaching observations, not a latency guarantee.

Figure data and text version

Stage	Milliseconds
Ingress	25
Feature reads	55
Control evaluation	180
Commit and response	45
Remaining budget	-45

Figure 03 / 12

A slow minority changes the tail

The constructed distribution separates the median from high percentiles. Percentiles are order statistics over the same request population; summing stage p99 values is not generally the service p99. This chart is a teaching profile, not a measured production distribution. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Percentile	Latency ms
p50	214
p75	274
p90	336
p95	427
p99	702

Figure 04 / 12

Retries increase attempted work

1280 original requests produce 2304 attempts under the stated average retry multiplier. The additional 1024 attempts consume capacity even when their financial effect must remain idempotent. This simple model omits recursive retry storms across multiple layers.

Figure data and text version

Attempt type	Count
Original requests	1,280
Additional attempts	1,024

Figure 05 / 12

Attempted demand versus capacity

The interval has capacity for 792 attempts against 2304 attempted requests. The difference is 1512 unserved attempts in this simplified window. An unserved attempt is not necessarily an unexecuted business action: reconcile the stable action identifier before retrying a financial effect.

Figure data and text version

Measure	Attempts
Attempted demand	2,304
Processing capacity	792
Served this interval	792
Unserved this interval	1,512

Figure 06 / 12

Overload leaves a durable backlog

The six intervals use explicitly varied arrival multipliers and constant capacity. The queue carries forward unfinished attempts. Real systems also need a maximum age, admission policy, and expiry semantics so delayed work does not execute after its business authority has ended. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Interval	Pending attempts
T1	821
T2	2,103
T3	4,306
T4	7,431
T5	9,404
T6	10,455

Figure 07 / 12

Throughput can stay flat while demand rises

The completion count is limited by available work and capacity. A saturated completion line with a growing queue is evidence of overload, not stable end-to-end service. Inspect waiting time and customer outcomes alongside throughput. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Interval	Attempts	Completions
T1	1,613	792
T2	2,074	792
T3	2,995	792
T4	3,917	792
T5	2,765	792
T6	1,843	792

Figure 08 / 12

Write state and pending effect together

This sequence describes a transactional outbox within one datastore boundary. The publisher can deliver more than once, so the consumer also needs duplicate handling. A durable local handoff does not create a universal exactly-once guarantee across an external payment system.

Figure data and text version

Event	Commit boundary	Evidence
Validate action	Before transaction	Stable action key and permitted operation
Write state and outbox	One local transaction	Both records commit or neither does
Publish effect	After commit	Delivery may be retried
Consume effect	Consumer boundary	Idempotent handling and stored result
Reconcile	Independent check	Expected and observed effects agree

Figure 09 / 12

Repeated delivery and financial effects differ

The example contains 2304 transport attempts for 1280 intended business actions. The protected effect count cannot be inferred from transport success alone. The table states the intended contract and the evidence needed to check it.

Figure data and text version

Measure	Illustrative value	Required evidence
Business actions	1,280	Stable unique action identifiers
Transport attempts	2,304	Delivery identifiers and retries
Permitted effects	1,280	One effect per authorized action
Observed effects	Must reconcile	Authoritative ledger or external record

Figure 10 / 12

An SLO budget is a measured allowance

For 100,000 eligible requests and a 99.9% illustrative SLO, the budget is 100 bad requests. This case observes 3000. Eligibility, success, and measurement windows must be fixed before interpreting the result. An SLO allowance never overrides a legal or financial correctness requirement.

Figure data and text version

Measure	Requests
Eligible population	100,000
Bad requests allowed	100
Bad requests observed	3,000
Remaining budget	-2,900

Figure 11 / 12

Dependency failures require scoped behavior

The dependency is external processor. The response depends on the operation, required control, and evidence available. The table gives illustrative behavior classes, not a universal fail-open rule.

Figure data and text version

Failure state	Illustrative response	Evidence to retain
Optional signal late	Use approved degraded policy	Missing signal and selected policy version
Required control unavailable	Hold the affected action	Pending owner and expiry
Unknown external outcome	Query and reconcile	Original external action identifier
Publisher retry	Repeat delivery safely	Stable business key and attempt history

Figure 12 / 12

Recovery ends with a reconciled population

The last successful health probe does not close the incident. Recovery must classify affected actions, resolve unknown outcomes, restore required controls, and verify money and records. A replay is allowed only under the action’s current authority and idempotency contract.

Figure data and text version

Stage	Completion evidence
Contain	A partner outage arrives during a traffic spike and a review staffing shortfall.
Classify	Known success, known failure, unknown outcome
Resolve	Repair or replay by stable action identifier
Verify	Counts, amounts, and control evidence reconcile
Close	Owners accept remaining exceptions explicitly

Connect the result to the system

Model the combined service path and verify scoped degraded behavior and recovery evidence.

Check the population, evidence, permitted action, and actual effect together. A balanced calculation can still use the wrong population; a successful response can still leave an unknown financial outcome. The case’s numerical result applies only to its stated assumptions.