Worked case · Reference condition · 12 figures

Model operating health

Monitor the model as part of a system

Figure 01 / 12

The population that monitoring actually sees

The population that monitoring actually sees — Model operating health. Count; model decisions in one day. Exact values are in the figure data below.
Count; model decisions in one day

38,220 of 39,000 source items enter this monitoring calculation. The missing 780 items are a coverage gap, not evidence of low risk. Reconcile stable identifiers and amounts where appropriate before interpreting the alert rate.

Figure data and text version
Population stateItems
Included in monitoring38,220
Absent from this run780

A model can retain its score distribution while its inputs, labels, or downstream actions fail. Monitoring should join technical quality with mature business outcomes.

The reference case starts with the stated population and a functioning evidence path. The owner is risk operations.

All amounts, rates, capacity limits, and outcomes in this case are synthetic. The three conditions are separate assumptions for comparison. A better result in the response condition is not measured proof that the proposed control causes that improvement. The figures expose the calculation and its limits; a real deployment needs its own evidence.

Read the result

The daily source population is 39,000 items, but 780 are outside the completed monitoring run. The included population creates 459 hits and 376 unique cases. With 95 cases already open and capacity for 460, the queue closes at 11. Coverage, duplicate work, and staffing are separate causes; reducing one number does not prove that the overall control improved.

Model inputs and calculated values

Inputs below are the case-specific values. Each figure states the condition-specific assumptions and units used in its calculation. Calculated values are rounded for display.

InputValue
population39,000
alertRate0.012
capacity460
backlog95
Calculated valueResult
population39,000
covered38,220
missing780
raw459
duplicates83
cases376
opening95
resolved460
closing11
capacity460
Figure 02 / 12

From scenario hits to unique cases

From scenario hits to unique cases — Model operating health. Count; hit and case units differ. Exact values are in the figure data below.
Count; hit and case units differ

459 raw hits become 376 cases after removing 83 repeated references to the same case under the stated merge rule. Deduplication should reduce duplicate work while retaining the underlying events and reasons. It must not merge unrelated activity merely because values look similar.

Figure data and text version
StageCount
Raw scenario hits459
Duplicate references83
Unique cases376
Figure 03 / 12

The queue balance is an accounting identity

The queue balance is an accounting identity — Model operating health. Cases per day. Exact values are in the figure data below.
Cases per day

Opening backlog 95 + arrivals 376 − completed cases 460 = closing backlog 11. Completion is capped by both available work and the stated capacity. This identity is useful even when average handling times are uncertain.

Figure data and text version
MovementCasesDefinition
Opening backlog95Unresolved at window start
New cases376Unique arrivals in this window
Completed460Reached a defined completion state
Closing backlog11Unresolved at window end
Figure 04 / 12

Backlog accumulates across uneven days

Backlog accumulates across uneven days — Model operating health. Cases unresolved at day end. Exact values are in the figure data below.
Cases unresolved at day end

This deterministic six-day example applies a stated daily arrival multiplier and a constant completion capacity. Unused capacity does not carry forward as completed work. The series is a workload illustration; it omits variable case durations and specialist routing. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version
DayClosing backlog
D111
D20
D30
D4104
D558
D60
Figure 05 / 12

Arrivals and completions need separate plots

Arrivals and completions need separate plots — Model operating health. Cases per day. Exact values are in the figure data below.
Cases per day

The service line cannot exceed the work available that day. A team can complete more cases than arrive while clearing an opening backlog. Conversely, stable staffing can coexist with a growing queue when arrivals remain higher than capacity. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version
DayArrivalsCompletions
D1376460
D2320331
D3451451
D4564460
D5414460
D6263321
Figure 06 / 12

An illustrative age profile of open work

An illustrative age profile of open work — Model operating health. Cases; constructed snapshot. Exact values are in the figure data below.
Cases; constructed snapshot

The closing backlog is 11 cases. The 50/30/remainder split is an explicit illustrative age allocation, not a distribution inferred from the arrival model. Production age buckets must come from each case’s actual receipt and status history.

Figure data and text version
Age bucketOpen cases
Under one day6
One to three days3
Over three days2
Figure 07 / 12

Completion outcomes are not criminal labels

Completion outcomes are not criminal labels — Model operating health. Completed review tasks; synthetic disposition allocation. Exact values are in the figure data below.
Completed review tasks; synthetic disposition allocation

Of 460 completed review tasks, 78 are referred for further assessment, 55 need another evidence action, and 327 close under the stated procedure. These are operational outcomes. None is a probability of money laundering or a substitute for a reporting decision.

Figure data and text version
Review dispositionTasks
Referred for assessment78
Further evidence action55
Closed under procedure327
Figure 08 / 12

Known-event tests assess specific coverage

Known-event tests assess specific coverage — Model operating health. Synthetic coverage test. Exact values are in the figure data below.
Synthetic coverage test

The injected test set contains 100 known test events designed to exercise monitor the model as part of a system. The system surfaces 92. That 92% detection result measures this constructed test set only; it does not establish population-wide detection of illicit activity.

Figure data and text version
MeasureCountInterpretation
Injected known events100Defined test population
Detected by the scenario92Expected evidence reached the control
Not detected8Investigate data, logic, and delivery
Real-world illicit prevalenceUnknownNot inferred from this test
Figure 09 / 12

Data quality has several independent dimensions

Data quality has several independent dimensions — Model operating health. Count; overlapping field checks. Exact values are in the figure data below.
Count; overlapping field checks

Each row is a different field requirement over the same source population. Completeness alone does not establish that values are accurate or current. The case uses explicit illustrative missing counts to show how data quality can affect scenario coverage.

Figure data and text version
Field requirementPresentAbsent
Party reference38,805195
Event time38,610390
Counterparty context37,8301,170
Figure 10 / 12

Type the relationship before drawing an inference

Type the relationship before drawing an inference — Model operating health. Typed evidence relationships. Exact values are in the figure data below.
Typed evidence relationships

This evidence map distinguishes a customer relationship, a transfer, and a case reference. The links support monitor the model as part of a system; they do not imply common ownership or intent. A shared data point is a lead whose meaning depends on source, time, and context.

Figure data and text version
FromToRelationship
Model operating healthCounterparty AObserved transfer
Model operating healthProfile recordDeclared business
Counterparty ACase recordEvidence reference
Profile recordCase recordContext for review
Figure 11 / 12

Case clocks start from defined events

Case clocks start from defined events — Model operating health. Illustrative internal timing. Exact values are in the figure data below.
Illustrative internal timing

A legal deadline, an internal response target, and an evidence-expiry date can start from different events. The hours here are internal teaching targets only. They are not BSA, sanctions, consumer-protection, or other statutory deadlines.

Figure data and text version
EventRelative timeOperational meaning
Source eventT0Activity occurred
Data arrivalT0 + 2 hoursThe monitoring system learned it
Case createdT0 + 3 hoursWork entered an owned queue
Internal review targetT0 + 27 hoursIllustrative 24-hour target from case creation
DispositionRecorded separatelyUse actual decision and reporting records
Figure 12 / 12

The end-to-end delivery contract

The end-to-end delivery contract — Model operating health. Operational control path. Exact values are in the figure data below.
Operational control path

The scenario is incomplete until the intended evidence reaches an owned case. For monitor the model as part of a system, verify source coverage, hit creation, queue acceptance, reviewer access, and final disposition separately. A green job status proves only that a job reported completion.

Figure data and text version
BoundaryAcceptance evidence
Source to scenario38220 included source items; 780 missing
Scenario to case459 hits linked to 376 unique cases
Case to reviewerRequired evidence visible under the reviewer role
Reviewer to outcomeDisposition, rationale, and any separate reporting decision retained

Connect the result to the system

Monitor input coverage, action delivery, queue age, and outcome maturity together.

Check the population, evidence, permitted action, and actual effect together. A balanced calculation can still use the wrong population; a successful response can still leave an unknown financial outcome. The case’s numerical result applies only to its stated assumptions.

Sources and further reading

The chapter sources support the concepts and scope. They do not prescribe the synthetic model rates.

  1. scikit-learn: model evaluation metrics
  2. scikit-learn: probability calibration
  3. Federal Reserve SR 26-2: revised model-risk guidance (2026)