Product changes that remove abuse incentives

Figure 01 / 12

Start with the assigned population

This teaching experiment assigns 5,000 eligible units to comparison and 5,000 to treatment. The units are eligible customer accounts. Mandatory controls are outside the experimental choice. Eligibility and assignment are defined before observing outcomes.

Figure data and text version

Assigned arm	Units
Comparison	5,000
Treatment	5,000

A product can repeatedly grant value because its cancellation or restoration rules are unclear. Fixing the state model may reduce the incentive without adding friction to every customer.

Measure the defined loss path, legitimate refunds, support load, and customer completion under a valid comparison.

All amounts, rates, capacity limits, and outcomes in this case are synthetic. The three conditions are separate assumptions for comparison. A better result in the response condition is not measured proof that the proposed control causes that improvement. The figures expose the calculation and its limits; a real deployment needs its own evidence.

Read the result

The comparison arm has 231/5000 adverse outcomes (4.62%) and the treatment arm has 151/5000 (3.02%). The absolute difference is -1.60 percentage points, with an illustrative large-sample 95% interval from -2.35 to -0.85. Interpretation depends on assignment integrity, outcome maturity, independence, and the actual decision being evaluated.

Figure 02 / 12

Retain events and nonevents in both arms

The defined adverse outcome occurs 231 times in comparison and 151 times in treatment. Each row includes every assigned unit under this complete-outcome illustration. Omitting nonevents or unresolved cases changes the denominator and the estimand.

Figure data and text version

Assigned arm	Adverse outcome	Other outcome
Comparison	231	4,769
Treatment	151	4,849

Figure 03 / 12

Absolute and relative effects answer different questions

Treatment minus comparison is -1.60 percentage points. The relative change is -34.63% when the comparison rate is nonzero. A percentage-point difference and a percentage change must not share the same label.

Figure data and text version

Measure	Value	Unit
Comparison rate	4.62	Percent
Treatment rate	3.02	Percent
Absolute difference	-1.6	Percentage points
Relative change	-34.632	Percent of comparison rate

Figure 04 / 12

Each arm’s rate has sampling uncertainty

The 95% Wilson intervals are [4.07, 5.24]% and [2.58, 3.53]%. These intervals use independent Bernoulli sampling assumptions. Shared customers or merchants can violate that independence and require an appropriate clustered analysis.

Figure data and text version

Arm	Observed %	Lower 95%	Upper 95%
Comparison	4.62	4.072	5.238
Treatment	3.02	2.581	3.532

Figure 05 / 12

Uncertainty in the difference

The normal-approximation 95% interval for treatment minus comparison is [-2.35, -0.85] percentage points. It is a teaching large-sample interval, not a replacement for the analysis appropriate to the design, low counts, sequential monitoring, or multiple comparisons.

Figure data and text version

Difference measure	Percentage points
Lower bound	-2.351
Point estimate	-1.6
Upper bound	-0.849

Figure 06 / 12

A fixed outcome horizon prevents premature comparison

Both arms use the same explicitly constructed maturation fractions. Earlier data can undercount the final outcome even when assignment is correct. Stopping at the first favorable partial result also changes the statistical interpretation unless the monitoring design accounts for it. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Observation age	Comparison events	Treatment events
Day 1	46	30
Day 3	92	60
Day 7	150	98
Day 14	196	128
Day 30	231	151

Figure 07 / 12

Assignment and received treatment differ

5000 units are assigned to treatment, while 4800 receive the modeled action. The primary assignment-based comparison retains all assigned units. Restricting to action recipients selects a post-assignment subgroup and can introduce bias.

Figure data and text version

Population	Units
Assigned comparison	5,000
Comparison action observed	4,900
Assigned treatment	5,000
Treatment action observed	4,800

Figure 08 / 12

Correlated units reduce independent information

The design-effect illustration is 1 + (m − 1)ρ with cluster size m = 6. At the case’s assumed ρ = 0.025, the design effect is 1.125 and n/design-effect is about 8,888.9. This approximation assumes a simple equal-cluster design; it is not a universal effective-sample-size formula. Horizontal positions are the labeled observations or scenarios; equal spacing does not imply equal numerical increments.

Figure data and text version

Intracluster correlation	Approximate effective n
0.00	10,000
0.01	9,523.8
0.03	8,888.9
0.05	8,000
0.10	6,666.7
0.20	5,000

Figure 09 / 12

Segment totals must reconcile to the whole

The two synthetic segments partition each arm. Their outcome counts add back to the original totals. Segment effects can differ from the overall effect, but small cells, selection, and multiple comparisons must be considered before interpreting a subgroup result.

Figure data and text version

Segment	Comparison n	Comparison events	Treatment n	Treatment events
Segment A	2,500	150	2,500	60
Segment B	2,500	81	2,500	91

Figure 10 / 12

The primary outcome is only one decision input

The primary target is Defined unrecovered misuse loss. A release also considers customer friction, queue capacity, required controls, and the uncertainty of the result. A favorable loss metric cannot justify an action that violates a separate obligation.

Figure data and text version

Dimension	Required definition
Primary outcome	Defined unrecovered misuse loss
Customer impact	Completion, delay, complaints, and access
Operating cost	Review effort and capacity in the same window
Mandatory controls	Outside randomized relaxation
Decision owner	risk operations

Figure 11 / 12

Specify the analysis before observing the result

The protocol fixes assignment, exclusions, metrics, outcome maturity, and the comparison method. Changing these after seeing results can make an ordinary noise fluctuation look like an improvement. Amendments need a documented reason and a clear distinction between confirmatory and exploratory work.

Figure data and text version

Stage	Timing	Record
Define estimand	Before assignment	close the loop into product design
Assign units	Experiment start	Stable unit and arm
Collect outcomes	Fixed window	Same definition for both arms
Analyze	At planned maturity	Chosen method and uncertainty
Decide	After review	Net effect, guardrails, and limitations

Figure 12 / 12

What this example establishes and omits

The arithmetic shows how the stated counts become rates, differences, and intervals. It does not prove a causal effect in production. Causal interpretation requires a valid assignment process, appropriate handling of interference and missing outcomes, and analysis consistent with the design.

Figure data and text version

Property	In this example	Production requirement
Assignment	Assumed valid	Verify implementation and unit integrity
Outcomes	Complete at final window	Resolve delayed and missing outcomes
Independence	Simple interval assumption	Account for clustering or interference
Cost	Not included in rate difference	Measure net economics separately

Connect the result to the system

Measure the defined loss path, legitimate refunds, support load, and customer completion under a valid comparison.

Check the population, evidence, permitted action, and actual effect together. A balanced calculation can still use the wrong population; a successful response can still leave an unknown financial outcome. The case’s numerical result applies only to its stated assumptions.