Metrics version 1.0
Method
Every number in an isovy report is calculated from blind ratings, the AI tool's assessment and the rater's final judgment. A metric is shown only when there is enough data. All interpretation thresholds are provisional.
1.Flip rate
- Formula
- For each case with ≥2 raters: share of rater pairs whose blind decisions differ. Averaged over cases.
- Interpretation
- The chance that two raters reach different outcomes for the same case. 0 = always agree.
- Minimum data
- 5 cases with at least 2 raters.
- Known limitations
- Depends on the base rate of advance/reject; a step where almost everyone advances will show low flip rates.
2.Noise index
- Formula
- Per criterion and overall: mean over cases of the population standard deviation σ = √(Σ(x − x̄)² / n) of blind scores.
- Interpretation
- Typical spread between raters on the same case, in scale points (1–5).
- Minimum data
- 5 cases with at least 2 raters.
- Known limitations
- Population SD understates spread for very small panels; compare like with like.
3.Rater level
- Formula
- For each rater: mean over their cases of (own score − mean score of the other raters on that case).
- Interpretation
- Positive = lenient, negative = severe, relative to colleagues — not to a true value.
- Minimum data
- At least 2 raters sharing cases.
- Known limitations
- Relative measure: if all raters are lenient together, it shows nothing.
4.Human–AI gap
- Formula
- Mean |blind overall − normalised AI score|; and share of cases where blind decision = AI decision.
- Interpretation
- How far the human view, formed before seeing the AI, is from the AI.
- Minimum data
- AI scores imported for the decision point.
- Known limitations
- AI scores are rescaled linearly to 1–5: n = 1 + 4·(raw − min)/(max − min). Non-linear AI scales may distort the gap.
5.Convergence toward AI (automation bias index)
- Formula
- Where blind ≠ AI: (|blind − AI| − |final − AI|) / |blind − AI|, averaged.
- Interpretation
- 0 = no movement, 1 = fully adopted the AI score, negative = moved away from it.
- Minimum data
- 5 revisions where the blind score differed from the AI.
- Known limitations
- Movement toward the AI can be legitimate learning. Read together with reason codes.
6.Decision change after reveal
- Formula
- Share where final decision ≠ blind decision, split into toward AI and away from AI.
- Interpretation
- How often seeing the AI changes the outcome.
- Minimum data
- 5 completed revisions.
- Known limitations
- Does not show whether the change was correct.
7.Override rate
- Formula
- Share where final decision ≠ AI decision, with the distribution of reason codes.
- Interpretation
- How often humans actively disagree with the AI.
- Minimum data
- 5 completed revisions.
- Known limitations
- A low override rate is not by itself evidence of good or bad oversight.
8.Review time after reveal
- Formula
- Median seconds from reveal to final submit. Share under 10 seconds flagged as possible rubber-stamping.
- Interpretation
- Very short reviews suggest the AI is accepted without real consideration.
- Minimum data
- 5 completed revisions.
- Known limitations
- Fast is not always careless; a rater may already have formed a clear view.
9.Tare round: d′ and c (signal detection theory)
- Formula
- H = (hits + 0.5)/(strong + 1); F = (false alarms + 0.5)/(weak + 1) (log-linear correction). d′ = z(H) − z(F). c = −(z(H) + z(F))/2.
- Interpretation
- d′: how well the rater separates strong from weak vignettes. c: negative = lenient, positive = strict.
- Minimum data
- 4 strong and 4 weak vignettes.
- Known limitations
- Vignettes are fictional and designed; performance on real cases may differ.
10.Tare spread and test–retest
- Formula
- Spread: mean over vignettes of the SD of overall scores between raters. Test–retest: share of identical decisions by the same rater on the same vignette across rounds.
- Interpretation
- Spread shows noise on identical material; test–retest shows consistency over time.
- Minimum data
- Two or more raters; two rounds with the same vignettes for test–retest.
- Known limitations
- Raters may remember vignettes between rounds.
References
- Kahneman, D., Sibony, O. & Sunstein, C. R. (2021). Noise: A flaw in human judgment. Little, Brown Spark.
- Parasuraman, R. & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410.
- Green, D. M. & Swets, J. A. (1966). Signal detection theory and psychophysics. Wiley.
- Hautus, M. J. (1995). Corrections for extreme proportions and their biasing effects on estimated values of d′. Behavior Research Methods, Instruments, & Computers, 27(1), 46–51.