Confusion matrix
The four outcomes of any pass or fail check, named for conversation quality, and which cell each headline metric is computed from.
Every pass-or-fail check produces four outcomes. Every metric in this vocabulary is a ratio of some of them.
| Really failed | Really passed | |
|---|---|---|
| Check said fail | True positive. Caught it. | False positive. Flagged a clean one. |
| Check said pass | False negative. Missed it. | True negative. Correctly quiet. |
What comes from where
| Precision | True positives over all flags raised. |
|---|---|
| Recall on fail | True positives over everything that really failed. |
| Specificity | True negatives over everything that really passed. |
| Prevalence | Everything that really failed, over the total. |
The cell nobody measures
False negatives. They are invisible by construction: the check said pass and nobody looked again, so the conversation leaves no trace of having been missed.
The only way to count them is to have people review conversations the check cleared. A review queue fed solely by flags measures three cells and guesses the fourth.
This page as markdown