Check you can trust the scores

Read the agreement figures for your own workspace before you act on a score.

Use this before you tell somebody their agent scores 71, so you know how much weight that number can carry.

The steps

1. Open Scoring quality in the left rail. It opens on Do we agree with people.
2. Read the three cards across the top. Agreement with people is how often the judge lands where a person landed. Score areas you can act on is how many areas clear the line. Repeat scoring is how far a score moves when nothing about the conversation changed.
3. Open What this licenses on a card. It says in plain words what the figure lets you claim, and what it does not.
4. Read the Scope line under the cards. It says which conversations the agreement figures were measured over.
5. Read Every score area, worst first. Each row carries Matches people, Flag accuracy, Catch rate, Really failing, how many results were compared, and a status.
6. Turn on the filter for areas checked against a person, to hide the rows nothing has been compared against yet. A blank in this table means Not measured, never zero.
7. Click a row for Why it reads this way, which says what put that status there.
8. Read The daily rerun, and how much a person has scored. It says whether the nightly rerun is passing, and how many examples a person scored the whole screen is checked against.
9. For the working behind any figure, open For engineers: the maths.

What to do with what you find

Act on a score area whose row says it lands where people land, and which has enough compared results behind it. Treat an area with no examples a person scored as Not measured. The judge still scores it, but that score is not evidence yet.

Repeat scoring sets your floor for change. A move smaller than that figure is not movement, and quoting it as an improvement will not survive a second look.

If an area you need is unmeasured, the fix is examples. Record Agree or Disagree on results as you read conversations, and the count on this screen rises.