Skip to content

Prove a fix worked

Show that a change moved the number, over the same time range and the same traffic.

Use this when somebody wants evidence a number moved, rather than your word for it.

  1. Write down the day and time you shipped.

    Every figure here is bucketed in UTC, and the charts label them that way.

    The Metrics screen: whether any agent has been sliding and what that was measured on, the changes that landed inside the period, and one card per measure over time.
  2. Open Overview, then Metrics indented under it.

  3. Pick a bucket size.

    By hour for a change that landed today, By day for one a week old.

  4. Read What changed in this range.

    Anything listed is drawn as a dashed line across every chart.

  5. Find the card for the measure you expect to have moved.

    The link at its foot opens that one measure in depth.

  6. Read the cuts by channel and by agent, and The worst of them.

  7. Read the drift band at the top of the screen.

    It tests each agent's weekly score for a sustained direction, not a step.

  8. Read the stamp on your agent's row.

    Sliding, Improving, Holding, Not enough yet, or Not comparable.

  9. Cross-check on Abandonment.

    Its reason rows do the arithmetic against the period immediately before.

What the drift band asks for

The testMann-Kendall on the weekly readings themselves, not on a fitted line
ReadingsTen whole weeks, and the week in progress is not one of them
Behind each readingAt least five conversations, or the week is not a reading
The bar it must clearA two-sided p of 0.05 or better, and a total move of two points
Not comparableThe span crosses a change to what we check, so the two halves cannot be read against each other

Reading it right

  • Two figures of the same length, over the same traffic, with the change marked between them, is a claim you can defend. One figure after the change is not.
  • A step in a chart is one week. The drift band is the question of whether it held, and it refuses to answer on fewer than ten readings.
  • If the cut by agent shows one agent carrying the whole move, you changed one agent and not the product. Say which.
Note: You cannot write your own mark into What changed in this range, and no control here compares two builds of one agent.

Read next

This page as markdown