How it works

The six stages a conversation passes through, from arriving to being scored.

A conversation passes through six stages between arriving and being scored. Each stage has a queue of its own, so a slow one never holds up the stage in front of it, and each one can be scaled on its own.

IngestA delivery arrives. We store the body exactly as it came and answer the platform straight away, then turn that body into events in the one shape the rest of the pipeline reads.
AssembleEvents for one customer are put in order and grouped into conversations. A conversation that has finished is closed here, and closing it is what asks for a score.
MetricsReply times, talk time, counts per client and per agent, and the recording when the platform gives us one to fetch. Your alert rules are tested against the new numbers here.
JudgeThe rules run against the conversation. Which of them run on which conversations is the subject of What gets scored.
DeepThe nightly pass. It plans the reading the day left undone, sends it at the providers' cheaper batch prices, and records what it could not afford. The same stage tags why people gave up, and re-scores a fixed set of conversations to catch the judge drifting.
HousekeepingEverything on a clock: retention scrubs, compaction of the rolled-up numbers, alert re-checks, the weekly client report, and asking the platforms we poll whether anything has finished.

Nothing is scored until the conversation is finished

Half a conversation scores badly for the wrong reason, so we wait. The platform usually tells us a conversation ended. When it does not, a conversation that has been quiet for long enough is closed on our side, and it is scored from then.

A conversation that reopens after it was closed supersedes its own score. The work already done for the old version is dropped rather than mixed with the new one.

Two speeds, on purpose

Reading a conversation with an AI judge costs money, and reading it within seconds costs more than reading it overnight. So the fast path takes a share of traffic as it lands, which is what makes a bad release visible the same hour, and the nightly pass takes the rest at batch prices, which is what makes full coverage affordable.

The nightly pass works one local day at a time, in the workspace's own timezone, and it reconsiders the last few days on every run. A night that was missed heals itself on the next one.

A stage that fails says so

A job retries a few times. A job that never lands is written down with its arguments and the error, against the stage it failed on, so a conversation with no score can be told apart from a conversation nobody sent.

These six are the whole pipeline. There is no seventh stage, and no hidden step between them: the names above come from the code that runs them.

Read next

  • Why an outside judge · Why the thing scoring your agent should not be the thing that built it.
  • Agreement with people · How we measure whether our scores match what your reviewers would have said.
  • What gets scored · Why the fast rules run on every conversation and the ones that cost money run on a share.
This page as markdown