Designing for AI judgement when the evidence is incomplete

Two ways to be wrong, both expensive

Researcher Assistant supported US criminal background screening, where records are matched against identifiers such as name, date of birth, address, and SSN. The problem was that court records do not contain those identifiers consistently. One jurisdiction might return a full date of birth, another only a birth year, and another nothing at all.

Researchers were making liability-bearing decisions from evidence that was incomplete by nature. Attribute a record incorrectly and a candidate can lose a job they should have had, with FCRA exposure behind it. Miss a real record and the client may hire the person they were screening to avoid.

In regulated work, design decisions are risk decisions whether or not anyone in the room calls them that.

The risk was spread evenly across work that was not

Researchers cross-checked identifiers by hand and held the reasoning in their heads, following written and often ambiguous guidelines. Nothing in the workflow distinguished a record that contradicted every identifier from one where the court file simply did not contain enough information to say.

Both cases cost the same to open and the same to think about. Because the reasoning lived in the researcher's head rather than in an artefact, little survived the decision that could be audited afterwards.

When every case looks equally decidable, attention is spent uniformly on a problem that is anything but.

The interface was part of the autonomy decision

The next phase of the programme would allow the agent to work independently under oversight, gated on 99.9% accuracy. That figure could only come from researchers overriding the agent when it was wrong.

Any asymmetry in the interface would corrupt the evidence: a recommendation shown before the underlying comparison, an accept that costs one click and a reject that costs four, or a score that flattens the reasoning into a number. Designing the tool and designing the evidence base for autonomy were the same job.

If disagreeing costs more than agreeing, you are not measuring accuracy. You are measuring compliance.

Defining rules that closed named failure modes

I translated the problem into a small set of rules, each connected to a specific way the workflow could fail.

1. Target identifiers, never full records. The agent states a status for each identifier it examined instead of issuing one conclusion for the record as a whole.

2. Make unmatchable a status, not a low score. Missing evidence is a fact about the source, not a weak signal about the candidate.

3. Make overriding cost the same as accepting. The interface must not turn agreement into the path of least resistance.

4. Show reasoning, do not summarise it away. The researcher needs to see what was compared against what, not only a sentence characterising the comparison.

Evidence before verdict

Researchers wanted a recommendation up front. A recommendation shown first becomes the thing a person confirms rather than the thing they check.

The final direction put the identifier comparison first and any record-level conclusion beneath it, traceable to the evidence above. This was slower on obvious cases, but it made uncertainty visible where the stakes were highest.

What outlived my time on the project

Most of the project happened in conversations I was not in: a model team tuning thresholds, engineers implementing the specification, researchers working cases. The useful output was not a set of screens. It was the small set of documents that let other people make the call I would have made.

• The status vocabulary: defined what the agent was allowed to assert.

• The provenance split: kept candidate-provided information separate from independently developed information.

• The evidence display specification: settled what the agent showed before showing a conclusion.

• The override and reason taxonomy: turned disagreement into usable evidence for the model team, QA, and future autonomy decisions.

On an AI product, the design work is largely deciding what the system is permitted to claim.

Reflection

Researcher Assistant was not simply an interface for reviewing records. It was a decision system operating where evidence could be incomplete and the cost of being wrong was unevenly distributed.

The work made that uncertainty visible, protected the researcher's judgement, and created a more defensible path toward AI-assisted screening. The most important outcome was not that the agent appeared more intelligent. It was that the product became more honest about what the evidence could and could not support.