Designing the evidence AI needs to improve

The accuracy target depended on a record that missed the resolution

The Researcher Assistant needed to meet a demanding accuracy target before operating independently. But the historical data did not reliably capture how researchers had resolved cases. Researchers investigated cases and produced reports, yet their final determinations and reasoning were not consistently written back into the training records. The organisation had accumulated expertise without retaining it in a form the model team could reliably use.

The design challenge was to make expert review produce a reusable record: what was correct, why it was correct and how it differed from the original proposals.

Use disagreement to focus expert attention

A disagreement provided a concrete question for an expert to resolve. It also created an opportunity to capture the evidence or interpretation that distinguished two competing answers. That became the basis of the first release: surface disagreements, resolve them at field level and require an explanation alongside the correction. The intended benefit was richer training material. Whether that material improved model performance still required evaluation. This focus also set a boundary. Cases where the researcher and assistant agreed stayed outside the queue, including cases where both might be wrong.

Decision 1: hide the source of each answer

A direct “researcher or AI?” choice would have produced a convenient label. It also risked allowing trust or suspicion toward the assistant to influence the review. I designed the interaction around unattributed candidate values. The expert selected the correct value or supplied another answer. The system then derived which source was correct from that action.

This separated the expert’s correction from the attribution needed by the model team. Original sources remained in the record for later analysis. The trade-off was reduced context during review: experts could not identify patterns in the assistant’s errors as they worked. That analysis had to happen afterwards. Blinding was intended to reduce source-related bias; we did not run a comparison that established its effect.

Decision 2: make “both wrong” a usable outcome

A binary verdict would have forced an expert to select an existing answer even when neither was correct. The workflow therefore supported three outcomes: the researcher was correct, the assistant was correct, or neither was correct. Every review also needed a correct determination and an explanation. Identifying an error alone would not provide the replacement answer the training record needed. This increased the work required from experts and gave the model team a more complex label set. We accepted that cost to avoid recording a knowingly incorrect answer as correct.

Decision 3: resolve individual fields and preserve their history

A single case could contain agreement on one attribute and disagreement on another. A case-level verdict would conceal that distinction. I specified separate resolution for severity, disposition and nature of offence. Each field retained its candidate values, correction and history. This gave engineering a defined unit of interaction and gave the model team a more precise record of what had changed.

Preserving the original proposals also supported later error analysis. Correcting a value did not erase the evidence that had led to the review.

Decision 4: prioritise disagreements while acknowledging the coverage gap

We limited the initial queue to disagreements because those cases exposed competing interpretations and gave experts a focused task. The cost was a known blind spot. Shared errors would not enter the queue, so this workflow could not establish the quality of the entire corpus. Agreement cases were planned for a later phase. In retrospect, I would have included a small sampled review stream earlier, allowing the team to check whether agreement was masking systematic errors.

Give the platform a purpose beyond training-data repair

The immediate brief was to improve training data. I also identified a longer-term use for the same review structure: correcting disputed fields before background reports reached clients and candidates. That shaped the decision to preserve corrections and their histories. The record could support training immediately while providing a foundation for a future report-quality workflow.

My contribution included the blinded review model, the field-level resolution specification, the value-history requirements and the proposal for this second use. Together, they connected interface decisions to engineering implementation, model evaluation and a potential continuing operational role. The report-quality workflow remained a proposal. It had not started when I left.

What the work demonstrated

The platform ran alongside the Researcher Assistant during training, with experts resolving real disagreements.

Demonstrated during useStill unresolved at handoverExperts could resolve disputed fields without source labels.Whether blinding changed review behaviour or accuracy.Selecting a candidate or supplying a replacement produced the expected training labels.Whether adjudicated data improved model accuracy.The review workflow operated on real disagreements.Whether it could sustain the volume needed for the broader objective.A future report-quality use had been defined.Whether that proposed workflow would deliver operational value.

Scaling depended on the Researcher Assistant launching and generating disagreements in sufficient volume. That launch had not happened when I left, limiting our ability to establish the downstream effect. The delivered outcome was a working expert-review mechanism with a defined correction record. Demonstrating sustained model improvement required further data, retraining and evaluation.

What I would change

• I would assess the historical data before committing to an accuracy target. Understanding what the records contained would have made the repair work part of the original scope and sequence.

• I would also establish a source of review cases independent of the assistant’s launch. Reconstructing historical disagreements or using an expert panel on sampled cases could have kept the workflow productive while the deployment dependency remained unresolved.

• Finally, I would sample agreement cases from the beginning. A queue built around disagreement can support focused correction, but it needs complementary checks to expose shared mistakes.

The lasting lesson for me was to design the evidence, evaluation and operating dependencies together. Capturing better expert decisions was a necessary step; proving their effect required a plan that could continue beyond the interface.