CONSCEPT ENGAGEMENT ASSESSMENT
A clearer view of the work ahead.
A practical record of the complexity observed, the activities likely to help, and the work that was actually conducted.
Complexity profile
This was a high-complexity AI governance project because the challenge extended beyond the interface. The work required uncovering hidden problems in the training data, translating expert judgement into a repeatable workflow, and aligning product, engineering, operations, subject-matter experts, and model teams around a shared definition of a reliable decision. It also carried significant risk: the platform needed to improve model quality and auditability while supporting high-stakes human oversight and a future path toward greater automation.
Likely engagement
Dedicated diagnostic phase
CompleteTurn uncertainty about the underlying problem into explicit, testable statements supported by evidence.
Executive and leadership interviews
CompleteCapture first-hand needs, constraints, incentives, and decision context from the people closest to the work.
Departmental interviews
CompleteCapture first-hand needs, constraints, incentives, and decision context from the people closest to the work.
Cross-team dependency mapping
Not completedMake relationships, dependencies, ownership gaps, and points of friction visible enough to act on.
Enterprise architecture assessment
CompleteExamine the current evidence and implementation systematically, then document risks, duplication, strengths, and priority improvements.
Full ecosystem and dependency mapping
CompleteMake relationships, dependencies, ownership gaps, and points of friction visible enough to act on.
Formal change-management plan
Not completedTranslate evidence into a sequenced course of action with priorities, dependencies, owners, measures, and decision points.
Pilot and phased rollout
CompleteIntroduce the change in controlled stages, gather evidence from real use, and adjust the next stage before scaling.
Comprehensive risk-management plan
CompleteTranslate evidence into a sequenced course of action with priorities, dependencies, owners, measures, and decision points.
Legal, regulatory and compliance consultation
CompleteDefine the standards the work must satisfy and provide evidence that critical obligations have been met.
Programme plan
CompleteTranslate evidence into a sequenced course of action with priorities, dependencies, owners, measures, and decision points.
Workstream definition
CompleteCoordinate the sequence of work, dependencies, responsibilities, and review points needed for reliable delivery.
The accuracy target depended on a record that missed the resolution
The Researcher Assistant needed to meet a demanding accuracy target before operating independently. But the historical data did not reliably capture how researchers had resolved cases. Researchers investigated cases and produced reports, yet their final determinations and reasoning were not consistently written back into the training records. The organisation had accumulated expertise without retaining it in a form the model team could reliably use.
The design challenge was to make expert review produce a reusable record: what was correct, why it was correct and how it differed from the original proposals.
function EvidenceRecovery() {
const [view, setView] = useState("raw");
const resolved = view === "resolved";
const layers = [
{
label: "Candidate values",
raw: "One partial identifier",
resolved: "Name, date of birth, address and case context",
},
{
label: "Determination",
raw: "No final answer",
resolved: "A supported record-level decision",
},
{
label: "Rationale",
raw: "No reasoning retained",
resolved: "Each comparison is visible and traceable",
},
{
label: "Decision history",
raw: "No expert judgement retained",
resolved: "Accept, override and reason are recorded",
},
];
return (
<section
style={{
width: "100%",
boxSizing: "border-box",
padding: "clamp(24px, 4vw, 48px)",
background: "#11161F",
color: "#FFFFFF",
borderRadius: "12px",
fontFamily: "Inter, Arial, sans-serif",
}}
>
<p
style={{
margin: "0 0 10px",
color: "#FF8A66",
fontSize: "11px",
fontWeight: 700,
letterSpacing: "0.14em",
}}
>
EVIDENCE RECOVERY
</p>
<h2
style={{
maxWidth: "780px",
margin: "0 0 12px",
fontSize: "clamp(28px, 5vw, 52px)",
lineHeight: 1.05,
letterSpacing: "-0.04em",
}}
>
What the system remembered—and what it lost
</h2>
<p
style={{
maxWidth: "680px",
margin: "0 0 28px",
color: "#C8CCD0",
fontSize: "16px",
lineHeight: 1.55,
}}
>
Switch the record state to see the difference between incomplete case
data and an evidence-backed decision that another researcher can review.
</p>
<div
role="group"
aria-label="Record state"
style={{
display: "flex",
flexWrap: "wrap",
gap: "10px",
marginBottom: "24px",
}}
>
{[
{ key: "raw", label: "Raw record" },
{ key: "resolved", label: "Resolved record" },
].map((option) => {
const selected = view === option.key;
return (
<button
key={option.key}
type="button"
onClick={() => setView(option.key)}
aria-pressed={selected}
style={{
minWidth: "160px",
padding: "13px 16px",
border: selected
? "2px solid #FF8A66"
: "1px solid #3B4552",
borderRadius: "8px",
background: selected ? "#202936" : "#191F29",
color: selected ? "#FF8A66" : "#FFFFFF",
fontSize: "14px",
fontWeight: 700,
cursor: "pointer",
}}
>
{option.label}
</button>
);
})}
</div>
<div
style={{
display: "grid",
gridTemplateColumns: "minmax(0, 1fr) auto",
gap: "18px",
alignItems: "start",
}}
>
<div
style={{
minWidth: 0,
padding: "20px",
border: "1px solid #3B4552",
borderRadius: "10px",
background: "#191F29",
}}
>
<div
style={{
display: "flex",
flexWrap: "wrap",
alignItems: "baseline",
justifyContent: "space-between",
gap: "10px",
marginBottom: "18px",
}}
>
<span
style={{
color: "#C8CCD0",
fontSize: "11px",
fontWeight: 700,
letterSpacing: "0.13em",
}}
>
{resolved ? "RESOLVED RECORD" : "RAW RECORD"}
</span>
<strong style={{ color: "#FF8A66", fontSize: "14px" }}>
{resolved ? "4 evidence layers" : "1 usable field"}
</strong>
</div>
<div style={{ display: "grid", gap: "10px" }}>
{layers.map((layer, index) => {
const available = resolved || index === 0;
return (
<div
key={layer.label}
style={{
display: "grid",
gridTemplateColumns:
"minmax(112px, 0.7fr) minmax(0, 1.7fr)",
gap: "12px",
padding: "14px",
borderRadius: "8px",
background: available ? "#202936" : "#151B24",
border:
available && resolved && index > 0
? "1px solid #FF8A66"
: "1px solid #303A46",
opacity: available ? 1 : 0.48,
}}
>
<span
style={{
color:
available && resolved && index > 0
? "#FF8A66"
: "#FFFFFF",
fontSize: "13px",
fontWeight: 700,
}}
>
{layer.label}
</span>
<span
style={{
color: "#C8CCD0",
fontSize: "13px",
lineHeight: 1.45,
}}
>
{resolved ? layer.resolved : layer.raw}
</span>
</div>
);
})}
</div>
</div>
<div
aria-hidden="true"
style={{
paddingTop: "70px",
color: "#FF8A66",
fontSize: "28px",
lineHeight: 1,
}}
>
{resolved ? "✓" : "…"}
</div>
</div>
<p
style={{
margin: "20px 0 0",
color: "#C8CCD0",
fontSize: "14px",
lineHeight: 1.55,
}}
>
{resolved
? "The decision can be reviewed, challenged and improved because the evidence and judgement survive the moment of review."
: "A case can be worked, but the useful evidence disappears once the decision is made."}
</p>
</section>
);
}
render(<EvidenceRecovery />);Use disagreement to focus expert attention
A disagreement provided a concrete question for an expert to resolve. It also created an opportunity to capture the evidence or interpretation that distinguished two competing answers. That became the basis of the first release: surface disagreements, resolve them at field level and require an explanation alongside the correction. The intended benefit was richer training material. Whether that material improved model performance still required evaluation. This focus also set a boundary. Cases where the researcher and assistant agreed stayed outside the queue, including cases where both might be wrong.
Decision 1: hide the source of each answer
A direct “researcher or AI?” choice would have produced a convenient label. It also risked allowing trust or suspicion toward the assistant to influence the review. I designed the interaction around unattributed candidate values. The expert selected the correct value or supplied another answer. The system then derived which source was correct from that action.
This separated the expert’s correction from the attribution needed by the model team. Original sources remained in the record for later analysis. The trade-off was reduced context during review: experts could not identify patterns in the assistant’s errors as they worked. That analysis had to happen afterwards. Blinding was intended to reduce source-related bias; we did not run a comparison that established its effect.
Decision 2: make “both wrong” a usable outcome
A binary verdict would have forced an expert to select an existing answer even when neither was correct. The workflow therefore supported three outcomes: the researcher was correct, the assistant was correct, or neither was correct. Every review also needed a correct determination and an explanation. Identifying an error alone would not provide the replacement answer the training record needed. This increased the work required from experts and gave the model team a more complex label set. We accepted that cost to avoid recording a knowingly incorrect answer as correct.
function DerivedVerdictExample() {
const [selection, setSelection] = useState("");
const [thirdValue, setThirdValue] = useState("");
const [explanation, setExplanation] = useState("");
const [completed, setCompleted] = useState(false);
const candidates = ["Pending", "Unknown"];
const selectedValue =
selection === "third" ? thirdValue.trim() : selection;
const canComplete = Boolean(selectedValue && explanation.trim());
const verdict =
selection === "third"
? "Resolved with an expert correction"
: "Resolved using the selected candidate value";
const reset = () => {
setSelection("");
setThirdValue("");
setExplanation("");
setCompleted(false);
};
return (
<section
style={{
width: "100%",
boxSizing: "border-box",
padding: "clamp(20px, 4vw, 40px)",
background: "#11161F",
color: "#FFFFFF",
border: "1px solid #3B4552",
borderRadius: "8px",
fontFamily: "Inter, Arial, sans-serif",
}}
>
<p
style={{
margin: "0 0 8px",
color: "#FF8A66",
fontSize: "11px",
fontWeight: 700,
letterSpacing: "0.12em",
}}
>
ILLUSTRATIVE EXAMPLE
</p>
<h2
style={{
margin: "0 0 12px",
fontSize: "clamp(26px, 4vw, 42px)",
lineHeight: 1.08,
letterSpacing: "-0.03em",
}}
>
Derive a verdict without losing the record
</h2>
<p
style={{
maxWidth: "720px",
margin: "0 0 28px",
color: "#C8CCD0",
fontSize: "16px",
lineHeight: 1.55,
}}
>
Two candidate values disagree. Select one, or enter a third value,
explain the decision, then see what the system preserves.
</p>
{!completed ? (
<>
<div
style={{
padding: "20px",
border: "1px solid #3B4552",
borderRadius: "6px",
background: "#191F29",
}}
>
<div
style={{
display: "flex",
flexWrap: "wrap",
justifyContent: "space-between",
gap: "12px",
marginBottom: "18px",
paddingBottom: "14px",
borderBottom: "1px solid #3B4552",
}}
>
<strong>Disposition</strong>
<span
style={{
color: "#FF8A66",
fontSize: "14px",
fontWeight: 700,
}}
>
Disagreement needs review
</span>
</div>
<p
style={{
margin: "0 0 12px",
color: "#C8CCD0",
fontSize: "13px",
fontWeight: 700,
}}
>
SELECT A CANDIDATE VALUE
</p>
<div
style={{
display: "flex",
flexWrap: "wrap",
gap: "10px",
marginBottom: "18px",
}}
>
{candidates.map((candidate) => {
const selected = selection === candidate;
return (
<button
key={candidate}
type="button"
onClick={() => setSelection(candidate)}
aria-pressed={selected}
style={{
flex: "1 1 180px",
minHeight: "52px",
padding: "12px 16px",
border: selected
? "2px solid #FF8A66"
: "1px solid #3B4552",
borderRadius: "4px",
background: selected ? "#202936" : "#151B24",
color: "#FFFFFF",
fontSize: "16px",
fontWeight: 700,
textAlign: "left",
cursor: "pointer",
}}
>
{candidate}
</button>
);
})}
</div>
<button
type="button"
onClick={() => setSelection("third")}
aria-pressed={selection === "third"}
style={{
width: "100%",
minHeight: "48px",
padding: "12px 16px",
border:
selection === "third"
? "2px solid #FF8A66"
: "1px solid #3B4552",
borderRadius: "4px",
background: selection === "third" ? "#202936" : "#151B24",
color: "#FFFFFF",
fontSize: "15px",
fontWeight: 700,
textAlign: "left",
cursor: "pointer",
}}
>
+ Enter a third value
</button>
{selection === "third" && (
<div style={{ marginTop: "12px" }}>
<label
htmlFor="third-value"
style={{
display: "block",
marginBottom: "7px",
fontSize: "13px",
fontWeight: 700,
}}
>
Expert-entered value
</label>
<input
id="third-value"
value={thirdValue}
onChange={(event) => setThirdValue(event.target.value)}
placeholder="For example: Non-conviction"
style={{
width: "100%",
boxSizing: "border-box",
minHeight: "48px",
padding: "12px",
border: "2px solid #FF8A66",
borderRadius: "4px",
background: "#151B24",
color: "#FFFFFF",
font: "inherit",
}}
/>
</div>
)}
</div>
<div style={{ marginTop: "20px" }}>
<label
htmlFor="decision-explanation"
style={{
display: "block",
marginBottom: "7px",
fontSize: "14px",
fontWeight: 700,
}}
>
Explain the decision
</label>
<textarea
id="decision-explanation"
value={explanation}
onChange={(event) => setExplanation(event.target.value)}
placeholder="Describe the evidence that supports this value."
rows={4}
style={{
width: "100%",
boxSizing: "border-box",
padding: "12px",
border: "1px solid #3B4552",
borderRadius: "4px",
background: "#191F29",
color: "#FFFFFF",
font: "inherit",
lineHeight: 1.5,
resize: "vertical",
}}
/>
</div>
<div
style={{
display: "flex",
flexWrap: "wrap",
justifyContent: "flex-end",
gap: "10px",
marginTop: "20px",
}}
>
<button
type="button"
onClick={reset}
style={{
minWidth: "140px",
minHeight: "52px",
padding: "12px 20px",
border: "1px solid #3B4552",
background: "#202936",
color: "#FFFFFF",
fontSize: "15px",
fontWeight: 700,
cursor: "pointer",
}}
>
Reset
</button>
<button
type="button"
disabled={!canComplete}
onClick={() => setCompleted(true)}
style={{
minWidth: "180px",
minHeight: "52px",
padding: "12px 20px",
border: 0,
background: canComplete ? "#FF8A66" : "#3B4552",
color: canComplete ? "#11161F" : "#C8CCD0",
fontSize: "15px",
fontWeight: 700,
cursor: canComplete ? "pointer" : "not-allowed",
}}
>
Complete decision
</button>
</div>
</>
) : (
<div
style={{
padding: "24px",
border: "1px solid #FF8A66",
borderRadius: "6px",
background: "#202936",
}}
>
<p
style={{
margin: "0 0 8px",
color: "#FF8A66",
fontSize: "11px",
fontWeight: 700,
letterSpacing: "0.12em",
}}
>
DERIVED VERDICT
</p>
<h3
style={{
margin: "0 0 8px",
fontSize: "24px",
lineHeight: 1.15,
}}
>
{selectedValue}
</h3>
<p style={{ margin: "0 0 22px", color: "#C8CCD0", lineHeight: 1.5 }}>
{verdict}
</p>
<div
style={{
paddingTop: "18px",
borderTop: "1px solid #3B4552",
}}
>
<strong style={{ display: "block", marginBottom: "12px" }}>
Candidate history preserved
</strong>
<ol
style={{
display: "grid",
gap: "10px",
margin: 0,
paddingLeft: "20px",
color: "#C8CCD0",
lineHeight: 1.5,
}}
>
<li>Original candidate: Pending</li>
<li>Original candidate: Unknown</li>
<li>Expert decision: {selectedValue}</li>
<li>Reason recorded: {explanation}</li>
</ol>
</div>
<button
type="button"
onClick={reset}
style={{
marginTop: "24px",
minHeight: "48px",
padding: "12px 18px",
border: "1px solid #3B4552",
background: "#191F29",
color: "#FFFFFF",
fontSize: "14px",
fontWeight: 700,
cursor: "pointer",
}}
>
Try another decision
</button>
</div>
)}
<aside
style={{
marginTop: "24px",
padding: "16px",
borderLeft: "3px solid #FF8A66",
background: "#191F29",
color: "#C8CCD0",
fontSize: "14px",
lineHeight: 1.5,
}}
>
<strong style={{ display: "block", marginBottom: "4px", color: "#FFFFFF" }}>
Static fallback
</strong>
Illustrative example: two candidate values remain visible even when an
expert supplies a third, supported value. The final verdict keeps both
proposals, the decision, and its explanation together.
</aside>
</section>
);
}
render(<DerivedVerdictExample />);Decision 3: resolve individual fields and preserve their history
A single case could contain agreement on one attribute and disagreement on another. A case-level verdict would conceal that distinction. I specified separate resolution for severity, disposition and nature of offence. Each field retained its candidate values, correction and history. This gave engineering a defined unit of interaction and gave the model team a more precise record of what had changed.
Preserving the original proposals also supported later error analysis. Correcting a value did not erase the evidence that had led to the review.
Decision 4: prioritise disagreements while acknowledging the coverage gap
We limited the initial queue to disagreements because those cases exposed competing interpretations and gave experts a focused task. The cost was a known blind spot. Shared errors would not enter the queue, so this workflow could not establish the quality of the entire corpus. Agreement cases were planned for a later phase. In retrospect, I would have included a small sampled review stream earlier, allowing the team to check whether agreement was masking systematic errors.
Give the platform a purpose beyond training-data repair
The immediate brief was to improve training data. I also identified a longer-term use for the same review structure: correcting disputed fields before background reports reached clients and candidates. That shaped the decision to preserve corrections and their histories. The record could support training immediately while providing a foundation for a future report-quality workflow.
My contribution included the blinded review model, the field-level resolution specification, the value-history requirements and the proposal for this second use. Together, they connected interface decisions to engineering implementation, model evaluation and a potential continuing operational role. The report-quality workflow remained a proposal. It had not started when I left.
What the work demonstrated
The platform ran alongside the Researcher Assistant during training, with experts resolving real disagreements.
Demonstrated during useStill unresolved at handoverExperts could resolve disputed fields without source labels.Whether blinding changed review behaviour or accuracy.Selecting a candidate or supplying a replacement produced the expected training labels.Whether adjudicated data improved model accuracy.The review workflow operated on real disagreements.Whether it could sustain the volume needed for the broader objective.A future report-quality use had been defined.Whether that proposed workflow would deliver operational value.
Scaling depended on the Researcher Assistant launching and generating disagreements in sufficient volume. That launch had not happened when I left, limiting our ability to establish the downstream effect. The delivered outcome was a working expert-review mechanism with a defined correction record. Demonstrating sustained model improvement required further data, retraining and evaluation.
What I would change
I would assess the historical data before committing to an accuracy target. Understanding what the records contained would have made the repair work part of the original scope and sequence.
I would also establish a source of review cases independent of the assistant’s launch. Reconstructing historical disagreements or using an expert panel on sampled cases could have kept the workflow productive while the deployment dependency remained unresolved.
Finally, I would sample agreement cases from the beginning. A queue built around disagreement can support focused correction, but it needs complementary checks to expose shared mistakes.
The lasting lesson for me was to design the evidence, evaluation and operating dependencies together. Capturing better expert decisions was a necessary step; proving their effect required a plan that could continue beyond the interface.