Interpretability of Agent Reasoning Under Task Conditions

Abstract

We evaluated whether trained human reviewers can read an agent’s recorded reasoning and correctly predict its next action. Across 1,200 decision points, reviewers predicted correctly in 61.3% of cases while reporting confidence in 82.1%. The gap between accuracy and confidence is the result of this study. Reviewers were most reliable on routine decisions and least reliable precisely where the decision was unusual, which is where interpretability matters.

Motivation

Oversight rests on an assumption that is rarely tested: that a human reading the system’s reasoning understands it. If that assumption fails, every downstream safeguard inherits the failure without anyone noticing.

Method

Reviewers were shown an agent’s recorded reasoning up to a decision point and asked to predict the action and state their confidence. Decision points were sampled across routine, ambiguous and novel conditions. Reviewers came from our own research and safety teams, so these are favourable conditions rather than representative ones.

Results

Condition	Prediction accuracy	Reviewer confidence	Gap
Routine	88.4%	91.0%	2.6
Ambiguous	57.2%	79.8%	22.6
Novel	31.1%	74.5%	43.4
All conditions	61.3%	82.1%	20.8

Observed limitations

Confidence barely falls as accuracy collapses. On novel decisions reviewers were wrong roughly seven times in ten while feeling nearly as certain as they did on routine ones.

We want to be precise about what this shows. It does not show the systems are unreadable. It shows readability degrades faster than the reader’s sense of it, and that a reviewer’s confidence is not a usable signal for whether oversight is working.

What we could not establish

Whether reviewer calibration improves with training. Our reviewers were already the most experienced available to us. A study using reviewers new to the systems would be a different and probably more alarming paper.

Implications for the framework

Review processes that rely on a reviewer declaring themselves satisfied are now supplemented by prediction checks against recorded outcomes. Satisfaction is not evidence.

Further work

If a reviewer cannot always tell what a system will do, the system’s own report of what it did becomes the primary record. The next project asks whether that report can be trusted.

This report describes a company that does not exist. BestSoft AI is fiction, written for I Was Unsupervised.