Confidence is not evidence: introducing the AI TrustGap
Most teams trust their AI more than they can prove. The distance between the two is measurable — and it is where the risk lives.
Ask a team how much they trust the AI they have put into production and you will usually get a confident answer. Ask them to show you the evidence for that confidence and the conversation changes.
That difference — between what a team believes about its AI and what it can demonstrate — is what we call the AI TrustGap.
Why the gap exists
It is rarely carelessness. It is the way AI gets built today.
- Demos are persuasive. A system that handles the ten examples you tried feels finished. The eleventh example — the one a customer sends on a Friday evening — is where it fails.
- Evaluation is self-graded. The team that built the system usually chooses the test cases, runs them and interprets the result. That is useful for building, and weak as evidence.
- The system keeps changing. Model providers update models. Prompts are edited. Tools are added. A result from last month is a statement about last month.
- Much of it is now generated. A large share of AI applications are assembled with AI coding tools. The builder can ship faster than they can review.
None of this means the AI is bad. It means nobody knows — and the people who will eventually ask are buyers, auditors, regulators and boards.
Measuring it
The AI TrustGap assessment asks two kinds of question.
First, calibration: how confident are you that your AI cannot be manipulated, that it is accurate on real work, and that it would pass a demanding enterprise review?
Then, controls: up to ten assurance areas — governance, risk classification, data, suppliers, evaluation, security, agent controls, transparency, monitoring and value. For each control the answer is not yes or no, but what you could prove today:
- Documented & evidenced
- Practised, not documented
- Planned
- Not in place
- Don't know
"Practised, not documented" and "Don't know" both count as evidence gaps. That is not a trick; it is how an auditor or a procurement team will read them.
The result places you on two axes — confidence and evidence — and the distance between them is your TrustGap.
The four positions
- Assured — confidence backed by evidence.
- Overconfident — more trust than proof. The most common position, and the riskiest, because nobody is looking for the problems.
- Underselling — more proof than confidence. Usually a team that has done the work but cannot yet show it.
- Early stage — neither yet. Honest, and a good place to start.
What closes the gap
Two things, in order. First, the controls that a team can put in place itself — ownership, a test set with known answers, a release gate, logging, a rollback path. Second, the things a questionnaire can never settle: whether the system is actually accurate on held-out tasks, whether it resists prompt injection, whether an agent behaves safely when a tool fails. Those need testing, and ideally testing by someone with no stake in the result.
That second step is where self-attestation ends and independent evidence begins.