The system recommends that a sales rep send an inquiry to a particular operator first. Three weeks later that operator runs into trouble and the deal is lost.
Was the recommendation wrong?
Most reviews would say yes. But if, on the day it was made, the operator looked fine — steady flight volume, valid certificates, no recent refusals — then it was right at the time. What went wrong was not the recommendation, but something that happened later that nobody could have seen.
The reverse is just as true. A recommendation that got lucky is logged as a "hit" in review, and the system grows more confident in the logic behind it — even if that logic had no real basis at the time.
Hindsight quietly contaminates evaluation
The problem is the data. When people and systems evaluate recommendations, they look at today's database: every field filled in later, every error corrected later, every outcome learned later.
Judging a three-week-old decision against today's data is like letting it peek at the answers during the exam. The "hit rate" you get is higher than the real one, and more misleading: it rewards luck and hindsight, not judgment.
If a system has to prove it is reliable through "how often it was adopted and how often it was right", that bias is fatal. It will make you trust something that is not actually trustworthy.
Keep two timelines, separately
The fix is to record two times in the data:
- When a fact became true: the day the operator's certificate lapsed, the day an aircraft changed operators
- When the system learned it: the day that fact was collected, entered, confirmed by a person
The two are usually different. A certificate may lapse at the start of the month and the system only learns at the end; the real operator of a flight may only become clear when the quote email comes back.
With both timelines, a review can ask a precise question: at the moment the recommendation was made, what did the system already know? Evaluate only with what was known at that moment, and the conclusion is fair.
It matters more than it sounds
This is not just about tidier review reports.
Whether a recommendation deserves trust ultimately comes down to its track record. If that track record is computed with hindsight, management will trust AI in the wrong places and let go where it should be careful.
Recording "what was known when" also settles a second issue: every relation carries its evidence, confidence and review status, and an inference only becomes a fact once a person confirms it. Then every number in the system can answer two questions: where it came from, and when it became something we believe.
A system that makes recommendations first has to be able to evaluate itself honestly. And the first step of honesty is not letting it see the answers during the exam.