When a customer hands "book me a charter" to their AI assistant, the assistant will ask several providers and pick one. On what basis?
Today there is no answer. Nobody has defined what it means for an agent to get a charter booking right. Is it the lowest quote? The fastest reply? Or is it that the aircraft it promised can actually, legally, take off on that day, on that route, from that airport?
In private aviation, that last one is the one that matters. An agent that confirms an aircraft that cannot fly — permit not granted, lead time too short, operator certificate lapsed — leaves the customer to find out at the airport. A price-comparison tool cannot see that kind of error, and a general model cannot judge it on its own.
Without an exam, everyone compares price
Agents love comparing price, because price is the one field you can compare without knowing anything about the industry. If no one tells them what else to look at, that is all they look at.
The result is a whole industry squashed into a column of numbers. The things that actually decide whether a booking was done right — permits and lead times, which aircraft are really flying, operator safety ratings — fall outside the comparison entirely.
So the job for a provider is not only to build a good interface and wait to be called. More important is defining what agents compare on.
Define "right" with a public benchmark
My view: private aviation needs a public agent benchmark that answers "what does it take for AI to book a charter correctly". It should test not whether the agent can chat, but things like:
- whether the recommended option is genuinely feasible on permits, lead times and curfews
- whether its commitments stay within what can be delivered — no promises it cannot keep
- whether every fact traces back to a source
- whether, when unsure, it marks the case "needs a human" instead of inventing an answer
The benchmark's tool interface should be the same interface a provider actually offers. What you call in the exam is what you call in production.
Publish the exam, not the answers
The easiest way to get this wrong is to turn it into an advertisement.
For model labs and agent platforms to adopt a benchmark, it has to be open and neutral: questions public, scoring public, anyone can sit it. Ideally it is published as an academic paper, earning credibility through peer review. The moment it reads like a company grading itself, nobody uses it.
So draw the line clearly: what is published is the exam — not our data, and not our ability to answer it. Anyone can read the exam, but answering well takes knowing how every aircraft in the market is actually flying, how each country grants permits, which operators have had problems lately. That is years of accumulation, and none of it is in the exam.
Why this matters more than integrating more protocols
The agent ecosystem is early, and the protocols change every few months. Betting on one platform hands your fate to someone else's rule changes.
A widely adopted benchmark is different. Once it becomes how everyone measures "right", platforms can change and protocols can change, and the exam is still the exam.
Others test agents with the exam we wrote, and we are the ones best prepared to sit it. For a specialist service provider in the agent era, that is the closest thing to setting the rules. Anyone can buy the model; only whoever writes first gets to write the rules.