Curtis Kadohama is Chief Product Officer at PerfectQuote, where he has spent the last six years leading product strategy for group benefits technology.
| Summary: Trustworthy AI is honest about what it is: a tool with real limits, not a magic wizard that always has the right answer. In an industry where mistakes carry real financial consequences, the products worth relying on are the ones that focus on their strengths, respect guardrails and spotlight potential shortcomings. |
Every vendor pitch I sit through these days leads with the same two words: AI-powered.
Half the room nods in agreement. The other half asks what’s actually underneath. In an industry where the numbers on a page impact a family’s health coverage and costs, the second group is right to question.
The SEC has opened enforcement actions against companies for overstating their AI capabilities to investors, a trend commonly called “AI washing.” If securities regulators are policing exaggerated AI claims, it’s a safe bet the buyers evaluating your product are equally skeptical.
The stakes explain the skepticism. A mispriced renewal or a misread plan detail costs employers and employees real money, and shortcuts such as AI washing don’t hold up when put to the test against a real book of clients and their range of needs.
So here’s the position I’ve landed on, and the one I think more people building AI products in insurance should be willing to say out loud: the best AI is transparent.
Not transparent in the sense of open-source code or published model prompts. Transparent in the sense that it’s honest about what it is, honest about what it can’t do and designed to help people rather than simply impress.
Is AI a tool, or a magic wizard?
AI is a tool, and a genuinely powerful one. I don’t think that’s overstated. But underneath the branding, it’s still a piece of engineering with inputs, outputs and failure modes, not unlike rate sheets, carrier data feeds or sets of business rules. It doesn’t reason the way a person reasons, and it doesn’t explicitly recognize when it may be incorrect.
That distinction matters more in insurance than in most industries. Regulators agree: roughly half of U.S. states have adopted the NAIC’s model bulletin on insurers’ use of AI systems, which requires companies to explain and govern the AI they use rather than treat it as a black box.
This goes beyond a mere compliance footnote; it represents the industry formally agreeing that the people using these tools are professionals whose judgment is the actual product they sell. A broker’s client pays for someone who can look at a renewal and know when something doesn’t smell right, not for a spreadsheet.
If a product team lets “well the AI has decided…” become an acceptable answer to “why does this number look off,” it has quietly told its most experienced users their judgment matters less than a model’s output, a trade-off I’ve watched other teams make in the name of moving fast.
Calling AI “just a piece of technology” sounds like modest hedging, but it’s actually the framing that lets you use it well. You wouldn’t trust an OCR tool to tell you whether a plan design is competitive, but you’d trust it to get the deductible and the out-of-pocket max onto the page correctly. Using AI as a tool helps you allocate more of your time and energy towards finding the most relevant and cost effective plan designs for your clients.
The best AI in insurance is honest; when properly used it can provide suggested guidance to reinforce and complement, not replace, critical thinking and contextual analysis.
Why is AI only as good as the patterns it can find?
Here’s the part that gets glossed over in most AI marketing, including some of our own industry’s: a model doesn’t inherently understand insurance. It recognizes patterns in the data it has access to, in order to establish a baseline of context and understanding for a given use case, and is supported by specific direction, guardrails and guidance. It then provides the most informed output that it can generate, bounded by the quality and breadth of its data and guidance provided, and continuously improves and becomes more accurate (i.e., “intelligent”) over time.
I think about this every time we evaluate a new AI capability internally. Being impressive in a demo is easy. The real test is what happens when the model sees something it hasn’t seen before.
Insurance is full of exactly that kind of variation:
- Every carrier uses its own language to represent coverages.
- Every state layers its own rules and regulations on top of federal guidelines and carrier-specific offerings.
- Every group’s employees have unique risks, preferences and corresponding coverage needs.
A model trained mostly on common cases will look brilliant in routine quoting situations; meanwhile, it will quietly guess at unknown factors. The guessing is the dangerous part, because it doesn’t announce itself. It just produces an answer that looks exactly as confident as a correct one.
This is why I’m skeptical of any AI capability (ours included) that gets sold on the strength of its best-case demo rather than its worst-case behavior. “What can it do?” is the easy question. The one worth asking any vendor, and the one we ask ourselves, is what it does when it’s outside its comfort zone, and whether it knows that’s what’s happening.
A model that can say, in effect, “I’m not confident here,” is more valuable than one that never hedges, even though the second one demos better. Confidence is not the same thing as competence, and in an industry where the numbers have real financial consequences for real people, that difference is the whole ballgame.
Trust should attach to the specific task an AI capability is built and tested for, not to the label “AI-powered.”
What does honest AI look like in practice?
Saying AI should be honest about its limits is one thing. Building that honesty into a product is another.
Take a tool that reads a scanned carrier proposal and pulls the plan details into a quote. Most of the time it accurately recognizes the deductible and the copay without assistance. Sometimes the document is low quality, the layout is unfamiliar, or a carrier uses ambiguous terminology. A transparent product flags the related outputs for a human to review and verify, rather than quietly guessing and moving on.
That’s a small design choice with a real cost. Admitting a tool doesn’t always know the answer is a harder sell in a demo than pretending it always does. But it’s the difference between guidance that a broker can actually rely on and a risky liability requiring constant verification, which diminishes the intended value and time savings.
Similar logic applies to a company’s product positioning. Publishing a capability’s strengths and core applications, along with guidelines for responsible usage and potential risks, can convey a stronger trust signal than a page full of confident claims. It’s easy to claim to have AI that is accurate and foolproof, but more trustworthy to provide transparency.
This is not suggesting that products should hide AI-assisted outcomes. Let users know when assumptions are made or supporting data is ambiguous, resulting in suggested guidance rather than concrete directives, especially in a business where trust is earned line by line.
But there’s a difference between transparency of outputs and a product experience that spotlights new technologies for the sake of marketing buzz. The former builds confidence. The latter is theater, and professionals doing serious work quickly see through it.
What should we ask of anyone building AI right now?
None of this reduces my optimism around the potential for AI in insurance technology. It’s the opposite. I think the honest version of this story is actually the more exciting one, because it’s the version that survives real-world use.
The tools that will maintain trust and engagement in five years share key characteristics:
- Quietly and reliably taking repetitive, pattern-bound work off a professional’s plate.
- Recognizing their strengths, weaknesses and limits; while respecting guidelines, guardrails and context.
- Avoiding the need to spotlight themselves to prove their value.
The dishonest version—AI as inevitability, as magic, as a word you slap on a press release—has a shorter shelf life, though may initially receive press and attention. Buyers in this industry are experienced, skeptical, and increasingly adept at recognizing the difference between real capabilities and empty hype.
My challenge to anyone building AI products right now, whether within insurtech or other highly regulated industries involving financial decisions and oversight, is simple: build the version you’d be comfortable explaining honestly to the most skeptical person in the room.
If you can’t explain what the AI actually does, and where it can fall short, it probably isn’t ready to carry the weight that “AI-powered” implies.
Curtis Kadohama is Chief Product Officer at PerfectQuote, where he has spent the last six years leading product strategy for group benefits technology.
More perspectives from the PerfectQuote team: perfectquote.io/resources