Ask an AI tool what your business paid in processing fees last month and you'll probably get an answer in seconds. It may be specific down to the cent, neatly formatted and delivered with complete confidence.

For payments teams, that's both the appeal and the problem. Getting an answer quickly is useful, but only if you know you can trust it.

We wanted to understand what actually makes the difference, so we tested the same AI model against 80 payments questions covering authorization, fees, sales, interchange and disputes. We started with raw processor files and progressively improved the data through six levels of readiness, eventually reaching governed calculations.

The model stayed the same. The questions stayed the same. What changed was the data available to answer them.

And one of the more interesting things we found was that simply making payments data cleaner didn't solve the problem nearly as quickly as you might expect.

What happened when we asked AI for one month of processing fees?

Here's one example that I think illustrates the issue particularly well.

We asked the AI to calculate one acquirer's June 2026 processing fees by category. At this point in the test, the data had already been cleaned, standardized and structured into one queryable layer. In other words, we had already done a lot of the work most teams think of when they talk about getting their data "AI-ready."

The AI came back with $191,246.32.

The signed-off figure was $225,000.

That's roughly a 15% difference, which is obviously significant. But what makes this example more concerning is that there was nothing about the response that would have told you something was wrong. It wasn't a rough estimate or an answer accompanied by a warning. It was a precise number, down to the cent, delivered just as confidently as the correct answer would have been.

Naturally, our next thought was: maybe the AI just needs more context.

So we gave the AI the definitions. It still got the answer wrong.

We gave the model the written fee definitions: the same type of documentation you might give a new analyst so they understand exactly what should and shouldn't be included in the calculation.

The result?

$191,246.32. Again.

Not approximately the same number. The exact same number, down to the cent.

This is where the distinction between giving AI access to a definition and actually enforcing that definition becomes important. Just because the model can read your documentation doesn't necessarily mean it will apply that logic consistently every time it answers a question. It still has to interpret the definition and determine how to use it.

The result only changed when the AI stopped working out the calculation itself and instead called a certified tool: a tested calculation built around the definition the business had already agreed on.

At that point, it returned $224,719.49, within tolerance of the $225,000 signed-off figure.

That may sound like a technical distinction, but for a payments or finance team it's a pretty practical one.

The bigger concern is how often the wrong answer looks right

This example made us curious about something beyond overall accuracy.

When AI gets a question wrong, does it know that it's struggling? Or does it still give you an answer that looks perfectly usable?

That matters because an obvious failure is relatively easy to deal with. If an AI tool says it can't answer your question, you know you need to investigate. The bigger business risk is getting a polished number, assuming it's right and using it in a report, forecast or conversation with finance.

So we tracked what we called the silent failure rate: the percentage of numeric questions where the AI returned a number that didn't meet our standard, without signaling that anything was wrong.

This was probably one of the findings that surprised us most.

Once the payments data was clean and standardized, the AI answered 97% of the numeric questions we asked it. That sounds great until you look at the second number: 80% of those questions resulted in a wrong number with no warning.

Adding agreed definitions helped considerably, bringing the silent failure rate down to 47%. But it didn't disappear until those definitions were actually enforced through certified calculations.

Why does clean data make the problem harder to spot?

There's a strange middle ground here that I think payments teams need to be aware of.

With raw processor files, AI often struggled visibly. It might fail to interpret the files, take longer to respond or simply tell you it couldn't answer the question. Those limitations are frustrating, but at least you know they're there.

Once we standardized the data, the experience improved dramatically. The AI became faster, more responsive and much more willing to answer.

And that's potentially where a lot of companies are today. They've spent time bringing processor data together, cleaning it up and connecting an AI tool to it. They ask a question and get back a fast, specific, polished answer, so it feels like the hard part is over.

Our results suggest that's exactly the point where you should be testing more carefully.

How can you tell if your AI's payments answers are right?

If you're experimenting with AI on payments data today, I wouldn't start by measuring how many questions it can answer. I'd start by testing whether you can trust the answers it gives you.

The study's lesson is simple: measure the answers, not the answer rate. A few practical steps:

• Test against a known figure. Pick something your team already knows to be true (a monthly processing fee total, effective rate or another figure that has been reconciled and signed off) and ask the AI to reproduce it. Then look at where the answer starts to break down.
• Don't treat confidence as a signal. One of the clearest findings from our testing was that a wrong answer can look exactly like a right one.
• Check whether definitions are enforced, not just available. If the AI simply has access to documentation explaining how your company calculates a metric, that's different from having the calculation itself governed and consistently enforced.
• Start with one workflow. Pick one workflow that has a clear owner and a real business decision behind it. Find out where the AI stops being reliable, fix that piece and then expand from there.

That process will tell you much more about whether your payments data is actually ready for AI than the quality of the chatbot experience will.

Frequently asked questions

Is the AI model the problem?

Not in this study. The model was held fixed at every level (Claude Sonnet 5, with the same settings and question wording), so the change from 5% to 98% came from the data alone. The study doesn't compare models or claim model choice is unimportant. It isolates what data preparation can move.

What is a certified tool?

A certified tool is a predefined, tested calculation built on a governed metric definition. Instead of working out a metric like effective rate on its own, the AI calls the tool, so the agreed definition is applied the same way every time.

Does standardizing payments data fix AI accuracy?

It helps, but not on its own. In the study, curating data into one standardized layer raised accuracy from 6% to 21% and cut cost and response time sharply. The largest gains came from agreed definitions and from enforcing them through certified tools.

How does Harmonize help with AI accuracy on payments data?

Harmonize brings data from 30+ payment providers into one standardized system of record and holds the agreed definitions for the metrics payments teams rely on. That gives AI tools a governed foundation to work from, so answers start from calculations your experts have validated rather than the model's own interpretation.