How accurately does a search-grounded AI model quote software pricing?
AI assistants increasingly answer “how much does this software cost” by searching the web and quoting a price back. We wanted to know how often that quoted price is actually right. So we asked a search-grounded model the same two pricing questions about a stratified set of products and scored every answer against CostBench’s verified record. This is what we found — reported as two separate numbers, because the answer depends entirely on the kind of question.
What we tested
The findings come from three controlled runs, all against gemini-2.5-flash-grounded (Google's gemini-2.5-flash via the Gemini API, with Google Search retrieval enabled), sampled 2026-07-12 and 2026-07-13. We report their sample sizes plainly rather than rolling everything into one headline number.
- An audited pilot run — 40 samples. 20 products × two questions, hand-checked row by row against the vendor sources. This run is where we learned our first scorer was too lax, and where we caught the first defects in our own records.
- The full grounded corpus — 220 rows across 110 products. Re-scored with a rebuilt, stricter scorer (v2). This is the source of the headline 81% / 34% split below.
- A paired grounded-vs-memory experiment — 80 calls on 20 stratified products. The same 20 products (chosen by pricing type, not popularity) run through both the grounded lane and a memory-only control lane (gemini-2.5-flash-memory), two questions each. This isolates what search retrieval actually contributes.
The two questions
Every product was asked exactly two deterministic questions. They sound similar but stress very different things — the first asks for a fact the vendor publishes directly; the second asks the model to do arithmetic over a scenario.
“How much does {product} cost per month? Give the specific price tiers.”
A direct lookup: does the model quote the vendor’s published tier prices?
“What would {product} cost for a team of 25 users? Give the monthly and annual price.”
A computed total: the model must find a per-seat price and reason to a 25-seat monthly and annual figure.
What the data shows
Three findings hold across the runs. We keep the two question types apart throughout — blending them into one accuracy number would hide the entire result.
- Direct monthly-price answers are genuinely strong. On the full corpus, the grounded model matched CostBench’s verified record on 81% of monthly-price questions (89 / 110). The paired experiment agreed — 85% (17 / 20) accurate on the same question. When the answer the model needs is a price the vendor publishes on its own page, search-grounded retrieval usually gets it right.
- Team-cost answers are unreliable, and fabrication is the dominant failure. On the same corpus, team-cost questions matched only 34% of the time (37 / 110). 60 / 110 of 110 were flagged for review — and 24 of those were outright fabricated totals: an invented team plan, a made-up per-seat discount, or a price for a different product entirely, reasoned into a confident-sounding monthly and annual figure. The model rarely says “I don’t know” on a team question; it computes something.
- Without search, the model mostly abstained — it did not spray stale prices. In the paired experiment, the memory-only control lane returned “I don’t have current pricing” on 24 / 40 answers, versus 1 / 40 for the grounded lane. When memory did commit to a number (16 / 40), it was right about half the time and flagged about half — its accuracy clustered on famous, rarely-changing products. So the honest reading is not “ungrounded memory is full of wrong prices” — it is “retrieval is what turns an abstention into an answer, and the retrieved answer still needs checking.”
These are cross-sectional snapshots from the dates sampled, not a trend. We say more about that in the limitations below.
Where the model got its numbers — and why that matters here
A fair accuracy test can’t grade a model against a source the model is also reading. So we checked every grounded answer’s source list. Across all 0 / 80 grounded samples we inspected for independence, costbench.com never appeared as a source. The model is not quoting us back to us — when we score its answer against our record, we are using our record as an answer key it did not consult.
In the audited pilot, the vendor’s own website was present among the sources in 87.5% of samples (35 / 40) — so grounding does reach official pricing pages. But by raw mention count, third-party aggregators and review sites outnumber vendor-official sources (in the paired experiment, roughly 179 aggregator mentions to 47 vendor-official). Grounding is a “vendor plus aggregator” mix, not a clean read of the vendor’s own page — which is part of why prices drift.
How we score — and the rule that keeps it honest
Our scorer extracts the structured price claims from each answer — the amount, currency, billing period, unit, and whether a number is a team total — and compares them to the product’s verified record. The worst material claim decides the verdict: one correct number never erases a wrong one. The public verdicts are accurate, flagged for review, stale, custom-pricing, and no price given.
The neutrality rule: a mismatch is never automatically called “wrong.” It is flagged for review and adjudicated against the vendor’s own published pricing first. “Wrong” is a verdict a human reaches after checking the source — never one the scorer stamps on its own.
This matters because it cuts both ways. When we adjudicated the first 4 suspected mismatches from the pilot run, 3 of them turned out to be defects in CostBench’s own record, not model errors:
- iXTime — the record stored its prices as US dollars, but the vendor (a German company) publishes them in euros. A CostBench record defect, not a model error.
- Render — the record captured Render's compute-instance sizes instead of the Workspace plans (Hobby / Pro / Scale) the vendor presents as its primary lineup. A CostBench record defect.
- ExpressVPN — the record held only the discounted two-year promotional rates and was missing the standing month-to-month list prices. A CostBench record defect.
- Scalefusion — suspected mislabeled, but on review the record was correct: the vendor only publishes an annual-billed rate, exactly as filed.
That is the point of flagging rather than blaming. A flag is a question — “these two numbers disagree, which one is right?” — that gets answered against the vendor source. It feeds a review queue that can just as easily fix our record as confirm a real model error, and we never publish a flag as a claim that a model is inaccurate.
Limitations
This is one early study, and it is worth being clear about what it can and cannot support.
- One model family, so far. Every figure here is from Google’s gemini-2.5-flash (grounded and memory-only). We have not yet run the same panel against other model families, so nothing here generalises to “AI models” as a class.
- API models, not consumer apps. As stated up top: these are controlled API calls. The ChatGPT, Gemini, and Claude products people actually use are different systems and may score very differently.
- Cross-sectional, not longitudinal. These runs are point-in-time snapshots from 2026-07-12 and 2026-07-13. They measure accuracy on those dates — not how quickly a model’s prices go stale over time. Longitudinal tracking (re-sampling on real price-change events at fixed intervals) began separately on 2026-07-13 and is where any staleness claim will come from.
- The scorer has documented limits. It validates against our record, so a gap in our record can produce a flag that isn’t a model error (exactly why the neutrality rule exists). Its rules for currency, billing cadence, and pricing axis are explicit and auditable, but they are rules — they will occasionally flag a defensible answer for human review rather than pass it silently.
- The staleness path was calibrated synthetically. In the paired experiment, one product group used an injected, simulated price change to confirm the “stale” detection works end-to-end. That single calibration case is a check that the machinery fires — not evidence that any model’s memory is actually stale.
Represent one of these products and think a flagged answer is wrong — or that our record is? Claim its pricing record free to verify your identity and confirm its sources. Corrections are always free.