Testing suggests Google's AI Overviews tell millions of lies per hour
The report says Google’s Gemini-powered summaries still produce millions of wrong answers each hour despite stronger results in later testing.
- The New York Times conducted an analysis with Oumi, finding Google's AI-generated Overviews are accurate 90 percent of the time using the SimpleQA benchmark for generative models.
- Following the update, Gemini 3 improved accuracy to 91 percent, up from 85 percent when Oumi tested Gemini 2 last year.
- Despite high accuracy, Google processes roughly 5 trillion queries per year, potentially exposing users to more than 57 million inaccurate answers each hour.
- Discrepancies between overviews and their sources rose to 56 percent after the update, up from 37 percent, as researchers found linked pages often fail to substantiate claims.
- While Google disputes the testing, The New York Times found approximately one in 10 overviews contains false information, and firms acknowledge the technology's tenuous relationship with the truth.
15 Articles
15 Articles
Study: Google's AI Overviews show millions of wrong answers every hour
Aside from the hallucinations, immense energy requirements, and potentially negative mental health effects, generative artificial intelligence still has an issue with accuracy. However, that hasn’t stopped major tech companies like Google from rolling out features like AI search summaries to its users. Most of the results seem fine at first glance and usually feature multiple source citations, but that doesn’t mean the product works perfectly. A…
Testing suggests Google's AI Overviews tell millions of lies per hour
Looking up information on Google today means confronting AI Overviews, the Gemini-powered search robot that appears at the top of the results page. AI Overviews has had a rough time since its 2024 launch, attracting user ire over its scattershot accuracy, but it's getting better and usually provides the right answer. That's a low bar, though. A new analysis from The New York Times attempted to assess the accuracy of AI Overviews, finding it's ri…
Google's IA research generates millions of wrong answers every day, a problem that is added to the hallucinations, huge energy consumption and negative effects on mental healthA recent study reported by the New York Times revealed that Google's "AI Overview" feature offers correct summaries from reliable sources in 9 out of 10 cases. But if 90% seems to be a sufficient score, the failure rate accumulates in a few minutes. This represents some...
Coverage Details
Bias Distribution
- 80% of the sources are Center
Factuality
To view factuality data please Upgrade to Premium
