Human Feedback Types
strongRubric Rating
Directly usable for protocol triage.
"Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini."
HFEPX · Eval paper review
Does the Citation Matter? Quality, Grounding, and Physician-Rubric Accuracy of Google's AI Overviews for Medical Information
Sam Agnoli, Brigid Reilly, Lily Barna, Emőke-Ágnes Horvát
Published
Oct 29, 2026
Citations
0
Trust level
Moderate
Usefulness score
65/100 (Medium)
Extraction confidence
70% (Moderate)
Derived from extracted protocol signals and abstract evidence.
Rater population
Domain Experts
Signals refreshed
Oct 29, 2026
Should you rely on this paper?
This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.
Use this for comparison and orientation, not as your only source.
Best use
Secondary protocol comparison source
Use if you need
A secondary eval reference to pair with stronger protocol papers.
What to verify
Validate the evaluation procedure and quality controls in the full paper before operational use.
Main weakness
No major weakness surfaced.
Human feedback signal
Detected
From extracted signals
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
65/100
Moderate-confidence candidate
Useful as a secondary reference; validate protocol details against neighboring papers.
If you are doing eval pipeline work, start here
Abstract
Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini. This may be especially consequential for health information, given that Google's traditional algorithm was designed to prioritize quality sources, and inaccurate information carries health risks. We evaluated 3,611 benchmark health questions, comparing each AIO against that question's top five traditional results on source selection and quality. We also evaluated the AIOs' statement grounding in their cited sources and how accurate AIOs were against physician-written rubrics. Overviews appeared for 96.4% of questions but drew on substantially different evidence than traditional search: only 44.4% of the top five results were cited by the corresponding AIO, which instead overrepresented commercial health publishers (e.g., WebMD, Healthline) at the expense of academic and government sources (e.g., Cleveland Clinic, National Health Service). AIO-cited pages scored lower than traditional results on four of six well-established credibility criteria. The grounding of AIO statements to their sources was poor: after correcting for classifier error, an estimated 56.7% of cited statements were supported by their cited page, and a further 20.8% of all statements were factual claims with no citation at all. When AIOs were graded against physicians' rubrics, they earned 38.2% of all available points (e.g., accuracy 54.2%, completeness 21.6%), and among the subset of AIOs whose physician rubric anticipated a specific inaccurate or potentially harmful claim, 22.2% of AIOs contained the harmful content. Notably, the accuracy of AIOs was unrelated to the grounding of their statements. We recommend AIO literacy interventions for users and holding AIOs at least to Google's own health-ranking standards.
What we could verify
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Evaluation Modes
strongAutomatic Metrics
Includes extracted eval setup.
"Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini."
Quality Controls
missingNot reported
No explicit QC controls found.
"Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini."
Benchmarks / Datasets
missingNot extracted
No benchmark anchors detected.
"Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini."
Reported Metrics
strongAccuracy
Useful for evaluation criteria comparison.
"When AIOs were graded against physicians' rubrics, they earned 38.2% of all available points (e.g., accuracy 54.2%, completeness 21.6%), and among the subset of AIOs whose physician rubric anticipated a specific inaccurate or potentially harmful claim, 22.2% of AIOs contained the harmful content."
Rater Population
strongDomain Experts
Helpful for staffing comparability.
"Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini."
Benchmarks and datasets
No benchmark or dataset names were extracted from the available abstract.
Reported metrics
accuracy
Human feedback details
- Uses human feedback
- Yes
- Feedback types
- Rubric Rating
- Rater population
- Domain Experts
- Unit of annotation
- Multi Dim Rubric
- Expertise required
- Medicine
Evaluation details
- Evaluation modes
- Automatic Metrics
- Agentic eval
- None
- Quality controls
- Not reported
- Evidence quality
- Moderate
- Use this page as
- Secondary protocol comparison source
Research brief
Metadata summary
Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Key takeaways
- Google's AI Overviews (AIOs) are the first result users encounter when searching the web, shifting the work of selecting and evaluating sources from the user to Gemini.
- This may be especially consequential for health information, given that Google's traditional algorithm was designed to prioritize quality sources, and inaccurate information carries health risks.
- We evaluated 3,611 benchmark health questions, comparing each AIO against that question's top five traditional results on source selection and quality.
Researcher actions
- Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
- Validate inferred eval signals (Automatic metrics) against the full paper.
- Use related-paper links to find stronger protocol-specific references.
Caveats
- Generated from abstract + metadata only; no PDF parsing.
- Signals below are heuristic and may miss details reported outside the abstract.
Recommended queries
Contribution summary
- We evaluated 3,611 benchmark health questions, comparing each AIO against that question's top five traditional results on source selection and quality.
- Overviews appeared for 96.4% of questions but drew on substantially different evidence than traditional search: only 44.4% of the top five results were cited by the corresponding AIO, which instead overrepresented commercial health…
- The grounding of AIO statements to their sources was poor: after correcting for classifier error, an estimated 56.7% of cited statements were supported by their cited page, and a further 20.8% of all statements were factual claims with no…
Why it matters for eval
- We evaluated 3,611 benchmark health questions, comparing each AIO against that question's top five traditional results on source selection and quality.
Researcher checklist
-
Human feedback protocol is explicit
Detected: Rubric Rating
-
Evaluation mode is explicit
Detected: Automatic Metrics
-
Quality control reporting appears
No calibration/adjudication/IAA control explicitly detected.
-
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
-
Metric reporting is present
Detected: accuracy