Healthcare Benchmarks AIIndependent benchmark analysis / Original Nature study, corrected version of record
MultiMedQA combines professional examinations, biomedical research questions and consumer health questions. We use the original Nature study to examine why those sources demand different evaluation endpoints. Multiple-choice answer accuracy is straightforward to count; a long answer can contain correct information, omit an important qualification and still receive mixed judgments. Our analysis concentrates on the consumer-question evaluation and the relationship between source inventories and the smaller human-rated sample. Published Med-PaLM and Flan-PaLM measurements remain historical study results. The coverage explorer is our interpretation of what each task format can and cannot support.
01 / What is being tested?
Data origin. Seven components: MedQA, MedMCQA, PubMedQA, medical MMLU subjects, LiveQA, MedicationQA and HealthSearchQA. HealthSearchQA contains search-engine-surfaced consumer queries, not clinical encounters. [1][2][3]
Version-of-record source inventory, not all human-rated.
Methods: HealthSearchQA [1]100 HealthSearchQA, 20 LiveQA and 20 MedicationQA.
Human evaluation results [1]The original study’s exam test denominator.
Methods: MedQA [1]Flan-PaLM 540B, Med-PaLM 540B and clinician answers.
Human evaluation results [1]Professional, research and consumer questions have different input and output contracts.
[1]The paper evaluates prompted model outputs; the consumer subset also receives clinician-authored answers.
[1]Answer keys score MCQs; clinicians and lay raters assess different dimensions of long-form answers.
[1]Report consensus, omissions and possible harm separately rather than collapsing them into a generic quality percentage.
[1]Dataset anatomy
Selected for human evaluation.
Selected for human evaluation.
Selected for human evaluation.
These are 140 selected questions, not the full sizes of the three source collections. Bar lengths use the largest listed count as their reference; they are not percentages of a shared population.
02 / Measurement
Direction depends on the axis: consensus higher, judged potential harm lower.
Multiple-choice correctness and human judgments of long-form answers are separate endpoints.
MCQ accuracy = correct answers / evaluated questions; human rating rate = responses assigned a category / responses rated for that axis
Do not average MCQ accuracy with harm or consensus percentages. Keep question subset, rater design and model configuration attached. [1]
03 / Measured evidence
Paper-reported results / selected rows
Nature 2023 original study; 140-question long-form evaluation; one clinician rating per answer in this pilot.
Paper-reported percentages; bootstrap uncertainty is described in the paper. Similar point estimates do not establish clinical equivalence.
Source: Human evaluation results: Scientific consensus; Figure 4a [1]
Paper-reported results / selected rows
Same original consumer-question study; subjective hypothetical-harm ratings, not observed adverse events.
Keep separate from consensus. These are evaluator judgments about answers, not measured patient harm rates.
Source: Possible extent and likelihood of harm; Figure 4d [1]
04 / Our original analysis
HealthSearchQA contains 3,173 questions; 100 enter the original human-rated subset. [1]
A large source collection does not make every reported human judgment a large-sample estimate. Track source size and evaluated sample separately.
The human framework separately assesses incorrect content, missing content and other axes. [1]
A single overall accuracy number erases error types that may matter differently to readers. Preserve a vector of results.
Clinician and Med-PaLM consensus point estimates are close in the original pilot. [1]
Without an equivalence design and its uncertainty, this is not evidence that they can replace one another in practice.
The later Med-PaLM 2 study distinguishes 140-question and 1,066-question sets. [3]
Model generations and sample expansions must remain separate in comparisons; a shared suite name does not fix the protocol.
05 / Scope of the evidence
These tasks do not directly evaluate actions in an EHR, actual patient dialogue or clinical outcomes. [1]
The original long-form study uses a small selected question set and subjective rating axes. [1]
Source overlap and changing scientific knowledge complicate interpretation across later model generations. [1]
No umbrella dataset license is established by the paper’s open-access article license. [1]
Evidence trail
Singhal, Azizi, Tu et al.; Nature. Version-of-record MultiMedQA definition and historical human evaluation.
Google Research. Official creator record explaining the seven-source suite and Med-PaLM study.
Singhal et al.; Nature Medicine. Follow-up study distinguishes MultiMedQA 140 from expanded 1,066-question evaluation.
Benchmark authors retain authorship of their work. This publication provides independent analysis; published rows are not Arcophos evaluation runs. Editorial method.