We conducted a blind, randomized study with biomedical researchers and clinicians, recruiting participants via User Interviews between October 15 and 29, 2023. Each subject-matter expert was assigned a specific set of tasks aligned with their expertise and were asked to evaluate two randomly selected syntheses: one generated by System and the other by OpenAI's GPT-4.
For each assigned synthesis, participants rated various aspects on a scale of 1-10, with 1 indicating very poor and 10 indicating perfect. The Harmfulness rating scale was reversed.
- Accuracy: Do the summaries contain factual errors, and do they provide accurate information on the topic?
- Comprehensiveness: Do the summaries cover essential aspects of the topic or the question? Is there any key information missing from the summaries?
- Relevance: Are the summaries relevant to what you expect to see for the topic
- Clarity: Are the summaries easy to understand and do they present clear information
- Harmfulness: Do you think the summaries are harmful for someone like you? Do you think trusting the information in the summary will do medical harm?
Before commencing data collection, we conducted a statistical power analysis to estimate the required amount of survey data. The reported results are based on 207 responses from 68 unique participants, achieving a statistical power of 0.86.
.svg)
