ChatGPT provided “largely appropriate” responses to questions about cardiovascular disease prevention, according to a research letter published in JAMA.
Researchers put together 25 questions about fundamental concepts for preventing heart disease, including risk factor counseling, test results and medication information. They posed the questions to the AI chatbot three times and each set of responses was graded by a clinician.
The clinician rated the responses as appropriate, inappropriate or unreliable if the chatbot’s responses varied in each set. The reviewers also evaluated the responses in two contexts: first as a patient-facing platform and then as draft responses to patient messages sent to a clinician for review.
The study found 21 of the 25 questions were considered appropriate in both contexts, while four were graded inappropriate in both scenarios. In three of the four sets of inappropriate responses, all three answers were incorrect, while one set had only one inappropriate response.
WHY IT MATTERS
The researchers noted several limitations in their analysis. At this time, the chatbot isn’t designed for medical use, and cardiovascular disease prevention can’t be covered in only 25 questions.
In future studies, they said researchers could use more reviewers to evaluate responses or set up a formal system for grading responses that didn’t rely as heavily on a clinician’s subjective opinion.
“Findings suggest the potential of interactive AI to assist clinical workflows by augmenting patient education and patient-clinician communication around common CVD prevention queries. For example, such an application may provide conversational responses to simple queries on informational platforms or create automated draft responses to patient electronic messages for clinicians,” they wrote. “Whether these approaches can improve readability should be explored, because prior work has indicated low readability of certain online patient educational materials for CVD prevention.”
THE LARGER TREND
There are a number of companies touting chatbots for healthcare purposes. Wysa, which makes a chatbot that guides users through cognitive behavioral therapy for concerns like low mood, anxiety and stress, raised $20 million last summer.
Another company behind a mental health chatbot, Woebot Health, scored a $9.5 million investment from Leaps by Bayer nearly a year ago. In 2021, Woebot received FDA Breakthrough Device Designation for its postpartum depression digital therapeutic.
Symptom checking chatbot Ada Health, which announced a $30 million Series B extension in February 2022, recently launched a digital COVID-19 care journey that was developed with Pfizer.
