This article was originally published in Knowable Magazine.
Death from colorectal cancer can be prevented by regular screenings. Controlling high blood pressure could prolong the lives of the nearly 500,000 Americans who die from this disease each year. Vaccinations help prevent tetanus, which could otherwise be lethal.
Clearly, preventive medicine can make a big difference to health.
And yet most people don’t get the preventive care that could save their lives. Indeed, as of 2015, only 8 percent of U.S. adults 35 and older had received all immunizations, cancer screenings, and other high-priority services recommended for them.
Researchers seeking to change that are borrowing a page from Facebook, Google, and other tech companies. By rapidly comparing small differences in how they communicate with patients—a process known as A/B testing—health-care workers can quickly learn what works and what doesn’t. The approach has already delivered several actionable improvements, though not everyone is convinced of its value.
Tech-oriented companies use A/B testing to make decisions about marketing slogans, web-page colors, and lots of other options. The key is randomization, meaning that people are randomly assigned to see different versions of whatever is being tested. Does a bigger “Subscribe” button on a website generate more clicks than a smaller one? Does one headline over a story capture more readers than another?
Leora Horwitz, an internist and a health-services researcher at NYU Langone Health, and her colleagues adopted this technique—which they call rapid randomized controlled trials—to learn how to improve the delivery of health-care services. Randomized controlled trials, or RCTs, are widely used in medicine, typically to test new drugs or other disease treatments. For example, patients may be randomly assigned to receive either a new drug or the current standard treatment, then followed for months or years to assess whether the new drug works better. But those trials are slow and expensive, in part because researchers have to recruit people willing to be in a medical experiment.
Rapid RCTs, by contrast, are not used to study new treatments, so nobody has to be recruited to participate. Rather, Horwitz’s goal is to improve health-care delivery through quick trials in which one can repeatedly test and fine-tune changes to health-care delivery based on what researchers learn from each test.
“We are randomizing what we’re doing so that we can quickly and accurately assess whether what we are doing is working,” says Horwitz, who wrote about the approach in the 2023 Annual Review of Public Health.
For example, Horwitz and her colleagues wanted to figure out how to get patients to book appointments to address care gaps—preventive services that are overdue. Because of the huge number of patients, physicians’ offices can’t contact everyone by telephone or through the online portal that NYU Langone uses to communicate with patients. So the health system needed to understand what type of reminders were most effective.
In the A/B test, patients with care gaps were divided into two sets: those who had signed up for an online-portal account and those who had not. Patients in each set were then sorted into different groups based on their health-care history. Patients who, based on past behavior, were unlikely to initiate appointments on their own were put in higher-risk groups; those who had eventually booked their own appointments in the past were assigned to lower-risk groups.
In one part of the test, several thousand patients who had no portal account were randomized so that some received a telephone-call reminder and others did not. Patients who received a phone call booked appointments to address 6.2 percent of the care gaps, compared with just 0.5 percent among those who were not called.
In another part of the test, some patients with portal accounts received a reminder message through that channel, while others did not. Of those who received the message, 13 percent scheduled the needed services, compared with 1.1 percent of those who were not contacted.
Importantly, the experiments revealed that a phone-call reminder was the most effective way to reach the subgroups of patients who were high-risk and the least likely to get their preventive services without a nudge. Shortly after the test results were known, NYU Langone prioritized all of its highest-risk patients to receive telephone reminders and greatly expanded its capacity for sending messages through the patient portal.
“When we learn something, we apply that to all of our messaging quickly,” Horwitz says. That immediately extends what they’ve learned to tens of thousands of people. “That’s gratifying.”
NYU Langone’s A/B testing is why many of the medical center’s female patients are now receiving short messages to remind them to schedule their mammograms. The researchers used rapid RCTs to test the wording on reminders sent through the online portal: Would shorter messages get better results? Indeed, patients who received a 78-word reminder scheduled nearly twice as many mammograms as those who received the old 155-word message.
In another investigation, to find out how to boost vaccination rates among very young children, Horwitz and her team turned to rapid randomized tests that compared one-text and two-text reminders to parents against no text reminder at all. Only the two-text reminder—one sent at 6 p.m., the other sent at noon two days later—made a difference, tripling the number of appointments scheduled. Most appointments were made after the second text, suggesting that this booster reminder was what triggered the parents to act.
Though it’s still new to the health-care sector, the idea of rapid RCTs is catching on. One research team—an economist, a physician, and a public-policy expert, none of whom was affiliated with Horwitz’s group—used the technique to learn how to increase the use of preventive-care services by Black men, the U.S. demographic group with the lowest life expectancy.
They recruited more than 1,300 Black men from Oakland, California–area barbershops and flea markets, asked them to fill out a health questionnaire, and gave them a coupon for a free health screening. A pop-up clinic, staffed with 14 Black and non-Black male doctors, was set up to provide the screenings, and the participating men were randomly assigned to a Black or a non-Black doctor. The result: Black men assigned to Black physicians were more likely to get diabetes screenings, flu vaccinations, and other preventive services than those assigned to non-Black doctors.
Some experts doubt that rapid A/B testing will ever become commonplace in health care. Darren DeWalt, a physician who directs the Institute for Healthcare Quality Improvement at the University of North Carolina, likes the concept, but he thinks most health-care organizations will avoid it for ethical reasons, possibly because people tend to disapprove of randomization, even in the context of something as innocuous as appointment reminders. “People in this country don’t like the idea that they are randomly allocated to something, even something as simple as that,” DeWalt says. “There’s a lot of suspicion around researchers in health care.”
Others criticize A/B testing as tinkering at the margins. Pierre Barker, the chief scientific officer for the nonprofit Institute for Healthcare Improvement in Boston, believes that significant improvements in health-care delivery require an in-depth analysis of the problem to be solved, which may require many changes to the system. By contrast, rapid randomized controlled trials focus on a single, discrete change—say, the words used in a telephone script—rather than a broader effort to understand why patients don’t get preventive services and what can be done to change that.
“The attractiveness is how fast it can move, more than the size of the impact,” he says. “I remain to be convinced that you can get more than a small incremental change” from rapid randomized controlled trials.
It is true that the majority of NYU Langone’s care gaps were not resolved by the new reminders, says Horwitz, but the tests did provide information that led to hundreds of potentially lifesaving services being performed. That is what convinces her that the health-care industry should embrace rapid randomized trials.
“If you were working for a web company or an airline or any other industry, you would randomize as a matter of course—this is the standard practice,” she says. “But it is still very foreign in health care, and it shouldn’t be.”
