Bold and Wrong: Doctors Often Too Confident with Diagnoses

Duration: 8:14

When it comes to diagnosing difficult cases, some doctors are much more confident than they should be. In a study by Meyer and colleagues in JAMA Internal Medicine in August 2013, researchers presented physicians with easy-to-diagnose cases, and difficult-to-diagnose cases. Despite performing poorly diagnosing the difficult cases, doctors reported confidence levels nearly as high as with the easy ones. The higher the confidence level of individual clinicians in the study, the less likely they were to order a test or seek a consult—a pattern that has clear implications for outcomes and professional liability. CRICO interviews study authors Ashley Meyer, PhD, and Hardeep Singh, MD, MPH, as well as Harvard patient safety researcher Tom Sequist, MD, about the latest thinking in the use of physician practice traits to minimize risk.

This podcast is an episode of Patient Safety Updates. You can find other episodes and subscribe using the links to the left.

Transcript for the Podcast

Physicians project certainty for good reasons. A confident doctor bolsters worried patients and directs the care process. But when it comes to diagnosing difficult cases, some doctors are much more confident than they should be. In a study by Meyer and colleagues in JAMA Internal Medicine in August 2013, researchers presented physicians with easy-to-diagnose cases, and difficult-to-diagnose cases. Despite performing poorly diagnosing the difficult cases, doctors reported confidence levels nearly as high as with the easy ones. By and large, they did not seek more diagnostic resources like tests or consults for the more difficult cases.

“When physicians were given extremely hard cases where their certainty levels should have been really low—we know that their accuracy dropped significantly—we also know that their certainty level was really high.”

Hardeep Singh was co-author on the study. Dr. Singh is a patient safety researcher at the Houston Veterans Affairs Center for Innovations in Quality, Effectiveness and Safety, and associate professor at Baylor College of Medicine. He notes that the higher the confidence level of individual clinicians in the study, the less likely they were to order a test or seek a consult—a pattern that has clear implications for outcomes and professional liability.

“When your accuracy is low and your uncertainty is really high and the case is very difficult, you really should be asking for more help, whether it is from your friend or colleague down the street or like a curbside consult or actual referral to a specialist or some diagnostic test that you need to get. What we found was that was not happening. So it was almost like when physicians most needed help, they actually didn’t seek it out.”

The researchers studied more than one hundred general internists from across the United States. Each was given four validated case vignettes of varying difficulty. One of the easy vignettes involved a 60-year-old man with crampy lower abdominal pain of three week’s duration, anemia, and weight loss. The patient also had a positive occult blood test.
A difficult diagnostic case in the study presented a 68-year-old man who came in with fever, fatigue, arthralgias, and shoulder pain. He had no joint swelling, but the patient had pain in his shoulders on abduction to 90 degrees. The study’s lead author, Ashley Meyer, is a cognitive psychologist at Baylor.

“A lot of the researchers in the medical area tried to figure out what people’s confidence was from the notes they left in a patient record, for example. But cognitive psychologists tend to measure this stuff more directly, so they actually ask ‘what is your confidence?’ And they have people rate their confidence on a scale of zero to 10, for example, and then they can compare it statistically with how they're are actually performing.”

One of the metrics used in the study is “diagnostic calibration,” which is simply the relationship between the accuracy of a diagnosis and the level of confidence in that accuracy, as expressed by the clinician.

“Even though the physicians only get five percent of the really difficult cases correct, which is almost none—pretty much everyone is getting it wrong—they are all rating their confidence at about a six out of 10, so rather high.”

Meyer says it isn’t exactly the way medical professionals have been trained to view their care processes. But diagnostic calibration is key to understanding the dimensions of the problem.

“I think the idea of diagnostic calibration is not necessarily a natural one. I don't know that they have been taught to assess their performance, and especially not in those terms, and so in our study when we had them rate their confidence as they're making decisions. That may not be entirely natural. They may just be used to thinking through it, making the decisions and moving on. …It may be something that becomes more important as we discover why diagnostic error happens in the first place because maybe they're not really assessing these things as they move along.”

The demographics of clinician subjects, such as age, ethnicity, and geographic region were not related to accuracy, confidence, or the ordering of further diagnostic testing. Meyer says that further research is needed to learn of any relationships between overconfidence itself and other personality traits of an individual clinician. In the Harvard medical system, Dr. Tom Sequist is studying how the characteristics of a physician’s practice relate to quality of care. Dr. Sequist is an internist at Harvard Vanguard Medical Associates. One of the variables in Dr. Sequist’s research is the individual doctor’s risk tolerance.

“There is a little bit of literature that shows that some ways of assessing physician personality can actually link to how a physician practices medicine. There is something called the Jackson Personality Inventory, which actually is a series of questions, short survey for physicians to answer, and you can give them a score based on how risk adverse they are and how risk tolerant they are. In the emergency department setting for patients who present with chest pain, you can show that the physicians who are more risk adverse are more likely to admit patients to the hospital who present with chest pain; emergency department physicians we’re talking about. The physicians who are more risk tolerant are more likely to discharge patients home directly from the emergency department and not admit them.”

In addition to an individual physician’s risk tolerance, another practice style measurement connected to patient safety involves the clinician’s level of stress response from uncertainty. Also, physicians could get feedback that compares their habits to those of other similar clinicians. Dr. Sequist says these kinds of tools may give practicing clinicians valuable insight as they evaluate how they manage their overall practice.

“Do we tend to order lots of test results compared to our other physician colleagues when evaluating a particular condition? Do we tend to not order enough tests when evaluating a condition compared to our physician colleagues? And there may or may not be a right answer in some of these scenarios, but we believe that it is important for you to know how this feature of your personality is really affecting how you practice medicine.”

Ashley Meyer says the field of cognitive psychology may offer additional ideas for improving diagnostic performance. For example, “de-biasing” is a mechanism to change behavior or take precautions based on knowing your pre-existing biases. But Meyer agrees that a mechanism for measuring traits like over-confidence, and feeding the information back to clinicians, may hold the most promise.

“Right now, they don't get feedback that we know of about their diagnostic performance, and there are people in other professions, other areas of expertise, such as weather forecasters. They get constant feedback until they're able to better calibrate themselves. Physicians sometimes get a patient who may be a more difficult case and they diagnose them, they send them off on their way, and sometimes they never hear what happens and so, unfortunately, they are not getting the feedback that they need to calibrate. Figuring out a way to deliver this feedback would be an ideal solution.”

Nowhere is the need to solve diagnostic errors more acute than in the ambulatory setting. Relatively little is known about what kind of practice characteristics and outcome measures are most valuable for improving care. But ambulatory medicine is where most malpractice cases now arise—and the most prevalent allegation there, is diagnostic error.