Local Neighborhoods, Lone Gunmen, And Exam Scores: Physician Empowerment In The Age of Accountability

I’ll admit it readily.  I googled my wife.  It was the spring of 2005, I sat down and quite brashly tracked her online footprints after our first date.  It was no less beguiling than the scent of perfume, just as palpable, and after a brief hour online, I was smitten.  I was floored to find out that, unlike some of our other high powered colleagues, she had not mentioned even one of her many accomplishments in the three hours I spent with her.  She seemed more interested in the slice of strawberry shortcake we ordered from Tart-to-Tart.

The internet has redefined persona.  And it certainly has redefined how we view professionals we trust with our lives, finances, and assets. Beginning as early as 1999, it was assumed that any legitimate professional should have some kind of internet footprint—and with that initial assumption, a community mandate emerged that any legitimate professional can be and should be reviewed and rated by the community of users searching the web.  The internet has transformed professional accountability into a real time, interactive dynamic.

For physicians, the internet has enabled online tracking of performance, satisfaction and allows multiple users to share information about the same provider.  Physicians and health providers are now fair game for a wide range of online rating scales.  The terrain is fairly uniform, most of the scales relying on some combination of objective and subjective criteria (wait times, bedside manner, communication, scheduling time, access) while also collating user (i.e. patient) reviews that are often highly specific, detail-rich, and quite subjective.  Some of the most frequented and/or better known physician review websites include:  HealthGrades (suitably titled, considering that massive extinction level events like MCAT scores and grades in organic chemistry are the evolutionary bottlenecks that whittle down the population of pre-medical students so that only the allegedly “fittest” survive admission to medical school), Vitals (ironically titled since physicians rarely collect vitals on their own, have been known to fumble with blood pressure cuffs, and rely almost exclusively on nursing and triage staff to collect them), Angie’s List (not an escort service, and you should know it is more legitimate than Craigslist, includes fewer serial killers per region enrolled as users, and is widely used for vetting a vast array of services from rug cleaning to scoping your colon), and Yelp (the “barbaric yawp”, I mean “yelp”, of reviewing physicians by the masses), to name a few.

Conducting my own empiric study, using HealthGrades as the intervention of choice, I subjected a number of Stanford and UCSF internal medicine and medicine subspecialty providers to online review—the cohort was divided evenly between outstanding providers and average providers, based entirely on my own experience working with them in clinic or in the inpatient setting.  I also added a few “troubled” providers from both institutions, each of whom I had observed having less than ideal interactions with patients and/or their families (yes, they exist even at top centers of excellence like Stanford and UCSF).   The results, though clearly anecdotal, were illuminating to say the least:

-less than fifty percent of the providers I considered to be stellar (with whom I would trust my own family) were ranked highly on HealthGrades

-some of the “troubled” providers (who clearly should have pursued investment banking, and not medicine) received shockingly high scores

-older veteran providers (i.e. those who had been practicing for thirty or more years) often had no reviews and no scores

-the extremes seemed to be missed, with veteran providers having zero reviewers and very recent graduates also having zero reviewers

-inpatient providers were almost universally not reviewed unless they had an outpatient clinic

-the absolute number of reviews for each physician was shockingly low, ranging from three to five for the typical physician I searched

In summary, at least based on my own anecdotal, retrospective gloss, HealthGrades proved to be a not so sensitive litmus for detecting excellent physician performance, and seemed woefully underpowered, particularly given the complexity of the various criteria it tries to evaluate.  I conducted a similar experiment with Yelp.  Yelped physician reviews were often quite detailed (and indeed entertaining at times) and were often supported by far more reviews (better power).  That being said, the overall review framework still seemed disjointed and, like the typical Yelp experience with restaurants, very much hit-or-miss.   Physicians seem very much vulnerable to the local neighborhood bias (where it seems obvious that you sent an e-mail to local fans stating “Dear friends, neighbors and loved ones, please log on and rate me as extraordinary…”) or to the lone gunman bias (where the one patient who waited an hour to see you due to no fault of your own, has now used the internet to attribute all ills–their own and the world’s– to you and your office.)

Suffice it to say, the system is flawed and health researchers and health mavens have been pontificating since the mid 2000’s on the ills of the system.  One study out of Tufts concluded that most of the physician rating sites are far too kind, primed to collect information skewed towards positive performance, and also noted that there seemed to be less overall interest in reviewing physicians.  (J Gen Intern Med. 2010 Sep;25(9):942-6. Epub 2010 May 13. Lagu et. al. do a remarkable job of reviewing and analyzing the various physician review websites, providing useful data helping to characterize the portals themselves as well as the nature of their findings.  Lagu’s data suggest an even more extreme version of local neighborhood bias, where the physician herself writes an anonymous review replete with publication data, biographical data, and glowing accolades.)  Other commentators insist that the large majority of their colleagues within a given region are not being reviewed, leaving one to conclude that online doctor reviews perhaps only offer insight into the performance of the more tech savvy physicians or the performance of physicians caring for tech savvy patients.

No doubt, physician reactions have been varied.  On perhaps the more extreme and aggressive side, Medical Justice, a group seemingly devoted to protecting the reputation of physicians, proposes a number of unconventional and hyperlegal approaches by which the physician can combat lone gunman bias.  Most notably, among the various legal contraptions it sells to defamation-wary physicians, Medical Justice offers a template agreement which purportedly any physician can use to prevent a patient from posting comments online; reportedly, the patient agrees in advance, presumably as a precondition to receiving medical services, that they will not post comments on the internet about the medical service provided.  The enforceability of these clauses is highly questionable:  (i)  like similarly unenforceable anti competition clauses, the clauses seem improperly broad;  (ii) on first blush, the patient waivers appear to be potentially improper restraints on the exercise of first amendment rights;  (iii) the waivers are likely obtained under circumstances where patients will not understand the true nature of the restraints to which they are consenting; and (iv) contrary to the assertions of Medical Justice supporters, the restraint does not actually shift focus to the internet providers.  In fact, the waivers may pose far less danger to the review websites themselves, many of which may enjoy statutory protection from suit if they are merely acting as forums for information exchange and are not otherwise soliciting collection of improper information (see 47 U.S.C. § 230;  see also Nemet Chevrolet. Ltd. v. Consumeraffairs.com. Inc., 591 F.3d 250, 255 (4th Cir. 2009);  cf. Fair Housing Council v. Roommates.com, LLC, 521 F.3d 1157, 1174 (9th Cir.2008)).  In reality, Medical Justice’s waivers will likely acutely target individual patients, having the intuitively paradoxical result of holding an unsuspecting, lay patient liable while the website that published the review (and arguably multiplied any alleged damage to reputation) sits comfortably behind the protections of 47 U.S.C. § 230.

On the other end of the physician spectrum, the proposals are equally controversial though arguably several notches more credible.  In 2007, UCSF’s Bob Wachter, noted hospitalist (an understatement considering he is often credited with founding the hospitalist movement, assuming there was an actual “big bang” from which myriad highly skilled, white caped practitioners began roaming the wards, admitting patients, and improving the galaxy with quality control measures) proposed publishing physician board scores:

“Yes, you heard me right: I’d like the Board to tell me whether the doc was in 5th percentile on the certifying exam or the 87th. It doesn’t pass the smell test to say that we consider both these board certified docs to be undifferentiate-able. In this new era of transparency, if we physicians would want that information before choosing a doc for ourselves (and I sure would), then I believe that patients should have access to it as well.”

Click here for the full blog post.

I had coffee with Dr. Wachter (still can’t call him Bob…) a few months ago.  I have known him since attending medical school at UCSF (where he is Chief of Medical Services, Chief of the Hospitalist Service, and holds the Lynne and Marc Benioff Endowed Chair in Hospital Medicine). He mentored me on a first of its kind major policy commentary I published on Telemedicine that was published in the NEJM in 2008.  Suffice it to say, I have the utmost respect for him.  In person, over a cappuccino in the lovely UCSF café overlooking the park, he explained to me his rationale for the otherwise shocking board score proposal—that if physicians do not seize control of the review process, current online physician review sites will pervert the process and physician accountability will be driven by the random and sporadic nature of what is posted online by individual reviewers. The Tufts study and my own ad hoc, retrospective review of Stanford and UCSF providers seem to confirm Dr. Wachter’s concern, at the very least as a general principle.

I have no doubt that Dr. Wachter, as one of the fiercest, self-proclaimed iconoclasts of health reform, is acutely aware that his board score disclosure proposal poses a dangerous precedent.  As he is on the governing board of the ABIM, his views are certainly not to be underestimated or discounted. 

To that end, I offer my mentor and colleague a few thoughts in response.  First, licensing and/or certification scores are not a reasonable metric for evaluating physician performance or patient satisfaction over the long term.  As with any highly trained profession, much of what is learned after licensing and/or certification affects the quality of a practicing physician.  My own experience with my colleagues is that it is often the average scoring physician that takes the extra step to question what they are doing, think about whether the prescribed steps in a clinical algorithm make sense for a particular patient, and really understand the patient’s overall clinical picture.  While of course there are physicians with high licensing scores who are excellent physicians, there are as many with lower but passing scores (keeping in mind the internal medicine licensing exams are rigorous) who are excellent physicians as well.  The internal medicine licensing examination (which you need to practice) and/or board exam (which you do not need to practice as a physician) are both multiple choice tests;  medicine is not a multiple choice practice, and in fact when it is practiced in that manner, it often leads to unfortunate results. 

Harvard’s Bruce Landon observed (in a 2008 Archives editorial) that:  “Currently, there is little research relating performance on written cognitive examinations with actual clinical practice, so it is not clear to what extent performance on such examinations is reflective of individual physician practice. . .Nevertheless, the meaning of specific knowledge deficits on examinations is unclear, because, in practice, many physicians have available professional colleagues and consultants as well as additional information sources that they can use in real time to help them evaluate patients and formulate treatment plans. Therefore, knowledge deficits in the artificial setting of an examination do not imply that physicians will make mistakes in caring for patients.”  Landon’s full editorial was balanced and did not draw conclusions one way or the other;  it was written in response to the 2008 Holmboe et al. study published in the same issue.   

Second, and perhaps more importantly, disclosure of physician licensing and/or board scores would set a troubling precedent for physician empowerment and indeed seems to run directly contrary to the notion that such scores will somehow become the great leveler for physicians and avoid the irrational online forces of patient reviews and HealthGrades.  The reality is that physicians are arguably one of the most politically and economically disenfranchised groups in America.  The vast majority of physicians work an enormous number of hours and shoulder considerable personal and professional risk every hour of every day they work.  It is probably the best kept secret in the American, media-driven understanding of medicine (an understanding premised on Sanjay Gupta gallivanting the globe, House “bah humbugging” his way through the halls of a swank modern hospital,  and now The Doctors talk show…well…how do I say this politely…I’ll reserve comment) that modern day physicians are in truth in most instances meagerly compensated.  In fact, physicians are probably among the lower if not the lowest paid members of your skilled health provider team when objective criteria (length of training, nature of work performed and critical decisions made, hours worked, and risks assumed) are used to evaluate adequacy of compensation.  Disclosure of individual physician examination scores would be yet another disenfranchising force in the ongoing physician struggle for a more unified and effective lobbying voice and more competitive compensation.  Physicians would suddenly be stratified into high scorers and low scorers, and inevitably insurance companies, hospitals, and other members of the health community would use this information (most likely improperly without real correlation to the performance of the actual physician) to make compensation decisions, administrative decisions, and potentially, reimbursement decisions. 

Specific examination scores, as I view it, yield a noisy, unusable data set.  High scores can be as distorting as local neighborhood bias, and low scores as distorting as the lone gunman bias.  If licensing and/or board scores are to be used as any kind of equalizer for improving physician accountability, the task is to ensure that the vast majority of physicians employed maintain a required knowledge base after passing their applicable licensing exams, and to ensure that the applicable examination itself is a grueling and challenging exam.  Creating an additional mandate to disclose individual exam scores seems to be an inefficient allocation of resources and energy that could be better spent in tackling the more challenging issues of how to better empower and equip physicians in the age of accountability.

Posted in Health Policy | Tagged | 3 Comments

3 Responses to Local Neighborhoods, Lone Gunmen, And Exam Scores: Physician Empowerment In The Age of Accountability

  1. dr411 says:

    Thanks for the thoughtful post. I think that publishing board scores for patients to read can be a good thing. Patients have no sense of who is a ‘good doctor’ or not, and so are often stuck with a provider who doesn’t know the most up to date medicine. This is a quantifiable score that we can use to help get the best doctor for us. It makes sense doctors are scared of this, who would want to have a bad score published? But this is to help the patients not the doctors.

  2. sanjiv says:

    We received numerous e-mails in response to the post. The following quote is from an ABIM board member who e-mailed me in connection with the board score disclosure proposal:

    “No where near becoming ABIM policy….The default setting is to avoid this issue (too tough), and I’m guessing that is the wrong call. The Board has committed to a policy of increasing transparency when the science supports it. We’ll see…The lawyers watch over this stuff very carefully — for example, when the board required MOC, part of the rationale for grandfathering the older docs was that the lawyers told us that we had an implicit agreement with past diplomates that their boards were good for life. So any more to reveal more data, whether test scores or something else, would have to be handled very carefully.”

  3. V.Venkatesan says:

    As a patient I would like to know the Board scores of the doctors that I am vetting. Obviously I won’t use that as the sole criterion for picking my doctor. I would check with his/her current patients, references within the medical community and any other information I can get. Right now we are flying by the seat of our pants in picking a doctor. We need all the information we can get including board scores, any malpractice suits, disciplinary actions etc. It is too important a decision to make without all available information.

Leave a Reply