How Valid Is "Anecdotal" Information about Clinical Information Systems?
Posted Jan 11 2011 12:00am
From Stan: “Re: article in Applied Clinical Informatics on why people discount personal experiences with HIT. I think your readers might find the free download worth reading. Best regards from Switzerland.” Frrom Mr. HIStalk: The author, Jon D. Patrick from the University of Sydney, Australia, took heat for publishing user reports of ED system problems. His editorial says the problem reports of experienced system users are dismissed as unscientific anecdotes to protect IT interests, the organization’s investment, or its executives from criticism instead of treating those reports as an early warning system. While I don’t buy the idea that user IT perceptions should always be taken at face value, he’s right about the weird dynamic: the IT department and all the suits who signed off on the deal shoot the messenger because they are emotionally invested in it. They honestly believe that complaining users are troublemakers or fools who aren’t blessed with their big-picture vision (specific, serious IT problems are often dismissed on the basis of the greater good, of course). That’s like blaming a patient for daring to develop a post-surgical infection since it makes the surgeons look bad (which wouldn’t surprise me either).
There is no question that anecdotal evidence can be flawed, in part for the reasons described above. Those individuals intimately involved in the selection of the system tend to be "emotionally invested" in it and are often loathe to criticise it. Moreover, even if they have misgivings that they will share with you, they may be afraid that any criticisms of it may filter back to the vendor and they may be penalized. Vendor contracts often prohibit public criticism of the system on the part of hospital users.
The term anecdotal evidence is often viewed as an oxymoron in the sense that personal accounts can be biased and inconclusive. For me, it's more a case of exactly whom I am talking to. If it's someone I have known for years, I will place a high degree of confidence in a telephone discussion about the positive and negative aspects of a system. I would never trust the account of an LIS if the person queried does not work with it directly every day. Paradoxically, the higher up the position on the hospital totem pole, the less reliable their account of a system may be. That's because the hospital "suits" are less apt to use the system every day. The same rule applies when you are making a lab site visit about a particular LIS. Request permission to visit the lab during the evening or night shift. The medical technologists working such shifts are usually not an integral part of the leadership team and will often be more forthright about their evaluation of the system.
Also keep in mind that there are various dimensions of a "lousy" LIS such as software bugs, system design, system functionality, user support and training, system response time, and vendor financial viability. Some of these problems can be addressed by contractual or hardware issues. It takes a sustained conversation to differentiate between all of these issues.