As a doctor bombarded almost daily with advertisements and journal publications raving about artificial intelligence, I’ve often wondered how much of it is actually true. As a challenge, I posed two straightforward, common questions to two major AI sources designed for doctors. The first was Dr. Oracle, sponsored by the massive data collection and analytics company Oracle. The second was Open Evidence, sponsored by the Mayo Clinic and closely tied to the New England Journal of Medicine.

My first question: When are SNRIs (antidepressants like Effexor and Cymbalta) better to use in depression than SSRIs (Prozac, Celexa, Zoloft)? The answers from the two AI services were surprisingly different. Dr. Oracle could only offer one solid reason and one “maybe” for choosing an SNRI. In addition, its primary reference was 17 years old! Open Evidence, on the other hand, provided five valid scenarios for using an SNRI. An added bonus: its main reference was much more recent — only six years old.

My second question: When is mastectomy advised for breast cancer? It seemed like a reasonable question, given that breast cancer is the most common cancer in women, affecting one in eight. Fortunately, both AI platforms basically agreed on the appropriate indications. However, Open Evidence’s references were seven years newer. Also, on my first attempt with Dr. Oracle, its answer was incorrect and based on a 24-year-old reference! Sensing something was off, I asked the question again — and it got it right the second time.

My final impression of medical AI available to healthcare professionals is that while it’s decent at providing references (albeit sometimes outdated), it’s only accurate about 50% of the time — and that’s after checking a second source and rephrasing the question. To complicate matters, AI tools available to the general public, such as ChatGPT, tend to be even less accurate. My advice to patients seeking answers to their health concerns: use AI as a very basic guideline, but remember it is not definitive. Your doctor is your best source — they are the only ones equipped to interpret the often ambiguous answers.

Dr. Charlie Barnett is a contributor at KnoxTNToday for a weekly column, DocTalk, providing his expertise on health and wellness management.