Introduction of ChatGPT into Ophthalmology Grand Rounds at Bascom Palmer Eye Institute
Last month we celebrated Thank a Resident Day. As Bascom Palmer’s residency program director, I tried to think of ways to show our trainees my appreciation. It dawned on me that Grand Rounds would be a great venue to share a supportive message in front of our live and virtual global audience of approximately 400 ophthalmologists.
Our Grand Rounds are fairly traditional. Three house staff present cases without revealing the diagnosis, at least initially. The junior residents in attendance are called upon at random to describe the clinical findings in the photos shared by the presenters, offer a differential diagnosis, and propose next steps in evaluation. On this particular day, I decided to further show my appreciation to the residents by letting them off the hook. ChatGPT was invited to fill in for them.
ChatGPT (OpenAI) is an artificial intelligence (AI)-powered chatbot created to mimic human conversation. Recently, the integration of chatbots in medical education has gained popularity, with their potential to provide immediate feedback, enhance clinical reasoning, and provide a novel way to approach complex medical cases. Dr. Eduardo Alfonso, our department chair, was the brainchild behind this experiment and suggested trialing it at Grand Rounds. We were all curious to know how ChatGPT would perform under the intense pressure to deliver high level accurate information to our astute audience.
The Bascom Palmer Experience
Context is important to maintain the state of a conversation with a chatbot, so our first step was to “train” our chatbot. Prior to Grand Rounds, we built an appropriate prompt to help generate a response at the appropriate level for the audience we were addressing. We toyed with variations of the prompt in order to establish the best method for getting results displayed as we wanted it, using a display format that could easily be presented during the talk. Ultimately, we engineered a primer to establish the appropriate role-playing scenario for ChatGPT to assume as it generated standardized responses.
Case after case, ChatGPT was asked to generate a differential diagnosis and appropriate testing based on clinical findings identified by the presenters. The responses generated by ChatGPT were then displayed in real time for all attendees to see and comment on. In one case we added additional context, including the patient’s geographic location when symptoms began, to see if it could improve the response (it did!).
Responses were robust and succinct in an organized table format. Additionally, the information was largely in agreement with the information provided by the case presenters. ChatGPT was able to learn exactly what level of information we desired to see and in what way. This ability to learn as the conversation proceeds separates language models from other sources of information, like an internet search engine, but there are limitations.
Potential Limitations in Medicine
Though ChatGPT was “trained” via a variety of sources on the internet, it was not intended to relay factual information. The chatbot has not been updated on information beyond 2021, and the chatbot does not access a database of facts to draw from to provide medical information. The user, without knowing the exact sources of the chatbot’s information, must fact check its responses, particularly when the stakes are high (e.g., involving medical decision making).
Even more concerning is artificial intelligence’s ability to deliver confident responses that are not grounded in or justified by its training data, called an artificial hallucination. The chatbot can wax prolifically providing a convincing argument to the user that supports its hallucination (even using fake sources). This further highlights medical professionals’ critical role in the clinical decision-making process and the importance of responsible use of technology in medical education and clinical practice. To mitigate the limitations described above, we instructed our chatbot to emphasize the importance of verification of its answers by a medical professional through a disclaimer at the end of its responses during Grand Rounds.
One potential solution to address the limitations related to up-to-date and accurate information is to control the information from which the chatbot draws. This would require ongoing monitoring and curation to ensure the accuracy and reliability of the information provided by ChatGPT.
Chatbots’ Future in Medical Education, Clinical Practice
Moving forward, we are excited about the potential of chatbots to revolutionize medical education and clinical practice. As technology continues to advance, the integration of chatbots into clinical workflows may become commonplace.
We look forward to further exploring the capabilities of ChatGPT in the context of ophthalmology and other medical specialties, and its potential to enhance patient care.
Imagine what would happen if one were to integrate an AI program that accurately interprets clinical findings on ophthalmologic exam and then feeds those findings to a chatbot that draws on curated, well-vetted sources to deliver a differential diagnoses and a diagnostic/treatment plan. That could be a powerful clinical tool in the physician toolbox.
In conclusion, the integration of chatbots in medical education is an innovative way to provide immediate feedback, enhance clinical reasoning, and provide a novel way to approach complex medical cases. Our limited experience demonstrates the potential of ChatGPT in the context of ophthalmology grand rounds, and we look forward to further exploring its capabilities in medical education and clinical practice.
If anything, it did a great job subbing in for our residents during Grand Rounds and gave our trainees a much-deserved break during Thank a Resident week.
Chris R. Alabiad, MD, is a professor of clinical ophthalmology and ophthalmology residency program director at Bascom Palmer Eye Institute, ranked for more than two decades by U.S. News & World Report’s Best Hospitals as the #1 eye hospital in the country. Special thanks to Dr. Giselle Ricur, Joshua Reyes, and David Taylor Gonzalez for their contribution to this piece.