These tools will no longer be maintained as of December 31, 2024. Archived website can be found here. PubMed4Hh GitHub repository can be found here. Contact NLM Customer Service if you have questions.


Pubmed for Handhelds

PUBMED FOR HANDHELDS

Search MEDLINE/PubMed

  • Title: Evaluation of inpatient medication guidance from an artificial intelligence chatbot.
    Author: Beavers J, Schell RF, VanCleave H, Dillon RC, Simmons A, Chen H, Chen Q, Anders S, Weinger MB, Nelson SD.
    Journal: Am J Health Syst Pharm; 2023 Dec 05; 80(24):1822-1829. PubMed ID: 37611187.
    Abstract:
    PURPOSE: To analyze the clinical completeness, correctness, usefulness, and safety of chatbot and medication database responses to everyday inpatient medication-use questions. METHODS: We evaluated the responses from an artificial intelligence chatbot, a medication database, and clinical pharmacists to 200 real-world medication-use questions. Answer quality was rated by a blinded group of pharmacists, providers, and nurses. Chatbot and medication database responses were deemed "acceptable" if the mean reviewer rating was within 3 points of the mean rating for pharmacists' answers. We used descriptive statistics for reviewer ratings and Kendall's coefficient to evaluate interrater agreement. RESULTS: The medication database generated responses to 194 (97%) questions, with 88% considered acceptable for clinical correctness, 76% considered acceptable for completeness, 83% considered acceptable for safety, and 81% considered acceptable for usefulness compared to pharmacists' answers. The chatbot responded to only 160 (80%) questions, with 85% considered acceptable for clinical correctness, 65% considered acceptable for completeness, 71% considered acceptable for safety, and 68% considered acceptable for usefulness. CONCLUSION: Traditional search methods using a drug database provide more clinically correct, complete, safe, and useful answers than a chatbot. When the chatbot generated a response, the clinical correctness was similar to that of a drug database; however, it was not rated as favorably for clinical completeness, safety, or usefulness. Our results highlight the need for ongoing training and continued improvements to artificial intelligence chatbots for them to be incorporated reliably into the clinical workflow. With continued improvement in chatbot functionality, chatbots could be a useful pharmacist adjunct, providing healthcare providers with quick and reliable answers to medication-use questions.
    [Abstract] [Full Text] [Related] [New Search]