In a recently published paper, researchers describe how ChatGPT came close to passing the notoriously rigorous U.S. Medical Licensing Exam (USMLE).
"ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement," researchers wrote in their published paper. "Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations."
The USMLE is actually three tests in one, and ChatGPT's ability to answer its questions demonstrates that these AI bots may one day be valuable for medical training and perhaps making certain diagnoses.
Researchers from the startup Ansible Health evaluated ChatGPT using USMLE example questions for which the answers were not available on Google; hence, they knew that ChatGPT would generate unique solutions based on the data it had been trained on.
Over the course of the three exams, ChatGPT scored between 52.4% and 75%. (the pass mark is usually around 60%). In 88.9% of its responses, it generated at least one major discovery, defined by the researchers as something "new, non-obvious, and clinically valid"
"Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation," the authors of the study noted in a press release.
ChatGPT also showed remarkable consistency in its responses and was able to provide explanations for each one. It also surpassed the 50.3% accuracy of PubMedGPT, a bot specifically trained on medical literature.
ChatGPT is a large language model, or LLM, a type of artificial intelligence. These LLMs are specifically designed for written responses, and with massive quantities of sample material and advanced algorithms, they can predict which words should fit together in a sentence, much like the predictive text mode on your smartphone.
ChatGPT does not actually "know" anything, but by evaluating a vast amount of web content, it is able to build plausible-sounding phrases on any topic.
It's important to remember that the data used to train ChatGPT will contain errors; if you ask the bot, it will admit that additional effort is required to enhance the accuracy of LLMs. In the foreseeable future, this technology will not replace medical practitioners.
Nonetheless, the potential for processing online knowledge is definitely enormous, particularly if these AI bots continue to improve over the years. Instead of replacing humans in the medical field, robots could serve as indispensable assistants.
The findings were published in PLOS Digital Health.
