Impressive AI text generator still has notable limitations

At the end of November, a new artificial intelligence chatbot was released for public use. Just 2 weeks later, Clifford Stermer, MD, found an intriguing new use for the technology in his rheumatology practice, which he shared on TikTok.

In the videoopens in a new tab or window, Stermer typed a prompt into ChatGPT, developed by OpenAI, requesting that the chatbot write a letter to a medical insurance company to explain why a patient with systemic sclerosis should be approved for an echocardiogram. Seconds later, on camera, the program started writing a full letter, complete with appropriate heading and formatting.

Stermer, who owns One Rheumatology in Palm Beach Gardens, Florida, narrated the writing of the letter during the TikTok video. He highlighted the most impressive elements, such as the explanation of the treatment and use of references. He concluded the video by saying, “Amazing stuff. Use this in your daily practice, okay. It will save time. It will save effort.”

That video went viral almost immediately, he said, and now has more than 130,000 views. It was widely shared on other social media sites as well, including Twitter, where it sparked a wide-ranging conversationopens in a new tab or window about other uses of this tool in healthcare.

Beyond the ChatGPT Hype

Since ChatGPT was released on November 30, it has become one of the internet’s most referenced and tested programs in recent weeks.

“I was talking to some other doctors about it online, and there was a question about how can we use this to our advantage,” Stermer told MedPage Today. He said that insurance denial letters were one of the first things that came to his mind when he learned about the program. “This seems like one of the tasks that we need to do regularly that it may be able to help with.”

According to the OpenAI websiteopens in a new tab or window, ChatGPT was created using a language model trained to produce text, known as GPT-3.5. And it was designed to create texts, like insurance denial letters, using a method that models previously demonstrated texts created by humans “to guide the model toward desired behavior.”

After the video took off, Stermer realized that the program had more limitations than he first thought. For one, he said, the references in the letter were not realopens in a new tab or window. ChatGPT made them up.

In fact, Stermer has found that much of the information produced by the program has been inaccurate or only partially correct.

“The sources were kind of half right,” he said. “The authors were right, and then some of the articles were right, but it didn’t piece it together exactly. So it’s not something you could just send off to an insurance company, or really use at this point.”

“It wrote a beautiful text, and it kind of got the points across,” he said. “But it was still not where it needs to be.”

OpenAI acknowledges that ChatGPT is not connected to the internet, so it can produce incorrect answers. In fact, ChatGPT has limited knowledge of “events after 2021 and may also occasionally produce harmful instructions or biased content,” according to the organization’s website.

A Future of Possibilities

However, ChatGPT still holds some unique potential in easing a physician’s daily tasks, according to urologic oncologist Anobel Odisho, MD, MPH, of University of California San Francisco.

In particular, Odisho noted that repetitive writing is a big part of his job as a clinician, a practice manager, and a researcher.

“In each of those roles, we do some repetitive, low-value writing and administrative work which I think we can automate, either with ChatGPT or other tools,” Odisho told MedPage Today.

Odisho said he has used the program to generate first drafts of letters that he then reviewed and edited. He also used it to generate more patient-friendly descriptions of procedures or post-procedure instructions. He even used it to create a draft of an on-call schedule.

“I think the possible use cases are only limited by your creativity and ability to write a good chat ‘prompt,'” Odisho said.

But he also acknowledged that this technology has serious limitations as well. He emphasized that healthcare professionals should be careful about what information they share with this program, especially when it comes to patients.

“Do not trust it, it is not ready for primetime,” Odisho warned.

Ultimately, Odisho said ChatGPT is just the beginning of this kind of technology in medicine. His primary takeaway is that this is the future, and he believes all healthcare professionals should start getting comfortable with it.

“I think in the future you really will have two classes, one group that can seamlessly and fluidly utilize these tools effectively in their personal and work lives, and another that cannot, and will struggle to keep pace,” Odisho said.

Stermer agreed that ChatGPT is a preview of things to come, but right now he acknowledged it is not as useful for his practice as he first hoped.

“It’s more a look into the future of the possibilities, or how things are going to be done,” he said “But it does not replace medical expertise.”