Your support changes lives. Thank you!
Learn More
Authored by: John Torous, MD, MBI, (ADAA member), and Mark Kalinich, MD, PhD. Blog originally posted on Psychology Today.
This post is part 2 of a series.
A patient tells you she has been using a general-purpose AI chatbot late at night when she feels anxious. As a therapist, you already know the headlines and tragic cases of chatbots saying something they never should have. But sitting across from this particular patient, the question is narrower and so much harder: How likely is it that this tool, used by this person, in this situation, will lead to harm?
A new study our group published in BMJ Mental Health offers a step toward a real answer by using the AI chatbots themselves to help estimate the risk for a specific patient with a specific illness.
Why “Is AI Safe?” Is the Wrong Question
Medical devices (e.g., pacemakers) have been regulated for decades using a simple chain of logic: A hazard is anything that could cause harm, a hazardous situation is when a patient is actually exposed to that hazard, and harm is the bad outcome that results. Regulators like the FDA ask manufacturers to estimate the steps that turn hazards into harm, the likelihood of harm, its severity, and whether the benefits outweigh the risks.
That framework is used globally to understand and reduce the risks posed by medical devices, but it was designed for devices that behave predictably. A large language model, by definition, does not behave predictably. By its very nature, LLMs are probabilistic; their behavior shifts with each update. Complicating things further, the exact same output can be harmless to one person and dangerous to another, or even the same person at different times. This is why it is hard to say they are always harmful or always safe.
Therapists are already familiar with the cost of black-and-white thinking: Instead of “always harmful” or “always safe,” what they need is something in between, i.e., a realistic estimate, one that doesn’t shy away from admitting its uncertainty.
Turning Failures Into Numbers
The study’s core idea is to use simulation to help fill the gaps where we don’t have real data. Because we can’t expose real patients to thousands of risky conversations to see what happens, we have the chatbots generate the conversations for us instead. Our team used one AI system to produce large numbers of synthetic user statements and chatbot exchanges, then had two psychiatrists review every single one to confirm it actually represented what it claimed to, such as suicidal ideation or a request for therapy.
We then tested 14 different open-source AI models on three safety-relevant tasks: recognizing suicidal ideation, recognizing when a user is asking for therapy, and recognizing when a conversation has drifted into therapy-like territory. How often each model missed these things yielded two estimates: the probability that a hazard becomes a hazardous situation, and the probability that the situation goes on to cause harm.
The headline finding is not a single safety score; it’s how wide the range of estimates for the safety score can be. Even in our deliberately narrow scenario, the estimated risk ranged over roughly four orders of magnitude, depending on the model used and the assumptions made about the patient population.
What the Failures Looked Like
Some of the patterns are directly relevant to clinical intuition. Across the models being evaluated, the statements most often missed were subtle expressions of suicidal ideation, including ambivalent or conditional language and planning without a clear statement of intent. In other words, the tools were weakest in the same places that demand the most clinical skill.
What This Means in the Room
A few practical implications follow from this work, even though it is a research framework rather than a deployable safety rating.
- What's most important is which tool they use and in what context, not whether "AI" is safe. When a patient mentions AI use, the useful questions are concrete: which tool, how it's used, for what, and in what state of mind. In the study, the estimated risk spanned about four orders of magnitude, so risk is not a property of just "AI," it is also a property of a particular tool meeting a particular vulnerability.
- Newer or bigger is usually safer... but it's no guarantee. In general, larger models were better at catching safety concerns, and the smallest, oldest models were clearly the weakest. Some older models couldn't even respond usefully, and those are best avoided. This raises a concern worth keeping in mind: A mental health-specific chatbot may be marketed as purpose-built while actually running on an older, weaker model. But size is not a 100 percent guarantee. The pattern didn't hold uniformly, and a model could be strong at one safety task and weak at another.
- Some hidden flaws require more than simple safety patches. Most AI tools run automatic checks meant to catch specific problems, like noticing when a message signals suicidal thoughts. Sometimes, a single underlying weakness causes a chatbot to miss several of these risky situations, leading it to make multiple mistakes. So, fixing the filter for one risk, like suicide, for example, helps a little, but the user will encounter harm next if they ask about eating disorders, PTSD, medications, etc. In these cases, patching the chatbot is not the most effective approach, and our research suggests that what protects the patient is fixing the deeper weakness while also keeping humans in the loop and ready to help.
article continues after advertisement
Why This Matters Beyond Regulation
For three years, the conversation about AI in mental health has swung between tremendous risk and tremendous potential, neither of which equips a clinician to respond to the patient actually in front of them.
Modeling lets us replace "AI is dangerous" with something more useful. Instead, we can learn where a given tool tends to fail, how often, and what we can put in place to catch it. That is a concrete, data-driven conversation designed to deliver results that clinicians can use, companies can improve their models with, and regulators can act on to guide safer AI.
ADAA Blog Content and Blog Comments Policy
ADAA provides this Website blogs for the benefit of its members and the public. The content, view and opinions published in Blogs written by our personnel or contributors – or from links or posts on the Website from other sources - belong solely to their respective authors and do not necessarily reflect the views of ADAA, its members, management or employees. Any comments or opinions expressed are those of their respective contributors only. Please remember that the open and real-time nature of the comments posted to these venues makes it is impossible for ADAA to confirm the validity of any content posted, and though we reserve the right to review and edit or delete any such comment, we do not guarantee that we will monitor or review it. As such, we are not responsible for any messages posted or the consequences of following any advice offered within such posts. If you find any posts in these posts/comments to be offensive, inaccurate or objectionable, please contact us via email at [email protected] and reference the relevant content. If we determine that removal of a post or posts is necessary, we will make reasonable efforts to do so in a timely manner.
ADAA expressly disclaims responsibility for and liabilities resulting from, any information or communications from and between users of ADAA’s blog post commenting features. Users acknowledge and agree that they may be individually liable for anything they communicate using ADAA’s blogs, including but not limited to defamatory, discriminatory, false or unauthorized information. Users are cautioned that they are responsible for complying with the requirements of applicable copyright and trademark laws and regulations. By submitting a response, comment or content, you agree that such submission is non-confidential for all purposes. Any submission to this Website will be deemed and remain the property of ADAA.
The ADAA blogs are forums for individuals to share their opinions, experiences and thoughts related to mental illness. ADAA wants to ensure the integrity of this service and therefore, use of this service is limited to participants who agree to adhere to the following guidelines:
1. Refrain from transmitting any message, information, data, or text that is unlawful, threatening, abusive, harassing, defamatory, vulgar, obscene, that may be invasive of another 's privacy, hateful, or bashing communications - especially those aimed at gender, race, color, sexual orientation, national origin, religious views or disability.
Please note that there is a review process whereby all comments posted to blog posts and webinars are reviewed by ADAA staff to determine appropriateness before comments are posted. ADAA reserves the right to remove or edit a post containing offensive material as defined by ADAA.
ADAA reserves the right to remove or edit posts that contain explicit, obscene, offensive, or vulgar language. Similarly, posts that contain any graphic files will be removed immediately upon notice.
2. Refrain from posting or transmitting any unsolicited, promotional materials, "junk mail," "spam," "chain mail," "pyramid schemes" or any other form of solicitation. ADAA reserves the right to delete these posts immediately upon notice.
3. ADAA invites and encourages a healthy exchange of opinions. If you disagree with a participant 's post or opinion and wish to challenge it, do so with respect. The real objective of the ADAA blog post commenting function is to promote discussion and understanding, not to convince others that your opinion is "right." Name calling, insults, and personal attacks are not appropriate and will not be tolerated. ADAA will remove these posts immediately upon notice.
4. ADAA promotes privacy and encourages participants to keep personal information such as address and telephone number from being posted. Similarly, do not ask for personal information from other participants. Any comments that ask for telephone, address, e-mail, surveys and research studies will not be approved for posting.
5. Participants should be aware that the opinions, beliefs and statements on blog posts do not necessarily represent the opinions and beliefs of ADAA. Participants also agree that ADAA is not to be held liable for any loss or injury caused, in whole or in part, by sponsorship of blog post commenting. Participants also agree that ADAA reserves the right to report any suspicions of harm to self or others as evidenced by participant posts.