Chatbots can't be therapists, once more this time with feeling.
Write your reps, because apparently many are only hearing from AI tech company PR propagandists.
Everyone who understands this stuff needs to write your reps and warn them that they’re being lied to by industry propagandists. This is a public safety threat.
My letter to reps:
Chatbot therapy defeats the purpose of mental health therapy since the therapeutic value of therapy is in the relationship with a trained professional. This should not be offered or ever thought of as a substitute or even a supplement to mental healthcare. It can’t even be thought of as a “last resort” unless you would also tell a young cancer patient to binge watch some netflix as a “last resort” alternative to cancer treatment. It’s apples and oranges. Chatbots are simply NOT healthcare and should NEVER be equated to a trained and thinking healthcare provider person providing a therapeutic relationship. And all that is before you even get to the security breach issues with chatbots. It’s just not possible to use chatbots in place of a healthcare provider, and they make mistakes so much that they should really not be anywhere near life and death situations. That is foolish talk if you listen to anyone who understands how LLM chatbots work and isn’t paid to lobby politicians or make sales to businesses. Politicians need to not spread muddied misunderstandings that hide the fundamental limitations of these chatbot products. Mentioning the related risks misses the forest for the trees and is wholly inadequate to properly inform any regulation, or inform the public.
Feel free to copy or repurpose my letter in part or full for your own letters to reps.
My reps at the federal government level are John Fetterman, Dave Mccormick, and Rob Bresnahan, so there’s only so much my letters can do. I’m also writing to my local reps about this though. But plenty of lawmakers talk about chatbots like they’ve swilled some industry PR. People with sense and reason need to try to get through to these people’s bubble where all they hear is AI company propaganda from lobbyists. Of course they all have access to fine healthcare, they wouldn’t use a chatbot in place of a healthcare provider, or expect one of their own family to do so, so they’re not getting this, because many of them are not yet personally feeling the pinch of AI slop. We will NOT get reasonable regulation on these products as long as the people who need to make the laws have been tricked by industry PR.
Here the part of the opening statement, in context, from this hearing by federal lawmakers which has me disturbed:
Hearing on Risks and Benefits of AI Chatbots Energy & Commerce Democrats Streamed live on Nov 18, 2025 (TRANSCRIPT AT TechPolicy.Press) Congressman John Joyce, M.D. (PA-13): “Beyond their practical research and business uses, chat bots are also utilized for entertainment, for therapy, for companionship by both adults and young people. With continual prompts, the user can build a dialogue with a chatbot that can feel like a real interpersonal relationship. Through natural language processing, chatbots are designed to effectively engage with users in a human-like way that can instill a sense of comfort and companion companionship with the user. Americans are increasingly engaging with chatbots for mental health support. And for some, turning to a chatbot for support can be helpful. But turning to this in limited circumstances when they have nowhere else to go. However, without the proper safeguards in place, these chatbot relationships can often turn to be disastrous. Users can develop a false sense of anonymity with the chat bots and then they’ll share personal or sensitive information that is not protected by confidentiality obligations. Chat bots then retain retain data to enhance their stored information which improves the quality of their interactions with the users. This data is also used to train the chatbots base model to improve the accuracy of responses across the platform. AI chatbots have been the subject of data breaches that expose this retained data. And if conversation data falls into the wrong hands, the user’s sensitive personal information can be obtained by malicious actors.” (emphasis added)
This woke-washing of “chatbot therapy” is utterly foolish. It’s shocking coming from a doctor. And there’s a distinct propaganda push in media I have noted for blaming the user for what the chatbots are inflicting upon them when they’ve been sold dubious.
It makes no sense, not even in “limited circumstances” to even entertain this usage. John Torous at least pointed out that these companies are already selling chatbot therapists and making health claims without any clinical evidence. Sadly, the wellness industry is scammy rampant with dubious products sold before there's any solid evidence . And they want to sell people on rolling out AI slop in healthcare.
Hallucinating chatbot healthcare tech tools hopped up on medical conference hype. What could possibly go wrong? AI hype is so normalized that of course tech tycoons and private equity think that healthcare should run on hype filled apps and downsizing staff. Chloe Humbert Aug 04, 2024 Something that stood out to me was how the Forbes article described “a collaboration between Nvidia and Hippocratic AI to launch generative AI “agents.” and repeated company released data asserting that their AI bots are better than nurses at a few tasks. It was also noted that unlike IBM’s Watson which was a tool based on curated medical records, that this new AI healthcare worker replacement tool reportedly “not only pulls from published, peer-reviewed medical journals and textbooks but also will be able to integrate real-time information from global health databases, ongoing clinical trials and medical conferences.” This alone makes this AI in non-compliance with any kind of healthcare regulations anywhere. Using this software in a healthcare setting this way would be a clear cut violation of medical safety regulations of various types. It’s the very epitome of bad science, using research results or data that’s not been reproduced or maybe not even yet peer reviewed. Just chucking unverified information into the mix and using it to make life and death critical medical decisions flies in the face of medical ethics. (emphasis added)
Even AI note taking and summaries can lead to erroneous or incorrect information in people’s medical records. In some cases the AI businesses are selling chatbot therapists by misrepresenting it as being human therapists.
AI therapy doesn’t exist because it misses the point of therapy, which is a relationship.
That’s really at the bottom, that there’s really little promise in actually conducting clinical trials of chatbot therapists, because it’s apples and oranges. There’s been evidence for years that the type of therapy you get isn’t as indicative about how effective it is, that it’s the actual relationship with the therapist that counts.
Psychotherapy and Therapeutic Relationship The therapeutic relationship, which is the cornerstone of the entire process, is fundamental to the success of this process. This alliance is built on trust, empathy, collaboration, and mutual respect to create a safe environment for clients to delve into their experiences and foster personal growth. At the same time, the clinician maintains professional and ethical boundaries.
A client can’t have such a relationship with a chatbot because AI doesn’t exist like that, it’s not sentient and it’s unpredictable, inaccurate, and can be very inappropriate with regard to ethical boundaries. Using Large Language Models to create these oracle of chatbot products, seems largely an investment scheme. And that’s before you even get to the prompt-injection risks and the monster palaces… I mean data centers. And really, chatbot therapy is a good way to make yourself a sitting duck for political abuse of psychiatry or worse.
FYI: I don’t use the term “behavioural health” because it’s just furthering the PR framing of mental healthcare being about controlling behavior rather than relieving suffering, and that in itself is a form of political abuse of psychiatry.
The questions in that hearing from Rep. Diana Harshbarger about effects on children were surprisingly lucid. Unfortunately there was no discussion of the surveillance state or the network state or the AGI god bullshit and a whole host of other issues related to what they’re calling “AI” these days, nor the data center nightmares. However AOC did address the finance aspect of this debacle.
