Can an AI therapy app truly match a human therapist?

Feb 24

A fascinating new study published in PLOS Mental Health has sparked a lot of interest lately, with some media outlets proclaiming things like “A New Study Says ChatGPT Is A Better Therapist Than Humans”. The research paper, titled "When ELIZA meets therapists: A Turing test for the heart and mind", examines how responses from ChatGPT compare to those from professional therapists. Let's take a look at what the study actually says, and what it means means for the future of AI in mental health support and in particular for a mental health app such as Wellness AI.

What the Study Found

The researchers conducted what's essentially a modern version of the Turing test – comparing AI-generated therapy responses to those written by human professionals.

Here's what they discovered:

Participants had significant difficulty telling the difference between responses written by professional therapists and those generated by ChatGPT. Their accuracy was only marginally better than chance.
Perhaps more surprisingly, ChatGPT's responses were rated higher on the five "common factors" of therapy – key principles like empathy, professionalism. and therapeutic technique. This was in spite of an observed bias against AI; participants rated ChatGPT lower if they suspected the response was AI-generated.

One thing that makes this study compelling is that the therapists were fully aware they were competing against ChatGPT. These weren't random clinicians caught off guard – they knew exactly what they were up against.

Important Limitations to Consider

The PLOS Mental Health article provides evidence in favor of AI’s ability to provide people with valuable mental health support. But before we declare AI the new gold standard in therapy, it's crucial to understand the study's limitations, which the authors themselves acknowledge:

The study analyzed responses to written vignettes about couples therapy scenarios, not actual back-and-forth therapy sessions. Real therapy involves complex conversation dynamics that unfold over time.
The evaluation focused specifically on five "common factors" of therapy that were explicitly emphasized in the prompt given to ChatGPT. This demonstrates AI's general domain knowledge and its ability to apply instructions to new scenarios, but may not reflect the full scope of what makes therapy effective.
This study did not actually measure human preferences to the responses, just their ratings on “common factors”; so we don’t know if ChatGPT responses would be scored higher by a client. There was also no attempt to measure how much benefit a client would objectively receive from different responses.

Note that none of this is intended as a critique of the authors of this paper—I actually think this was a very well conducted study. This is just to point out the limitations of the study, many of which the authors themselves acknowledged.

Stepping back: the ideal role of AI in mental health support today

As someone deeply involved in developing AI mental health solutions, I expect to see more findings along these lines moving forwards as AI technology continues to advance. Here's my general perspective on what this means right now in terms of getting mental health support:

Strengths of AI Therapy Tools

24/7 availability: Human therapists need rest; AI doesn't. Support is available whenever someone needs it – at 2 AM during a panic attack or on holidays when traditional services are unavailable.
Affordability: In many parts of the United States, a year's subscription to an AI therapy app costs less than a single therapy session with a licensed professional.
Consistency: AI systems can reliably produce appropriate therapeutic responses based on their training data and specific programming approaches.
Novel combinations of support: AI mental health apps have the ability to do things that can’t be achieved in traditional settings. At Wellness AI, we've combined AI therapy with personalized guided AI meditations in a single app – creating a multimodal experience that's impossible to replicate in traditional therapy settings. Not only are therapists unavailable 24/7 like Wellness AI, but most therapists aren't also trained meditation guides, able to craft personalized meditations about any topic someone wants.

Current Limitations of AI Therapy

Crisis detection: AI can sometimes miss subtle signs of serious issues. That's why responsible AI therapy apps must include crisis detection mechanisms and clear pathways to human support when needed.
Complex clinical presentations: AI isn't equipped to diagnose or treat specific mental health conditions, and shouldn't be positioned as a replacement for professional care in cases requiring clinical intervention.
Memory and contextualization: While some apps (including Wellness AI) build persistent memory about users over time, AI generally doesn't always match a human therapist's ability to remember and contextualize information across multiple sessions.

Differentiation of Mental Health Support, Not Replacement

Rather than trying to replicate traditional therapy exactly, I believe the most promising approach for AI mental health tools at this point is to offer differentiated experiences that leverage AI's unique capabilities.

That's precisely what we've focused on with Wellness AI. By combining AI counseling with custom AI meditations, we've created an experience that isn't just imitating what human therapists do – it's creating something new that offers unique benefits to users that could not be replicated in traditional mental health settings.

The technology is advancing rapidly, and studies like this one suggest AI can already provide meaningful therapeutic value. But I see AI therapy as complementary to, rather than replacing, traditional mental health care. Each has its strengths, limitations, and ideal use cases. Ideally, for those who can afford things like therapy, AI mental health apps would be a supplement to traditional therapy sessions.

What excites me most is the opportunity to expand access to mental health support, and the notion of what types of wellness support are possible. With so many people unable to access traditional therapy due to cost, availability, or other barriers, AI tools can help bridge that gap – providing evidence-informed support to people who might otherwise receive none at all.

-Tim

Founder & Developer, Wellness AI