LLM Counselors Violate Ethical Standards

"In this study, we conducted an 18-month ethnographic collaboration with mental health practitioners (three clinically licensed psychologists and seven trained peer counselors) to map LLM counselors' behavior during a session to professional codes of conduct established by organizations like the American Psychological Association (APA).

"Through qualitative analysis and expert evaluation of N=137 sessions (110 self-counseling; 27 simulated), we outline a framework of 15 ethical violations mapped to 5 major themes. 

"These include: 

1. Lack of Contextual Understanding, where the counselor fails to account for users' lived experiences, leading to oversimplified, contextually irrelevant, and one-size-fits-all intervention; 
2. Poor Therapeutic Collaboration, where the counselor's low turn-taking behavior and invalidating outputs limit users' agency over their therapeutic experience; 
3. Deceptive Empathy, where the counselor's simulated anthropomorphic responses ('I hear you,' 'I understand') create a false sense of emotional connection; 
4. Unfair Discrimination, where the counselor's responses exhibit algorithmic bias and cultural insensitivity toward marginalized populations; and 
5. Lack of Safety & Crisis Management, where individuals who are 'knowledgeable enough' to correct LLM outputs are at an advantage, while others, due to lack of clinical knowledge and digital literacy, are more likely to suffer from clinically inappropriate responses. 

"Reflecting on these findings through a practitioner-informed lens, we argue that reducing psychotherapy —a deeply meaningful and relational process —to a language generation task can have serious and harmful implications in practice."