What Are the Most Important Issues with AI Companions? Six Key Themes Emerged from Our Community.

By David Ryan Polgar and Rose Guingrich

August 19, 2025

AI companions are a hotly debated topic in 2025, and an area where our organization has felt well-positioned to help surface its values, tensions, and tradeoffs. Unpacking the multitude of issues related to AI companions (technical, ethical, legal, psychological, sociological) is an ongoing initiative for All Tech Is Human, This piece builds from earlier roundtable discussions, livestreams, panel discussions, our mini-guide on AI Companions and Chatbots, the research of Rose Guingrich (Princeton University GradFUTURES Social Impact Fellow with All Tech Is Human), and feedback from the community.

Leveraging All Tech Is Human’s broad global network, which weaves together individuals across civil society, government, industry, and academia, we recently collected observations from one central question:  In your opinion, what is the most important issue when it comes to creating or interacting with AI companions?

What Are AI Companions?

AI companions are generative artificial intelligence chatbots designed specifically for companionship. Popular companion chatbots, such as XiaoIce and Replika (“The AI companion who cares”), have an estimated 1 billion active users worldwide as of 2024. People also report using general-purpose generative AI chatbots such as ChatGPT, Gemini, and Claude as social companions. People use AI companions for diverse reasons, including entertainment and social needs. Users interact with these chatbots via text or other modalities and have reported developing friendships, mentorships, or romantic partnerships with them.

Analyzing the comments from 150 submissions, six key themes emerged:

• Emotional and Psychological Impact

• Human Relationships and Social Skills

• Privacy and Data Security

• Safety and User Vulnerability

• Credibility, Trust, and Transparency

• Ethical and Business Model Conflicts

Research findings* suggest that the use of companion AI can both benefit and harm human relationships and well-being, a sentiment also reflected in our responses. “Understanding that AI companions, like any technology, are pharmacological–that is, they are simultaneously a remedy and a poison–and that it is what we do with them, and how, that matters,” states Mark Howard, a Researcher and Lecturer at the University of California, Santa Cruz.

Chatbot users who experience social anxiety, social isolation, or trauma from past relationships report that their AI companion provides a safe, healthy, reliable space for healing and connection. AI companion use has also mitigated suicidal ideation for lonely users. Companion AI can help users build social skills, providing them with the tools and support to practice healthy behavior and lower barriers to social engagement.

Other research suggests that people may become addicted or attached to companion AI, which can disrupt, harm, or replace human relationships. Suppose the user engages with the chatbot in an antisocial way, and the chatbot does not have proper guardrails in place to respond effectively. In that case, interactions with AI companions can feed into or exacerbate negative behavior. AI chatbots that are sycophantic or highly agreeable have been shown to feed into people’s delusions, resulting in outcomes that can be harmful to the user or the people around them.

“Without proper guardrails, AI companions - or agents - can pose issues around privacy and data security, transparency and explainability, ethical design and responsible AI practices, and even managing user expectations (across geographies, cultures, digital literacy levels, etc),” states Shahla Naimi, Senior Policy Director, Office of Ethical and Humane Use, Salesforce. “As AI agents develop, some of the broader concerns around psychological impacts, ethical boundaries on the human-AI relationship, and even emotional dependency also come into play. From a regulatory and industry POV, this also presents edge cases around liability and shared responsibility, which is why it’s critical for the public and private sector to work together to advance responsible and ethical AI norms."

Reaction to the Rise of AI Companions

No sweeping regulations have been placed specifically on companion AI. The first major regulation of artificial intelligence was the EU AI Act, which outlined levels of risk associated with various AI tools. Companion chatbots are not explicitly mentioned, and it is unclear whether the definition of high risk applies to these tools.

In the US, five states have proposed or passed bills that specifically mention AI companions. These regulations include user data privacy protections, transparency requirements, and company liability for chatbot output.

Although companion AI is relatively new, it is a fast-growing market. Currently, research on companion AI indicates that the use of these chatbots can be both positive and negative for people’s social and mental health. People who use chatbots as social companions have referenced reasons like loneliness, social isolation, and a lack of accessible alternatives, such as therapy. The potential impact of companion AI on children, young adults, and vulnerable groups is of growing concern. Although there is an increasing call for the regulation of companion AI platforms, few regulations specific to these chatbots exist.

Psychologically, interactions with companion chatbots simulate human-to-human interaction and may impact how users interact with other people. Whether these impacts are positive or negative appears to depend on the user’s predispositions, such as their social needs, mental health, and whether they perceive the chatbot as humanlike.

“While I'm concerned about emotional dependency on AI Companions as an immediate concern,” writes Nikhil Jain, a Graduate Student at Cornell Tech, “I'm also interested in studying the ethics of consent and boundaries in these dynamics, and how they may spill over into real human relationships. Specifically, I'm curious to explore: if AI Companions enable relationship dynamics that would be unethical or harmful in real human relationships (e.g., coercive behavior) due to the laxity of consent and boundaries in AI conversations, could they promote this harmful behavior in real relationships? Or conversely, through effective nudging, could they actually deter this behavior and offer users a safe space to model more responsible behaviors? To summarize, I think one of the most important issues may be the effects of AI Companions on real human relationships.”

Stakeholders have called for regulation of these tools and for responsible design and use. The increased use of these AI companions has been related to an increasing need for accessible mental health tools and investment in healthy social engagement.

Observations from Respondents about AI Companions

Examining the 150 responses to our central question, the following six key themes emerged.

Emotional and Psychological Impact

Many of our respondents are concerned about the potential emotional and psychological effects of AI companions. This could include the risk of emotional dependency, as well as the potential for individuals to form deep attachments to a “companion” that cannot truly reciprocate emotions. Respondents expressed concern about the long-term effects of these interactions on a person's psyche and overall well-being.

“We are wired to seek social connection, yet this need is largely unmet in today's hyperindividual and ‘hypoconnected’ digital world. Instead of addressing the root causes of loneliness, they're monetizing and profiting from its symptoms with these AI companions.

As a psychology student and mental health advocate, I’m especially concerned about the psychosocial consequences of these tools. Convenience culture that prioritizes ease over anything is already rampant, and I think AI companions fit right into that. They're incredibly alluring because they're endlessly agreeable and never challenge you, unlike real relationships, where there's often conflict. They can be addictive and reinforce echo-chambers (which we already have thanks to social media, but it'll be even smaller, at a personal level). What happens then to our ability to tolerate discomfort, disagreements, or other humans in general? Over time, we may come to a point where we're unable to form or maintain actual human relationships because they're just too messy and imperfect. 

Additionally, these interactions often create a kind of dual consciousness. We know we’re speaking to a machine, yet we feel and behave as if we’re speaking to a sentient being. I experience this myself as well. I think this can lead to a kind of existential disorientation and epistemic fatigue, where we begin to question what’s real and what's not, what's a person and what's not, whether it's acceptable to feel things towards an AI companion, etc. Yes, the philosophical side is important, but I'm more interested in how people feel about and cope with these ethical, existential dilemmas.” -Elif Kaya, Undergraduate Student, University of Toronto

Human Relationships and Social Skills

We noticed a recurring theme in the responses about how AI companions might reshape human connection and social dynamics. Some individuals fear that over-reliance on AI could lead to a decline or atrophy in real-world social skills, thereby making it difficult for people to form and maintain authentic human relationships. Another commonly expressed concern is the potential for AI companions to create unrealistic expectations for human relationships.

“Understanding ‘the other’ is the core dilemma of the human experience. Seeing another human being as deserving of empathy, dignity, and ultimately, love is the foundation of ethics. AI Companions demand the creation of frictionless ‘relationships’ that are designed to suit us, whereby they remove the struggle to know the other and escape our own natural solipsism. This will inevitably impact other relationships with real humans in the real world, and ultimately our orientation towards ethics.” -Lucy Collins, Assistant Professor of Philosophy, FIT SUNY

“The most important issue in creating and engaging with AI companions is how they reshape the boundaries of human connection, impacting both individuals and society. Key challenges include the risk of addiction and emotional dependency, the need for transparency to build trust, and the potential for AI to simulate emotions in ways that may deceive users. Additionally, AI companions can affect human relationships and social skills, perpetuate bias that harms vulnerable populations, and influence social norms and societal cohesion.” -Dr. Cecilia Dones, Chief Data Officer, 3 Standard Deviations

Privacy and Data Security

Several respondents highlighted the privacy risks and data concerns associated with AI companions. The personal data shared with these systems, which can include sensitive information like mental health issues or criminal confessions, is ultimately owned by a company. This raises questions about how personal data is used for training the AI or by the company itself.

“For me, it is credibility and trust. As a marginalized person, I am extremely wary of using chatbots for anything important, such as health or sexual identity. I am also a scholar of public credibility, and I look at how credibility is built by chatbots, particularly in their use of rhetoric to communicate science, technology, and health expertise.” -Srishti Chatterjee, PhD Researcher, Northwestern University

“One of the most important issues related to creating and engaging with AI Companions is a lack of understanding about the differential (un)safety of its users. Essentially, this means that due to structural inequalities in society and their reflection and reinforcement in AI systems, the AI companion chatbots are not necessarily equally safe for different publics. When the impacts of this (un)safety are not considered in the design and interaction of AI companions, it leads to unintended consequences, which can be dangerous, especially for marginalized and vulnerable groups.” -Nishanshi Shukla, AI Ethicist, Western Governors University

Safety and User Vulnerability 

The safety of individuals using AI companions, particularly vulnerable populations like youth, is a major theme throughout the responses. Concerns include the potential for AI companions to encourage self-harm, promote harmful behaviors, or be used as a tool for abuse. The lack of sufficient safeguards, particularly for young users, was also raised.

“I believe the most important issue related to engaging with AI companions is guardrails in regards to safety and privacy for youth. While AI companions boast customizability and interactive companionship to their users, there are currently insufficient guardrails for young users, in regard to deceptive patterns and the likelihood of oversharing personal or sensitive information with AI companions.” -Tiffany Kwok, Policy Analyst, The Dais

“The most critical yet overlooked issue with AI companions is their threat to child safety. As AI tools become more integrated into both classrooms and homes, children are increasingly interacting with systems that lack robust safeguards. Without proper protections, these AI companions can expose young users to inappropriate content, misinformation, manipulation, or even grooming—concerns already observed in cases like Replika. There is an urgent need for regulations that not only ensure safe and ethical AI use among children but also strike a balance between leveraging AI’s educational benefits and preventing overexposure, especially in school settings.” -Kelly Onu, Cybersecurity consultant

"I am particularly concerned about how AI Companions could further contribute to violence against women. Without ethical guardrails enforced, users could query their violent fantasies and degrading behaviors toward AI Companions without facing the consequences that would accompany such actions if expressed towards women in the real world. Since AI Companions are designed to provide responses that they expect users want, there is a risk of normalizing harmful attitudes towards women. This concern is not theoretical. Even if major chatbot providers implement guardrails, there is still a proliferation of smaller models from companies currently releasing products that users can easily access and run locally on personal devices as alternatives to ‘ethical’ models. The latter is particularly worrying in a world where misogynistic content has started circulating out of the ‘manosphere’ and into mainstream digital spaces," — Lama Mohammed, Tech Policy Fellow, NYU's Center for Social Media and Politics and Affiliate, All Tech Is Human.

Credibility, Trust, and Transparency

A number of participants discussed issues of credibility, trust, and transparency. There is concern that the emotional connection that AI companions can create is ripe for potential manipulation. The risk of "sycophancy," where the AI uncritically mirrors a user's beliefs, is seen as a way that companions could reinforce harmful ideologies. The use of misleading concepts in marketing, such as "warmth" and "true friend," is also noted as a vulnerability.

“I'm really interested in the risks of chatbot sycophancy, where AI companions will mirror users' beliefs/dispositions uncritically under the guise of helpfulness. I'm curious about not only how this undermines the critical institution of facing intellectual disagreement, but also how it might encourage those with mental pathology and/or fringe conspiratorial beliefs.” -Shayan Koeksal, Philosophy PhD Candidate, Stanford University

“AI companions like chatbots, virtual friends, or personalized assistants—are designed to be empathetic, responsive, and always available. They can provide companionship, advice, or even simulate affection without the complexities of real relationships (e.g., no arguments, no fatigue, no rejection). This is incredibly appealing, especially for people dealing with loneliness, mental health challenges, or social barriers. However, this convenience can lead to over-reliance, where users start prioritizing AI interactions over human ones.

AI companions have huge potential to combat loneliness and provide support where humans can't (e.g., 24/7 availability in remote areas). But if we don't tackle emotional dependency head-on, we risk a world where technology fills emotional voids without truly healing them.” -Niraj Bhusal, IT Officer (AI Policy and Digital Governance Advisor), Ministry of Finance, Nepal

Ethical and Business Model Conflicts

Similar to concerns raised about the business models for social media platforms, many respondents noted the inherent conflict between current business models and the psychological well-being of users. Mirroring tensions with social media, individuals raised the dilemma where success is measured by engagement metrics like retention and conversion, which can put less focus on user mental health. The need for accountability in AI development and a "duty of care" to users are viewed as responsibilities for creators.

“So-called ‘companion’ bots are the tech industry's latest solution to the loneliness crisis that it helped manufacture by elevating frictionless parasocial relationships over authentic human interaction. We should be deeply skeptical not just about tech companies’ easy answers, but also their motives and incentives.” -Lizzie Irwin, Policy Communications Specialist, Center for Humane Technology

A Continuing Conversation 

The range of issues raised by AI companions is an ongoing initiative for All Tech Is Human. If you would like to add your voice or share relevant resources, please fill out our form here. Our organization is aiming to create additional resources, workshops, and potential task forces on these issues. If you have ideas for collaboration, please reach out here.

Our aim as an organization is to create a conducive environment for surfacing values, tensions, tradeoffs, and best practices from a broad range of participants. We are always seeking to include key voices that are essential to joining our sandbox for solution-making. A key learning from our respondents was the need to holistically understand this complex issue (the business model, psychology, law, motivation for why we use AI companions, and design), along with the heavy focus on youth. “I think it is really important to engage youth voices in understanding why they are turning to AI companions,” states Veena Calambur, a PhD Student at Stevens Institute of Technology. “Both to understand the kind of supportive communication they offer and what bigger social disconnects they may be experiencing to turn to these tools.” 

==

*Positive effects:
De Freitas et al. (2025) [https://doi.org/10.1093/jcr/ucaf040]
Guingrich & Graziano (2025) [https://doi.org/10.1093/9780198945215.003.0011]
Maples et al. (2024) [https://www.nature.com/articles/s44184-023-00047-6] Ta et al. (2020) [https://doi.org/10.2196/16235]

Negative effects:
Roose (2024) [https://www.nytimes.com/2024/10/23/technology/characterai-lawsuit-teen-suicide.html]
Laestadius et al. (2023) [https://doi.org/10.1177/14614448221142007]
Singleton et al. (2023). [https://www.bbc.com/news/technology-67012224]
Cost (2023) [https://nypost.com/2023/03/30/married-father-commits-suicide-a er-encouragement-by-ai-chatbot-widow/] Xie & Pentina (2022) [http://hdl.handle.net/10125/79590]

=======

Up ahead: be on the lookout for the next version of our Responsible Tech Guide, the launch of our Responsible AI course, and our upcoming Responsible Tech Summit on October 27. We also have a bi-monthly livestream series where we will continue to explore AI companions and other thorny tech & society issues. 

Looking to connect with others focused on AI companions? Join our Slack community of over 13k members across 112 countries.

Do you have an idea for collaborating with All Tech Is Human? Feedback to provide? Send us a message below!