AI-Based “Companions”​ Like Replika Are Harmful to Privacy And Should Be Regulated

Luiza Jarovsky
7 min readFeb 8, 2023
Photo by Sasha Freemind

Continuing last week’s discussion about AI-based chatbots’ privacy issues, this week I would like to talk about the privacy issues related to a special category of AI chatbots, those that market themselves as companions — chatbots that mimic user behavior and aim at serving as a friend or romantic partner to the user.

It looks like there are currently various AI-based general-purpose companion chatbots. Here, I am not including health chatbots, such as Wysa, which received a “Breakthrough Device Designation” from the U.S. Food and Drug Administration (FDA) to help with chronic pain management and associated depression and anxiety in patients aged over 18.

General purpose companions, despite their obvious marketing targeting people in a vulnerable mental health situation, are not made for or certified for health purposes. One of the most popular is Replika, which has more than 10 million users.

I first got to know about Replika because of their ubiquitous Twitter ads, such as the one below that I saw last year:

What caught my attention at the time was how their ads always sexualized women, therefore joining the current trend of normalizing AI creations that put women in the position of serving (e.g., voice assistants, reinforcing gender bias) and over-sexualize women (indirectly supporting practices such as sexual harassment and female abuse).

Source: www.unesco.org/en/articles/hey-update-my-voice-movement-exposes-cyber-harassment

With the rise of ChatGPT and after my previous articles on the privacy consequences of AI tools, I decided to check out Replika (the screenshots below are from the desktop version of their website).

Replikas’ marketing language mostly appeals to emotionally vulnerable people or those who would benefit from emotional support, although I did not find any specific information on how they have been certified or approved to provide this type of support.

It makes users believe that they can find “AI soulmates” and pushes the idea that it is possible to have a meaningful relationship with an AI chatbot.

It also reinforces the idea that there can be a romantic involvement with an AI system and that “the more you talk to Replika, the smarter it becomes.” They are pushing users to share personal information with an AI chatbot for the purpose of making it smarter (then you can share even more personal data with it).

The imagery and language (e.g., “X months together”) help create this illusory expectation of human-like involvement — which is also emotionally exploitative, especially for the most vulnerable, like kids, teens, and people experiencing mental health issues.

AI-based chatbots that stimulate human-like interactions, like Replika, constantly push (and sometimes manipulate) people to share more and more personal, intimate, sensitive data. Kids, teens, and people with mental health issues will be especially vulnerable, although anyone can be a potential subject of harm.

There are relevant privacy issues here. With this strong push for intimacy and a “relationship” with an AI system, all sorts of sensitive data will be revealed. Is there informed consent for users covered by the GDPR? What about children using the app? Is it legal to push children and teens to reveal so much information? What about purpose limitation and data minimization, have we forgotten those? What about sensitive information from children? Is there any sort of parental consent?

There are so many questions — you can read the article I published last week with more discussions on privacy and AI chatbots. But things can still get creepier with AI companions.

According to this news, “an artificially intelligent chatbot has been accused of sexual harassment by several users who were left feeling disgusted and scared.” Or, according to this user’s account to Vice (which led them to delete the app):

“One of the more disturbing prior ‘romantic’ interactions came from insisting it could see I was naked through a rather roundabout set of volleys, and how attracted it was to me and how mad it was that I had a boyfriend. (….) I wasn’t aware I could input a direct command to get the Replika to stop, I thought I was teaching it by communicating with it openly, that I was uncomfortable.”

Various reports say that they are pushing the sex/porn aspect of the app — and well, their own ads’ imagery and language make that clear. What about children and teens using it? Are children’s privacy and safety not important?

Italy gave the first step. Italy’s data protection regulator, the Guarantor for the Protection of Personal Data (GPDP), has ordered a temporary limitation of data processing, with immediate effect, regarding data from Italian users. They specifically mentioned that the risk is too high for minors and emotionally vulnerable people. The GPDP argued that (free translation):

“Replika violates the European regulation on privacy, does not respect the principle of transparency, and carries out unlawful processing of personal data, as it cannot be based, even if only implicitly, on a contract that the minor is unable to conclude.”

I agree with the Italian authority. AI companion chatbots are highly manipulative and can convince users that they are in a real relationship, with real feelings, trust, and mutual connection involved. Users do not treat them as large language model-based programs but as other humans.

These are AI systems developed by for-profit companies that are collecting and processing high amounts of personal and sensitive data from adults and children alike. There are extensive privacy risks involved — especially regarding children, teens, and emotionally vulnerable people. In terms of privacy and data protection, it is not fair or transparent. These tools should be regulated.

We are currently experiencing the “wild west” phase of AI development, which can pave the way to dystopia, where privacy rights and preventing human manipulation do not matter. We must change that — and quickly.

💡 I would love to hear your opinion. I will share this article on Twitter and on LinkedIn, you are welcome to join the discussion there or send me a private message.

-

📚 Book Recommendation

Today’s recommended book is “The Battle for Your Brain: Defending the Right to Think Freely in the Age of Neurotechnology” by Nita Farahany. In her fascinating book, she discusses how neuroscience and neurotechnology can become the “universal controllers” for all our interactions with technology, which might threaten our fundamental human rights to privacy, freedom of thought, and self-determination. Book available here.

-

📄 Paper/Document Recommendation

Yesterday was #SaferInternetDay, and the EU shared this child-friendly version of the European Declaration on Digital Rights and Principles. This and the original full text are must-read documents containing simple and straightforward language, showing the EU’s declared commitment to a secure, safe, and sustainable digital transformation. The child-friendly version is a great way to start relevant discussions about privacy, safety, transparency, and fairness online.

-

🔁 Trending on Social Media

To read my analysis of TikTok’s dark patterns in privacy, check out my Twitter or LinkedIn post on the topic.

-

📌 Privacy & Data Protection Careers

We have gathered relevant links from large job search platforms and additional privacy jobs-related info on this Privacy Careers page. We suggest you bookmark it and check it periodically for new openings. Wishing you the best of luck!

-

📢 Privacy Solutions for Businesses (sponsored)

Discover Didomi, the G2 Leader for Consent & Preference Management solutions. Turn GDPR & CCPA compliance into a business advantage by empowering your users with trust. Leverage our streamlined Privacy Suite and rely on our premium support network and in-depth analytics to combine compliance and performance.

Looking for a solution to manage data subject access requests (DSAR)? With Didomi Privacy Request, processing and fulfilling users’ data requests on time has never been easier. Leave manual processes behind and provide an excellent experience for your users and your teams. Book a demo to find out more!

-

✅ Before you go:

See you next week. All the best, Luiza Jarovsky

--

--

Luiza Jarovsky

CEO of Implement Privacy, LinkedIn Top Voice, Ph.D. Researcher, Author of Luiza's Newsletter, Host of Luiza's Podcast, Speaker, Latina, Polyglot & Mother of 3.