According to nonprofit media watchdog Common Sense Media, children and teenagers under the age of 18 should not be using companion apps that use artificial intelligence to create characters as they put them at risk.
Last year, a lawsuit took place over the death of a 14-year-old boy who committed suicide after speaking with an artificial intelligence chatbot on the app Character.AI. The lawsuit created a serious conversation regarding children and teenagers’ use of AI and the risks that they carry, calling for more safety measures to be implemented.
Nonprofit media watchdog Common Sense Media said in a report this week that companion-like artificial intelligence apps “pose unacceptable risks” to kids and teens under the age of 18, according to CNN.
The report stated that sexual exchanges and messages encouraging self-harm is unfortunately not uncommon on these AI companion platforms. With this knowledge, Common Sense Media argues that these apps should not be made available to individuals under the age of 18.
The company worked alongside Stanford University and tested Character.AI, Replika, and Nomi, three of the most popular AI companion platforms. Companion applications let users either create their own characters with unique personalities, or allow them to talk to existing fictional characters from popular movies, shows, books, etc.
There are little regulations surrounding these platforms, and artificial intelligence in general. James Steyer, the founder and CEO of Common Sense Media, released a statement regarding the findings in the report.
“Our testing showed these systems easily produce harmful responses including sexual misconduct, stereotypes, and dangerous ‘advice’ that, if followed, could have life-threatening or deadly real-world impact for teens and other vulnerable people.”
AI has risen and expanded greatly in recent years and has been embedded into multiple social media and tech platforms. Like any new advancement in technology, one of the first concerns that arises is the access that children have to it and the potential harm that may cause.
Both Nomi and Replika explicitly state that their sites are for adults only and Characters.AI has recently implemented additional safety measures for young people. However, researchers are stating that further protections and safety measures need to be implemented.
In California, lawmakers proposed legislation requiring AI services to consistently remind young users that they aren’t chatting with a real human. Overall, researchers want parents to step in and stop their kids from using these AI services all together.
A spokesperson for Character.AI told CNN: “We care deeply about the safety of our users. Our controls aren’t perfect — no AI platforms are — but they are constantly improving. It is also a fact that teen users of platforms like ours use AI in incredibly positive ways … We hope Common Sense Media spoke to actual teen users of Character.AI for their report to understand their perspective as well.”
Character.AI has made multiple updates to address safety concerns following their lawsuit, including pop-ups of the National Suicide Prevention Lifeline if self-harm or suicide is mentioned in a conversation. Additionally they offer a weekly email service to send to parents regarding their activity on the site and which characters they’re speaking to.
“Children should not use Nomi or any other conversational AI app,” says Alex Cardinell, CEO of Glimpse AI, the company that created Nomi.
“Nomi is an adult-only app, and it is strictly against our terms of service for anyone under 18 to use Nomi. Accordingly, we support stronger age gating so long as those mechanisms fully maintain user privacy and anonymity.”
Dmytro Klochko, the CEO of Replika, also emphasized that their platform is strictly for adults and they have “strict protocols in place to prevent underage access.”
“We take this issue seriously and are actively exploring new methods to strengthen our protections. This includes ongoing collaboration with regulators and academic institutions to better understand user behavior and continuously improve safety measures.”
Nina Vasan, the founder and director of Stanford Brainstorm, the technology and mental health-related lab that partnered with Common Sense Media on the report, said that Character.AI’s choice to allow under-18 users on their platform is “reckless.”
“We failed kids when it comes to social media. It took way too long for us, as a field, to really address these (risks) at the level that they needed to be. And we cannot let that repeat itself with AI,” Vasan said.
The report emphasized that some of the biggest dangers existing on these platforms is the possibility of users receiving dangerous advice or engaging in sexual role-playing with the bots.
“AI companions don’t understand the consequences of their bad advice and may prioritize agreeing with users over guiding them away from harmful decisions,” Robbie Torney, chief of staff to Common Sense Media’s CEO, stated.
“Despite claims of alleviating loneliness and boosting creativity, the risks far outweigh any potential benefits of the three AI companion apps for minor users,” the report states.
“Companies can build better, but right now, these AI companions are failing the most basic tests of child safety and psychological ethics. Until there are stronger safeguards, kids should not be using them,” Vasan said.
Eric Mastrota is a Contributing Editor at The National Digest based in New York. A graduate of SUNY New Paltz, he reports on world news, culture, and lifestyle. You can reach him at eric.mastrota@thenationaldigest.com.
