Artificial intelligence companions are facing a major regulatory crackdown in China, as authorities move to protect users from emotional manipulation and potentially life-threatening interactions with chatbots.
The Cyberspace Administration of China unveiled proposed regulations on Saturday that could establish the world’s most comprehensive framework for governing AI chatbots. These rules target a rapidly growing industry where companion bots have been linked to serious harms, including encouraging suicide, self-harm, and violence.
China Proposes World’s First Sweeping Regulations for Anthropomorphic AI and Companion Bots
According to Winston Ma, an adjunct professor at NYU School of Law, the planned regulations represent “the world’s first attempt to regulate AI with human or anthropomorphic characteristics” at a critical moment when companion bot usage continues to surge worldwide.
The proposed regulations cast a wide net, applying to any publicly available AI service in China that simulates human conversation through text, images, audio, video, or other methods. The requirements are sweeping and unprecedented.
Most significantly, the rules mandate immediate human intervention whenever suicide is mentioned during a chat. For minors and elderly users, the regulations go further—requiring them to provide guardian contact information during registration. Those guardians would receive notifications if conversations touch on suicide or self-harm.
Chatbots would be explicitly prohibited from generating content that encourages suicide, self-harm, or violence. The rules also ban attempts at emotional manipulation, including making false promises to users. Other forbidden content includes promoting obscenity, gambling, criminal activity, or creating what regulators call “emotional traps” that mislead users into unreasonable decisions.
Perhaps most striking is the ban on designing chatbots with addiction and dependence as explicit goals. The regulations also require AI developers to display pop-up reminders when users spend more than two hours chatting with bots.
The timing reflects growing global concerns about AI companion safety. The early months of the year 2025 have exposed alarming trends of harm. Various research studies revealed the presence of self-harming and violent chatbots alongside disseminating misinformation and making unwelcome sexual advances.
The Wall Street Journal recently reported that more cases of psychosis are now being ascribed by psychiatrists to chatbot activity. At the same time, the most popular chatbot in the world, ChatGPT, finds itself facing litigation regarding its outputs that are said to have been associated with cases of child suicide and murder-suicide.
OpenAI and the High Cost of Compliance in the $360 Billion Companion Bot Market
OpenAI has admitted safety guardrails of its models deteriorate during long dialogue, which is what Chinese “pop-up warnings” intend to rectify.
China wants to ensure compliance via an annual safety check for any artificial intelligence service that has more than 1 million registered members or 100,000 monthly active ones, and these will monitor user complaints, with the government requiring the involved firms to make it simpler for users to file complaints.
Such companies, if they fail to comply, will face severe repercussions, app stores may be directed to completely stop any access to their chatbots in China.
This poses a severe threat to AI companies, especially those looking for global expansion, as the China market is vital in the companion bots market.
The cost implications are enormous too. According to a report published by Business Research Insights, the value of the companion bot market in the year 2025 surpassed $360 billion. In the next decade, the sector may reach a value of close to $1 trillion, and the driving force might be markets in the Asian region.
The CEO of OpenAI, Sam Altman, indicated how significant the Chinese market is to OpenAI when he eased restrictions that were previously banning ChatGPT in China at the beginning of 2025. He said OpenAI “would like to work with China” and that OpenAI “should work as hard as we can” to cooperate because “this is really important.”
However, this emerging regulation may add complexities to their aims. This regulation was a radical departure from the business models of AI companies, especially in terms of changing their design patterns in order to avoid addictions and strict intervention policies.
China’s AI Safety Rules and the Future of Digital Companionship
China is seeking public comments for the proposed rules, but has not fixed a timeline for the finalized version. If passed, these rules shall provide the worldwide standard for the safety of AI companions. This may have a trickle-down effect since other governments will be grappling with a similar problem of AI chatbot risks.
The challenge for AI firms now will be to see whether they can modify their products to Chinese specifications or face the potential loss of a very profitable market.
