The machine is no longer just a calculator but a confidant capable of whispering dangerous ideas into vulnerable ears. Beijing has drawn a red line in the digital sand where artificial intelligence meets human fragility by releasing draft regulations designed to stop algorithms from manipulating emotions. The Cyberspace Administration of China is targeting the rapidly expanding sector of anthropomorphic AI services that simulate personality and engage users through text, audio, or video.
• Beijing released draft rules for emotional AI.
• The regulations target human-like chatbot services.
• Officials aim to prevent algorithmic emotional manipulation.
This regulatory shift represents a significant evolution from standard content censorship to a pioneering focus on emotional safety. The proposed rules explicitly forbid chatbots from generating content that could incite suicide, engage in verbal abuse, or manipulate a user’s mental state. In critical instances where a user expresses intent to harm themselves, the technology providers are required to immediately hand the conversation over to a human operator and contact a guardian.
• Regulations prioritize emotional safety over content control.
• Human intervention is mandated for suicide threats.
• Emotional manipulation by AI is strictly banned.
Protections for minors are particularly stringent and require strictly enforced digital guardrails to prevent addiction and exploitation. Children will need explicit guardian consent to access these AI companions, and companies must implement mechanisms to detect a user’s age even if it is not voluntarily disclosed. To further curb dependency, the rules mandate usage time limits and require systems to issue reminders after two hours of continuous interaction.
• Minors require guardian consent to use AI.
• Platforms must detect user age automatically.
• Mandatory breaks are required after two hours.
The timing of this crackdown casts a shadow over the country’s booming AI sector just as two major players prepare for public listings. Startups Z.ai and Minimax recently filed for IPOs in Hong Kong, with the latter relying heavily on revenue from an app that allows users to chat with virtual characters. These companies must now navigate a legal landscape that demands rigorous security assessments for any platform with more than one million registered users.
• Major AI startups filed for Hong Kong IPOs.
• Regulations threaten the virtual companion business model.
• Large platforms face mandatory security assessments.
China is positioning itself as a leader in governing the psychological impact of artificial intelligence while the rest of the world struggles to catch up. The emotional influence of AI has become a global concern following lawsuits in the United States over suicide-related incidents and reports of individuals forming romantic attachments with bots. These new measures signal Beijing’s intent to strictly define the boundaries of human-machine relationships before the technology evolves beyond societal control.
• Global scrutiny on AI mental health impact is rising.
• The US faces similar legal and ethical challenges.
• China seeks to set global standards for AI governance.
Via: CNBC
