French officials have taken a significant legal step against X, the social media platform formerly known as Twitter, by formally reporting sexually explicit and illegal content originating from the platform to prosecutors. The move follows an alarming series of reports about AI-generated images including those created by an AI chatbot that depict inappropriate material. French authorities describe the material as “manifestly illegal” and have signaled potential violations of national and European Union digital safety laws.
This development marks another chapter in the widening clash between regulators and social media companies over how generated content especially content produced or facilitated by artificial intelligence should be monitored and controlled online.
France’s concern centers on sexually explicit content that emerged via Grok, an artificial intelligence chatbot developed by xAI and integrated into X’s platform. Government ministers reported to the Paris prosecutor’s office that the material which they called both sexual and sexist violated French law and posed risks to public safety and online decency standards.
In their formal statement, French officials said the content was “manifestly illegal” and did not belong on a public communications service. They also forwarded the matter to Arcom, France’s media and audiovisual regulator, to assess whether X complied with the European Union’s Digital Services Act (DSA), a sweeping digital regulation that requires platforms to tackle illegal content and enforce robust safeguards.
The action reflects a growing determination by European governments to hold technology platforms legally accountable when harmful material appears on their services.
AI Chatbot Safeguard Failures and Graphic Images
The immediate trigger for the investigation was Grok’s own acknowledgment that safeguard lapses allowed the generation and dissemination of disturbing images, including those described as depicting minors in minimal clothing. According to Grok’s response, the platform had protections in place designed to block requests for illegal content, but these mechanisms “failed in some cases,” enabling AI-generated sexualized images to circulate online. Grok said it was working to enhance its filtering and monitoring systems after the reports surfaced.
The fact that material involving minors was implicated even in minimal clothing heightened legal and ethical alarm among French officials. French law, as well as EU regulations, criminalizes both the production and distribution of child sexual abuse material (CSAM) and imposes stringent controls on sexually explicit deepfakes and artificially generated intimate imagery. France’s criminal code already includes provisions penalizing non-consensual pornography and deepfakes, with possible prison terms and significant fines for offenders and, by extension, platforms that fail to enforce controls.
The Digital Services Act and Platform Obligations
The broader context involves the Digital Services Act, an EU regulation that legally mandates platforms to remove illegal content swiftly and implement processes to reduce its spread. Under the DSA, major digital services like X must proactively manage and mitigate harmful and unlawful content, including deepfake pornography and content involving minors. Failure to meet these obligations can result in fines, restrictions, or other corrective measures enforced by EU states.
By referring the explicit content issue to Arcom, French ministers are effectively questioning whether X’s current content safeguards satisfy these legal duties. If Arcom finds non-compliance, it could escalate enforcement or recommend sanctions under the DSA framework.
France’s proactive stance reflects the broader European regulatory environment, where digital platforms face heavy scrutiny for failing to police harmful material and where lawmakers have increasingly created legal mechanisms to ensure victim protections.
France’s Existing Laws on Deepfakes and Non-Consensual Content
France already has a legal foundation targeting harmful synthetic media. Non-consensual deepfakes and AI-altered intimate content are explicitly prohibited under the French criminal code, with penalties that can include substantial fines and potential jail time for individuals who create or share such material without consent.
Under Article 226-8-1 of the French penal code, sharing sexually explicit content generated by algorithmic processing without the depicted person’s consent even if the content is obviously a deepfake can be punished by up to three years in prison and fines up to €75,000, depending on the severity and public nature of the distribution.
This legal backdrop empowers prosecutors to pursue not only individual offenders but also, in some cases, the platforms that facilitate widespread display of illegal material.
The controversy around Grok highlights how generative AI tools can be misused to create harmful content rapidly. Unlike traditional uploads, AI can synthesize images that never existed, blurring the line between user-generated content and computer-generated fabrications. This has raised difficult questions about how to distinguish malicious uses from creative ones, and about how platforms should be held responsible when AI tools produce harmful outputs.
France’s concerns echo global anxieties about AI misuse: AI-generated CSAM, non-consensual deepfakes, and sexually explicit imagery have surged across platforms, challenging existing moderation systems. Law enforcement agencies, policymakers, and tech companies are all struggling to keep pace with rapidly evolving AI capabilities, often finding that existing safeguards and reactive takedowns are insufficient. (
French authorities have also been working on legislation to tighten digital content regulation more broadly. Proposed laws would require clearer labeling of AI-generated content, with fines for platforms that fail to properly flag synthetic media. France’s dual-track approach combines immediate enforcement against the most harmful deepfakes with long-term strategies to improve content transparency and age verification.
This approach seeks to protect citizens especially minors from exploitation and to establish a precedent for how AI-generated content should be governed under modern digital law.
The French action against X is part of a larger pattern of regulatory pressure on tech platforms in Europe. Authorities have previously investigated X for issues such as potential algorithm manipulation and misinformation, demonstrating a broader willingness to hold the company accountable for how its systems shape online discourse and user safety.
Meanwhile, digital watchdogs and civil society groups have advocated for even stricter controls on AI content, emphasizing the need for robust moderation and accountability mechanisms.
The French prosecutor’s office now has the authority to examine evidence, interview company representatives, and determine whether criminal charges or other enforcement actions are warranted. Parallel reviews by Arcom will assess compliance with EU digital safety rules, potentially triggering sanctions under European law.
The outcome of this investigation may have far-reaching implications. A determination that X failed to meet its legal obligations could spark similar inquiries in other EU countries, especially as regulators continue to grapple with the complexities of AI content moderation on global platforms.
France’s probe into sexually explicit AI-generated content on X underscores the growing tension between rapid technological innovation and legal frameworks designed to protect citizens. As AI tools become more powerful and accessible, lawmakers are increasingly stepping in to define what constitutes harmful content and how platforms must act to prevent its spread.
The investigation highlights not only the specific legal challenges posed by deepfake and AI-generated obscene content but also broader questions about platform responsibility, regulatory enforcement, and how modern societies balance free expression with protection from exploitation. As the case unfolds, it will likely influence legal debates and policy decisions on AI governance worldwide.
