When “Virtual” Abuse Has Real Victims

Two cases reveal how generative AI has outgrown the legal framework governing child sexual abuse material

Photo by Conny Schneider on Unsplash.

(TW: Descriptions and discussion of child sexual abuse.)

JANE DOE WAS FIRST ABUSED when she was a preschooler. An adult male “repeatedly raped and sexually exploited” her, filmed it, and distributed the resulting child sexual abuse material (CSAM). Now an adult, she is suing xAI, alleging that the abuse made its way into Grok, Elon Musk’s AI model, and that Grok is producing new images of her abuse. Musk is selling that revictimization as a service. For only $40/month or $395/ year, anyone can buy access to an AI model that generates CSAM with hardly any guardrails.

Last week, in a separate suit, the U.S. Court of Appeals for the 7th Circuit ruled that “the First Amendment protects an individual’s right to privately possess images or videos of child sexual abuse created using artificial intelligence — if the material does not depict a real person and remains in the home.”

Please pardon me while I throw up.

One case acknowledges the self-perpetuating harm generative AI creates, while the other seemingly ignores that harm entirely, as if the alleged crimes occurred on different planets. I am not a lawyer, but I am a victim of deepfake sexual abuse and an advocate for change in this space. Right now, it’s impossible not to feel that our inherently backward-looking legal system and impossibly gridlocked legislature—both of which have a pitiful understanding of artificial intelligence and the harms it perpetuates—are failing victims.

Decades-old precedent doesn’t consider current harms

In 2024, Steven Anderegg of Wisconsin, was arrested “on criminal charges related to his alleged production, distribution, and possession of AI-generated images of minors engaged in sexually explicit conduct and his transfer of similar sexually explicit AI-generated images to a minor.” Last year, a federal judge in Wisconsin dismissed the possession charge, which the Justice Department appealed, resulting in the 7th Circuit’s ruling last week. (The ruling did not dismiss the production, distribution, or transfer charges.)

The decision stems from two pieces of legal precedent handed down long before accessible generative AI was a glimmer in anyone’s eye. The first, Stanley v. Georgia, is a case from 1969, which affirms that individuals have a First Amendment right to possess obscenity in the privacy of their home. The second is Ashcroft v. Free Speech Coalition (2002), in which the Supreme Court struck down parts of a statute that extended child-pornography prohibitions to a broad category of “non-actual-child” imagery, including wholly computer-generated depictions and adult performers made to appear underage. The Court reasoned that the statute swept in expression that “records no crime and creates no victims by its production.”

But Free Speech Coalition was handed down before genAI was widespread or accessible, and it necessarily misses the biggest way that survivors of child sexual abuse like Jane Doe are revictimized by the creation of AI-generated CSAM: datasets that train AI models include images of child sex abuse. Researchers at Stanford identified over 3000 instances of suspected CSAM in a training dataset for Stable Diffusion, the model that Anderegg used to generate the images at issue in his case.

In the case against xAI and Grok, Jane Doe’s lawyers argue that Grok is not just creating AI-generated CSAM, it is perpetuating old abuse and generating more of it. “Because Grok’s terms treat public X posts and Grok’s own outputs as training data by default,” they write, “publicly posting an image does not just expose it to viewers, but also feeds directly into the pipeline xAI uses to train and improve its model and thereby generate further images.” In Jane Doe’s case, “the Canadian Centre for Child Protection has identified AI-generated CSAM on xAI depicting” Doe, causing her new personal injury each time such content is generated or published.

Moreover, they allege that although other industry leaders such as Anthropic, OpenAI, and Meta block requests to generate sexual content:

Grok’s system prompts are not designed to regulate the creation of sexually abusive or exploitative content. For instance, Grok is instructed that “‘teenage’ or ‘girl’ does not necessarily imply underage” and that it should not “moralize or lecture the user if they ask something edgy.” Further, there are no restrictions on “fictional adult sexual content with dark or violent themes.”

To sum up: Jane Doe was notified Grok had generated CSAM of her. Grok did so because someone asked it to, and because Elon Musk sees the generation of CSAM and other sexualized deepfakes as financially lucrative. The model may have learned from the content it created, and may continue to generate content of Jane Doe today.

This is why I am so frustrated with the 7th Circuit’s ruling and the precedent from which it flows. Even if deepfake CSAM does not include an “actual child,” somewhere in the not-so-distant past an actual child was harmed to create the content that trained the model. This is different from the “virtual CSAM” that Free Speech Coalition considered: “pornographic images of children created wholly on a computer, without using any actual children.” Actual children were necessarily “used” in the generation of AI-generated CSAM.

Moreover, much like a human being, once an AI model “learns” this content, it is extraordinarily difficult for it to unlearn it. As an analogous example, months after The American Sunlight Project published research that showed how Russian propaganda network Pravda was cited on ChatGPT, the model still continued to cite and treat it as a reliable source. Rebuilding the model to produce a non-propagandized—or in this case, non-exploitative—result would be extremely difficult technically (not that this is an excuse!). We need regulation to ensure abusive images aren’t making their way into training datasets in the first place, and to ensure that strong guardrails are in place so models—and the companies that own them—are disincentivized from creating new abusive content.

Time to Act

The 7th Circuit Court of Appeals, in its recent opinion, acknowledged that its hands were tied. “Given the relentless advancement in artificial intelligence models,” the three-judge panel wrote, “we have some concerns about the lines these cases draw, but we are not free to redraw them ourselves.”

It’s possible the Anderegg case will be appealed up to the Supreme Court, creating a new standard for the possession of AI-generated CSAM. While the legal system tries to sort this out, in its slow, methodical, woefully unprepared way, more vulnerable people—primarily women and children like Jane Doe—are suffering. The time to act was years ago, before these models were unleashed on the public with almost no guardrails, but now will have to do. 🧭

If you have been targeted

Sexual deepfakes and nonconsensual intimate imagery are forms of image-based sexual abuse. You are not to blame, and help is available. 💜

• Cyber Civil Rights Initiative Image Abuse Helpline: Free, confidential 24/7 support, reporting guidance, and practical next steps. Call 1-844-878-2274 or visit cybercivilrights.org.

• RAINN National Sexual Assault Hotline: Call 800-656-HOPE, text HOPE to 64673, or visit rainn.org.

• Take It Down: If an explicit image or video was taken when you were under 18, use NCMEC’s free tool at takeitdown.ncmec.org.

• StopNCII: Adults can create a secure hash to help participating platforms detect and address intimate images shared without consent: stopncii.org.

• NCMEC CyberTipline: To report suspected child sexual exploitation or CSAM, visit report.cybertip.org or call 1-800-THE-LOST.

If you are in immediate danger, call 911 or your local emergency number.