This investigation is difficult to fit into one post and documents the gruesome reality and ethical compromises generative AI startups are dealing with but it's deeply important so please bear with me
For months, we've been reporting about Civitai, a startup that lets people create and use AI models for AI-gen imagery. If you go to Civitai's site, it looks like it's mostly for generating sci fi scenescapes, futuristic images of creatures, things like this.
If you go a level down, however, you'll see the vast majority of uses are to create nonconsensual porn of celebrities and real people. It has introduced a "bounty" system that lets people generate models that many users then use for porn
Civitai has generally ignored us when we've tried to report on them and their (very poor) content moderation. Last month, we broke news that a16z invested in them.
Our reporting eventually led us to a company called OctoML, which is a company that provides compute for services like Civitai. Basically, OctoML runs the generative AI that's on Civitai and thus has a good sense of what users are doing on Civitai's platform. OctoML has received $132 million in funding and is a giant in the space
We were leaked a cache of internal documents at OctoML which provides the basis for the story we published today.
Those documents included details of what users are actually making on Civitai, and the internal discussions this spurred at OctoML, because many employees were deeply concerned with what they found.
Most notably, Civitai users were generating what an OctoML cofounder said "could be categorized as child pornography." The Slacks and other data show that majority of content on Civitai is porn (much of it nonconsensual), despite what the company tries to show to the public
He and other employees worried about what legal risk OctoML was exposing themselves to with these two issues. They debated this repeatedly in a Slack room called #ai_ethics. Many employees had deep misgivings about what they were enabling.
*content warning for next couple posts*
Discussion of CSAM
A cofounder of OctoML said he personally had seen examples of possible CSAM being generated using fairly innocuous, safe for work prompts. Meaning Civitai was spitting out possible CSAM when the user wasn't even trying to make it: “A fairly innocent and short prompt ‘[girl: boy: 15], hyperdetailed’ automatically generated unethical/shocking content—read something could be categorized as child pornography," he said
Discussion of CSAM
These are examples of prompts users gave Civitai, that OctoML then processed and created images of. First screenshot is from just a few hours of data
As I mentioned, OctoML discussed at length the problematic nature of their contract with Civitai. Ultimately, OctoML decided to keep working with Civitai, but decided to stop promoting that relationship. OctoML had previously done a roundtable event with them. In a case study on its site, OctoML used Civitai as an example but decided not to name the company for "PR reasons"
On top of all of this, OctoML is not actually doing the generation on its own servers. It uses AWS. So this possible CSAM was actually being generated by Amazon.
@jasonkoebler It's not CSAM though. CSAM is created by abusing a child. AI is generating a fictional, not existing character. Which is...fine. The only way that would even be an issue is if the AI training data was using actual CSAM or images of real kids. However, most likely the AI training data is using consensual legal adult pornography.
We need to stop conflating CSAM which affects and harms REAL kids (therefore real victims) with material of fictional people that doesn't exist (and therefore non-existent victims)
