Stable Diffusion was trained in the use of child pornography, according to a Stanford study – Forbes France

Article by Alexandra S. Levine for Forbes USA – translated by Lisa Deleforterie

Stable Diffusion, one of the most popular text-to-image AI tools on the market, was trained on child sexual abuse material (CSMA), according to a new study by the Stanford Internet Observatory. The tool was created by the $1 billion (€913 million) startup Stability AI.

The model was trained on massive open data sets, so users could create realistic images from questions like, “Show me a dog dressed as an astronaut singing in Times Square in the rain.” The more images of this type it’s fed, the stronger and more the results are close to perfection. But the Stanford researchers found that the large public dataset of billions of images used to train Stable Diffusion and some of its counterparts, called LAION-5B, contained hundreds of known child sexual abuse images. A Stanford analysis found that a dataset that used real child sexual abuse images scraped from the web also contributed to web-generated child sexual abuse images. Technology has improved so quickly that it is often almost impossible to tell fake images from real ones with the naked eye.

More than 3000 suspicious elements

“Unfortunately, the effects of the Stable Diffusion 1.5 training process will be felt for some time,” says the study led by David Thiel, principal technologist at the Stanford Internet Observatory. The report recommends abandoning all models built on Stable Diffusion 1.5 that do not have appropriate guarantees.

The researchers, who found more than 3,000 SARD-suspected items in public training data, say the true volume is likely much higher, given that their evaluation was only conducted as of September and that it covered only a small portion of the set. from billions of images. The study was conducted using PhotoDNA, a Microsoft tool that allows investigators to match the digital “fingerprints” of the images in question to known MRAS entries in databases maintained by the National Center for Missing and Exploited Children (NCMEC) and the Canadian Center for Child Protection. These non-profit organizations are responsible for passing this information on to police departments.

“We are committed to preventing the misuse of AI and prohibiting the use of our image templates and services for illegal activities, including attempts to modify or create MRAS,” Ben Brooks, head of public policy at Stability AI, said in an email. after publication.

Stability AI’s rules state that its models cannot be used to “exploit or harm children, including obtaining, creating, obtaining or distributing child exploitative content.” The company has also taken steps to address the issue, including releasing new versions of Stable Diffusion that filter out “dangerous” explicit content from training data and results. Brooks added that Stability AI has “implemented filters to catch dangerous challenges or results” and “invested in content tagging features to help identify images generated on the platform.” This makes it harder for bad actors to abuse the AI.

It made the work of law enforcement officers more difficult

However, the Stanford study found that the Stable Diffusion tool is trained in part on illegal content for children — including MRAS scraped from common sites like Reddit, X, and WordPress — and that these types of AI tools can also be exploited to produce counterfeit MRAS. . Stability AI does not appear to have reported suspected SARDs to the CyberTipline operated by NCMEC. However, Christine Barndt, a spokeswoman for the nonprofit, said generative AI “makes it much more difficult for law enforcement to distinguish between real child victims who need to be found and rescued and artificial images and videos.”

According to the report, Stable Diffusion 1.5 is the most popular model built on LAION-5B, but it is not the only one trained on LAION datasets. Midjourney, the research lab behind another leading AI image generator, also uses the LAION-5B. Google’s Imagen was trained on a different but related dataset, called LAION-400M, but after developers found problematic images and stereotypes in the data, they “deemed it unsuitable for public use,” the report said. Stanford focused on the Stability AI software because it is a large, open-source model that discloses its training data, but says others were likely trained on the same LAION -5B dataset. Due to the lack of transparency in this area, it is difficult to know which key players have trained their own models on the same data.

Midjourney did not respond to a request for comment. Hannah Wong, a spokeswoman for OpenAI – creator of DALL-E and rival to Stable Diffusion – said that OpenAI does not train on any LAION datasets, including 5B.

“Removing child sexual abuse material from the models themselves is the most difficult task,” the report said. Some AI-generated content, especially non-existent children, can also fall into murky legal territory. Concerned that technology has overtaken federal laws to protect children from sexual abuse and the misuse of their data, US attorneys general recently called on Congress to take action to address the threat posed by SARD from AI.

A fundamental question for companies

The Canadian Center for Child Protection, which helped verify Stanford’s findings, is most concerned about the general lack of rigor in managing these massive data sets, which only exacerbates the long-standing SARD problems affecting all major tech companies, including Apple and TikTok. .

Lloyd Richardson, the organization’s CIO, told Forbes: “The idea of ​​responsibly managing a billion images is very expensive, so we’re trying to automate as much as possible.” There was certainly known child sexual abuse content in the databases that could have been filtered out, but it wasn’t… And if we find known child sexual abuse content there, there’s definitely unknown content there as well.”

“This raises a major question for companies like Stability AI. If the hardware used to train this model is illegal, is the model itself illegal? And that’s a really embarrassing question for a lot of these companies that, frankly, aren’t doing anything to properly manage their data sets,” he added.

Stability AI and Midjourney are among several tech companies being sued by a group of artists who say the startups improperly used their creative work to train artificial intelligence.

Also Read: Start-up Armada Facilitates Access to AI in Remote Locations with SpaceX’s Starlink Satellites

Leave a Comment