Digital ‘watermarks’ will not save us from deepfakes and AI-generated fraud

Hello and welcome to Eye on AI.

It is fast becoming apparent that we are living in a new age of inauthenticity, midwifed in no small measure by generative AI. This week brought news of robocalls in New Hampshire using a voice impersonating President Joe Biden—almost certainly created with easily available voice cloning AI software— urging Democrats to “save their vote” for November and not to vote in today’s Republican primary. New Hampshire has open primaries, meaning voters do not have to be registered party members to vote, and some Democrats had hoped to hurt Donald Trump’s chances of securing the GOP nomination by casting ballots for his rival Nikki Hailey.

State election authorities are investigating the robocalls but there is little the government can do to prevent AI-aided dirty tricks such as this. It seems likely that with more than 4 billion people globally eligible to vote in various elections this year, not least the U.S. presidential vote, voice cloning is on track to superpower election interference. It is also enabling frightening new scams in which fraudsters use voice cloning to impersonate the victim’s loved ones in distressing situations (usually held captive by kidnappers or arrested and in need of bail money). More significant for Eye on AI’s business readers, perhaps, are scams in which con artists impersonate a CEO or company director and ask a finance or corporate treasury executive to make an urgent payment to “seal a secret acquisition deal”—that just happens to send the money straight to an offshore account the fraudsters control. None of this seems to have dissuaded investors from piling into synthetic media companies such as AI voice cloning pioneer ElevenLabs. The London-based startup just raised an additional $80 million from some of Silicon Valley’s best-known venture capital funds, including Andreessen Horowitz and Sequoia Capital, in a deal that values it at $1.1 billion.

Many people have suggested that “digital watermarking” will help turn off the spigot on the firehose of falsehood AI generates. But another story from this week shows why that is unlikely to be the case. Samsung’s new Galaxy S24 smartphone (which my Eye on AI colleague Sage Lazzaro reported on in Thursday’s newsletter) comes with generative AI photo editing tools. Realizing that this would raise concerns about the ease of creating manipulated scenes and deepfakes, the company decided that all images created with its AI editing tools would bear a visible digital watermark—an icon with four star shapes—in the lefthand corner. But, as journalists for Gizmodo quickly reported, the digital watermark can be easily removed using (checks notes) the phone’s own AI editing tools.

Now, the S24 also automatically adds metadata to the image file that indicate AI effects have been added. This metadata may be a bit harder to scrub clean than removing the watermark. But probably not that hard. And this is why digital watermarking, which has now been rolled out by Adobe, Microsoft, and others, is no silver bullet for the problem of deepfakes and other kinds of generative AI fraud. It is simply too easy for bad actors to figure out ways to remove the watermarks or bypass the watermarking process. (Compared to images, there is even less consensus about exactly how to watermark AI-generated text and audio. OpenAI was working on a cryptographic-based watermark for the text ChatGPT produces, but it has not been implemented in production, and some researchers have already figured out ways to get around it.)

This brings me back to my conversation from a few weeks ago with Getty Images CEO Craig Peters. First, Peters related an astounding statistic that attests to the urgency of the problem: There have been more images produced with AI in the past 12 months than all the photographs taken in the history of lens-based photography. Think about that for a second. Peters says what’s needed is a layered approach to authenticity. He’s in favor of metadata that provides an indication of photo manipulation, but knowing metadata can be altered, he says this alone is insufficient. In addition, he says there should be a global effort to create a provenance standard that includes a cryptographic hash stored in an immutable database that anyone could check to verify if an image is AI-created or if it comes from an authentic source.

Currently, Adobe has been promoting a Content Authenticity Initiative that includes encrypted metadata to track the provenance of images, along with related “Content Credentials” for AI-generated ones. Both are based on a cryptographic standard called C2PA. Besides Adobe, hundreds of organizations have signed up to this standard, including most notably camera-makers like Nikon and Leica, as well as Microsoft, which labels all of Bing’s AI-created images using C2PA-compliant Content Credentials. But, as Peters notes, C2PA is itself a flawed standard and there have already been cases where people have managed to change the metadata of AI-generated images to make them appear legitimate. He says we need something better. Getty is working on it, he says—but it isn’t there yet.

So, no watermarking is not going to save us this election year. We’ll need to muddle through another year of post-truth, our skepticism ramped up to 11. But we better find a solution soon. Distrust is insidious and corrosive to democracy and society.

With that, here’s more AI news.

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

This story was originally featured on Fortune.com