A few days ago I read an article in Il Post about a South Korean YouTuber arrested for defamation after sharing an AI-generated audio that wrecked the reputation of two of his fellow actors. The same day I came across two very interesting pieces on the problems of AI-generated content: a hands-on guide to analysing and spotting deepfakes, and a reflection on the legal side.
The case
On 27 May, Kim Se-ui, a South Korean YouTuber with nearly a million subscribers, was arrested for defamation. In March 2025 he had published an AI-generated audio attributed to a deceased actress, Kim Sae-ron, using it to insinuate a relationship with actor Kim Soo-hyun back when she was still a minor.
The fallout for the actor? Lost contracts, cancelled productions, a postponed Disney+ series, and the actor ending up in psychiatric care.
If you work in OSINT, this news story is yet another warning of how increasingly hard it is to tell whether the video or audio in front of you is real or generated by an LLM.
Until a couple of years ago, producing realistic fake video and audio took resources and skills the general public didn’t have. Then, in May 2025, Google unveiled Veo 3, letting anyone get eight seconds of video with audio, dialogue and a soundtrack in two minutes, from a text prompt. You’ll probably remember the “prompt theory” wave on social media, with synthetic characters denying they had been generated by an LLM.
Today the web is awash with tutorials on the best tools and prompts to create content indistinguishable from the real thing. Not to mention how easy it is to clone someone’s voice and make them say anything at all. Ever tried Voicebox? Do it! It is impressive!!
We are well past the “you can tell by eye or ear” line: the obvious giveaways are getting rarer (hands with six fingers, the stiff alignment of eyes and lips, mechanical audio, and so on).
So what? What can we do to tell whether the video in front of us is real or artificial?
As I said, that same day I found a couple of interesting articles on exactly this.
An operational process
The first article is about a forensic workflow proposed by Massimo Iuliani of Amped Software.
Iuliani’s thesis is that you cannot establish a deepfake using a single tool. AI detection tools can only do part of the job: they are useful for triage — filtering large volumes, setting priorities — but they are not suited to producing “a source of evidence”, because they have limited explainability and error rates that vary by domain (a model tuned on Caucasian faces gets it wrong more often on Asian ones, and collapses on media compressed by WhatsApp or Telegram).
Analysing a deepfake should be approached as a forensic workflow, not as a single act performed by a single tool.
- The goal is to find traces of manipulation and produce explainable, reproducible and defensible results.
- AI detection alone is not a source of evidence: it is the starting point of the forensic examination, not a verdict.
- The scientific analysis of images and video rests on method, validation and consistency, especially when the results have to support an investigation or legal proceedings.
- A structured workflow must allow you, starting from an initial AI alert, to produce documented, corroborated and defensible conclusions.
- A robust workflow must combine AI triage, analysis of the metadata and of the video/audio format, checks on the video content (geometry, reflections, pixel-level errors, and so on).
In other words, Massimo too stresses that building a process must always take priority over using a single tool or piece of software. Once we have developed a process suited to our needs, we can look for the tools that support its individual stages. But the point stays the same: no single tool, on its own, is THE DEFINITIVE TOOL — the process is what counts.
Let’s apply the process to the Korean case
Let’s bring this method down to the case we started from.
A caveat: the Korean content is mostly audio, only partly images — screenshots of KakaoTalk chats — whereas Iuliani’s workflow is built for images, and audio and voice cloning forensics are a discipline of their own. But the method, that one can be transferred.
And the Korean case proves it better than any explanation.
Because here a single check did not settle it. The National Forensic Service, South Korea’s national forensic body, in November 2025 said it could not confirm whether the audio had been altered with AI: inconclusive. The Gangnam police, on the other hand, at the end of a broader investigation concluded that the recording was entirely fabricated with AI. Two forensic analyses, two different outcomes on the same file: this is exactly Iuliani’s point — no detector, on its own, is the verdict.
What moved the needle was not a tool handing over the smoking gun “it’s AI”, but the convergence of different analyses. On one side the audio, a voice cloned with AI; on the other a much more old-school, not-at-all “smart” manipulation. The police found that many of the KakaoTalk screenshots had been doctored by hand, swapping the name of the conversation partner with Kim Soo-hyun’s.
Two fakes, two techniques — voice clone and photo editing — not a single video. And that is exactly the point of the workflow: only by combining several tools, experience and intuition can you analyse the context — audio, images, metadata — and reach a solid conclusion. And often that is not enough. Often proving the truth is not enough: the reputational and moral damage done to the victim is frequently almost irreparable.
What the law says about deepfakes
A heads-up before we dive in: everything in this section refers to Italian law. Other countries have their own — often very different — rules on AI-generated content.
This is where Alessandro Milone comes in. In his article Il diritto penale alla prova dell’intelligenza artificiale: alcune riflessioni intorno all’art. 612-quater c.p., he explains how Law 132/2025 introduced art. 612-quater, which states that anyone who distributes, without consent, AI-falsified images, voices or videos capable of deceiving as to their authenticity, causing unjust harm, faces one to five years in prison. The offence is prosecuted on the victim’s complaint (querela).
There is also a new general aggravating circumstance, art. 61 no. 11-undecies, for when AI is an “insidious means”.
One distinction that matters. The rule targets whoever transfers, publishes or distributes, not whoever merely creates the deepfake: it takes distribution without consent and unjust harm. And here is a point that hits close to home: even simply re-sharing material produced by others can constitute the offence, if the harm and the lack of consent are there. Re-sharing a fake “just for a laugh” may not be so harmless.
Unjust harm is the constitutive element: without concrete harm to the person, 612-quater does not kick in — it is not enough that the image is fake. But mind what is meant by harm: it lies not only in distributing the fake content, but in the loss of credibility of the person portrayed, who finds themselves having to “prove their own innocence” before an image that, apparently, testifies the opposite. It is the reversal of the reputational burden: the fake weighs more than the denial.
Milone notes that the new article has a broader reach than the one on revenge porn (art. 612-ter): the protected interest is the dignity and moral freedom of the person. And when the fake targets a public authority because of the functions they exercise, prosecution is automatic (d’ufficio) — a sign that the legislator is also looking at political propaganda and disinformation.
Two slippery boundaries Milone flags, which apply to us too. The “capable of deceiving” requirement makes the line with satire and political criticism blurry: an obviously comic video deceives no one. And the harm must be not only caused but intended; proving that, given the unpredictability of AI systems, is no small thing.
It is worth pausing on why distribution is punished and not generation. There is a solid investigative reason: whoever creates a deepfake often works with local models using anonymisation techniques, and is hard to reach; whoever distributes it, on the other hand, is far easier to trace.
But do not read it as a “free-for-all” on creation. Milone recalls that a denigratory deepfake that starts circulating already constitutes defamation, aggravated by the use of a means of publicity (art. 595); a sexual one, revenge porn (art. 612-ter, applicable to artificial material too, as long as it is realistic); one used to deceive banks or institutions, fraud (art. 640); and propaganda can touch offences against public order (arts. 656 and 658). The new 612-quater targets distribution; what really stays out is the mere generation that never leaves the creator’s hard drive — or, as Milone notes, the case of someone who “only” fabricates the fake that is then distributed by others, escaping the new offence. A gap in protection, not a free pass.
Conclusions
The Kim Soo-hyun case is telling: it shows how a reputation and one’s peace of mind can be destroyed in a few seconds of plausible audio, and how hard it is to recover them once the truth comes out.
Even if we can no longer trust our senses alone, we are not defenceless: pixels, metadata and distribution patterns, put together, still tell the truth. There are processes, there are tools, but actually proving that a video was generated by an LLM is getting harder and harder, and it is not always punishable by law.
If the topic interests you, let’s keep the conversation going in the comments or on our Telegram channel.