Garrigues Digital_

The expansion of generative AI not only broadens the scope for criminal activity, but also calls into question the limits of the Spanish Criminal Code and tests the reliability of digital evidence, compelling a fundamental rethinking of both the prosecution of offences and the standards of authenticity in judicial proceedings. 

Generative artificial intelligence (AI) has transformed the manipulation of images, voice and video into a tool readily accessible to criminals of all kinds. “Deepfakes” are no longer a laboratory phenomenon: they are employed to perpetrate large-scale fraud, automated cyberattacks and identity theft, and may even be used to manipulate evidence in judicial proceedings. We examine the principal criminal phenomena, the difficulties faced by legal practitioners in prosecuting such conduct, and the challenges posed by the obtaining, authentication and assessment of digital evidence in a context where synthetic content is increasingly indistinguishable from reality.

The response to deepfakes under Spanish criminal law

Pending legislative reforms to adapt the current criminal law framework, in both its substantive and procedural aspects, to the new forms of criminal conduct arising from the advent of generative artificial intelligence and, in particular, deepfakes, it is essential, in order to delineate which acts may potentially give rise to criminal liability, to take as the starting point the EU standard set out in the definition provided by Article 3(60) of the Artificial Intelligence Regulation (AI Act), which defines a “deepfake” as “an AI-generated or manipulated image, audio or video content that resembles existing persons, objects, places, entities or events and would falsely appear to a person to be authentic or truthful”.

From a technical-criminal perspective, generative AI has drastically reduced the barriers to entry for the commission of a wide range of offences: computer crimes and cyberattacks, fraud and scams, forgery and identity theft, and even the potential manipulation of evidence in judicial proceedings. Such conduct, until recently the preserve of offenders with advanced technical capabilities, is now within the reach of any user owing to the Crime-as-a-Service (CaaS) model, which provides the offender, through clandestine markets such as the deep web, with either access to computer tools for generating synthetic content or the finished product itself, tailored to their specific criminal modality or criminal plan. In this manner, the capacity and sophistication of unlawful conduct is exponentially increased, both as regards its commission and its potential prosecution and proof at trial.

The pace of technological evolution has been extraordinary. Current deepfakes no longer exhibit the visible defects that characterised earlier generations of synthetic content. Photographs, video, audio, complete avatars and even live video calls may be indistinguishable to the average observer. Voice cloning has attained a level of sophistication that enables the reproduction of intonation, rhythm, pauses and even breathing patterns from very little original audio material (in some instances, merely a few seconds). This technical reality has direct legal consequences: the more perfect the simulation, the more difficult it becomes to detect the falsity and, consequently, the greater the potential harm of the conduct.

Cyberattacks, fraud and scams: AI as a multiplier of property crime

Artificial intelligence serves not only to generate false content with the appearance of reality but has also become a powerful instrument for the automation of cyberattacks and large-scale fraud. Tools specifically designed for unlawful purposes enable offenders without advanced technical knowledge to acquire kits purpose-built to launch massive and automated phishing campaigns or the indiscriminate sending of messages (email, WhatsApp, etc.) with the aim of infecting a computer or information system and subsequently demanding a financial ransom in order to recover said systems (ransomware). The emergence of generative artificial intelligence has resulted in a reduction in costs and greater accessibility for offenders lacking technical expertise. The principal and immediate consequence thereof has been the launch of massive campaigns against undertakings of all kinds, with a degree of sophistication previously reserved to organised crime groups.

In the specific field of criminality, generative AI is redefining and driving the commission of property offences (principally fraud) through three main methods: identity impersonation using voice or video deepfakes, algorithmically generated falsifications, and manipulation chatbots that exploit the trust of victims through social engineering techniques refined by AI.

In recent months, we have witnessed the following instances of increasing sophistication in cyber fraud:

CEO Fraud. The so-called “CEO fraud” — whereby offenders seek to mislead an employee of a company who has authority to effect payments into making a transfer to an account controlled by the offender under the guise of a purported instruction from a superior — is by far the most prevalent form of fraud targeting businesses. However, we are already confronted with cases in which the deception is no longer confined to sending an email from an account with a domain similar to a legitimate corporate domain, but extends to voice notes or telephone calls in which the offenders clone the voice of the relevant executive, or even video calls in which the supposed executive was in fact a synthetic avatar generated in real time by means of AI, a deepfake.

Ransomware. The second most frequent attack suffered by companies — the infection of a computer or information system with malware or malicious software for the purpose of subsequently demanding a financial ransom in order to recover said systems — is also being subject to increasing sophistication, principally as regards the means of access to the company’s information system, through the use of communications which, on occasion, incorporate algorithmically generated falsifications that are extremely difficult to detect and which are tailored to their intended recipient on the basis of both personal and professional information available from public sources or obtained through a prior phishing attack and the subsequent application of social engineering techniques refined by AI.

These criminal phenomena, together with many other so-called “cybercrimes” in which the use of generative AI and, in particular, deepfakes may have particular incidence, are not always readily capable of being subsumed within the existing offences of the Spanish Criminal Code (CC), such as the various forms of fraud — common fraud (Article 248 CC), computer fraud (Articles 249 and 251 bis CC) or procedural fraud (Article 250.117ª CC) — identity theft (Article 401 CC), documentary forgery (Article 390 CC) and computer forgery (Article 400 CC) or criminal damage (Articles 264 to 264 quater CC);

This compromises both the response that the criminal justice system can provide to such criminal conduct and the protection it can afford to those who fall victim to the use of technologies that exponentially increase the scale of fraud, the credibility of the deception, the automation of execution and the difficulty of detection and subsequent prosecution.

Procedural fraud, identity theft and evidence tampering by means of ‘deepfakes’

One of the most serious risks posed by deepfakes to our justice system as a whole is their potential use as false evidence in judicial proceedings, with the capacity to mislead the judge or tribunal. Documents, videos, audio recordings and images generated by AI may be presented before a judicial body with the appearance of authenticity, thereby compromising the integrity of the proceedings and, ultimately, the lawfulness of the court’s determination.

Procedural fraud by means of deepfakes constitutes an aggravated form of process manipulation: the perpetrator not only deceives the opposing party, but instrumentalises the judicial body itself as a means of obtaining an unjust decision and does so, moreover, by availing himself of an artifice (the deepfake) which, by virtue of its degree of realism, is particularly effective in achieving the intended deception. This heightened culpability of the attack upon the proper administration of justice by one who commits procedural fraud through the use of deepfake techniques ought to find a proportionate reflection in sentencing, whether through the specific criminalisation of such conduct or the introduction of an aggravated sub-type.

The increase in hearings and witness, expert and defendant appearances conducted by remote means, to the detriment of personal attendance, driven by recent legislative reforms, offers significant advantages in terms of logistics and procedural efficiency, but also entails substantial risks associated with the use of AI or deepfakes, and it is only a matter of time before we begin to witness cases that necessitate the systematic verification of the authenticity of all audiovisual evidence.

The evidential risk is compounded by the fact that any generative AI system accessible to the public can today create images, altered video, documents or invoices indistinguishable from genuine ones. This compels judges, prosecutors and lawyers to ask themselves systematically whether the evidence presented is authentic, has been modified, or is an outright algorithmic fabrication. The traditional practical presumption of authenticity attaching to audiovisual media is no longer tenable.

‘Deepfakes’ in judicial videoconferences. In Spain, the objective set by the Ministry of Justice is for remote appearances to increase progressively, to the detriment of in-person attendance. This decisive commitment to remote appearances widens the risk of fraud and requires the implementation of controls. Real-time deepfakes in videoconferences enable a person to impersonate another during a statement or an oral hearing. How can it be ensured that the person appearing remotely is truly who they purport to be? How can a synthetic avatar be detected during a judicial hearing conducted by videoconference?

To mitigate these risks, numerous proposals exist based on verification systems already in use in other domains, such as two-factor authentication, the use of qualified electronic certificates or dynamic biometric recognition (requiring random gestures or movements from the person appearing), and it would even be desirable to implement automated systems for real-time detection of synthetic content, so as to guarantee the identity of those appearing before the courts.

Difficulties in obtaining and assessing evidence. The chain of custody of digital evidence is compromised when metadata can be easily deleted or falsified. Watermarks (watermarks) implemented by certain platforms are vulnerable and do not constitute a sufficient safeguard. The early preservation of evidence, by means of qualified time-stamps, access logs and cryptographic digital fingerprints, is critical.

Expert computer forensic evidence will be increasingly necessary, but it presents inherent limitations: automated detectors of synthetic content yield significant rates of false positives (genuine content erroneously identified as false) and false negatives (deepfakes that go undetected). This technical uncertainty translates into evidential uncertainty within the proceedings.

The “liar’s dividend”. Another equally pernicious effect of the emergence of deepfakes is that any party to proceedings may deny the authenticity of legitimate audiovisual evidence by alleging that it constitutes AI-generated content. This possibility, as real as it is increasingly frequent, erodes general confidence in audiovisual evidence and may seriously impede the formation of judicial conviction. The courts will need to develop robust assessment criteria that reconcile the right of defence with the need to preserve the probative efficacy of digital media.

Digital identity theft. Beyond procedural fraud in the strict sense, deepfakes enable the complete usurpation of the identity of a third party in the digital environment. A person may impersonate a public official, a chartered professional or even a party to proceedings by means of a synthetic avatar indistinguishable from the original.

The convergence of all these risks (procedural fraud, evidence tampering, impersonation in remote hearings and identity theft) presents a landscape in which the integrity of judicial proceedings is vulnerable to technologically sophisticated attacks that are difficult to detect using current mechanisms. The response must necessarily be multidimensional: regulatory, technological and educational.

Difficulties in criminal prosecution: a crime without borders

The prosecution of offences committed by means of deepfakes confronts a set of structural obstacles that seriously impede the operation of criminal justice. The following table summarises the principal obstacles:

To these obstacles must be added the difficulty of obtaining the effective cooperation of technology platforms that host the content or provide the services used for the commission of the offence. Response times to judicial requests frequently exceed the useful time window for the preservation of evidence, and regulatory differences between jurisdictions complicate the enforcement of court orders.

Proposals: towards a criminal law framework adapted to the synthetic era

In light of the deficiencies identified, it is imperative to advance along the following lines of action:

1. Specific criminal offences. To create specific offences for non-consensual sexual deepfakes and for identity impersonation by means of synthetic content, thereby avoiding the current difficulties of subsumption within offences not designed for this reality.

2. Generic aggravating circumstance for the use of AI. To incorporate an aggravating circumstance for the use of artificial intelligence or deepfakes in the commission of offences, particularly fraud and procedural fraud, enabling an increase in the penalty where the use of synthetic technology heightens the culpability of the act or the gravity of the result.

3. Liability for tools with an unlawful purpose. To establish criminal liability for those who develop, distribute or commercialise tools whose exclusive or principal purpose is the production of unlawful synthetic content, distinguishing them from general-purpose tools.

4. Reinforced obligations for platforms. To impose effective obligations for the preservation of digital evidence, proactive cooperation with the judicial authorities and the immediate removal of unlawful synthetic content, with proportionate sanctions in the event of non-compliance.

5. Robust judicial authentication mechanisms. To implement identity verification systems within the digital judicial environment combining multi-factor authentication, dynamic biometrics and automated real-time detection of synthetic content, thereby safeguarding the integrity of remote hearings.

6. International cooperation. To promote a binding international instrument on AI and criminal law addressing the cross-border obtaining of data from AI systems for evidential purposes, adapted extradition mechanisms, expeditious mutual legal assistance, harmonisation of sanctions, criteria for authorship and complicity in the digital environment, and fundamental rights safeguards.

The procedural dimension: protocols for the assessment of synthetic evidence

The emergence of deepfakes demands a fundamental rethinking of the criteria for the admissibility and assessment of audiovisual evidence in criminal proceedings. The traditional presumption of authenticity attaching to audiovisual media has been overtaken by technological reality. The courts require new protocols adapted to this new reality.

In the first place, it is advisable that all audiovisual evidence of relevance to the proceedings be subjected to a preliminary authenticity screening whenever any party calls its origin into question. Such analysis may be conducted by means of forensic tools for the detection of digital manipulation, verification of metadata and examination of the chain of custody from the moment of capture to its incorporation into the court file.

In the second place, judicial training in the field of digital evidence must be intensified, and in particular concerning what a deepfake is, how it is generated, what its technical limitations are and what indicators may alert to its presence. Without such training, there is a risk that false evidence may be admitted and relied upon or, at the opposite extreme, that legitimate evidence may be rejected on the basis of a generalised and unfounded distrust.

In the third place, it is appropriate to explore the implementation of provenance certification systems for relevant audiovisual content that would enable the incorporation of verifiable provenance metadata at the point of content creation. Should such standards become widespread, they would be capable of providing an additional layer of confidence in the authenticity of digital evidence.

Finally, it is necessary to develop specific protocols for the challenge and cross-examination of audiovisual evidence. The defence must have effective access to independent experts capable of analysing suspect content, and procedural time limits must be adapted to the technical complexity required by the forensic analysis of potentially synthetic content.

A necessary technologically informed and internationally coordinated response

The increasing sophistication of deepfakes poses a significant threat to society in general and to businesses in particular, as the primary targets of cyberattacks and fraud attempts. This demands not only the implementation of protective measures and the provision of training and advice in these matters by businesses, but must also involve the legislature and all legal practitioners in acting not merely through the criminalisation of new conduct, which is undoubtedly necessary, but also in rethinking the evidential and prosecutorial mechanisms for offences committed through the use of generative AI in general, and deepfakes in particular.

The boundary between the real and the synthetic has become blurred, and with it the certainties upon which confidence in evidence, in identity and in procedural truth is founded. The response must be comprehensive, proportionate, technologically informed and coordinated at European and international level, and must foster a culture of verification that permeates the entire justice system, if we are to preserve the fundamental guarantees of criminal proceedings in the era of generative artificial intelligence.