1.1. The Unexpected Challenges of Detecting Deepfakes

Until recently, content generated by AI was characterised by relatively low quality and contained easily identifiable errors, such as distorted facial features, unnaturally rendered skin, or hands depicted with more than five fingers. However, advancements in machine learning techniques and the increasing computational power of modern systems have made contemporary deepfakes nearly indistinguishable from authentic content, exhibiting an exceptionally high level of realism.

It is important to note that not all deepfakes produced today achieve hyperrealistic quality, but many of these videos are consumed on mobile devices, where smaller screen sizes and lower resolution settings can mask imperfections and make distinguishing them from real content significantly more challenging.

Detection of deepfakes is a complex issue. Research shows that human ability to recognise deepfakes varies considerably, with detection accuracy ranging from 57% to 89%. This suggests that even in the most optimistic scenario, individuals fail to identify 11 out of every 100 deepfakes, while in the most pessimistic case, as many as 43 out of 100 deepfakes go undetected [4].

1.2. Deepfake Technology as a Tool for Cybercriminal Activity

Although the online dissemination of false content is not a novel phenomenon, the advent of deepfake technology has enabled cybercriminals to engage in malicious activities on an unprecedented scale. Deepfakes are employed in a range of criminal endeavours, including extortion, reputational damage, the manipulation of political processes, disinformation, and financial frauds. According to the assessment conducted by the authors of the publication ‘AI-enabled future crime’, deepfake technology may inflict the most substantial harm and yield the greatest potential profits for cybercriminals [5].

Cybercriminals frequently integrate deepfake technology with social engineering strategies to enhance the effectiveness of their activities. They exploit various human psychological vulnerabilities, such as the ease of eliciting trust, the propensity to act under time pressure, and susceptibility to suggestion. By doing so, cybercriminals can construct deceptive narratives and influence the behaviours of potential victims, ultimately aiming to gain access to sensitive information.

Deepfake technology poses threats not only to individual users but also to businesses, government institutions, and international organisations. By exploiting human beings’ natural tendency to trust the authenticity of perceived images and sounds, cybercriminals destabilise cognitive processes, affecting emotions, attitudes, and perceptions of reality. As a result, victims of such manipulation often make irrational decisions and engage in behaviours based on false premises, potentially leading to severe personal and professional repercussions.

In March 2022, as reported by the Reuters news agency, a deepfake of Ukrainian President Volodymyr Zelensky appeared on the social media platform X, calling on Ukrainian soldiers to lay down their weapons. It can be inferred that the purpose of this deepfake was to undermine morale and create confusion during an active armed conflict. Although the footage was almost immediately identified as inauthentic, its deceptive nature was not immediately apparent to all viewers – particularly older people or those less familiar with technological advances. This example illustrates how dangerous the spread of false content on social media can become in times of armed conflict and highlights the serious social and political repercussions that can result.

In recent times, the Russian Federation’s production and large-scale dissemination of deepfake content has become increasingly intense. Merely one day following the terrorist attack of 22 March 2024, a fabricated video emerged in which the image of Oleksiy Danilov, Secretary of the National Security and Defense Council of Ukraine, was integrated into a format resembling a professional television interview. This footage served as part of a broader disinformation campaign aimed at attributing responsibility for the incident to Ukraine. To create this material, archived footage from 16 March 2024 – originally depicting Kyrylo Budanov, the head of the Main Directorate of Intelligence of the Ministry of Defense – was repurposed by substituting his likeness. Despite propagandists’ efforts, the quality of the fabricated material remained low. Advanced voice cloning tools and lip sync techniques were employed to synchronise mouth movements with the manipulated statements; however, the final result was far from seamless. Visual distortions, unnatural synchronisation of lip movements with speech, and notable blurring – especially around the neck and mouth – remained discernible to vigilant observers. It must be acknowledged, however, that not all audiences are capable of detecting such subtle signs of interference. A previous attempt at using a similar disinformation strategy was recorded in October 2022, involving voice cloning to gain access to confidential information from the drone manufacturer Bayraktar. In this instance, perpetrators impersonated Ukraine’s Prime Minister, Denys Shmyhal, but their attempt was thwarted. This underscores the need for continuous improvement in methods of detection and countering such threats.

Recently, there has been a marked intensification in the use of fraudulent investment advertisements as tools for extracting personal data and financial resources. Such content appears on social media platforms, websites, and even as sponsored advertising materials. A defining characteristic of these schemes is the promise of rapid profits with minimal risk, often reinforced by the use of recognisable figures from the worlds of politics, business, entertainment, or finance.

Fraudsters increasingly produce complex audiovisual materials whose aesthetics and format resemble those of news programmes. They incorporate the likenesses of well-known presenters and journalists, who appear to endorse ‘investment opportunities’ or prompt viewers to take specific actions, such as clicking on a link or downloading an application.

According to information published in the British newspaper The Guardian, in early February 2024, the Hong Kong police reported that cybercriminals had employed deepfake technology to steal nearly £20 million. An investigation was launched following a report from an employee of a British company operating a branch in China, who informed the police that she had been coerced into transferring a significant sum of money into bank accounts designated by individuals posing as high-ranking company officials. Prior to executing the transfer, the employee had participated in a videoconference with the chief financial officer and other members of the management team. The investigation subsequently revealed that the individuals participating in this meeting were generated by AI [6].

Cybercriminals produce a diverse array of materials, each tailored to distinct target audiences. To promote fictitious investment ventures, they frequently use the likenesses of politicians and business leaders, intending to foster trust among individuals interested in traditional forms of investment. In contrast, content featuring celebrities and influencers are commonly deployed in advertising mobile applications – such as those simulating online casinos or games – primarily attracting younger audiences seeking entertainment. For older or unwell individuals, cybercriminals create deceptive ‘miracle’ drug or medical procedure advertisements, capitalising on the credibility associated with renowned physicians, athletes, or religious figures. Among the widely publicised cases reported by the media are as follows:

All the above examples underscore the growing significance of fraudulent investment advertisements as tools employed by cybercriminals. Given the substantial social harm caused by such offenses, it is imperative to conduct in-depth research into the mechanisms governing the creation, distribution, and reception of this type of content, as well as to implement effective educational programmes and advanced technological measures. Such actions are crucial for reducing the scale of losses suffered by potential victims and for enhancing security within the digital environment.

It should be emphasised that deepfakes and social engineering are mutually reinforcing phenomena, giving rise to a new generation of threats characterised by a high level of technological sophistication and effectiveness in manipulating human behaviour. Unfortunately, conventional protection methods, which predominantly rely on technological safeguards, have proven insufficient. Consequently, countering deepfakes necessitates an interdisciplinary approach that integrates psychological, technological, and legal expertise.

3.1. Deepfake Selection and Transcription

Given the objectives outlined in this study, a deepfake video observed on social media platforms in Poland between May and July 2024 was selected to serve as the central case study. The sample was limited to Polish social media platforms to tailor the analysis to the local cultural and social context. The sample selection in our study was intentional. This video was purposefully chosen to ensure that it provided a diverse and comprehensive representation of deepfake-related content, encompassing a wide range of narrative structures, emotional triggers, and psychological attributes. Transcript analysis, therefore, incorporated both qualitative and quantitative methodologies, offering insights into the persuasive techniques at play, including mechanisms rooted in emotional appeal, authority, and social proof.

In the second stage, the researchers conducted the transcription of deepfake videos. The study employed automatic speech recognition (ASR) models to transcribe audio from deepfake videos. These models, often underpinned by large language models (LLMs), are specifically designed to accurately transcribe audio recordings into written text.

It should be emphasised that one of the primary challenges faced by ASR models is their limited ability to process noisy audio recordings. In cases of low-quality audio, where background noise, music, or other interferences are present, transcriptions may be incomplete or inaccurate. Under such conditions, models often generate extraneous elements in the text, such as phrases like ‘subtitles sounds...’ or other artifacts caused by misinterpretation of background noise. Another significant challenge arises when interpreting words that are mispronounced or articulated in an unusual manner. ASR systems tend to substitute such words with alternatives that better align with the surrounding context. This can result in distortions, particularly when proper nouns, technical terms, or slang expressions are replaced with incorrect equivalents.

Furthermore, transcriptions may contain repetitive words, sentences, or even entire speech segments. These repetitions often stem from the model’s uncertainty regarding the interpretation of specific audio fragments or errors in the speech recognition algorithm. ASR models also struggle with recordings that feature shifts in accent or pronunciation. This issue becomes particularly pronounced when the speaker’s intonation changes or when an accent characteristic of another language is introduced, such as Polish speech interspersed with Russian or English accents. In such scenarios, models may generate what are referred to as ‘linguistic hallucinations,’ introducing foreign language fragments into the transcription [10].

3.2. Cialdini’s Persuasion Strategies

This study applied Robert Cialdini’s six principles of social influence – reciprocity, commitment and consistency, social proof, authority, liking, and scarcity – to analyse deepfake [11]. The deepfake’s transcript was analysed by a psychologist, who, drawing on their knowledge of influence techniques and professional experience, classified specific sections of the text as particularly persuasive. This assessment considered key persuasion mechanisms, such as authority, social proof, emotional appeal, and the principles of scarcity and reciprocity.

The principle of reciprocity refers to the innate human tendency to reciprocate benefits received from others, even in the absence of necessity. The analysis focused on identifying elements within deepfake videos that could evoke a sense of obligation or an inclination to reciprocate in the viewer. For instance, the videos may imply exclusivity in the presented information, potentially prompting viewers to reciprocate by further sharing the content.

The principle of commitment and consistency emphasises individuals’ tendency to maintain alignment between their actions and decisions over time. The study examined whether the deepfakes employed techniques designed to prompt initial low-commitment actions, such as liking or sharing content, which could subsequently foster greater engagement.

The principle of social proof is based on the influence of others’ behaviours on an individual’s decision-making process, particularly in new or ambiguous situations, where people tend to follow the actions of others as a guide. The analysis assessed whether the deepfake materials incorporated elements, such as positive comments, or references to perceived broad social support, which could amplify the message by creating the impression that the stance presented is widely accepted and endorsed.

The authority principle is based on the tendency of individuals to place trust in and act upon information provided by perceived experts or leaders. The analysis examined whether the deepfake materials featured prominent figures, such as politicians, scientists, or opinion leaders, leveraging their perceived authority to enhance the credibility and impact of the message.

The liking principle refers to the idea that individuals are more likely to be persuaded by those they perceive as likable or who share similarities with them. The analysis investigated whether the deepfake materials employed references to shared cultural and social values to strengthen their persuasive impact on recipients.

The principle of scarcity emphasises the perceived value of information by suggesting its limited availability. The study analysed whether the deepfake materials strategically employed techniques implying rarity, exclusivity, or urgency in the message, which could facilitate expedited decision-making or deepen viewer engagement through psychological pressure.

4.1. Deepfake Analysis

4.2. DEEP FRAME tool

In Appendix 1, we propose a DEEP FRAME tool designed for analysts that enables a comprehensive examination of deepfake content through an interdisciplinary approach. DEEP FRAME is a self-reported tool that includes a set of questions about both technical and psychological elements of deepfakes.

The proposed tool offers a wide range of benefits, serving as a valuable resource for interdisciplinary research on deepfake content by integrating both technological and psychological perspectives. The data and analyses it generates have the potential to advance significantly the development of more sophisticated algorithms for detecting manipulated content. Beyond its contributions to research and detection, the tool holds considerable promise for public education, as the insights it provides can support the design and implementation of effective awareness campaigns. Moreover, it strengthens our capacity to understand emerging threats and devise targeted strategies to mitigate their impact. By combining elements from computer science, psychology, and communication studies, it ensures a more thorough evaluation of the subtle and overt manipulations embedded in digital media.

The DEEP FRAME tool facilitates the systematic collection of knowledge about deepfakes. The collected data is categorised based on key parameters, such as the type of manipulation, the technologies used, and the context of publication. A technical module provides analysis of the quality of video and audio, potential artifacts of manipulation. A unique feature of the tool is its ability to conduct psychological evaluations of deepfake content. This module identifies persuasive techniques. Such analyses deepen our understanding of the manipulative mechanisms employed in deepfake content. The DEEP FRAME tool enables the monitoring of global trends in the use of deepfake technology, helping to identify emerging threats.

5.1. Limitations

The study was limited to a single case, which may not reflect the full diversity of situations occurring in the deepfake phenomenon. The results may be difficult to generalise to a larger population or other contexts. The results depend on the quality of the data available on the case being studied. Short deepfakes that last, for example, for 1 minute will have significantly fewer persuasive fragments compared to longer materials that last, for example, for 3 minutes. However, the authors of the study plan to conduct further in-depth research in the future on the phenomenon of using influence methods in deepfakes.