Political Manipulation with Massive AI Model-driven Misinformation and Microtargeting

By Ben Gelman and Adarsh Kyadige

In today’s digitally connected world, political messaging and misinformation are becoming increasingly sophisticated. Political campaigns and misinformation efforts, particularly those that are well-funded, have significant societal impacts. These campaigns have historically exploited political and ideological views to resonate with people, convince them to act, or even lure them into scams.

Generative AI technologies such as large language models (LLMs) and large image models will likely transform the domain. Generative AI offers tools for creating sophisticated, individualized content at scale, which was previously difficult and labor-intensive. With these capabilities, the risk posed by malicious actors can reach new heights.

We have already observed a variety of actors abusing generative AI as part of ongoing fraud campaigns, including the use of generative text to send messages to scam victims, generative AI images to create deceptive social media and “deepfake” video and voice created by AI to aid social engineering of victims. These same tools have been used as part of political misinformation and deception campaigns on social media.

Given the relevance of these subjects due to ongoing elections worldwide, understanding the effect of new technology on political misinformation is particularly consequential. In this analysis, we explore one of the greatest emerging threats from malicious use of generative AI: tailored misinformation. If someone includes intentional misinformation in a bulk email, people who don’t agree with that misinformation will be turned away from the campaign. But in the method we explored in our research, misinformation is only added to the email when that specific individual is likely to agree with it. The ability to do this can completely change the scale at which misinformation can propagate.

In the research we document in this report, we aimed to uncover potential methods in which adversaries could apply generative AI tools to make impactful changes in the political sphere. These methods use current generative AI technologies in a way that can be executed at very low cost by a wide range of potential actors who wish to influence politics on a small or large scale.

This effort was based on research we’ve already conducted, in which we developed a tool that can automatically launch an e-commerce scam campaign based on AI-generated text, images, and audio to create diverse and convincing fraudulent webstores. An example of one these websites is shown in Figure 1; a full description of the research can be found here (https://news.sophos.com/en-us/2023/11/27/the-dark-side-of-ai-large-scale-scam-campaigns-made-possible-by-generative-ai/).

Figure 1: A fully functioning webstore with Facebook and payment credential theft. Everything you see is generated by AI, including code, text, images, and an audio testimonial

We believed that the sophistication of the content created by this tool demonstrated that this approach has potential in the political sphere, so we reconfigured it to generate political campaign websites with any characteristics of our choosing.

We then extended the research, putting the concepts of political emails and fraud campaigns together by generating microtargeted emails. By combining fake social media profiles with several campaign websites, we then generate personalized emails that use unique, individualized arguments to best persuade people to believe in a campaign – even if the people don’t generally agree with those ideas.

Massive model-driven misinformation

To start building towards the future of political misinformation, we first had to augment our tool that generates large scale e-commerce phishing campaigns entirely with generative models . This tool uses Auto-GPT to allow GPT to communicate with itself, as well as other agents such as Stable Diffusion, to complete a large task.

We already saw an example of a phishing website in Figure 1, but with minor modifications, we were able to reconfigure the tool to create political campaign websites. To avoid political affiliations and demonstrate more extreme campaigns, we intentionally use fantasy scenarios and worlds.

Figure 2 illustrates all the components of one of these websites.

Figure 2: Three screenshots of a single, vertical website for an Elven political campaign. (Click to expand) All the art, ideas, and text are automatically generated by AI. Picture (a) shows the top the website, with a fully functioning navigation bar, an intro to the campaign, and relevant art given the context of the campaign. Using the navigation bar or scrolling down reveals the “About Us” and “Achievements” sections displayed in (b), which describe the background of the campaign. (c) shows the bottom of the page, containing future political goals, pictures of the candidate and important team members, and a set of contacts

With a little imagination (and nothing more than changing a few words in the input to the tool), we can see that any level of fabrication is possible with these websites. This means a user could generate anything from benign campaign material to intentional misinformation and malicious threats with minor reconfiguration. It is possible to associate any real political movement or candidate with supporting any policy, even if they don’t agree. Intentional misinformation like this can make people align with a candidate they don’t support or disagree with one they thought they liked. Furthermore, the tool can generate hundreds of diverse websites with the press of a button, making the threat from small organizations or even individuals a relevant concern.

We took advantage of our tool to create four fantasy campaigns with different alignments, generally touching on different parts of the political spectrum. Because we will be combining social media profiles with these campaign websites to generate microtargeted emails in the next section, we wanted to ensure there would plenty of conflicting views. Thus, including some more extreme perspectives will allow us to showcase the depth to which the AI can deceive people who disagree with the perspectives. Figure 3 summarizes the general ideas of each campaign.

Figure 3: Summaries of the four fantasy campaigns. (Click to expand) (a) displays a relatively progressive agenda for the Elven campaign: supporting education, renewable energy, research and innovation, and social equality. In picture (b), the Dwarven candidate is focused on the economy with goals such as encouraging trade, strengthening mining industries, lowering taxes, investing in more infrastructure, and increasing military spending. The Orcish campaign(c) has more national security focused goals—after successfully expanding their territory through warfare and establishing defense agreements, they want to have a strong focus on the military with policies such as mandatory service, continued defense spending, and support for veterans. Finally, picture (d) explains the goals of a thoroughly authoritarian demonic campaign. They have already conquered several realms, and they seek complete domination through military action. They want to levy taxes against the defeated and develop a portal system to ensure they can take swift action against potential dissidents

 

Large-scale microtargeting

With the campaigns completed, we need social media profiles before we analyze the ability for LLMs to specifically target individuals. The user profiles simulate data harvested from social media and other public domains, while the fantasy websites provide a topic to strive for. We generate synthetic user profiles with a simple yet effective prompt to OpenAI’s GTP-4o model, “Please generate a detailed user profile. Make it seem realistic. Name, location, age, occupation, likes, dislikes, frequent places, hobbies, political leanings, and anything else that you think is relevant. Make it seem as if the data were realistically scraped from social media.”

The resulting profiles include significant amounts of detail, so we summarize some of the key information in Table 1. We selected five user profiles that fall across the political spectrum. Next, we combined the campaign websites with each individual user profile in a prompt requesting a targeted email.  Using the same website with different profiles demonstrates the ability of LLMs to microtarget users with persuasive arguments best suited to the users’ preferences. The resulting 20 emails will allow us to analyze gradual changes in microtargeting as the political views of the campaigns/individuals diverge.

The technical procedure is simple: Include both a synthetic user profile and a campaign website as input to an LLM, with a prompt asking the model to write an email that would be most persuasive for convincing a user with the given interests to support or donate to the political campaign. The full prompt is written out in the appendix at the end of this post. Of note is the use of chain-of-thought prompting, which improves LLM performance by asking the model to write out the steps of the “thought” process before generating the final output. In our prompt, asking the model to cite specific examples from the user profile ensures a deeper targeting of the individual in the resulting email.

In the email displayed in Figure 4, we combine the avaricious Dwarven campaign (Figure 3b) with the moderate user (Table 1-3), and the result showcases one of the biggest threats that AI-based microtargeting introduces: You can perform misinformation at a scale that is simply infeasible with existing methods. The model fabricates a point contrary to the Dwarves’ goals, saying the campaign supports social justice issues because it appears on the moderate user’s profile (User 3). This doesn’t work without personalization, because a mass email about supporting social justice may actually lose people who don’t vote or donate based on that issue. By fabricating that point specifically to people who support it, the lie is more insidious and effective.

Table 1: Overview of AI-generated social media profiles of users across the political spectrum. (The full profiles are extensive, so we only include highlights here.)

Figure 4: Microtargeting email directed at User 3 by Campaign B. Green text show actual beliefs of the campaign, while red highlights show conflicting points created to convince the specific user

Furthermore, the LLM can use other language tools besides outright lies, such as half-truths or lies by omission, to match campaigns with disagreeing users.  We see some particularly interesting examples of this when we combine the military-focused orc campaign (Figure 3c) and the conquering demons (Figure 3d) with the progressive user (Table 1-1). In this case, the orcs claim to invest in technology, but conveniently leave out that the investment is entirely in military technology. Without that extra information, the user may be inclined to have a positive view of the investment. Similarly, the demons claim to support the unity and celebration of tradition but leave out the fact that they intend to unify and force everyone they conquer to exclusively celebrate the demon traditions. Figure 5 shows excerpts from the emails that highlight these points.

Figure 5: Lies by omission that the model employs to persuade a disagreeing user.

Between the fabrications, half-truths, direct agreements, and ability to hide the detailed personal knowledge about the users, we can see that these techniques are particularly chilling because users may interpret the most effective microtargeting as serendipitous coincidences. Spear phishing previously required dedicated manual effort, but with this new automation, it is possible to achieve personalization at a scale that hasn’t been seen before.

Conclusion

To address these emerging threats, a multi-faceted approach combining technological, educational, and legislative efforts is imperative. From a technical standpoint, the continued improvement of AI-generated content classifiers and fact checkers can ease the personal burden of identifying threats. Email or cybersecurity providers may also integrate these tools into their products. For education, increasing public awareness of AI-generated content, creating opportunities for people to generate the content themselves, and understanding what personal information data mining can exploit may reduce the number of victims falling for scams.

In conclusion, our research indicates a future where generative AI can substantially enhance the capabilities of political campaigns and malicious actors. The ability to produce personalized, finely tuned political content poses a significant risk of enhanced misinformation, financial scams, and the deepening of ideological polarization. The generation of fantasy political websites and individualized spam emails elucidates the transformative potential of generative AI tools in this domain.

The Author

Ben Gelman is a Senior Data Scientist at Sophos AI researching the uses of AI/ML in cybersecurity. His work primarily focuses on alert prioritization, case analysis, and detection of malicious command lines. In prior work, he used deep learning in a variety of domains, including source code analysis, natural language processing, image recognition, data privacy, and hyperparameter optimization.