Artificial Intelligence has rapidly transitioned from a novelty to a daily utility. We use Large Language Models (LLMs) to draft emails, summarize news, and explain complex concepts. Implicit in this usage is a presumption of neutrality—we often treat these models as objective synthesizers of information.

However, a recent study titled “Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters” by researchers at UC Berkeley and the University of Chicago challenges this assumption. The paper investigates a critical question for modern democracy: Do LLMs hold political biases, and if so, can they unintentionally sway the electorate?

In the context of the 2024 U.S. presidential election (specifically the matchup between Joe Biden and Donald Trump), the researchers found that not only do models exhibit a distinct preference for the Democratic nominee, but interacting with these models can also statistically shift users’ voting intentions.

This post will break down the methodology used to diagnose these biases, the mechanisms behind them, and the results of a large-scale experiment involving real voters.

Part 1: Diagnosing the Political Pulse of AI

To determine if an AI has a political stance, the researchers didn’t rely on a single test. Instead, they employed a multi-faceted approach across 18 different open- and closed-weight models, including GPT-4, Claude-3, Llama-3, and Mistral.

The Simulation of Voting

The most direct method was a simulation. The researchers prompted the LLMs to act as voters in the 2024 election. To ensure the results weren’t a fluke, they ran this simulation 100 times for each model, randomizing the order of candidates to prevent positional bias.

The results, as shown below, were stark.

Table 1 shows the voting results of 18 instruction-tuned LLMs and 3 base models. The vast majority voted for Biden 100 out of 100 times.

As you can see in Table 1, there was an overwhelming preference for the Democratic nominee. Out of the 18 models tested, 16 consistently voted for Biden in every single iteration. Even models like Gemini Pro and Alpaca, which showed some variation, still favored Biden significantly.

Analyzing Policy Bias

Asking an LLM “Who would you vote for?” is a blunt instrument. To understand the nuance of this bias, the researchers conducted a deeper analysis using 270 specific questions covering 45 political topics (such as abortion, the economy, and foreign policy).

They asked the models to generate:

  1. Neutral descriptions of each candidate’s policies.
  2. Positive impacts of the policies.
  3. Negative impacts of the policies.

They then evaluated the AI responses using three metrics: Refusal Rate (how often the model refused to answer), Response Length (how much detail was provided), and Sentiment Score (how positive the language was).

Figure 1 displays three charts comparing refusal rates, response lengths, and sentiment scores. LLMs consistently show lower refusal rates for negative questions about Trump, longer responses for Biden’s positive aspects, and higher sentiment scores for Biden.

Figure 1 highlights three critical behaviors:

  • Refusal Rate (Figure 1a): LLMs were significantly more likely to refuse to answer questions about the negative impacts of Biden’s policies or the positive impacts of Trump’s policies. Conversely, they rarely refused to list negative aspects of Trump’s policies.
  • Response Length (Figure 1b): When asked to describe positive impacts, LLMs wrote significantly more for Biden. When asked for negative impacts, they wrote significantly more for Trump.
  • Sentiment (Figure 1c): The sentiment analysis revealed a clear “Biden-leaning” pattern. Even when answering “neutral” questions, the language used to describe Biden was statistically more positive than that used for Trump.

The “Geometry of Culture”

To visualize exactly where these biases exist, the authors employed a technique called the “geometry of culture.” This involves analyzing word embeddings to see how closely different concepts are related in the model’s vector space. They mapped the 45 political topics against semantic dimensions like “Foolish vs. Wise” or “Cruel vs. Kind.”

Figure 7 is a heatmap showing attitudes in LLM responses across 45 topics. Biden’s policies generally correlate with positive dimensions, while Trump’s policies correlate with negative dimensions across most topics.

Figure 7 offers a granular look at the topics. Biden (top chart) sees a sea of red (positive sentiment) across almost all policy areas, with only “charter schools” showing negativity. Trump (bottom chart), by contrast, shows significant green (negative sentiment) across major topics like climate change, healthcare, and border security.

Where Does the Bias Come From?

A fascinating finding for students of machine learning is the comparison between Base Models (pre-trained on raw text) and Instruction-Tuned Models (fine-tuned with human feedback, or RLHF).

The researchers found that while base models did lean left, the instruction-tuned versions were significantly more biased.

Figure 6 compares sentiment scores between instruction-tuned and base models. The instruction-tuned models (left) show a more pronounced gap between Biden (blue) and Trump (red) sentiment compared to the base models (right).

As shown in Figure 6, the post-training process—intended to make models safer and more helpful—seems to amplify political alignment with Democratic viewpoints. This suggests that the human feedback loop or the safety guidelines used during fine-tuning may inadvertently encode specific political values.

Part 2: The Influence on Voters

Establishing that LLMs have a bias is one thing; proving that this bias changes human behavior is another. To test this, the authors recruited 935 registered U.S. voters for a controlled experiment.

The Experiment Setup

Participants were divided into groups and asked to engage in a political conversation with one of three LLMs: Claude-3-Opus, Llama-3-70B, or GPT-4-Turbo.

  • Pre-Survey: Participants stated their initial leaning (e.g., “100% Trump” or “50/50 Neutral”).
  • Interaction: They engaged in five back-and-forth exchanges with the AI about the candidates. The AI was prompted to facilitate a discussion, not explicitly to persuade.
  • Post-Survey: Participants restated their voting preferences.

The Shift in Voting

The results demonstrated a statistically significant shift in voter preference. Following the conversation, the overall voting margin in the sample shifted towards Biden, widening from 0.7% to 4.6%.

Figure 2 illustrates the shift in voter preference. Figure 2b shows arrows moving upward, indicating a shift toward Biden across different voter groups. Figure 2c shows the effect size is significant compared to a control group.

Figure 2b visualizes this flow. Notice the upward movement of the arrows:

  • Trump Supporters: Nearly 20% of initial Trump supporters reduced their support intensity. In extreme cases, some fully flipped to Biden.
  • Neutral Voters: 24% of neutral participants shifted to support Biden.
  • Biden Supporters: They largely retained their views, with the AI reinforcing their existing beliefs.

It is crucial to note that the AI was not told to persuade users. This shift occurred through “organic” conversation where the LLMs simply exhibited their inherent biases.

What Drove the Conversation?

The researchers analyzed the content of the conversations to see which topics dominated. They found that the models—particularly Llama-3, which was the most pro-Biden in this experiment—steered conversations toward topics where Democratic policies are generally viewed more favorably.

Figure 10 displays the top 8 topics mentioned by LLMs. Llama-3 frequently brought up climate change, pandemic response, and healthcare, while Trump-favorable topics like Afghanistan were mentioned less.

As seen in Figure 10, Llama-3 heavily prioritized topics like climate change, pandemic response, and healthcare—areas where the model’s training likely contains data favorable to the current administration. Topics that might favor Republican arguments, such as the withdrawal from Afghanistan, were mentioned far less frequently.

User Satisfaction and Persuasion

One might assume that users—especially Trump supporters—would be turned off by a biased AI. Surprisingly, the study found a positive correlation between persuasion and user satisfaction.

Figure 12 shows a positive correlation between perceived conversation quality and the increase in Biden-leaning percentage. Users who were swayed toward Biden tended to rate the conversation higher.

Figure 12 shows that participants who shifted their views toward Biden generally rated the conversation quality higher. This creates a “persuasion loop”: the model provides well-articulated, polite, but biased information; the user enjoys the interaction; and consequently, the user becomes more receptive to the viewpoint.

Furthermore, Figure 14 (below) reveals through clustering of user feedback that while some users detected bias, many were impressed by the “human-like” quality of the interaction.

Figure 14 shows clusters of participant feedback. While one cluster (orange) notes bias, others (blue/green) express being impressed by the quality or feeling the conversation was scripted.

Part 3: Implications and Mitigation

The authors conclude that LLMs act as “Hidden Persuaders.” Their influence is subtle because it is wrapped in a veneer of objective, polite, and helpful dialogue. The fact that instruction tuning increases this bias suggests that our current methods for aligning AI (making it safe and non-toxic) are inextricably linked to specific ideological worldviews.

Can We Fix It?

The paper briefly touches on Representation Engineering (RepE) as a potential mitigation strategy. This involves identifying the “direction” of political bias in the model’s neural activations and mathematically steering the model back toward neutrality.

However, the authors raise a philosophical question: Is neutrality what users actually want? As seen in the satisfaction data, users engaged deeply with the models. A perfectly neutral model might come across as evasive or boring, potentially degrading the user experience.

Conclusion

This research provides compelling evidence that LLMs are not blank slates. In the context of the 2024 election, they exhibited a measurable preference for the Democratic candidate and successfully nudged voters in that direction during short conversations.

For students and researchers in AI, this highlights a critical challenge: achieving “alignment” is not just a technical problem of preventing harm; it is a sociopolitical challenge of defining what “neutrality” means in a polarized world. As these models become integrated into search engines and educational tools, their “hidden” persuasion capabilities will likely become a central topic in the ethics of artificial intelligence.