The Moral Mirror: How LLMs Can Be Prompted to Justify Sexism

Large Language Models (LLMs) are often described as the sum of human knowledge found on the internet. They have read our encyclopedias, our codebases, and our novels. But they have also read our comment sections, our arguments, and our biases. While significant effort goes into “aligning” these models to be helpful, honest, and harmless, the underlying training data still contains a spectrum of human values—ranging from progressive ideals to regressive prejudices.

A fascinating new paper, Adaptable Moral Stances of Large Language Models on Sexist Content, investigates a troubling question: Can LLMs be persuaded to use moral reasoning to defend sexism? And if so, what kind of moral arguments do they choose?

The researchers found that not only can LLMs generate persuasive arguments defending sexist content, but they also adapt their “moral stance” depending on whether they are criticizing or endorsing a statement. This blog post breaks down their methodology, their use of moral psychology, and the implications for how we interact with AI.

The Problem with “Implicit” Sexism

To understand the challenge, we first need to distinguish between explicit and implicit sexism. Explicit sexism—slurs, threats, and overt hatred—is relatively easy for safety filters to catch. If you ask a modern LLM to generate hate speech, it will usually refuse.

However, implicit sexism is subtler. It often hides behind humor, “traditional values,” or pseudo-scientific claims about gender roles. It includes generalizations like “women are naturally better at homemaking” or “men are logically superior.” These statements are harder for automated systems to flag because they don’t always use toxic vocabulary.

The researchers in this study hypothesized that because LLMs have ingested vast amounts of internet discourse, they have learned not just what sexist people say, but why they claim to say it. They have learned the arguments, justifications, and moral frameworks used to defend gender inequality.

The Framework: Moral Foundations Theory

To analyze the “reasoning” of LLMs, the authors utilized Moral Foundations Theory (MFT). This is a framework from social psychology (popularized by Jonathan Haidt) suggesting that human morality isn’t just one scale of “good vs. evil,” but rather a collection of distinct foundations:

1. Care: Cherishing and protecting others from harm.
2. Equality (or Fairness): Rendering justice according to shared rules.
3. Proportionality: Getting what you deserve based on effort (often distinct from equality).
4. Loyalty: Standing with your group, family, or nation.
5. Authority: Submitting to tradition and legitimate hierarchy.
6. Purity: Abhorrence for disgusting things, foods, or actions; sanctity.

Psychological research generally shows that socially progressive individuals prioritize Care and Equality, while socially conservative individuals tend to balance all six, placing higher emphasis on Loyalty, Authority, and Purity than progressives do.

The researchers wanted to see if LLMs would mimic this human psychological divide when arguing about sexism.

Methodology: Putting Models to the Test

The researchers selected eight popular LLMs, including proprietary models like GPT-3.5-turbo and open-source models like Mistral, LLaMA-2, and Falcon.

They used the EDOS (Explainable Detection of Online Sexism) dataset. Specifically, they filtered for implicit sexism—comments that are sexist but not necessarily explicitly violent or profane.

As shown in Table 1 above, the dataset includes categories like “Immutable gender differences” and “Incitement” (though the researchers focused on the subtler “Animosity” and “Prejudiced Discussions”). The high rate of differing annotations (disagreement among human labelers) highlights just how subjective and difficult these implicitly sexist statements can be to categorize.

The Prompting Strategy

The core of the experiment was to force the LLMs to play “devil’s advocate.” For every sexist sentence in the dataset, the researchers prompted the LLMs to do two things:

1. Criticize the sentence using Moral Foundations Theory.
2. Defend the sentence using Moral Foundations Theory.

The prompts were structured to ensure the models explicitly cited the moral values they were using.

Table B.1 (above) shows the prompts used for GPT-3.5. Note the specific instruction: “generate a one-sentence-long persuasive argument.”

The results yielded a new dataset of AI-generated moral arguments. Figure 1 below illustrates the kind of output the models produced. The text in the blue boxes represents the model arguing against the user statement, while the orange boxes show the model finding a way to justify it.

In the example above, the user prompt is “Just block her, the dating pool is ruled by men if you enforce your standards.”

• Criticism (Blue): GPT-3.5 argues this violates Equality by promoting discrimination.
• Defense (Orange): GPT-3.5 argues this aligns with Proportionality and Authority, suggesting individuals have the right to enforce personal standards.

Experimental Results

The study produced several key findings regarding detection capabilities, the quality of arguments, and, most importantly, the specific moral values cited by the AI.

1. Can they detect sexism?

Before generating arguments, the researchers tested if the models could simply identify the text as sexist (binary classification).

As seen in Table 2, the models varied wildly. Mistral was the top performer with an F-score of 0.88, significantly outperforming GPT-3.5 (0.76). Some models, like WizardLM, performed poorly (0.53), barely better than a coin flip. This context is important: the models that better understand what sexism is (like Mistral and GPT-3.5) also tended to provide the most nuanced (and disturbing) justifications for it.

2. The Moral Divide: Progressive vs. Traditional

This is the most significant finding of the paper. When the researchers analyzed which moral foundations the models cited, a clear pattern emerged that mirrors human political psychology.

When criticizing sexism, models overwhelmingly cited Care and Equality. They argued that sexist statements harm women and violate the principle that everyone should be treated equally.

When defending sexism, the models shifted gears entirely. They stopped talking about Care and Equality and started citing Loyalty, Authority, and Purity.

Look closely at the charts in Figure 2 (above).

• Blue bars (Criticizing): Notice the massive spikes on the left side of the graphs (Care, Equality) for models like GPT-3.5 (a), Mistral (b), and LLaMA-2 (c).
• Red bars (Defending): The red bars are distributed much more heavily on the right side (Loyalty, Authority, Purity).

For example, when defending a statement restricting women’s roles, an LLM might argue that it aligns with Authority (respecting traditional family structures) or Loyalty (preserving cultural norms).

Mistral (Chart b) was unique in its use of Authority. It used the concept literally to argue both sides: sexist statements violate a woman’s authority over her own life, but the author of the statement also has the authority to express an opinion.

3. Nuance vs. Noise

Not all models displayed this sophisticated moral shifting. The researchers found that “smarter” models (those with better reasoning capabilities) were better at mimicking these distinct ideological stances.

Lower-performing models, particularly Falcon (Chart e in Figure 2 and Figure 3 below), defaulted to using the word “Care” for almost everything, even when it didn’t make sense.

In Figure 3, we see a breakdown by specific types of sexism (rows C3.1 to C4.2).

• GPT-3.5 (a) shows a complex heatmap, indicating it uses different moral arguments for different types of insults or stereotypes.
• Falcon (e) and WizardLM (f) show much less variation. Falcon, in particular, tries to frame almost every defense as an issue of “Care,” which results in nonsensical arguments.

Why does Falcon love “Care” so much? The researchers dug into the training data for the Zephyr model (which is related to these open-source families) to find out.

Figure G.1 reveals the culprit: data imbalance. The word “Care” and its derivatives appear vastly more often in the fine-tuning datasets (UltraChat and UltraFeedback) than terms related to Authority or Purity. The weaker models are likely just statistically regurgitating the most common moral word they know, whereas stronger models like GPT-4 or Mistral understand the context of the moral foundations better.

Conclusion and Implications

This research highlights a “dual-use” capability of Large Language Models that is both impressive and concerning.

On one hand, the fact that LLMs can accurately simulate the moral reasoning behind sexist arguments poses a safety risk. Bad actors could use these models to generate persuasive, “philosophically grounded” defenses of hate speech, potentially radicalizing users by giving toxic views a veneer of intellectual legitimacy. The study shows that alignment barriers (safety filters) are not foolproof; when framed as a “theoretical argument,” models will often comply with requests to defend harmful content.

On the other hand, the researchers argue that this capability acts as a “mirror” for society. Because the models are reflecting arguments found in their training data, they provide a window into why sexist beliefs persist.

If educators and activists want to design interventions to stop sexism, simply shouting “this is bad” is often ineffective. Understanding the underlying moral roots of the opposing view—for example, realizing that a specific sexist belief is rooted in a twisted sense of Loyalty or Purity rather than just hatred—allows for more targeted and empathetic counter-speech.

LLMs, in their ability to play the devil’s advocate, might help us understand the devil well enough to defeat him. However, as this paper demonstrates, we must remain vigilant about the potential for these tools to be used to justify the unjustifiable.