The term “fake news” has become a staple of modern vocabulary, but it is a clumsy instrument for a surgical problem. Disinformation isn’t just about truth versus fiction; it is about the intent to harm and the methods used to deceive. Whether it involves denying climate change or undermining public health during a pandemic, disinformation is a calculated effort to shift public perception.

To combat this effectively, we need to understand not just what is being said, but why and how. This is the core problem addressed in a recent paper titled “MIPD: Exploring Manipulation and Intention In a Novel Corpus of Polish Disinformation.”

The researchers behind this work argue that existing datasets are limited because they often focus solely on binary classification—labeling a text as simply “true” or “false.” To remedy this, they introduce the MIPD (Manipulation and Intention in Polish Disinformation) dataset. This massive corpus moves the field forward by annotating the specific manipulation techniques used (the “how”) and the malicious intentions of the authors (the “why”). While the dataset focuses on the Polish language—a critical linguistic region due to its proximity to the Russo-Ukrainian conflict—the methodology offers a universal blueprint for dissecting disinformation.

Background: The Anatomy of Disinformation

Before diving into the dataset, it is crucial to distinguish between misinformation and disinformation. The High-Level Expert Group (HLEG) of the European Commission defines disinformation as “false, inaccurate, or misleading information designed, presented, and promoted to intentionally cause public harm or for profit.”

There are two key components here:

  1. Intention: The specific goal of the creator (e.g., to polarize society or undermine trust in institutions).
  2. Manipulation: The rhetorical devices used to achieve that goal (e.g., cherry-picking data or appealing to fear).

Historically, the V4 countries (Poland, Czech Republic, Slovakia, and Hungary) have been particularly vulnerable to disinformation campaigns, especially those related to geopolitical conflicts like the war in Ukraine. By focusing on Polish, the 5th most spoken language in the EU, the authors provide a resource for a region on the front lines of information warfare.

The MIPD Corpus: A Multilayered Approach

The MIPD dataset consists of 15,356 web articles. Unlike many datasets that rely on crowdsourcing (like Amazon Mechanical Turk) for labeling, MIPD was annotated by professionals. Five native Polish speakers with years of experience in fact-checking and debunking organizations were employed to ensure high-quality, expert-level annotations.

The annotation process was rigorous, following a five-step methodology:

  1. Thematic Categorization: Determining the topic (e.g., COVID-19, Migration).
  2. Source Credibility: Evaluating the reputation of the publisher.
  3. Main Class: Labeling the article as Credible, Disinformation, Misinformation, or Hard-to-say.
  4. Manipulation Techniques: Identifying rhetorical tricks (Multilabel).
  5. Intention Types: Identifying the author’s goal (Multilabel).

Crucially, every article was annotated independently by two experts. If they disagreed, they met to reach a consensus. If no consensus was reached, the article was discarded. This resulted in a dataset with exceptionally high reliability.

The Landscape of Topics

The dataset covers ten specific thematic categories often targeted by disinformation campaigns. As shown in the figure below, the distribution of disinformation varies across topics. While categories like “Paranormal Activities” (PA) or “Pseudomedicine” (PSMED) heavily lean toward disinformation (DIS), others like “Women’s Rights” (WOMR) have a more balanced or credible-leaning distribution in this specific corpus.

Figure 2: Percentage of disinformative (DIS) and credible (CI) articles per thematic category.

Decoding the “How”: Manipulation Techniques

One of the paper’s most significant educational contributions is its taxonomy of manipulation. The experts identified distinct techniques used to mislead readers. Understanding these is essential for any student of media literacy or NLP.

The researchers annotated for techniques such as:

  • Cherry Picking (CHP): Selecting only data that supports a thesis while ignoring context.
  • Quote Mining (QM): Taking a short fragment of a speech out of context to distort its meaning.
  • Anecdote (AN): Using personal stories or rumors to discredit statistics.
  • Whataboutism (WH): Deflecting an argument by raising an unrelated issue.
  • Strawman (ST): Distorting an opponent’s argument to make it easier to attack.
  • Appeal to Emotion (AE): Using charged language to bypass rational thought.
  • False Cause (FC): Assuming causation based solely on correlation.
  • Exaggeration (EG): Overstating a phenomenon to make it seem catastrophic or insignificant.

The distribution of these techniques is not random. As the heatmap below illustrates, different topics attract different manipulation styles. For example, Exaggeration (EG) and Appeal to Emotion (AE) are pervasive across many categories, whereas Reference Error (RE)—citing fake experts or false quotes—is heavily utilized in pseudomedicine and conspiracy theories.

Figure 3: Percentage of different manipulation techniques per thematic category among articles with manipulation.

Decoding the “Why”: Intention Types

The second unique layer of this dataset is the classification of intent. The authors grouped malicious goals into categories such as:

  • Negating Scientific Facts (NSF): Challenging established science (common in Climate and COVID topics).
  • Undermining Public Institutions (UCPI): Eroding trust in government bodies.
  • Promoting Social Stereotypes (PSSA): Fueling homophobia, xenophobia, or antisemitism.
  • Weakening International Alliances (WIA): Sowing discord between allied nations.
  • Raising Morale of a Conflict’s Side (RMCS): Propaganda specifically designed to boost one side of a military conflict (prominent in Ukraine war coverage).

Figure 4: Percentage of different intention types per thematic categories among articles with malicious intention

The heatmap above highlights that while some intentions are topic-specific (like Raising Morale for the Ukraine war topic), others, like Promoting Social Stereotypes, appear across migration, LGBT+, and even 5G narratives.

Experiments and Results

With this high-quality data in hand, the researchers established baselines for three automatic classification tasks:

  1. Binary Disinformation Detection: Is it credible or not?
  2. Manipulation Classification: Which techniques are present?
  3. Intention Classification: What is the author’s goal?

They fine-tuned Polish language versions of BERT models (HerBERT and Polish-RoBERTa) and compared the results.

Task 1: Binary Disinformation Detection

For the simple task of flagging disinformation, the models performed exceptionally well. The Polish-RoBERTa-Large model achieved an accuracy of 96% and a weighted F1 score of 0.96. This suggests that modern Transformer models, when fine-tuned on expert-labeled data, are highly effective at distinguishing credible news from disinformation in a specific language domain.

Table 2: Results for disinformation detection task. Table shows accuracy, weighted F1 score, and F1 score on test data for pre-trained Polish BERT-based models.

Task 2: Detecting Manipulation Techniques

This task is significantly harder. It is a multilabel problem, meaning one article can contain multiple manipulation types (e.g., Exaggeration and Strawman simultaneously).

The results show a drop in performance compared to binary detection, which is expected given the complexity. The best model (PL-RoBERTa-L) achieved a weighted F1 score of 0.47.

Table 3: Results for manipulation techniques classification. Table shows F1 scores for each manipulation type.

It is important to analyze the breakdown by category in the table above. The models struggled immensely with Quote Mining (QM) and Leading Questions (LQ), scoring near zero. This is likely because these techniques are context-dependent and rarer in the dataset, making them difficult for the model to learn. Conversely, Exaggeration (EG) and Reference Error (RE) were easier to detect, likely due to distinct linguistic markers.

Task 3: Detecting Intention

Classifying intent was more successful than classifying manipulation, though still complex. The best model achieved a weighted F1 score of 0.71.

Table 4: Results for malicious intention type classfication. Table shows F1 scores for each intention type.

The model was particularly good at identifying Negating Scientific Facts (NSF) (\(F_1\) = 0.86), likely because the vocabulary used in anti-science narratives (climate denial, anti-vax) is very distinctive.

The LLM Showdown: BERT vs. GPT-4

In a fascinating addition to the study, the researchers tested whether General Purpose Large Language Models (LLMs) like GPT-3.5 and GPT-4 could detect disinformation “out of the box” (Zero-Shot), without the specific fine-tuning used for the BERT models.

They prompted the models in two ways:

  1. Without Definition: Simply asking, “Is this disinformation?”
  2. With Definition: Providing the HLEG definition of disinformation in the prompt.

Table 5: Results of the disinformation detection task for GPT-4 and GPT-3.5.

The results were revealing. GPT-4 achieved a weighted F1 score of 0.86 when provided with a definition. While impressive for a zero-shot attempt, it notably lagged behind the fine-tuned Polish RoBERTa model (which scored 0.96). This reinforces a key lesson in Machine Learning: for specialized, high-stakes tasks, a smaller model fine-tuned on high-quality, domain-specific data often outperforms a massive, general-purpose model.

Conclusion

The MIPD dataset represents a significant step forward in the computational study of disinformation. By moving beyond simple binary labels and dissecting the intent and manipulation within the text, the authors have provided a roadmap for a deeper understanding of information warfare.

Key takeaways for students:

  • Data Quality Matters: The use of professional fact-checkers rather than crowdsourcing resulted in a highly reliable dataset.
  • Disinformation is Layered: It is not just about “fake” content; it is about specific rhetorical techniques and malicious goals.
  • Fine-Tuning Wins: For specific languages and nuanced tasks, domain-specific fine-tuning still beats general zero-shot LLMs.

As information warfare continues to evolve, tools that can automatically identify not just the what, but the how and why of disinformation will become indispensable for maintaining the integrity of our digital information ecosystem.