The way we train Large Language Models (LLMs) is evolving. In the early days, it was all about next-token prediction on massive datasets. Then came the era of alignment, where we started telling models what we actually wanted them to do, primarily through Reinforcement Learning from Human Feedback (RLHF).

But if you look closely at how we “teach” these models, it feels surprisingly primitive compared to how humans teach each other. In RLHF, we often treat the model like a black box that spits out two answers, and we simply tell it, “Answer A is better than Answer B.”

Imagine a teacher grading a student’s essay by simply writing “7/10” or “Better than your last one” without pointing out a single spelling error, logical fallacy, or structural issue. That student would struggle to improve. Yet, this is exactly how we train state-of-the-art AI.

In the paper “Let Me Teach You: Pedagogical Foundations of Feedback for Language Models,” researchers from EPFL and the Allen Institute for AI argue that it is time to stop treating feedback as a mere signal and start treating it as pedagogy. By looking at decades of research in Learning Sciences, they propose a new framework, FELT, to systematize how we provide Natural Language Feedback (NLF) to AI.

The Problem with “Good vs. Bad”

Current alignment techniques have been incredibly successful. Models like ChatGPT and Claude are helpful and generally harmless because they have been tuned using human preferences. This usually involves a Reward Model (RM) trained on rankings—learning that humans prefer safe, polite answers over toxic ones.

However, these methods have a ceiling. A scalar score (e.g., +1 or -1) conveys very little information. It doesn’t tell the model why it failed. Was the answer factually wrong? Was the tone rude? Was it just boring?

To fix this, the NLP community has started moving toward Natural Language Feedback (NLF). Instead of a score, we give the model text: “You made a calculation error in step 2” or “Please be more concise.”

While promising, current NLF research is messy. Researchers often rely on “intuitive guesses” about what good feedback looks like. There is no standard system. Some papers tell the model to “critique itself,” others use “editorial comments,” and others use “simulated human feedback.”

The authors of this paper ask a simple question: Why are we guessing? Humans have been studying how to give effective feedback for centuries.

Bridging NLP and Learning Sciences

The paper begins by surveying the field of Learning Sciences—the study of how humans learn. It turns out that feedback is not just “information provided to a learner.” It is a complex ecosystem.

Research shows that for feedback to be effective, it usually needs to satisfy three conditions:

  1. Applicability: It must be actionable. The learner needs to know where they are going and how to get there.
  2. Learner Regulation: It must trigger a cognitive response. The learner has to process the feedback, not just receive it.
  3. Personalization: It must fit the learner’s current level of knowledge and the task’s difficulty.

The researchers mapped the questions currently being asked in NLP (like “How informative is feedback?”) to established concepts in Learning Sciences.

Figure 1: Connecting feedback research in NLP to foundations of feedback in the Learning Sciences.

As shown in Figure 1, the disconnect is clear. NLP tends to focus on the technical implementation (enhancing the model, revising generations), while Learning Sciences focuses on the why and how—the variables that influence the feedback loop, such as timing, source, and learner characteristics.

To understand the depth of this field, consider how varied the definitions of feedback are in pedagogical literature. It isn’t just one thing; it’s a mechanism that changes the gap between what a student knows and what they should know.

Table 2: Different pedagogical works’ definitions of feedback.

Table 2 highlights this diversity. From Ramaprasad (1983), who argues information isn’t feedback unless it actually changes the gap in performance, to Hattie and Timperley (2007), who view it as a cycle of “Feed Up, Feed Back, and Feed Forward.” The authors suggest that if we want better LLMs, we need to embrace this complexity.

The FELT Framework

To unify these two worlds, the authors introduce FELT: a framework consisting of Feedback, Errors, Learner, and Task.

This framework moves beyond the idea that feedback is just a dataset. It models the entire “classroom” environment in which the LLM operates.

Figure 2: FELT, the feedback ecosystem adapted for LLMs.

Let’s break down the components of the ecosystem illustrated in Figure 2:

1. The Task

In NLP, we often treat all prompts as equal “inputs.” In FELT, the Task is defined by its complexity, instructions, and answer type (open-ended vs. closed).

  • Why it matters: A simple arithmetic task requires immediate, corrective feedback. A creative writing task requires delayed, elaborated feedback. Treating them the same is inefficient.

2. The Learner (The LLM)

The “Learner” in this context is the model itself. In pedagogy, a student’s prior knowledge dictates how you teach them.

  • For LLMs: “Prior knowledge” is encoded in the model’s pre-training data and size. A 70B parameter model is a “smarter” student than a 7B model.
  • Feedback Processing Mechanism: How does the learner digest the critique? Is it through In-Context Learning (pasting the critique in the prompt)? Or is it through weight updates (Reinforcement Learning)?

3. Errors

Not all mistakes are created equal. The framework distinguishes between Error Type and Severity.

  • Pedagogical View: If a student makes a typo, you circle it. If a student misunderstands a fundamental concept, you need to re-teach the lesson.
  • LLM View: A hallucination is different from a reasoning error. Feedback should target the specific type of error rather than just penalizing the output generally.

4. Feedback

This is the lever we can pull. Feedback has three main levers:

  • Source: Who is giving the feedback? (Human, another Model, or a heuristic script?)
  • Timing: When is it given? (Immediate, or delayed to allow for “reflection”?)
  • Content: What does the feedback actually say?

A Taxonomy for Natural Language Feedback

The most actionable contribution of the paper is the deep dive into the Content of feedback. The authors argue that “textual feedback” is too vague. They propose a taxonomy to classify exactly what information a prompt contains.

They identify four non-overlapping areas of feedback content:

  1. Learner Status: What did the model do right or wrong? (“You missed the second constraint.”)
  2. Goal: What is the correct answer or target? (“The answer is 42.”)
  3. Procedural: Instructions on how to fix it. (“Try breaking the problem into steps.”)
  4. Peripheral: Extra context. (“This is a trick question commonly used in interviews.”)

The 10 Dimensions of Feedback

Furthermore, the authors define 10 dimensions that allow researchers to modulate feedback precisely. If you are designing a prompt to teach an LLM, you should be controlling for these variables:

  1. Granularity: Are you critiquing the whole paragraph, a sentence, or a specific token?
  2. Applicability of Instructions: Are you giving vague advice (“Be better”) or specific algorithms (“Use chain-of-thought”)?
  3. Answer Coverage: Does the feedback address the whole response or just the error?
  4. Target Coverage: How much of the “perfect” solution is revealed?
  5. Criteria: Are you judging based on factuality, style, or safety?
  6. Information Novelty: Is the feedback telling the model something it already knows, or introducing new data?
  7. Purpose: Is this to improve the score or to clarify the task definition?
  8. Style: Is the feedback formal, rude, or encouraging?
  9. Valence: Positive (reinforcement) or negative (correction)?
  10. Mode: Text, image, or code?

Why This Taxonomy Matters

Previous research has been inconsistent. As seen in Table 1 below, different papers claim to use “Natural Language Feedback,” but they are doing vastly different things.

Table 1: Past works using textual or otherwise augmented feedback.

Some papers use “Learner Status” (telling the model it was wrong), while others use “Procedural” feedback (telling the model how to think). Without a taxonomy, we cannot compare these papers fairly. The FELT taxonomy allows us to say, “Paper A worked better than Paper B because they used high-granularity Procedural feedback, whereas Paper B used vague Learner Status feedback.”

Implications: The Future of Feedback

So, what can we do with FELT? The authors suggest that this framework opens up entirely new avenues for research, moving us away from simple “RLHF with rankings” toward complex “LLMs as Learning Agents.”

Figure 3: Visual summary of several future research directions motivated by FELT.

Figure 3 outlines several future directions derived from the FELT components:

  1. Task-Based Processing: We can design different feedback loops for different tasks. We shouldn’t use the same “prompt engineering” for math as we do for creative writing.
  2. Error-Specific Penalties: By mapping error types (from the “Errors” component), we can train Reward Models that penalize hallucinations differently than they penalize stylistic errors.
  3. Feedback Timing & Reflection: Inspired by pedagogy, we can explore delayed feedback. Instead of correcting an LLM immediately, we might ask it to “reflect” on its answer first, then provide feedback on the reflection rather than the answer. This mimics how teachers force students to find their own mistakes.
  4. Personalized Learners: We can model LLMs as learners with specific traits. A “novice” model might need explicit, high-granularity feedback, while an “expert” model might only need a slight nudge (low granularity).

Conclusion

The transition from “training” to “teaching” LLMs is more than just a semantic shift. It represents a move from brute-force statistical correlation toward structured, pedagogical alignment.

The FELT framework provides the map for this territory. By understanding that feedback is an ecosystem involving the learner’s state, the task’s complexity, and the specific content of the critique, we can design more effective alignment strategies.

For students and researchers entering the field, the takeaway is clear: Don’t just throw data at the model. Analyze how you are teaching it. Are you being a helpful teacher providing actionable, procedural feedback? Or are you just grading papers with a red pen and no comments? The future of AI capabilities may very well depend on the quality of our pedagogy.