How to Teach LLMs Like Human Students: The PTD-SQL Framework

Introduction

Imagine you are studying for a difficult math exam. You open your textbook, but instead of just reading every page in order, you notice the chapters are divided by topic: Geometry, Algebra, Calculus, and Statistics. When you struggle with a specific type of geometry problem, you don’t practice by solving calculus equations. Instead, you perform targeted drilling—you find a set of geometry problems, study the specific formulas required for them, and practice until you master that category.

Human learning relies heavily on inductive reasoning—we categorize problems and apply specific thought patterns to solve them. However, when we use Large Language Models (LLMs) for complex tasks like converting natural language into SQL queries (Text-to-SQL), we often treat the model as a generalist. We feed it a prompt and hope it figures out the logic, regardless of whether the question requires a complex set operation, a nested query, or simple filtering.

This lack of “categorical thinking” in LLMs is exactly what the authors of the paper “PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL” set out to solve. They propose a framework that mirrors the human learning process: first, identify what type of problem this is, and second, use a specialized “drilling” strategy designed specifically for that problem type.

In this deep dive, we will explore how PTD-SQL works, the architecture behind its partition-and-drill strategy, and how this approach allows LLMs to achieve state-of-the-art results on benchmark datasets.

Background: The Text-to-SQL Challenge

The task of Text-to-SQL involves taking a natural language question (e.g., “Which students have a GPA over 3.5 and play on the soccer team?”) and converting it into an executable SQL query.

While early approaches relied on training encoder-decoder models specifically for this task, the rise of LLMs like GPT-4 has shifted the focus toward In-Context Learning (ICL). Techniques like Chain-of-Thought (CoT) prompting, where the model is encouraged to “think step-by-step,” have significantly improved performance.

However, existing methods often treat SQL generation as a monolithic task. A leading method, DIN-SQL, attempted to break the task down by complexity (easy vs. nested queries). While this was a step in the right direction, it missed a crucial nuance: SQL queries aren’t just “hard” or “easy.” They have structural categories. A problem requiring a UNION operator involves a completely different thinking process than a problem requiring a GROUP BY and ORDER BY clause.

PTD-SQL argues that to truly unlock LLM reasoning, we need to move beyond difficulty grading and move toward structural partitioning.

The PTD-SQL Framework

The core hypothesis of PTD-SQL is simple: employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type.

The framework operates in a three-stage pipeline, which can be visualized as follows:

1. Query Group Partition (QGP): Classifying the user’s question into a specific category.
2. Targeted Drilling Bank Construction: Offline creation of specialized “textbooks” (prompt banks) for each category.
3. Inference: Retrieving the most relevant examples from the correct bank to guide the LLM.

Let’s look at the overall flow of the system:

As shown in Figure 4, the process begins with the “QGP sub-task” (Stage a), where questions are sorted. This leads to the creation of specific “Shot Banks” (Stage b). Finally, when a new query comes in, the system automatically selects the right examples (Stage c) to generate the final SQL.

Phase 1: Query Group Partition (QGP)

SQL queries are unique because they have fixed patterns. Unlike general coding problems, SQL relies heavily on specific keywords to define logic. The researchers identified four distinct “problem groups” based on the keywords required in the ground-truth SQL:

1. Multi-set Problems: Questions requiring keywords like INTERSECT, UNION, or EXCEPT. These usually involve combining results from two different sub-queries.
2. Combination Problems: Questions requiring GROUP BY to aggregate data, often followed by ORDER BY or aggregate functions (like COUNT, MAX).
3. Filtering Problems: Questions that focus on WHERE clauses to sift through data based on conditions.
4. Other Simple Problems: Basic retrieval tasks.

The diagram below illustrates these partitions with concrete examples:

The Classifier: Fine-tuning over Prompting

How does the system know which group a new question belongs to? You might assume they simply ask GPT-4 to classify the question. However, the researchers found that even powerful LLMs struggle with accurate classification in a few-shot setting.

Instead, they adopted a more robust approach: Parameter-Efficient Fine-Tuning (PEFT). They took a smaller, open-source model (Llama-2-7b) and fine-tuned it specifically for the classification task using LoRA (Low-Rank Adaptation).

LoRA allows for efficient fine-tuning by freezing the pre-trained model weights and injecting trainable rank decomposition matrices. The forward pass looks like this:

Where \(W_0\) is the frozen weight and \(A\) and \(B\) are the trainable matrices. This approach is computationally efficient and, as the experiments showed, highly effective.

The table below highlights the performance gap between a fine-tuned small model and a massive model like ChatGPT using few-shot prompting:

As you can see, the Llama-2-7b model with LoRA achieved 85.0% accuracy in classifying questions, significantly outperforming ChatGPT’s 68.0%. This proves that for specialized discrimination tasks, a small, fine-tuned specialist often beats a large generalist.

Phase 2: Targeted Drilling Banks

Once the questions are partitioned, the system needs to teach the LLM how to solve them. This is where Targeted Drilling comes in.

The researchers constructed specific prompt templates for each of the four categories. These templates don’t just provide the answer; they provide a Chain-of-Thought (CoT) reasoning path tailored to that specific type of SQL logic.

The “Drilling” Concept

Think of this as a question bank for students. If a student is learning about Set Operations (Intersections and Unions), they are given a worksheet that explicitly demonstrates how to break down those specific problems.

Figure 1 shows this concept in action. The prompt provides examples of INTERSECT, UNION, and EXCEPT questions. When the LLM encounters a new question (e.g., “Find students who are female… OR male…”), it references this specific bank, triggering the memory of how to structure a UNION query.

Decomposition Strategies

The “thought process” included in these prompts is critical. The authors use a technique called Decomposed Prompting.

• For Multi-set Problems: The prompt encourages the model to break the question into sub-queries (e.g., “Subproblem 1: Find UK artists,” “Subproblem 2: Find English songs”) and then link them with a set operator.
• For Combination Problems: The prompt emphasizes identifying the grouping key (GROUP BY) and the sorting condition (ORDER BY).

Here is a side-by-side comparison of how the prompts differ for Multi-set vs. Combination problems:

Notice the distinct reasoning steps in Figure 3. The Multi-set prompt focuses on “Question Decomposition” into sub-parts. The Combination prompt focuses on “Schema Linking” and specific keywords like COUNT() and GROUP BY. This variability is key—using a generic reasoning template for all SQL problems leads to suboptimal results.

Phase 3: Few-Shot Selection and Inference

In the final phase, when a user asks a question, the PTD-SQL system:

1. Uses the fine-tuned Llama model to classify the question (e.g., “This is a Multi-set problem”).
2. Goes to the “Multi-set Targeted Drilling Bank.”
3. Selects the most relevant few-shot examples to include in the final prompt.

Mix-of-Matching Strategy

How does it select the “best” examples from the bank? The paper proposes a Mix-of-Matching strategy that combines two metrics:

1. Semantic Matching (Equation 4): This measures the meaning similarity between the user’s question (\(s\)) and the examples in the bank (\(s_i\)) using vector embeddings (OpenAI text-embedding-ada-002).

2. Syntactic Matching (Equation 5): This measures the overlap of specific tokens (words/keywords). This is crucial for SQL because the presence of words like “each,” “total,” or “highest” strongly implies specific SQL operations.

By selecting examples that are both semantically similar (same topic) and syntactically similar (same phrasing), the model is given the perfect context to generate the correct SQL query.

Experiments and Results

The researchers evaluated PTD-SQL on two major benchmarks: Spider (a standard cross-domain dataset) and BIRD (a large-scale database benchmark). They compared PTD-SQL against standard prompting methods and state-of-the-art frameworks like DIN-SQL and DAIL-SQL.

Main Performance

The results were impressive. PTD-SQL consistently outperformed or matched previous state-of-the-art methods.

In Table 2 (Spider-dev), PTD-SQL with GPT-4 achieved an Execution Accuracy (EX) of 85.7%, surpassing DAIL-SQL (83.6%) and DIN-SQL (80.6%). Even with weaker models like the open-source Deepseek-coder, PTD-SQL showed strong improvements.

Similar trends were observed on the BIRD dataset, which is known for being significantly harder and larger:

Here, PTD-SQL + GPT-4 reached 57.0% accuracy, a notable jump over DAIL-SQL’s 53.6%.

Breaking the “Capability Boundary”

One of the most fascinating findings of the paper is how PTD-SQL affects models of different capability levels. The authors analyzed performance across difficulty levels (Easy, Medium, Hard, Extra Hard).

Figure 5 reveals a pattern akin to human learning progression:

• ChatGPT (Blue line): Shows massive gains in the “Hard” category but drops off at “Extra Hard.” It’s like a student mastering the core curriculum but struggling with the most advanced problems.
• GPT-4 (Green line): Shows the biggest improvement in the “Extra Hard” / “Challenging” category. Because GPT-4 is already an “elite student,” the targeted drilling allows it to crack the toughest nuts that were previously out of reach.

This suggests that Targeted Drilling helps models push the boundary of their current capabilities.

Why Categorization Matters

Is the categorization really necessary? Could we just dump all the examples into one big bank? The researchers conducted an ablation study to find out.

Figure 6 illustrates the performance of different methods across the four problem groups. PTD-SQL (Green line) shows a clear advantage, particularly in the Multi-set and Combination categories (the top left and top right axes). These are the categories that require the most specific structural reasoning, validating the hypothesis that “drilling” specific logic patterns yields better results than generic prompting.

Conclusion and Implications

The PTD-SQL paper presents a compelling argument for categorical thinking in Large Language Models. By acknowledging that not all SQL queries are created equal, and by creating specialized learning paths (Partitioning and Targeted Drilling), the authors managed to achieve state-of-the-art performance.

The key takeaways from this work are:

1. Divide and Conquer: Breaking a complex domain like SQL into structural categories (Multi-set, Filtering, etc.) is more effective than simple difficulty grading.
2. Specialized Teachers: Using a small, fine-tuned model for classification is more reliable than asking a large model to classify zero-shot.
3. Targeted Context: Providing few-shot examples that mirror the logic of the target problem (not just the topic) significantly boosts reasoning capabilities.

This approach likely extends beyond just Text-to-SQL. Whether it’s math word problems, code generation in other languages, or complex logical reasoning tasks, the “Partition and Drill” framework suggests that LLMs, much like humans, learn best when they know exactly what kind of problem they are trying to solve.