Beyond Few-Shot: Making Meta-Learning Work for Large-Scale Tasks

Meta-learning—often called learning to learn—has revolutionized how we approach tasks with limited data. Classical methods like Model-Agnostic Meta-Learning (MAML) have shined in few-shot learning scenarios, where models adapt to new tasks using just a handful of examples. These techniques learn a set of initial parameters that enable rapid adaptation across tasks with minimal gradient updates.

But what happens when we leave the comfort of few-shot learning and venture into many-shot territory—where each task comes with thousands or even millions of examples? In standard deep learning, good initialization is still crucial: consider the widespread success of ImageNet pre-training. Could meta-learning find an even better, more general starting point for diverse large-scale problems?

Unfortunately, traditional meta-learning methods hit a computational wall. The same mechanism that makes MAML effective—backpropagating through the learning process—becomes prohibitively expensive when that process spans thousands of optimization steps. A single meta-update can require days of computation, making large-scale meta-learning impractical.

A 2021 paper from KAIST and Google, “Large-Scale Meta-Learning with Continual Trajectory Shifting,” directly confronts this problem. The authors propose a simple yet powerful method that breaks through the computational barrier by enabling frequent meta-updates without waiting for long, costly training trajectories to end. Their approach not only speeds up convergence but also leads to smoother, more generalizable model initializations.

In this post, we’ll explore the intuition behind large-scale meta-learning, unpack the core ideas behind Continual Trajectory Shifting (CTS), and see how this technique transforms efficiency and stability in meta-learning for large-scale domains.

---

The Scaling Problem in Meta-Learning

Meta-learning algorithms such as MAML and Reptile operate via two nested loops—an inner loop and an outer (meta) loop.

1. Inner Loop: For each task, starting from shared initialization parameters \( \phi \), the model optimizes via standard gradient descent for \( K \) steps to yield task-specific parameters \( \theta_K \).

2. Outer (Meta) Loop: The meta-learner then updates \( \phi \) based on how well \( \theta_K \) performs on its task. The meta-gradient points toward an initialization that would lead to better task performance in fewer updates.

For few-shot learning, \( K \) is small—just a few steps—so meta-updates can be done frequently.

However, in large-scale tasks like Aircraft or Stanford Dogs classification, convergence can require \( K = 1{,}000 \) or more inner steps. The meta-learner must wait for every task to finish before making a single meta-update. If we have \( T = 10 \) tasks per meta-batch, that’s over 10,000 gradient operations per meta-update. This delay drastically slows down meta-training.

Figure 1. In conventional meta-learning, the learner waits for full inner-learning trajectories (left). In CTS, meta-updates occur more frequently, enabling smoother convergence over heterogeneous tasks.

---

The Core Method: Continual Trajectory Shifting

The paper’s central insight is straightforward but powerful: break the dependency that forces the meta-learner to wait.

Imagine we update the initialization \( \phi \) after each inner step. After step \( k \), the task-specific parameters are \( \theta_k = U_k(\phi) \), derived from the current initialization. We compute a meta-update \( \Delta_k \) and adjust the initialization: \( \phi_{new} = \phi + \Delta_k \).

The problem: the current \( \theta_k \) is no longer consistent—it’s based on the old \( \phi \). To remain correct, we’d need to restart all \( k \) optimization steps for each task from the new initialization \( \phi_{new} \), which would be computationally impossible.

Continual Trajectory Shifting (CTS) solves this elegantly. Instead of recomputing trajectories, we shift the existing parameters by the same amount as the meta-update:

\[ \theta_k^{new} \approx \theta_k + \Delta_k \]

This approximation keeps all task-specific learners in sync with the evolving initialization, allowing the meta-learner to make a meta-update after every inner step. The optimization trajectories for tasks are gradually shifted—hence the name continual trajectory shifting.

Figure 2. Continual trajectory shifting interleaves frequent meta-updates with ongoing inner optimization.

Each inner step now consists of:

1. Taking an inner gradient update for all tasks.
2. Computing meta-update \( \Delta_k \).
3. Updating \( \phi \leftarrow \phi + \Delta_k \).
4. Shifting each task’s parameters by \( \Delta_k \).

This simple interleave drastically increases update frequency and accelerates convergence.

---

Why Does This Simple Approximation Work?

The authors justify CTS using first-order approximations. Let \( U_k(\phi) \) denote the parameters obtained after \( k \) optimization steps from initialization \( \phi \). They show:

\[ U_k(\phi + \Delta) \approx U_k(\phi) + \frac{\partial U_k(\phi)}{\partial \phi}\Delta + O(\beta^2) \]

Here, \( \beta \) is the meta learning rate. The Jacobian term \( \frac{\partial U_k(\phi)}{\partial \phi} \) expresses how sensitive the optimized parameters are to changes in initialization. In first-order meta-learning, this Jacobian is typically approximated by the identity matrix \( I \) when inner-learning steps are small, so:

\[ U_k(\phi + \Delta) \approx U_k(\phi) + \Delta \]

This validates the shifting rule: each task’s parameters can be updated directly alongside \( \phi \) by adding the meta-shift \( \Delta_k \).

Figure 3. First-order Taylor and Jacobian approximations support CTS’s shifting rule.

Error Analysis

This approximation introduces an error that scales with inner-learning rate \( \alpha \), meta-learning rate \( \beta \), and trajectory length \( k \):

\[ U_k(\phi + \Delta) = U_k(\phi) + \Delta + O(\beta \alpha hk + \beta^2) \]

The cumulative error grows roughly as \( O(\beta \alpha h k^2 + \beta^2 k) \).

Figure 4. Approximation error trends with learning rates and trajectory length. Smoother activations (Softplus) reduce error compared to ReLU.

While error increases with larger \( k \), the authors observe that CTS still performs well in practice—even under less ideal conditions. The explanation lies in a hidden benefit: meta-level curriculum learning.

---

An Accidental Curriculum: Meta-Level Regularization

Curriculum learning introduces tasks from easy to hard, allowing models to build understanding progressively. CTS inherently creates a curriculum at the meta-level.

1. Early Training (Small \( k \)) Updates are based on short trajectories. The meta-learner focuses on short-term improvements—an easier optimization landscape with fewer risky local minima. This “short-horizon bias” acts as a warm-up, helping avoid poor initialization regions.

2. Later Training (Large \( k \)) As training continues, \( k \) increases. Longer trajectories reveal more complex meta-loss surfaces, enabling the meta-learner to refine the initialization with nuanced task feedback.

Figure 5. As \( k \) increases, the meta-loss surface grows more complex. Shorter horizons simplify initialization search in early training phases.

This natural curriculum makes CTS robust even when its approximation error grows. Initially accurate shifts help the model move toward promising regions before longer steps fine-tune the result.

---

Experiments

The paper validates CTS through synthetic benchmarks and large-scale image classification tasks.

1. Synthetic Experiments

The authors first construct a toy distribution of eight heterogeneous tasks derived from a 2D function with multiple minima. They experiment with variants:

• Reptile: baseline meta-learning method.
• CTS (Ours): with continual trajectory shifting.
• CTS Accurate: a computationally expensive version that recomputes trajectories exactly.

Figure 6. Synthetic setup showing rotated and translated loss surfaces.

Findings:

• Long horizons are essential: Meta-training with small \( K \) leads to suboptimal performance even after many gradient steps—a direct demonstration of the short-horizon bias.
• Curriculum effect improves quality: CTS avoids bad local minima by starting with simple losses and gradually increasing trajectory length.
• Comparable outcomes despite approximation: CTS matches “Accurate” performance with dramatically lower cost.

Figure 7. CTS avoids poor minima via gradual trajectory expansion, outperforming baseline meta-learners.

---

2. Large-Scale Image Classification

CTS was evaluated on a diverse suite of vision datasets. Meta-training datasets: TinyImageNet, CIFAR100, Stanford Dogs, Aircraft, CUB, Fashion-MNIST, SVHN. Meta-testing datasets: Stanford Cars, QuickDraw, VGG Flowers, VGG Pets, STL10.

Figure 8. CTS dramatically accelerates meta-convergence compared to first-order baselines.

Results:

• Faster meta-convergence: CTS reaches lower training loss far earlier than competing methods like Reptile, Leap, and MAML variants.
• Better generalization: Across fine-grained and heterogeneous tasks, CTS yields higher test accuracy.
• Efficiency: CTS achieves superior performance with fewer cumulative inner-gradient steps.

Figure 9. CTS maintains higher accuracy per inner step, indicating its superior sample efficiency.

Ablation studies confirmed that trajectory shifting is critical. Removing the directed shift (“No Shifting”) or randomizing the direction eliminates CTS’s advantage.

Figure 10. Ablation demonstrates that CTS’s directed shifting drives improvements.

---

3. Improving ImageNet Pre-Trained Models

Finally, the authors tested whether CTS could improve on standard ImageNet finetuning—especially in limited data scenarios.

They meta-trained models starting from ImageNet initialization using subsets split by WordNet hierarchy and tested on nine classification datasets, each with 1,000 training examples.

Method	CIFAR100	CIFAR10	SVHN	Dogs	Pets	Flowers	Food	CUB	DTD	Avg.
ImageNet Pre-training	41.95	81.60	60.09	55.56	83.48	87.01	36.95	34.32	59.39	60.04
+ MTL	42.79	82.33	59.05	55.00	83.29	87.04	36.84	34.19	58.86	59.93
+ Reptile	47.98	84.58	62.39	56.97	84.25	87.22	37.35	35.44	58.98	61.68
+ CTS (Ours)	48.34	84.42	62.82	57.53	84.65	87.54	37.84	36.40	59.53	62.12

Table 1. CTS-enhanced initializations outperform standard ImageNet fine-tuning and other baselines in low-data regimes.

Figure 11. CTS yields the greatest improvements with small datasets, acting as a strong regularizer against overfitting.

These results show that meta-learning, recalibrated through CTS, can refine even state-of-the-art pre-trained models. The smooth initialization learned through CTS acts as an implicit regularizer, boosting generalization for scarce, fine-grained datasets.

---

Conclusion

Scaling meta-learning beyond few-shot tasks has long been hindered by the extreme computational burden of long optimization trajectories. Continual Trajectory Shifting cuts through this limitation with a first-order approximation that shifts inner-learning trajectories alongside frequent meta-updates.

By doing so, CTS:

1. Drastically improves efficiency — Frequent meta-updates accelerate convergence.
2. Finds better initializations — Implicit meta-level curriculum avoids poor local minima.
3. Outperforms strong baselines — Works better than multi-task learning, standard fine-tuning, and previous meta-learning methods.

This approach expands the practical reach of meta-learning to real-world, large-scale scenarios. The result is a universal initialization that adapts rapidly and reliably across diverse, high-dimensional tasks—bringing the promise of meta-learning closer to widespread application in mainstream deep learning.