Beyond Adam: How AI Can Learn to Optimize Neural Networks

Training deep neural networks remains one of the most frustratingly manual aspects of modern machine learning. Researchers meticulously tweak dozens of optimizer hyperparameters—learning rate, momentum, weight decay—hoping to hit the sweet spot where a model learns smoothly. Mistune just one, and your training can diverge or stagnate.

What if we could automate that entire tuning process? What if an AI could learn the art of optimization itself?

This is the promise of learned optimizers—algorithms that learn how to train neural networks by adjusting learning dynamics automatically, replacing traditional, hand-engineered approaches like SGD and Adam. Yet for years, progress in this direction has been hampered by a single, persistent problem: generalization. Learned optimizers often perform excellently when facing tasks similar to those they were trained on, but fail dramatically when applied to new architectures or data types.

A recent paper from OpenAI, “A Generalizable Approach to Learning Optimizers,” takes direct aim at this challenge. The researchers propose a paradigm shift: instead of trying to learn how to update model parameters directly, their system learns to update optimizer hyperparameters. This higher-level strategy allows the optimizer to generalize across tasks, architectures, and data modalities at a scale never seen before.

Their learned system—called the Learned Hyperparameter Optimizer (LHOPT)—achieves up to 2× speedups on ImageNet training and 2.5× speedup on large language models. Even more astonishingly, it generalizes from training tasks requiring only minutes of compute to real-world tasks requiring hundreds of GPU-days. This marks a breakthrough in designing self-improving optimizers that can match human tuning expertise while vastly reducing computational costs.

The Problem with Learning to Optimize

Earlier attempts to build learned optimizers focused on training neural networks that directly output parameter updates. Typically, these models use small RNNs that consume statistics like raw gradients or moving averages and decide how to modify each weight.

Unfortunately, this approach doesn’t scale or generalize. Raw statistics vary wildly across problems—what counts as a large gradient in one model could be negligible in another. These learned optimizers thus become highly task-specific and behave erratically on unseen architectures, loss functions, or scales. The result: optimized systems that work on toy problems but fail on realistic tasks.

The LHOPT Solution: A Generalization-First Approach

The OpenAI team takes a radically different angle. Their guiding principle is clear: design for generalization first. Every part of LHOPT—in its architecture, features, and reward design—serves that goal.

The method relies on a two-loop training structure, shown below.

Figure 1: Simplified view of LHOPT’s outer-inner training process. The inner loop performs standard model updates, while the outer loop uses reinforcement learning to adjust the optimizer’s hyperparameters.

1. The Inner Loop — Regular model training using an inner optimizer (in their experiments, a custom Adam-like variant).
2. The Outer Loop — Periodically, training pauses and an LSTM controller observes summary statistics—features like validation loss or gradient stability—and decides how to adjust the inner optimizer’s hyperparameters (learning rate, weight decay, etc.) before training continues.

By decoupling the outer loop from step-by-step gradient updates, LHOPT can reason at a higher level, making optimization decisions based on longer time horizons. It focuses not on immediate gain but on final performance.

The Senses: “Unitless” Features for Robust Perception

For the LSTM controller to reason across vastly different problems—from tiny MLPs to billion-parameter Transformers—it must see the world in a task-agnostic way. To achieve that, LHOPT uses unitless features: all inputs are expressed in normalized, relative form so that their numeric scales are comparable regardless of dataset or architecture.

Examples include:

• Log-ratio between training and validation loss (instead of raw values).
• Cosine similarity between gradients and momentums, indicating alignment rather than magnitude.
• Progress through training (a value between 0 and 1).
• Fraction of gradients clipped, a normalized stability signal.

One particularly clever invention is the CDF feature. Rather than using absolute values, the optimizer tracks how current statistics compare to their historical distribution. The system maps each observation to the [0, 1] interval via a Gaussian cumulative distribution function (CDF). For example, a CDF value near 1.0 indicates unusually high loss relative to the past. This enables the optimizer to detect plateaus and trends without knowing exact task-specific numeric values.

These features strip away dependence on raw units, allowing LHOPT’s policy to recognize patterns that generalize across very different learning environments.

The Hands: Relative Actions for Reactive Control

Just as the features are relative, so are LHOPT’s actions. Instead of assigning absolute hyperparameter values, the controller performs scaling—multiplying an existing hyperparameter by a factor (e.g., ×0.5 or ×2)—or logit shifting for parameters bounded between 0 and 1. These discrete relative actions improve training stability and prevent the model from memorizing fixed schedules.

Additionally, LHOPT includes a powerful exploration mechanism: checkpoint restarts. The controller can save model states, test risky hyperparameter changes, and, if instability occurs, revert to the saved checkpoint. This capability helps LHOPT explore aggressive optimization dynamics safely, improving its resilience and adaptability.

The Goal: Reinforcement Learning with a Self-Improving Reward

The outer loop is trained via reinforcement learning (PPO). Designing an appropriate reward function is crucial. Instead of directly using final validation loss—which can’t compare difficulty across tasks—the authors employ a dynamic baseline inspired by self-play.

For each task, LHOPT runs a baseline using an exponential moving average of its own past policy weights. It fits a power-law curve to the baseline’s learning trajectory. The LHOPT’s reward is the improvement over that curve. As training progresses, the baseline itself improves, creating a continually rising performance bar—effectively a self-improving learning process.

Putting LHOPT to the Test

The ultimate test for any learned optimizer is not reproducing its training results but generalizing to unseen real-world problems. The OpenAI team evaluated LHOPT on multiple challenging benchmarks far outside its training distribution, achieving impressive results.

Large-Scale Language Modeling

To push LHOPT to large-scale tasks, the researchers used it to train a 760M-parameter GPT‑2‑Large model on one epoch of WikiText‑103—a setup with no hyperparameter tuning.

Table 2: Language model performance on WikiText-103. LHOPT dramatically outperforms the baseline AdamW.

Figure 3: GPT‑2 learning curves, showing LHOPT’s nearly 2× speedup despite scaling to a task hundreds of times larger than training tasks.

Interestingly, the LHOPT’s curve rises slower early on than AdamW’s—a deliberate trade‑off resembling human‑engineered schedules like cosine decay. It sacrifices immediate progress for better long-term performance, a hallmark of sophisticated optimization behavior.

ImageNet Classification with ResNets

Next, the team tested generalization to novel architectures on ImageNet using ResNet models—a domain entirely absent during LHOPT’s training.

Table 3: ImageNet results showing ~2× reduction in training time and superior performance versus tuned AdamW.

Figure 4: LHOPT’s learned hyperparameter schedules enable faster convergence on ImageNet—an unseen domain.

Although SGD still holds the highest raw accuracy, LHOPT’s success without any tuning illustrates robust cross-domain generalization.

Beyond Vision and Text: MLPerf Benchmarks

To truly stress-test generalization, LHOPT was applied to two completely different tasks from the MLPerf suite:

1. Neural Collaborative Filtering (NeuMF) — Recommendation on MovieLens 1M.
2. Deep Speech 2 — Speech recognition on LibriSpeech.

Table 4: LHOPT surpasses baselines even on a recommendation model—an unseen modality.

Figure 5: Neural collaborative filtering performance shows LHOPT adapting effectively to new domains.

And for speech recognition:

Figure 6: LHOPT generalizes to audio tasks using FP16 training—both unseen conditions.

Without any tuning, LHOPT successfully optimized models across modalities it had never encountered, from images and text to recommendations and speech.

Transferring a Learned Schedule to GPT‑3‑Scale Models

Perhaps the most compelling demonstration of generalization comes from schedule transfer. The team trained a smaller LHOPT on a lightweight language modeling task, recorded the hyperparameter evolution, and reused that fixed schedule on a well‑tuned large‑scale language modeling codebase—similar to what is used to train GPT‑3.

Figure 2: Scaling laws showing consistent 2–2.5× speedups, growing to ~3.6× at GPT‑3 scale.

Remarkably, a single schedule generated on a small task provided performance improvements across orders of magnitude of model sizes and compute budgets—proving that LHOPT captures hyperparameter relationships that remain valid at scale.

Figure 8: The learned hyperparameter schedule transferred to large models. Its complex trajectories outperform handcrafted decays like cosine or linear.

This direct transferability means that LHOPT’s learned knowledge—expressed as hyperparameter trajectories—can be reused by practitioners without running the full reinforcement-learning optimizer.

Why It Works: Decoupling and Design Choices

A few elements underlying LHOPT’s success stand out:

• Decoupled optimization layers: The policy network operates separately from per‑parameter updates, freeing it from task-specific statistics like gradients.
• Feature normalization: Every input statistic is scaled and clipped, preventing numerical instability and helping robustness.
• Relative actions and random initialization: These mechanisms force reactive behavior instead of memorizing fixed hyperparameter settings.
• Reward‑driven improvement: Competing against its own moving‑average baseline ensures continuous progress.

Together, these choices yield an optimizer that performs strong updates only when necessary and otherwise conservatively maintains stability—mirroring expert intuition.

Conclusion: Toward Self‑Improving AI Optimizers

This research shows that learned optimizers can finally break out of the sandbox. By focusing entirely on generalization, LHOPT learns policies that transfer across datasets, architectures, and even input modalities. The resulting 2× to 2.5× speedups on crucial benchmarks like ImageNet and large‑scale language models represent tangible gains—saving substantial compute time and cost.

LHOPT’s design opens the door to broader possibilities:

• Applying learned hyperparameter control to reinforcement learning, GANs, or fine‑tuning pipelines.
• Combining LHOPT with per‑parameter learned optimizers for hierarchical control.
• Discovering dynamic architectures that adapt hyperparameters continuously during training.

The takeaway is profound: optimization doesn’t have to remain an art form of manual trial and error. With systems like LHOPT, AI can learn to optimize itself, accelerating the path toward autonomous, efficient deep learning at every scale.