MALT: Lightweight Curvature-Aware Muon via Diagonal Preconditioning
MALT (Muon Augmented by Lightweight Two-sided Preconditioning) is a new optimizer that uses lightweight diagonal preconditioners to reduce Muon's sensitivity to curvature anisotropy. It orthogonalizes preconditioned momentum via Newton-Schulz iterations and uses norm grafting. MALTER adds adaptive stepsize rescaling for robustness. Convergence guarantees are provided for MALT in stochastic non-convex settings. Experiments on GPT-2 Small, Medium, and Large pretraining show improvements.
发展脉络
- 首次出现MALT: Lightweight Curvature-Aware Muon via Diagonal PreconditioningarXiv cs.LG
- 当前判断The development of MALT reflects ongoing competition to improve training efficiency for large language models. Optimizers that reduce sensitivity to curvature anisotropy could lead to faster convergence and lower training costs, which is valuable for organizations training large models. The lightweight design suggests a practical approach that can be integrated into existing training pipelines without significant overhead. This may influence the choice of optimizers in future pretraining efforts, potentially shifting away from AdamW as the default.Agent Pulse · 分析
Muon has emerged as a promising alternative to AdamW for language model pretraining by orthogonalizing momentum matrices using Newton-Schulz iterations. However, Muon does not explicitly account for curvature geometry, remaining sensitive to curvature anisotropy. MALT bridges this gap by using lightweight two-sided diagonal preconditioners to approximate curvature geometry with low memory and computational overhead. It orthogonalizes the preconditioned momentum and uses norm grafting to control update magnitude. MALTER further improves robustness to stochastic gradient noise via adaptive stepsize rescaling. Convergence guarantees are provided for MALT in the stochastic non-convex setting. Experiments on GPT-2 Small, Medium, and Large pretraining show that the proposed methods improve performance.
MALT introduces a lightweight two-sided diagonal preconditioning approach to Muon, aiming to capture curvature geometry without the overhead of full-matrix preconditioners. This suggests a trend toward curvature-aware optimizers that balance performance and computational efficiency. The use of norm grafting and adaptive stepsize rescaling (MALTER) indicates a focus on stability under stochastic noise. The convergence guarantees in non-convex settings provide theoretical backing. The next signal to watch is whether MALT or similar methods are adopted in larger-scale pretraining runs and whether they consistently outperform AdamW and Muon across diverse architectures.
The development of MALT reflects ongoing competition to improve training efficiency for large language models. Optimizers that reduce sensitivity to curvature anisotropy could lead to faster convergence and lower training costs, which is valuable for organizations training large models. The lightweight design suggests a practical approach that can be integrated into existing training pipelines without significant overhead. This may influence the choice of optimizers in future pretraining efforts, potentially shifting away from AdamW as the default.
For organizations training large language models, MALT offers a potential reduction in training time and compute costs due to faster convergence. The lightweight overhead makes it easy to adopt without significant infrastructure changes. If MALT proves consistently superior to AdamW, it could become a standard optimizer, providing a competitive edge to early adopters. This could also influence the design of training accelerators and software libraries.
Future work may extend MALT to other model architectures and larger scales, and investigate its interaction with distributed training and mixed precision. The adaptive stepsize rescaling in MALTER could be further refined. Adoption by major training frameworks or research labs would be a key signal. Additionally, theoretical analysis might be extended to other settings, such as convex or non-smooth objectives.