Making pre-training predictable, stable, and efficient.
I am a Ph.D. candidate at the Institute of Computing Technology, Chinese Academy of Sciences, and a research intern with the RedNote Dots pre-training team. I study the geometry of optimization trajectories, the spectral dynamics behind training phases, and the scaling rules that turn those mechanisms into reliable large-model recipes.
01 / Experience
Research shaped by production constraints, from optimizer behavior to compute-aware scaling.
Research Intern · RedStar / ACE Talent Program
Now Oct 2026 — Present
Contributing to a production-oriented overtraining scaling ladder for LLM pre-training—connecting controlled small-model studies with reliable decisions at target-model scale.
Research Intern · Qingyun Talent Program
Apr — Oct 2026
Worked on production optimizers, training stability, and scaling laws. The work linked effective learning-rate dynamics to transferable schedules and Hyperball optimizers.
Research Intern · Pre-Training Algorithms Team
Nov 2025 — Mar 2026
Worked on production pre-training optimizers and late-stage model merging, studying how the geometry of training trajectories can be converted into cheaper improvements after training.
02 / Research roadmap
I specialize in large language model pre-training, with a focus on optimization, training dynamics and stability, scaling laws, and model merging. My work follows a consistent loop: identify structure in large-scale training, explain it through optimization and spectral dynamics, then turn the mechanism into a robust recipe that survives scale.
Find the quantities that actually organize training: effective learning rate, sharpness, singular spectra, and checkpoint trajectories.
Signal → structureUse subspace structure to reason about Muon, Shampoo, momentum, and model merging—not as isolated tricks, but as geometric operators.
Structure → algorithmTranslate controlled experiments into stable schedules, optimizer choices, and overtraining ladders for production pre-training.
Algorithm → recipe03 / Publications
Seven projects across pre-training dynamics, optimizer geometry, and scalable training. * equal contribution
ELR view
Reframes the update scale through an effective learning rate that tracks angular parameter motion. The signal predicts loss more consistently across width and depth and motivates a minus-square-root schedule that transfers from speedruns to a 6B-MoE scaling ladder.
Fig. 5
Shows that late-stage merged checkpoints collapse onto a stable, nearly rank-1 flow even when raw updates oscillate. Extra-Merge extrapolates along that direction—improving models without another gradient step across GPT-2, LLaMA, Pythia-12B, and Muon settings.
Fig. 1
Identifies Stability of Singular Distribution (SoSD): normalized singular spectra settle early, at the same point that loss enters its slow phase. The effect persists across model families, schedules, weight decay, and AdamW/Muon, giving a spectral explanation of two-phase training.
Subspace view
Separates updates into a high-curvature dominant subspace and a descent-efficient bulk subspace, then suppresses the former while amplifying the latter. A block-wise, history-based implementation acts as a scalable optimizer plug-in and accelerates LLaMA pre-training.
Fig. 1
Explains why Muon’s best whitening granularity changes during training through a trade-off between group-wise whitening gain and grouping-induced norm cost. Group Muon makes head grouping an optimizer control and outperforms both full-QKV and fully head-wise variants.
Fig. 1
Builds a unified Bregman-divergence view of Shampoo’s Kronecker approximation. It shows how each divergence redistributes spectral error and compensates differently for finite-sample bias, then connects those choices to GPT-2 pre-training behavior.
Spectral view
Characterizes how Frobenius, von Neumann, and LogDet divergences allocate unavoidable Kronecker error across the covariance spectrum. The analysis motivates subspace-aware BregTop using the observed alignment between Hessian and covariance top eigenspaces.
05 / Contact
I am always interested in clear empirical questions about optimization, training dynamics, and scale.