
The figure above illustrates model performance under different complexity settings on simple compositional tasks (left) and image compositional tasks (right).
Recently, the Deep Learning Theory Team from the Institute of Natural Sciences and the School of Mathematical Sciences at Shanghai Jiao Tong University, led by Prof. Zhi-Qin John Xu and Prof. Yaoyu Zhang together with their students, conducted a systematic study on how complexity control influences compositional generalization in Transformer models. Their findings reveal that lower model complexity—achieved, for example, through smaller initialization scales or larger weight decay coefficients—significantly promotes the learning of reasoning-structured compositional rules, thereby enhancing out-of-distribution (OOD) generalization. As the core architecture of large language models, Transformers have achieved groundbreaking success in natural language processing and computer vision. However, achieving genuine compositional generalization in structurally simple yet compositionally challenging tasks remains a fundamental open problem. This work fills an important theoretical and mechanistic gap in understanding Transformer generalization. The study, titled “Complexity Control Facilitates Reasoning-Based Compositional Generalization in Transformers,” has been published in the leading journal IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).
Although Transformers demonstrate remarkable performance in language and vision tasks, they may still fail to generalize on simple compositional tasks under distribution shifts. Understanding the underlying causes of such failures and identifying effective solutions has long been a central research question in Transformer generalization.
To address this issue, the research team innovatively adopted small initialization as an analytical entry point. They systematically examined the relationship between the initialization factor γ and model generalization behavior, and further extended this framework to common regularization techniques, such as weight decay. This led to the proposal of a unified theoretical framework of complexity control.
Specifically, the study constructed interpretable synthetic compositional tasks based on anchor functions. Through carefully designed data partitioning, the authors predefined three potential solution types:
Generalization solution (low complexity): The model learns primitive-level functions and composes them to derive composite functions, achieving true OOD generalization.
Symmetric solution (medium complexity): The model memorizes composite function mappings but does not abstract underlying primitive functions.
Memorization solution (high complexity): The model overfits by memorizing all combinations in the training data.
The complexity of these three solution types increases sequentially. Experimental results show that: With large initialization scales, the model converges to high-complexity memorization solutions. With standard initialization, it tends to learn symmetric solutions. And with small initialization, it converges to generalization solutions.
Further, through token masking analysis, the authors revealed distinct internal mechanisms across different phases. Dimensionality-reduction visualizations show that:
Transitioning from memorization to symmetric solutions, the model begins to recognize the commutativity of composite functions and clusters equivalent structures in latent space.
Transitioning from symmetric to generalization solutions, the model learns to abstract primitive functions, enabling step-by-step reasoning-based computation.
The study also finds that Transformers trained with small initialization exhibit a clear neuron condensation phenomenon. Condensation is a common characteristic in nonlinear neural network training, where neuron weights align along a few dominant directions. This effectively reduces the network’s functional dimensionality, allowing a highly complex network to maintain low effective complexity while still fitting the data.
The authors further generalize small initialization together with weight decay under the unified concept of complexity control. Across synthetic compositional tasks, natural language tasks, and image compositional tasks, complexity-controlled models consistently demonstrate stronger generalization performance.
Zhongwang Zhang and Pengxiao Lin are co-first authors and Prof. Zhi-Qin John Xu is the corresponding author. Prof. Xu and Prof. Zhang are jointly appointed at the Institute of Natural Sciences and the School of Mathematical Sciences at Shanghai Jiao Tong University and is also a graduate of the inaugural 2008 class of the Zhiyuan College’s science program. Zhongwang Zhang is a fifth-year Ph.D. student in the team, and Pengxiao Lin is a third-year Ph.D. student.
In addition, the team previously published preliminary results of this work at NeurIPS 2024 under the title “Initialization is Critical to Whether Transformers Fit Composite Functions by Reasoning or Memorizing.”
Currently, the Institute of Natural Sciences and the School of Mathematical Sciences at Shanghai Jiao Tong University have established a strong research group dedicated to foundational deep learning theory. The team’s research focuses primarily on two major directions: the Frequency Principle and Neural Condensation.
This research was supported by the National Key R&D Program of China (Youth Scientist Project 2022YFA1008200), the National Natural Science Foundation of China (92270001, 12371511, 1242211, 12101402), Lingang Laboratory (LG-QS-202202-0), the Shanghai Science and Technology Major Project (2021SHZDZX0102), and computational resources from Siyuan-1 Supercomputing Platform of Shanghai Jiao Tong University, the Supercomputing Center of the School of Mathematical Sciences, and the Student Innovation Center.
About TPAMI
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) is one of the world’s premier journals in artificial intelligence, pattern recognition, and computer vision. Its latest impact factor (2023) is 18.6. TPAMI ranks among the top journals in computer science and engineering according to Google Scholar’s H-index. Known for its rigorous review standards and deep theoretical contributions, TPAMI holds authoritative influence in the field of artificial intelligence.