Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data

Дата публикации: 17-08-2026 20:26:00

Neural collapse (${\cal NC}$) is a phenomenon that emerges at the terminal phase of the training (TPT) of deep neural networks (DNNs). The features of the data in the same class collapse to their respective sample means and the sample means exhibit a simplex equiangular tight frame (ETF). In the past few years, there has been a surge of works that focus on explaining why the ${\cal NC}$ occurs and how it affects generalization. Since the DNNs are notoriously difficult to analyze, most works mainly focus on the unconstrained feature model (UFM). While the UFM explains the ${\cal NC}$ to some extent, it fails to provide a complete picture of how the network architecture and the dataset affect ${\cal NC}$. In this work, we focus on shallow ReLU neural networks and try to understand how the width, depth, data dimension, and statistical property of the training dataset influence the neural collapse. We provide a complete characterization of when the ${\cal NC}$ occurs for two or three-layer neural networks. For two-layer ReLU neural networks, a sufficient condition on when the global minimizer of the regularized empirical risk function exhibits the ${\cal NC}$ configuration depends on the data dimension, sample size, and the signal-to-noise ratio in the data instead of the network width. For three-layer neural networks, we show that the ${\cal NC}$ occurs as long as the first layer is sufficiently wide. Regarding the connection between ${\cal NC}$ and generalization, we show the generalization heavily depends on the SNR (signal-to-noise ratio) in the data: even if the ${\cal NC}$ occurs, the generalization can still be bad provided that the SNR in the data is too low. Our results significantly extend the state-of-the-art theoretical analysis of the ${\cal NC}$ under the UFM by characterizing the emergence of the ${\cal NC}$ under shallow nonlinear networks and showing how it depends on data properties and network architecture.

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	Optimization and Generalization of Gradient Descent for Shallow ReLU Networks with Minimal Width	0	3.84	17-08-2026
2	High-Dimensional Analysis of Gradient Flow for Extensive-Width Quadratic Neural Networks	0	8.7	17-08-2026
3	A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks	0	10.97	17-08-2026
4	Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks	0	5	17-08-2026
5	Optimal Approximation and Generalization Errors for Deep Convolutional Neural Networks	0	7.31	17-08-2026
6	Statistical Learning Theory for Neural Operators	0	10.21	17-08-2026
7	Finite Neural Networks as Mixtures of Gaussian Processes: From Provable Error Bounds to Prior Selection	0	4.23	17-08-2026
8	Reparameterized Complex-valued Neurons Can Efficiently Learn More than Real-valued Neurons via Gradient Descent	0	11.86	17-08-2026
9	Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes	0	7.52	17-08-2026
10	End-to-End Deep Learning for Predicting Metric Space-Valued Outputs	0	10.66	17-08-2026

Классификация: . Схожих патентов: 0. Схожих новостей: 10. Тональность: 0. Информативность: 4.36. Источник: jmlr.org.