Stochastic Gradient Methods: Bias, Stability and Generalization

Дата публикации: 17-08-2026 20:26:00

Recent developments of stochastic optimization often suggest biased gradient estimators to improve either the robustness, communication efficiency or computational speed. Representative biased stochastic gradient methods (BSGMs) include Zeroth-order stochastic gradient descent (SGD), Clipped-SGD and SGD with delayed gradients. The practical success of BSGMs motivates a lot of convergence analysis to explain their impressive training behaviour. As a comparison, there is far less work on their generalization analysis, which is a central topic in modern machine learning. In this paper, we present the first framework to study the stability and generalization of BSGMs for convex and smooth problems. We introduce a generalized Lipschitz-type condition on gradient estimators and bias, under which we develop a rather general stability bound to show how the bias and the gradient estimators affect the stability. We apply our general result to develop the first stability bound for Zeroth-order SGD with reasonable step size sequences, and the first stability bound for Clipped-SGD. While our stability analysis is developed for general BSGMs, the resulting stability bounds for both Zeroth-order SGD and Clipped-SGD match those of SGD under appropriate smoothing/clipping parameters. We combine the stability and convergence analysis together, and derive excess risk bounds of order $O(1/\sqrt{n})$ for both Zeroth-order SGD and Clipped-SGD, where $n$ is the sample size.

Схожие новости

#	Наименование новости	Тональность	Информативность	Дата публикации
1	Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent	0	6.66	17-08-2026
2	Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descent	0	6.38	17-08-2026
3	A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization	0	8	17-08-2026
4	Convergence of Decentralized Stochastic Subgradient-based Methods for Nonsmooth Nonconvex Optimization	0	8.78	17-08-2026
5	Optimization and Generalization of Gradient Descent for Shallow ReLU Networks with Minimal Width	0	3.84	17-08-2026
6	The Sample Complexity of Parameter-Free Stochastic Convex Optimization	0	5.7	17-08-2026
7	Near-optimal Delta-convex Estimation of Lipschitz Functions	0	9.71	17-08-2026
8	Graph-based Clustering Revisited: A Relaxation of Kernel k-Means Perspective	0	10.94	17-08-2026
9	Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes	0	7.52	17-08-2026

Классификация: . Схожих патентов: 0. Схожих новостей: 9. Тональность: 0. Информативность: 6.3. Источник: jmlr.org.