On the discrepancy principle for stochastic gradient descent
Tim Jahn, Bangti Jin
Abstract
Abstract Stochastic gradient descent (SGD) is a promising numerical method for solving large-scale inverse problems. However, its theoretical properties remain largely underexplored in the lens of classical regularization theory. In this note, we study the classical discrepancy principle, one of the most popular a posteriori choice rules, as the stopping criterion for SGD, and prove the finite-iteration termination property and the convergence of the iterate in probability as the noise level tends to zero. The theoretical results are complemented with extensive numerical experiments.
Topics & Concepts
MathematicsStochastic gradient descentRegularization (linguistics)Applied mathematicsA priori and a posterioriConvergence (economics)Inverse problemDescent directionGradient descentMathematical analysisComputer scienceEpistemologyArtificial intelligencePhilosophyEconomic growthEconomicsMachine learningArtificial neural networkSparse and Compressive Sensing TechniquesNumerical methods in inverse problemsStochastic Gradient Optimization Techniques