HalpernSGD: A Halpern-Inspired Optimizer for Accelerated Neural Network Convergence and Reduced Carbon Footprint

This research aims at focusing attention on Halpern iteration, a technique that can be exploited to define optimizers in neural network settings to outperform numerous current state-of-the-art methods. More specifically, we introduce HalpernSGD, an innovative network optimizer that leverages Halpern’s method to enhance the rate of convergence of the Stochastic Gradient Descent (SGD), reducing the carbon footprint associated with neural network training. HalpernSGD exhibits a quadratic rate of convergence compared to the one of SGD, without compromising the accuracy of the model. We compared it with SGD and Adam through experiments that demonstrate HalpernSGD’s superior efficiency by significantly reducing the number of epochs required for convergence, thereby lowering energy consumption and carbon emissions. The study also identifies potential improvements in Adam’s approach to stability and convergence, suggesting a future direction for developing combined optimizers.