Adaptive Perturbation Radius For Sharpness-Aware Minimization

Abstract

Sharpness-Aware Minimization (SAM) is an optimization method that improves the generalization of deep neural networks by encouraging convergence toward flatter regions of the loss landscape. However, SAM relies on a fixed perturbation radius, denoted by ρ, which may not remain suitable throughout training as the geometry of the loss surface changes. This thesis investigates adaptive strategies for controlling ρ using sharpness-related indicators derived from first-order loss variation and directional curvature information. The proposed approaches are evaluated on image classification tasks using convolutional neural networks trained on CIFAR datasets. Experiments compare stochastic gradient descent, fixed-radius SAM, and adaptive variants of SAM. The results show that the proposed metrics capture meaningful aspects of training dynamics, while the adaptive-radius strategies require further refinement to consistently improve generalization over fixed-radius SAM.