Predicting if Executing Applications are Near Completion

Abstract

Predicting the running time of applications is essential for High-Performance Computing (HPC) job schedulers and resource managers. However, accurately estimating the execution time of previously unseen applications or those operating on variable-sized datasets remains a challenging task. When an application’s running time is underestimated and exceeds its allocated time budget, or when a higher-priority application arrives, it can result in costly delays and resource wastage within HPC environments. In response to these challenges, we introduce AWTY, a comprehensive workflow designed to predict if an executing application is close to completion. AWTY leverages historical application profiles to gain insights into the characteristics of an application’s final phases. This data is then used to train specialized classifiers capable of determining whether an executing application has entered its concluding stages. Our approach incorporates both single-application classifiers, tailored for previously encountered applications, and general classifiers, suitable for applications not previously observed. Our evaluation shows that AWTY exhibits promising accuracy in predicting whether applications are in their final stage of execution.