Abstract:
Predicting the running time of applications is essential for High-Performance
Computing (HPC) job schedulers and resource managers. However, accurately estimating the execution time of previously unseen applications or those operating on
variable-sized datasets remains a challenging task. When an application’s running
time is underestimated and exceeds its allocated time budget, or when a higher-priority application arrives, it can result in costly delays and resource wastage within
HPC environments.
In response to these challenges, we introduce AWTY, a comprehensive workflow designed to predict if an executing application is close to completion. AWTY
leverages historical application profiles to gain insights into the characteristics of
an application’s final phases. This data is then used to train specialized classifiers
capable of determining whether an executing application has entered its concluding
stages.
Our approach incorporates both single-application classifiers, tailored for previously encountered applications, and general classifiers, suitable for applications not
previously observed. Our evaluation shows that AWTY exhibits promising accuracy
in predicting whether applications are in their final stage of execution.