HUMAN-IN-THE-LOOP MACHINE LEARNING ARCHITECTURE FOR HIGH-RISK APPLICATIONS

Abstract

The rise of Artificial Intelligence (AI) and Machine Learning (ML) has had a profound impact on various industries. As self-learning and autonomous systems are increasingly relied upon to make decisions, the safety and well-being of humans and society at large become more reliant on these technologies. In high-risk and safety-critical applications, such as the health industry and autonomous driving, incorrect predictions by AI can have disastrous consequences. It is therefore crucial to ensure the safe use of AI in these contexts. One way to address the challenges associated with machine learning for sensitive applications is to utilize a Human-In-The-Loop approach. By combining the strengths of AI and human expertise, this approach aims to minimize the burden on professionals while improving the reliability of the system. The core idea is to automate decision-making for the majority of instances, while flagging difficult-to-classify instances for human experts to evaluate. This approach not only reduces the burden on professionals but also ensures the safety and reliability of the system. In this paper, we present a novel two-layer Human-In-The-Loop architecture designed for sensitive applications. Our approach is based on the central principle of learning from mistakes to improve the reliability of the system. Specifically, we propose a model that can forecast the type and class of errors for future estimations improvement. We demonstrate the feasibility of our architecture by conducting an empirical study focused on two distinct applications: agitation detection in dementia patients, Arabic speech recognition. Our two-layer classification system aims to improve the overall performance of the classifier by increasing its sensitivity, while directing difficult-to-classify instances to human experts for further evaluation. Overall, our proposed architecture provides a promising solution for sensitive applications that require high levels of reliability and accuracy. By combining the strengths of AI and human expertise, we believe our approach can significantly improve the safety and well-being of individuals and society at large.