Metacognition for artificial intelligence system safety – An approach to safe and desired behavior
Bonnie L. Johnson
Abstract
Advances in computational thinking and data science have led to a new era of artificial intelligence systems being engineered to adapt to complex situations and develop actionable knowledge. These learning systems are meant to reliably understand the essence of a situation and construct critical decision recommendations to support autonomous and human–machine teaming operations. In parallel, the increasing volume, velocity, variety, veracity, value, and variability of data is confounding the complexity of these new systems – creating challenges in terms of their development and implementation. For artificial systems supporting critical decisions with higher consequences, safety has become an important concern. Methods are needed to avoid failure modes and ensure that only desired behavior is permitted. This paper discusses an approach that promotes self-awareness, or metacognition, within the artificial intelligence systems to understand their external and internal operational environments and use this knowledge to identify potential failures and enable self-healing and self-management for safe and desired behavior.