Machine Learning (ML) and broadly Artificial Intelligence (AI) technologies have caused significant disruptions by automating several aspects of human and/or machine functioning. From automotive and healthcare to legal and retail experience, the disruption is significant and is expected to be sustained in the future. ML/AI technologies use data to learn rules/patterns associated with tasks; these rules (e.g., Bayesian conditioning) are then used to make decisions which be used to actuate without needing human supervision (referred to as automation in this article). These automations, while being “accurate” (i.e., making the right decision and actuating as expected) on a majority of instances, are highly-unlikely to achieve sustained 100% precision across the entire space of actions.
In part, this lack of perfection can be attributed to a combination of the lack of:
data relating to rare events
the context in which seemingly correct decisions could lead to adverse effects downstream
feedback mechanisms to point out which corner cases need changes in rules to make correct decisions in the future
Knowing that these widely deployed technologies will not be perfect, now or in the foreseeable future, requires us as a community (i.e., academic researchers, developers, industry leaders, and technology adopters) to think about how we can design, introduce and operationalize “failsafe protocols.” Particularly, where human experts can serve as “AI-chaperones,” allowing for AI technologies to bloom (akin to children enjoying company in adult supervision), while also telling the AI models where the judgment was wrong, and what is the course of retraining and reengineering to avoid incorrect actuations in future (akin to children being told to correct behavior if needed). Just as children evolve to become well-behaved adults in socially complex situations when chaperoned, AI systems with “human AI-chaperons” could also evolve into reliable, complex decision-making engines that earn the long-term trust of its adopters.
Developing AI technologies with “failsafe protocols,” allowing humans to avert rare but catastrophic mishaps, will be a significant avenue for innovation in the future. The following challenges will have to be overcome in developing fail-safe protocols when using AI technologies.
First, identifying when to engage human experts, i.e., adopters (users needing the technology) to enumerate to developers as many possible states of failures and agree on when to “hand-off” operations based on human judgment. There is a responsibility on the adopter’s end to seek data on as many routine and corner cases of operations as possible and continue to measure such data even after deployment.
Second, arriving at a consensus on when models have to be retrained to handle new states of failures that are learned during operations following deployment. To achieve this objective, both adopters and developers will have to collaborate on defining performance metrics that will warrant an “update” to the underlying rule sets that are used to actuate.
Finally, introducing high degrees of interpretability in model outputs allow humans to quickly identify why the model arrived at a particular decision and what are the alternative actions any human can take to avoid a catastrophic outcome. Achieving high degrees of interpretability requires developers to have a deep algorithmic understanding of AI algorithms and provide “simple English readouts” of features and how they contribute to decision-making capabilities. An example of human-engagement proven to be essential in seeking regulatory approvals for automation technologies is self-driving cars. Despite self-driving cars being able to “drive themselves,” not only are humans expected to be behind the wheels by law, but are also expected to engage with the car’s controls at varying intervals of time or when the car’s automation system is unsure of how to actuate based on scanned data.
Automating simple tasks will be low-hanging fruits to convince adopters that these technologies can “make fast disruptions.” However, as the complexities of tasks requiring automation increases, we assert that significant coordination between developers (either in research or industry settings) and adopters (i.e., customers/consumers) is needed to introduce “failsafe protocols,” where human engagement to avert rare, but damaging outcomes is possible. Introducing this notion will likely earn users’ trust, and most importantly, introduce industry-standards for what constitutes dependable ML/AI-based automation systems. We conclude this article with the enthusiasm that “safe and dependable” ML/AI-based automation systems are here to stay and will become ubiquitous across all industry domains.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the respective institutions or funding agencies.