Many industries expect that artificial intelligence (AI) will transform the way they operate and do business within the next five years. When such industries effectively deploy artificially intelligent systems, these will act autonomously upon certain events, which inevitably implies that fewer and fewer decisions will be made by humans. However, that must not be seen as a negative development, since it will allow humans to focus on higher level strategic and conceptual decisions, as they can leave the granular level of detail to the autonomous agents. 

As machines make more and more decisions, one obvious question needs to be answered: Are their decisions reliable? This can be answered by checking the outcomes and quantifying how often the AI system has made the right decision. But the machine will make a mistake every now and then. While we often say that “mistakes are human,” the level of tolerance applied to humans does not carry over to machines. That said, people expect machines to be at least close to error free. 

 

What if a machine makes a wrong decision? 

While there seems to be a general expectation that machines are without errors, people also acknowledge that nothing is completely without fault. Therefore, it is fair to say that even according to the general consensus, machines will at some point make a wrong decision. 

When a person makes a wrong decision, often our first reaction is to talk to that person and understand how he or she made the wrong call. While talking to a machine may, in many cases, be more of a challenge, we tend to a have a similar reaction. We want to know how the machine arrived at the wrong decision. We want to be able to trace back every step that led the machine up to doing so. As natural as this may sound, many state-of-the-art AI systems do not offer the option to explain their decisions. Those AI systems are called black box models. 

 

Black box versus interpretability: choose, but choose wisely

If we want to understand the AI systems that we deploy, then why do we implement black box solutions? In short, it’s because black box models tend to be better predictors across a wide range of applications. 

To quote Yann LeCun, NYU professor of AI and one of the global thought leaders on deep learning: given the choice between a 90% accurate explainable model and a 99% accurate black box model, most industrial experts would opt for the black box model. From his point of view, that implies that the industrial user will just want some sort of reassurance: knowing that the model will work well, not necessarily how. 

However, not all experts in the field agree, and for a diverse set of reasons. At first, an improvement from 90% to 99% accuracy may be convincing. Yet there will be a set of industrial applications where those numbers are still a dream. If black box models increase accuracy from 60% to, say, 70%, which is not unrealistic, then the 70% accurate model may still not be trusted by users. 

Beyond that, a drastic performance improvement of black box over explainable machine learning models does not always substantiate either depending on data availability, data quality and how suitable the model is for the specific domain. But still, if we can get the 99% accurate black box model, we would use it, right? 

 

And what if every case matters? 

Artificial intelligence has penetrated our everyday lives, and we rely on AI-based tools often without being aware. Many of us rely on smartphone map navigation, on internet search, on the gig economy to get a ride or an overnight stay or on the availability of scooters and city bikes downtown. None of these examples would work without AI engines under the hood. 

Most users would classify these apps as sufficiently accurate: for instance, when they use smartphone navigation, they can assume that they will flawlessly arrive where they need to be. Just every so often, the route proposed by the app doesn’t turn out to be optimal, but beyond maybe some instantaneous anger the user will keep using and trusting the app. 

The apps cited here are tools that make our lives more convenient, and for such tools we are willing to forgive mistakes as long as they are rare. But how about when we apply for a mortgage and our credit is denied in spite of having an impeccable payment track record? We would want to know why our credit is being denied, and we would most likely not be satisfied by the answer that the “algorithm has it right for 99.9% of all cases.” 

The distinction is that each decision to receive or be denied credit may have a profound impact on that individual’s life. As soon as each case connects to a human life, different standards apply. Negative impact from false algorithmic decisions must be minimized and explainability becomes a necessity rather than a nice-to-have. This picture can get more pronounced, such as in medical AI, where AI decisions could mean life or death.

 

When AI needs to be explainable, what can be done? 

Well, this is where it can get technical. Roughly, three approaches can be pursued. At first, intrinsically explainable models can be deployed. These models often have a lower accuracy than black box models, even though the science is evolving fast. 

Second, surrogate models can be deployed that are only used to try and explain the black box model. While often successful, these surrogate models may be flawed themselves and be locally unstable. Finally, it is also possible for models to learn explanations along with decisions, but that requires that a big data set of explanations is available. 

In short, there is no winner-take-all solution (yet). What really matters is that whenever life-changing decisions are made by algorithms, there is some degree of model explainability and that those explanations can be used by the experts who manage the models to overrule single decisions. It is encouraging to see that societies are increasingly aware of this need and that legislation in this direction is being enacted in various countries. 

 

Do the process industries need explainable AI? 

Does every case ever matter in the process industries? In brief: yes. There are three main areas where individual faulty predictions could have a huge impact. The most obvious area is process safety. Today, deployment of black box AI in safety applications is restricted and, in my opinion, that that will not change any time soon. We need to make sure that we do everything we can to avoid loss of life or grave injuries due to process accidents. Should we be so unfortunate to be part of a process accident we would be required to investigate the cause. This includes the AI involved in the series of events leading up to the accident. 

Similar considerations can be held to address environmental concerns. It would be unacceptable to cause a toxic wastewater accident due to an impenetrable decision taken by a black box algorithm. Finally, product quality may often require explainable AI as well. When an algorithm steers a plant in a direction such that it produces out-of-specification product the responsible plant manager and engineers will want to investigate how the algorithm got there and how it could be tweaked to avoid the same result in the future. 

While there will be some checks in place beyond the AI algorithm’s predictions to make sure that out-of -spec material doesn’t get shipped to the customer, its mere production may imply a huge financial loss due to wasted raw materials and energy, and failure to meet product demand and deliver on time. 

 

Then the process industries are all about interpretable models? 

There is ample room for deployment of black-box-driven AI systems in the process industries. The process industries are in some way a microcosm of our broader society. Black box AI systems have ample opportunity to enhance or facilitate the engineer’s work. 

Imagine AI assisting an engineer who designs a plant, suggesting optimal plant design at every level of detail. This will drastically increase the engineer’s productivity — and the engineer is still in control in case the AI drifts. Or, imagine AI assisting process engineers in analyzing the root cause of an event in the plant without engineers having to manually sift through years of data history. 

 

To learn more about the AspenTech POV on AI, watch our newest video here.