Explainable AI: The data

scientists’ new challenge

Mouhamadou-Lamine DiopFollow
Jun 14, 2018
Authors: Lamine Diop and Jean Cupe

Before we start, let me mention that in this article I will use

without regards the words explainability and interpretability to
describe the same concept.

That being said, let’s imagine a situation where someone would

ask for a bank loan and the computer, powered by a
specialised Machine Learning algorithm in credit risk assessment,
denies the demand. The employee, being unable to explain the
reasons of such a categorical answer, would surely be left in a very
embarrassing position.

In a context where advances in Machine Learning are reaching

critical areas such as medicine, criminal justice systems and
financial markets, this kind of situation will surely tend to be more
frequent. In fact there is a growing concern around the
acceptance of AI agents and trust issues due to their
lack of explainability. For many years, priority has been given
to the performance over the interpretability leading to huge
advancements in several fields including computer vision, natural
language processing and sequence modeling. However, the crucial
questions driven by the reluctance to accept AI-based decisions
may lead to a whole new dynamic where explainability may be
among the key measures for evaluating models. And in this article,
I will discuss some concrete reasons why it would be a major
differentiator. Some approaches to develop powerful algorithms
while still retaining a good degree of interpretability will be also
suggested.
“ Having an accurate model is good, but
explanations lead to better products. “

Data science improvements have always been driven by the search

of the model with the best performance to address any problem.
From a mathematical point of view, the search of the model
focuses on the minimization of a cost function or the maximization
of a likelihood function. Thus the performance of the model is
measured almost exclusively on the results we can get according to
some rightly chosen metrics. This tendency has led to more and
more sophisticated algorithms to the expense of explainability. In
fact, some data scientists may even assert that “ What makes
machine learning algorithms difficult to understand is
also what makes them excellent predictors ”. They are
complex and that is what makes them look like “ black boxes
” for most field practitioners.

Explainability vs Accuracy seen through main modelization techniques

The previous image depicts 3 levels of interpretability for machine

learning algorithms:

 High interpretability: This level includes

traditional Regression Algorithms, Decision Trees and
Classification Rules. Basically they approximate monotonic
linear functions.
 Medium interpretability: This level includes more
advanced algorithms such as Graphical Models. They are
constrained to the approximation of monotonic non linear
functions.
 Low interpretability: This level includes advanced machine
learning techniques such as SVM, Ensemble Methods, and
Deep Learning. At best they provide feature importance
insights as a standard measure of interpretability.

Being explainability the measure of the degree to which a human

observer can understand the reasons behind a prediction made by
a model, it becomes increasingly more necessary to find the right
balance with the accuracy. It could be the key to making
algorithms as transparent as possible for day to day users. This
change of focus towards “ user centricity ” may probably make
the difference on the appropriation and acceptance of this
technology throughout a variety of domains, granting them a more
comfortable adaptation to carry out the tasks in a more realistic
way.

Although many core aspects of applied machine learning are the

same across different fields, they cannot be applied in every
industry. The machine learning approach is very different specially
in banks, insurance companies, healthcare providers and other
regulated industries. The reason is mainly that they are prone
to legal or even ethical requirements which tend to limit
more and more the use of black box models. As an illustration, I
may site the section 609(f)(1) of the Fair Credit Reporting Act:
“ The consumer shall be provided all of the key
factors that adversely affected the credit score
of the consumer in the model used, the total
number of which shall not exceed 4. ”

Most importantly this clearly means one should provide insights

about how the response of the model was obtained. The immediate
consequence is reducing the scope of possible approaches to the
simplest ones due to their easy explainability, unless we find ways
to add contexts to the predictions of state-of-the-art algorithms.

Not to mention also that the tendency is more to the accentuation

of regulation constraints rather than their slow-down.

An ever growing list of legal and ethical considerations surrounding businesses

All those factors contribute widely to the general reluctance of the

industry to adopt and deploy advanced data products powered by
Machine Learning systems. In front of such restraints, researches
are more and more focused on the identification of ways to deal
with the interpretability of models.

As far as I know, there are mainly two approaches towards

achieving explainability of models: global
explainability and local explainability. The former aims
at making the entire process of decision making
completely transparent and comprehensive while the
latter focuses on providing explanations for each
decisions.
“ When we start a Data Science project, it may
now be a best practice to determine first the
degree of interpretability we would like to
achieve. “

In fact, deciding the degree of interpretaility at first place will

guide the choice of algorithms and techniques we might
implement. Concretely it means whether go for a simple model
and make it more powerful (to achieve global explainability), or
use a complex model that we could make more interpretable (to
achieve at least local explainability). The following table presents
a non-exhaustive list of techniques to deal with the
interpretability of a model.

Maybe you have noted that there are more details about local
explainability than global explainability. This is mainly due to the
huge efforts that have been made to explain complex models but
with a local boundary approach.
Finally, when we implement our algorithm with the right
approach, the last interrogation lies on how to evaluate the
interpretability of the model. There are 2 approaches that lead to a
better representation of the degree of understanding granted by
the final model:

 Simulate data with known characteristics and intrinsic

implications to confront the interpretation of the model to the
prior knowledge that we have in that context.
 Test the stability under data perturbations by putting a
little noise in the explicatives variables. Trustworthy
explanations likely should not change drastically for minor
those minor changes.

Hope that you enjoyed your journey through this yet highly
important “recent” concern. If you have any question or remark, I
will be glad to welcome it for further discussions. In my next
publication I will directly address this problematic through a real
world use case to get a better understanding of all the implications
above.