Use Explainable AI to Prove Image Classification Works

Integrated Gradients

Understanding the pixels that influence the classification decision is critical to provide a rationale.

Option B is the correct answer!

Integrated gradients¹ is a popular Explainable AI method for deep learning networks that results in a “heatmap” that shows which pixels are most important to prediction. It would allow an analyst to cross check what pixels the model uses to identify a panda against “common sense.”

Why do Integrated Gradients Work?

Start with a Basic Model

We are going to start out with a very basic way of predicting pandas from images and build our way up from there to a deep learning model before we tackle integrated gradients.

We can start by coming up with predictors ("features") that determine if we are looking at a panda. One type of feature could be the number of colors in an image. If the image is solid white, you are more likely looking at a polar bear in a snowstorm 😊. Now I know you're saying - pandas are black & white, and they may be sitting in front of green grass; there is a lot more to classification than just color. I completely agree with you; classification of images is complex, but we are starting out with some critical concepts that we are going to need later.

Cool² – so we have a concept that we can use to make predictions. Going forward, I am going to represent all of this as a formula, and I am going to use the following notation:

• $P(defect)$ : The probability that a defect has occurred.
• $\sigma$ : A function that converts the inputs³ into the probability of defect.
• $w_0$ : The importance of the number of colors to the classification decision.
• $x_0$ : The number of colors detected in the image.

For the rest of this section, I am just going to add a bullet point with an explanation each time I need to add some type of math symbol.

So what we have so far is:

To give another example of a “handmade” feature, examine the size of the largest color in the image. Maybe a 1-pixel speck of white is noise, but if the majority of the image is white it might represent part of a panda.

• $w_1$ : The importance of the size of the color to the classification decision.
• $x_1$ : The size of the main color in the image.

Ok, so now we're getting somewhere. If there are multiple colors in the picture, and if the main color in the image is a panda color, then the combination of those two features might increase the probability that we have a panda in the image.

What if there is something other than color or the size of color (hint: there is) that could explain whether a defect could happen? We could explain that with some general “catch all” number called a bias⁴ .

Examine a Baseline for that Model

If we are going to start to examine our basic model and we are going to “explain” the model to a group of executives, we might first start talking about what the probability of getting a panda is without looking at anything related to color (we only look at the bias in this case). Then, we might want to talk about how the model is impacted once we start to see a lot of black or white in the picture. Likewise, we might talk about how the probability adjusts as you start to see an increase in the “size of a panda color.”

When we do this, we start to examine the idea of missingness. It is the idea that we want to compare what happens when there is an input to what happens when there are no inputs.

Extend the Analogy to Neural Networks

It becomes much harder to “explain” an interpretation of the pixels because the model has layers of activations (“neurons”).

One solution to this looks at gradients⁵ (“direction of maximum gain”) of an output category (e.g. “panda detected”) relative to the image that was fed into the model. This is similar to the concept that we just explored where we changed the number of colors and the size of the color to get a sense as to where there is maximum change in prediction.

The challenge with this approach is that neural networks are prone to a problem called saturation. The gradients of the input features (e.g. the white pixel in the panda image) may be small even if the neural network depends heavily on that pixel. Because it is small, we don’t get a sense as to which pixels are important.

Integrated gradients solves the saturation problem this by looking at several defined steps to get the gradients. But defined steps against what? This is where the baseline that we discussed earlier is used.

Ok one step further now. What kind of baseline should we use?

Although there are other articles that focus on optimal baselines⁶, the following are some initial baselines worth exploring:

• White background for black & white images or images that are black
• Black background for other types of images

How does this relate to the exam question?

Consider the example above where a deep learning model (Inception v1) is classifying a panda⁷. Integrated gradients is not only identifying the head, nose, and eyes, but it also identifies pixels in the grass which helped do the classification. Since grass isn’t essential to what makes a panda… “ a panda”, feeding more panda images which do not have grass into the model might be a way to improve accuracy.

TAKEAWAY

The certification guide discusses integrated gradients broadly under:

In fact, Google Cloud documentation actually gives an overview of Explainable AI.

Once you understand integrated gradients, it is easy to see that this is the obvious answer. Integrated gradients highlights the pixels that are relevant to the classification decision, and it does so in a way that can be easily interpreted by management.