Model Interpretability with Occlusion Mapping – ‘An AI Tells us What it Knows When We Poke it in the Eye’

Machine learning underpins technology solutions of ever greater consequence to society and Model Interpretability is seen as increasingly important and necessary by regulatory bodies, academia and technology businesses in the market-place. Model Interpretability is the burgeoning field of inquiry that seeks to understand and explain the inherent mechanisms in artificial intelligence systems in particular deep learning models and why they make the decisions that they do.



Researchers have identified various motivations for gaining insight into the inner workings of an artificial intelligence. Motivations include the need for faith in the system – do we believe that the real world deployment of the artificial intelligence will perform as specified? An artificial intelligence is almost always created in a simplified version of the real world in which it will ultimately have to work. Trust – can we trust its decisions? Do we believe the technology will make sensible decisions given poorly specified decision criteria, unforeseen or implicit factors? Legal and political efficacies – A person might challenge the denial of their application for services claiming prejudice or there may be property damage, injury or loss of life caused by an artificial intelligence. Where is the fault and who is to blame? How do automated decision technology providers anticipate and address these challenges?

Initial efforts in Model Interpretability research circa 2008 focused on methods for visualizing the “brain” activity of an artificial intelligence or how knowledge and perception was represented inside its “brain” – the set of node thresholds and weighted connection strengths of and between all of its “brain” cells. These methods were about as useful as it would be to insert probes into a persons brain and measure electrical signals as a way to understand what the person is thinking. In 2009 Dumitru Erhan, now a senior research scientist at Google had a paper published in the academic literature in which he showed a technique using mathematical methods to calculate the inputs that would maximize the internal signal at some location in the AI’s brain [1]. For an artificial intelligence tasked with understanding images this method could calculate images that would maximize the activation of some part of the AI’s brain. For the first time scientists could see what the AI was seeing, or rather what images the AI’s brain would respond to the most. In 2014 Google scientists used this method to create their DeepDream software.

The idea of synthesizing inputs to an artificial intelligence lead to the idea of modifying inputs that the artificial intelligence was ordinarily expected to understand and then to measure how the modifications would affect the response of the system. In 2013 Matthew Zeiler, now the founder and CEO of Clarifai developed a method to test if object recognition in images by an artificial intelligence really was recognizing the objects or was giving answers based on other details in the images [2]. The idea was to occlude the relevant object in an image by overwriting it with a rectangular grey patch and then to test if the artificial intelligence could detect the object. This occlusion sensitivity analysis has been generalized as a Model Interpretability method now called Occlusion Mapping.

For an artificial intelligence that must calculate the probability that an input image belongs to a certain category, occlusion maps are computed by capturing the classification probabilities when an input image is partially occluded by a rectangular patch of some color, some size and at some location in the image. For a grid of occlusion locations over the image, classification probabilities can be visualized as heat-maps that highlight image regions important to classification accuracy as in Figure 1. In a sense we are partially blinding the artificial intelligence in a slightly different way many times and measuring how each “poke in the eye” affects what the AI sees. In the example in Figure 1 an artificial intelligence is designed to classify skin lesions as types of skin cancer. The image on the left is the source image showing the occluding patch in its first position. The heat-map on the right shows specifically which parts of the image are important for the decision the system makes. The dark regions show where the probability of Melanoma is low when we occlude that region suggesting that there is important information in that part of the image.

table.images { text-align: center; margin: 1em 0 2em 0; } table.images td { border: 1px solid black; padding: 2px; } table.images td.small-images img { height: 150px; width: 180px; } table.images td.table-legend img { height: 30px; width: 600px; } table.images img { height: 220px; }

Melanoma Occlusion Map
Figure 1

In Figure 2 the darker shades are spread out over the entire skin lesion indicating that the important information in the image is not in one specific location but is distributed. The artificial intelligence must make its decision on the lesion as a whole rather than look for specific tell tale features. This ability to make gestalt decisions based on a holistic interpretation has proved difficult for artificial intelligence researchers and engineers to program explicitly in rule based software. Machine learning approaches however exhibit a remarkable capability to encode such functions.

Melanoma Occlusion Map
Figure 2

Occlusion Mapping can be used as an analysis tool for inspecting the progress of an artificial intelligence system during the learning procedure. Figure 3 shows three examples of Occlusion Mapping on Melanoma cases for a machine model as it learns. The first row shows the source images with occluding patches in their first position. Succeeding rows show occlusion maps after 100, 500, 900, 1300 and 1700 training epochs respectively. The initial occlusion maps are dark and mostly course grained indicating low classification probabilities and poor localization of discriminative structure. As learning progresses the images become brighter and more fine grained indicating higher classification probabilities and increasing sensitivity to structure in the source images.

Figure 3

Human civilization has unquestionably come to depend on its technological systems for the coordination of our society and the orderly conduct of our daily lives. As we push forward we will be driven inexorably to an ever increasing reliance on automated decision systems. There is increasing regulatory, social and commercial pressure to understand and explain as exactly as possible how these systems work and why they make the decisions that they do. It has been discovered that significant insight into the operation of an artificial intelligence can be garnered by modifying the inputs that the system would ordinarily be expected to interpret and then observing the effects of the modification on the performance of the system. Occlusion Mapping is one such method and is a robust technique that yields informative visualizations of a machine learning models response to regions of its input and the structure of discriminative information in source images applied to the model.


Written by Allan Campbell, RMIT University, Silverpond



  1. Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
  2. Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818{833. Springer, 2014.

Share this post

3 responses to “Model Interpretability with Occlusion Mapping – ‘An AI Tells us What it Knows When We Poke it in the Eye’”

  1. Normally I do not learn post on blogs, however I wish to say that this write-up very forced me to take a look at and do so! Your writing style has been amazed me. Thanks, quite great article.

  2. broadband says:

    Do you mind if I qսote a couple of yօսr articles as long as I
    provide credit and sources back to your weblog? My
    website is in the exact same area of interest as үours and my visitors would definitely benefit from some of the information уou provide herе.
    Please let me know if thіs ok with you. Cheeгs!

Leave a Reply

Your email address will not be published. Required fields are marked *