The Design of AEVis

The development of AEVis with the machine learning team that won first place in the NIPS 2017 non-targeted adversarial attack and targeted adversarial attack competitions, which aim at attacking CNNs . Despite the promising results , the experts found inefficient and inconvenient, especially the explanation of the model outputs. In their research process, a step introduced by adversarial examples. Understanding why an error has been made and further design a more effective method. The experts thus desire a tool that can assist them in understanding the prediction process of the target CNN.

Requirement Analysis

We have identified the following high-level requirements based on previous research and discussions with two experts (E$`_1`$ and E$`_2`$) from the winning team of the NIPS 2017 competition.

R1 - . Both experts expressed the need for extracting the datapaths of examples, serves as the basis for analyzing why the adversarial examples misclassified. In a CNN, different neurons learn to detect different features , for the prediction of an example. E$`_1`$ said that only analyzing the datapath can greatly by allowing them to only focus on critical neurons In addition to the datapaths for examples, E$`_1`$ emphasized the need for extracting for normal examples simultaneously.

However, their prediction results are different. The experts are interested in how they diverge to different predictions. For example, E$`_2`$ commented, “I want to know whether there are some critical ‘diverging points’ for the or gradually layer by layer through the network.” To this end, E$`_2`$ to compare the datapaths of normal source examples and adversarial examples. Triggered by E$`_2`$, E$`_1`$ added that it was interesting to compare the datapath of an adversarial example (e.g., a panda image that is misclassified as a monkey) with that of normal target examples (e.g., normal monkey images). Such comparisons help understand how these very different images “merge” into the same prediction (e.g., the monkey). The need for visual comparison is consistent with the findings of previous research .

R3 - Exploring datapaths at different levels. In a large CNN, a datapath often contains millions of neurons and connections. Directly presenting all neurons in a datapath will induce severe visual clutter. E$`_1`$ commented, “I cannot examine all the neurons in a datapath because there are too many of them. I often start by selecting an important layer based on my knowledge and examine the neurons in that layer to analyze the learned features and the activation of these neurons. The problem is when dealing with a new architecture, I may not know which layer to start with. Thus, I have to examine a bunch of layers, which is very tedious.” an overview of the datapath with visual guidance to facilitate experts in selecting the layer of interest. The requirement of providing an overview of a CNN aligns well with previous research . Although the overview of a datapath facilitates experts in finding the layer of interest, it is not enough to diagnose the root cause of the wrong prediction. The experts said that the overview of a datapath and the detailed neuron activation helps them identify the most important neurons that lead to misclassification. it is to provide a multi-level exploration mechanism that allows experts to zoom into the neurons of interest gradually. Previous research also indicates that visual analytics for deep learning benefits from multi-level visualization .

R4 - . Finding a diverging or merging point is not the end of the analysis. To develop effective defense methods, disclose how such divergence or happens. As the data flows from previous layers to the current diverging or merging point, a practical method of finding the root cause is tracing back to the previous layers and examining how neurons contribute to the neurons the diverging or merging point. E$`_1`$ commented, “When I find a neuron or feature map that performs very differently for an adversarial and a normal example, I’m interested in difference. For example, it is useful to know whether it caused by the neurons in the previous layer or even the neurons in a far-away layer due to the skip-connections  in modern CNNs.” we need to analyze how neurons contribute to each other in a DNN. Previous research also indicates that presenting the contribution among neurons is for understanding the outputs and roles of neurons .

System Overview

Driven by the requirements experts, we have developed a visual analysis tool, AEVis, It consists of the following two parts.

As shown in Fig. 1 , AEVis takes a trained CNN and the examples to be analyzed as input. The examples include Given the examples and the CNN, the datapath extraction module extracts the critical and their connections that are responsible for the predictions of the examples (Fig. 1 (b)). The extracted datapaths are then fed into the visualization module (Fig. 1 (c)), which supports the navigation and comparison of the datapaths from the high-level layers to the detailed neuron activation.

Background

In this section, we introduce how to generate adversarial examples.

Discussion

AEVis can illustrate the prediction mechanism of adversarial examples and help discover the to incorrect predictions. However, it has several limitations, which may shed light on future research .

Time complexity. The datapath extraction usually takes a few minutes and is computed offline.

Visual scalability. We have demonstrated that AEVis is able to analyze a state-of-the-art CNN (ResNet101), which has 101 layers and is much deeper than traditional CNNs (e.g., VGG-Net). More recently, When handling such deep employ a mini-map to help the expert track current viewpoint, which has proven effective in TensorFlow .

Generalization. AEVis aims the adversarial examples for CNNs because most research on adversarial attacks focuses on generating adversarial images for CNNs.

In addition to attacking CNNs, there are several initial attempts to attack other types of DNNs , such as recurrent neural networks (RNNs), autoencoders (AEs), and deep generative models (DGMs). In these types of DNNs, For example, Ming et al.  demonstrated that some neurons in an RNN were critical for predicting the sentiment of a sentence, such as the neurons for detecting positive/negative words. Such neurons and their connections form a datapath for an RNN. Thus, AEVis can be extended to help understand the root cause of adversarial examples for these DNNs. suitable datapath extraction and visualization methods for different types of DNNs. For example, to visualize the datapath of RNNs, we can first unfold the architecture of an RNN to a DAG , and then employ a DAG layout algorithm to calculate the position of each unfolded layer.

In addition to images, , such as adversarial documents and adversarial videos. To generalize AEVis to types of data, we need to change the visual hint for neurons (learned features and activation maps) according to the target data type. For example, analyzing adversarial documents, we can use a word cloud to represent the ‘learned feature’ of a neuron , select the keywords that strongly activate the neuron.

Conclusion

We have presented a robustness-motivated visual analysis machine learning experts