Salient deconvolutional networks - Supplementary material - ECCV 2016 - Paper ID: 1160
The supplementary material mainly consists of expanded versions of the figures included in the core submission. We also revisit the point about reversing LRN layers in AlexNet below. This html page is best viewed on firefox and chrome.
Table of Contents
- Effect of LRN
- Comparing SaliNet, DeSaliNet and DeConvNet
- Lack of neuron selectivity
- Foreground object selectivity
- Eight Deconvolutional Architectures
- Segmentation qualitative results
- References
Effect of LRN
To reverse LRN layers, we use BP-reversed LRN for SaliNet and DeSaliNet and Identity for DeConvNet. For SaliNet, this agrees with the network Saliency algorithm of Simonyan et.al [1]. For DeConvNet, this agrees with the original architecture of Zeiler et.al.[2]. The figure below shows that this choice doesn't matter much for DeSaliNet.
| Original Image |
Pool5, Identity |
Pool5, BP-reversed LRN |
FC8, Identity |
FC8, BP-reversed LRN |
Above are the DeSaliNet visualizations of Pool5 and fc8 neurons in AlexNet while setting LRN reverse to either the identity or BP-reversed LRN. The difference is small.
Comparing SaliNet, DeSaliNet and DeConvNet
The following figures provide more images like that in figure 1 in the paper.
Here we show visualizations from the fc8 layer in VGG-16 (just before the softmax operation). The maximally active neuron is visualized in each case. DeSaliNet results in crispier visualizations. They suppress the background while preserving edge information.
| Original Image |
 |
| DeConvNet |
| SaliNet |
| DeSaliNet |
The same images are next visualized using the maximally active neuron in FC8 layer of AlexNet.
| Original Image |
 |
| DeConvNet |
| SaliNet |
| DeSaliNet |
Lack of neuron selectivity
This section supplements figure 4 in the paper by including more images and more layers and by visualizing both AlexNet and VGG-16 models.
In the next figure, the bottleneck information is fixed to the one computed during the forward pass through AlexNet and the output of the deconvolutional network is computed by choosing e as: the most active neuron in the center (Max Neuron), a second neuron at random (Rnd. Neuron), or as a positive random mixture of all neurons (Rnd. Noise). Results barely differ between these three choices of e, particularly for the deeper layers. The first collection of images below shows this effect for the DeSaliNet architecture.
|
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
| Image |
 |
| Pool 1 |
| Pool 2 |
| Relu 3 |
| Relu 4 |
| Pool 5 |
| Relu 6 |
| Relu 7 |
| FC 8 |
The same result is shown below now for VGG-16 again using the DeSaliNet architecture.
|
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
| Image |
 |
| Pool 1 |
| Pool 2 |
| Pool 3 |
| Pool 4 |
| Pool 5 |
| Relu 6 |
| Relu 7 |
| FC 8 |
The same result is shown next for DeConvNets using VGG-16.
|
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
| Image |
 |
| Pool 1 |
| Pool 2 |
| Pool 3 |
| Pool 4 |
| Pool 5 |
| Relu 6 |
| Relu 7 |
| FC 8 |
Next for DeConvNets using Alexnet.
|
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
| Image |
 |
| Pool 1 |
| Pool 2 |
| Relu 3 |
| Relu 4 |
| Pool 5 |
| Relu 6 |
| Relu 7 |
| FC 8 |
Next for SaliNet using Alexnet.
|
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
| Image |
 |
| Pool 1 |
| Pool 2 |
| Relu 3 |
| Relu 4 |
| Pool 5 |
| Relu 6 |
| Relu 7 |
| FC 8 |
Next for SaliNet using VGG-16
|
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
Max Neuron | Rnd. Neuron | Rnd. Noise |
| Image |
 |
| Pool 1 |
| Pool 2 |
| Relu 3 |
| Relu 4 |
| Pool 5 |
| Relu 6 |
| Relu 7 |
| FC 8 |
We can see that all three visualizations do not meaningfully change in deeper layers when changing the chosen neuron e. But in shallow layers, particularly in the first layer, the chosen neuron has a significant effect on the visualization output.
Foreground object selectivity
Below we compare the response of DeConvNet, SaliNet, and DeSaliNet by visualizing the most active neuron in several layers of VGG-VD. SaliNet and DeSaliNet tend to emphasize foreground objects, whereas DeConvNet's response is nearly uniform. Note that the apparent spatial selectivity of shallow layers is due to the finite support of the neuron and is content independent. In the figure below, each layer in depicted using three columns - DeConvNets (Column 1), SaliNet (Column 2) and DeSaliNet (Column 3)
| Image |
Pool 1 |
Pool 2 |
Pool 3 |
Pool 4 |
Pool 5 |
ReLU 6 |
ReLU 7 |
FC 8 |
 |
Next are the same visualizations but for AlexNet.
| Image |
Pool 1 |
Pool 2 |
ReLU 3 |
ReLU 4 |
Pool 5 |
ReLU 6 |
ReLU 7 |
FC 8 |
 |
Eight Deconvolutional Architectures
In figure 3 of the main submission, we have compared eight deconvolutional architectures using a trilobite image. Below are ten more images visualized using the same eight deconvolutional architectures. The layer being visualized is pool5. First we show results using VGG-16.
|
RU∘RUBP |
RU |
RUBP |
Identity |
| MPBP |
 |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
Next we show results for AlexNet.
|
RU∘RUBP |
RU |
RUBP |
Identity |
| MPBP |
 |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
| MPBP |
| Centered nails |
Segmentation qualitative results
Segmentation results (random selection). For each image, the top row shows the GrabCut segmentation and the bottom row shows the output of the corresponding deconvolutional network derived from AlexNet.
| Original Image |
DeSaliNet |
SaliNet |
DeConvNet |
Baseline |
 |
References
- K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proc. ICLR, 2014.
- M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In Proc. ECCV, 2014