[Deep Learning Notes 08] Classic Neural Networks

LeNet-5

This network can be used for handwritten digit recognition.

• Input: 32x32x1 handwritten digit images
• Convolution: f=5, s=1, filters=6, image becomes 28x28x6
• Average pooling: f=2, s=2, image becomes 14x14x6
• Convolution: f=5, s=2, filters=16, image becomes 10x10x16
• Average pooling: f=2, s=2, image becomes 5x5x16
• Fully connected layer: 400 -> 120
• Fully connected layer: 120 -> 84
• Classification: 10-class output

About 60,000 parameters in total, using sigmoid as the activation function.

AlexNet

• Input: 227x227x3 images containing target objects
• Convolution: f=11, s=4, filters=96, image becomes 55x55x96
• Max pooling: f=3, s=2, image becomes 27x27x96
• Convolution same: f=5, padding=2, filters=256, image becomes 27x27x256
• Max pooling: f=3, s=2, image becomes 13x13x256
• Convolution same: f=3, padding=1, filters=384, image becomes 13x13x384
• Convolution same: f=3, padding=1, filters=384, image dimensions unchanged
• Convolution same: f=3, padding=1, filters=256, image becomes 13x13x256
• Max pooling: f=3, s=2, image becomes 6x6x256
• Fully connected layer: 9216 -> 4096
• Fully connected layer: 4096 -> 4096
• Classification output: softmax over 1000 classes

About 60 million parameters in total, using ReLU as the activation function.

VGG-16

• Input: 224x224x3 images
• Convolutional network
  • Convolution: f=3, s=1, same
  • Max pooling: f=2, s=2
  • Repeated 16 times
• Fully connected layers
• softmax

About 1.38 times one hundred million parameters. After each group of convolutional layers, the image is halved in size and the number of channels doubles.

ResNet

Residual Blocks

ResNet is built from residual blocks.

Thus $a^{[l+2]}=g(z^{[l+2]}+a^{[l]})$

Significance of ResNet

For ordinary neural networks without residual connections, the deeper the network, the harder it is for optimization algorithms to train, and as network depth increases, training error grows.

With ResNet, even when the network is very deep, the error rate does not increase.

Why ResNet Works

In a very deep network, if a layer suffers from vanishing gradients, i.e., W=0.

According to $a^{[l+2]}=g(z^{[l+1]}+a^{[l]})=g(w^{[l+2]}a^{[l+1]}+b^{[l+2]}+a^{[l]})$

Due to vanishing gradients, $w^{[l+2]}=0$, and if $b^{[l+2]}=0$, then $a^{[l+2]}=g(0+0+a^{[l]})=g(a^{[l]})$. Because the activation function is ReLU and $a^{[l]}$ has already been activated, $a^{[l+2]}=g(a^{[l]})=a^{[l]}$.

This is equivalent to skipping the part where gradients vanish.

Inception Network

The image above shows an Inception module. The idea is to stack along the channel dimension the outputs from convolutions with different kernel sizes into one large block that contains all convolutional features.

Because various convolution kernels are provided, the network can automatically select the best one or combination during training, avoiding the limitation of using only a single type of convolution kernel.

This is the structure of an Inception network: it replaces traditional convolutional layers with Inception modules.