[Deep Learning Notes 01] Neural Networks and Deep Learning

Contents

I. Introduction to Deep Learning
1.1 Neural Networks
1.2 Supervised Learning
1.3 About Deep Learning
II. Fundamentals of Neural Network Programming
2.1 Notation
2.2 Binary Classification—The Logistic Regression Method
2.3 Gradient Descent
2.4 Computation Graphs
2.5 Vectorization
2.6 About numpy and Common Functions

I. Introduction to Deep Learning

1.1 Neural Networks

The activation function ReLU (basic rectified unit, Rectified linear unit) starts at zero and then rises along a straight line.

Every unit in a neural network may be a ReLU or another nonlinear unit.

1.2 Supervised Learning

(1) Application areas of deep learning

In supervised learning, X is first provided as input, and then a function is learned that maps the input to output Y.

Deep learning now plays an important role in fields such as online advertising, image processing, speech recognition, machine translation, and autonomous driving. However, for each application, we must choose x and y appropriately to solve the specific problem.

(2) Some commonly used architectures

Price prediction: general-purpose standard neural network (Standard NN) Image processing: convolutional neural network, CNN (Convolutional NN) Sequence processing, such as audio: recurrent neural network, RNN (Recurrent NN) Language processing uses more complex RNNs Autonomous driving uses images and radar, so it employs more complex hybrid neural network architectures

(3) Structured and unstructured data

Structured data is data in databases. For example, price prediction uses standard databases, while advertising uses databases containing user information, advertising information, and so on.

Unstructured data includes speech, images, videos, and text. Compared with structured data, it is more difficult for computers to understand.

1.3 About Deep Learning

The main factors behind the rapid development of deep learning in recent years are described below.

With earlier algorithms, such as support vector machines in traditional machine learning, performance basically stopped improving as the amount of data increased. In the early days, little data was available, and algorithms improved somewhat as the amount of data grew. Later, however, society entered the data age, in which everyone generates a large amount of data every day. Although the amount of data kept growing, earlier algorithms did not produce better results.

At this point, training a small neural network with the data gives much better algorithm performance, and training a more complex, large neural network produces even better results. Although increasing the amount of data increases training time, large neural networks have now helped humanity achieve a great many results.

On the other hand, the quality of training with earlier SVMs depended more on manual design, whereas today’s neural networks are more general-purpose.

Another factor is computing power. Improvements in CPU and GPU computing capabilities also provide a foundation for the development of deep learning.

Algorithmic innovations have also greatly advanced deep learning. Many algorithms have been proposed to improve neural network performance and increase computation speed.

II. Fundamentals of Neural Network Programming

2.1 Notation

When you need to iterate over a dataset, try not to use a for loop.

For example, consider identifying whether an image of a cat is a cat.

First, we obtain an image composed of 3 matrices (RGB). We can extract all pixel values from each matrix and arrange them into a feature vector. The dimensions of the input matrix are therefore 3xheightxwidth.

The input is the feature vector, and the output is the label 0/1, indicating whether there is a cat.

In mathematical notation:

Input and output: $(x,y)$

Dataset: $m:\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(m)},y^{(m)})\}$

The input data $x^{(n)}$ is usually arranged as column vectors to form an input matrix.

$X=[x^{(1)},x^{(2)},...,x^{(m)}]$

The input matrix $X\in{R^{n_x\times{m}}}$ has dimensions $X.shape=(nx,m)$ .

Similarly, the output data $y^{(n)}$ is also arranged to form an output matrix.

$Y=[y^{(1)},y^{(2)},...,y^{(m)}]$

The input matrix $Y\in{R^{1\times{m}}}$ has dimensions $Y.shape=(1,m)$ .

2.2 Binary Classification—The Logistic Regression Method

(1) Binary classification

For a neural network problem such as object recognition in images, the output is $y\in(0,1)$ ; that is, the image either contains the target or it does not.

Given an image input feature vector $x$ , we expect to obtain a predicted value $\hat{y}$ that determines whether the image contains the target we want. The prediction process is equivalent to calculating a probability:

$\hat{y}=P(y=1|x)$

(2) The logistic method

Because this is a binary classification problem, the predicted result should be in [0,1]. A linear approach is therefore generally not used, because with a linear expression, Y can grow infinitely large. A common method is to apply a sigmoid function on top of the linear expression.

$\textcolor{red}{\hat{y}=\sigma(w^Tx+b)}$

Here, $w^T\in{R^{n_x}}$ is the coefficient vector of $x$ , $b\in{R}$ is a constant and an intercept, and $\sigma(z)=\frac{1}{1+e^{-z}}$ is the sigmoid function. Using $\sigma(z)$ converts the output of the straight line into an output in [0,1].

(3) Loss function Loss

The loss function Loss is the difference between the predicted value and actual value for a single sample

To train $w$ and $b$ , we need to define a loss function that describes how close $\hat{y}$ is to $y$ . In the logistic method, we use the following function as the loss function Loss.

$\textcolor{red}{L({\hat{y},y})=-(y\log\hat{y}+(1-y)\log(1-\hat{y}))}$

For the loss function, we want it to be as small as possible.

When $y=1$ , $L(\hat{y},y)=-\log\hat{y}$ . We can see that the larger $\hat{y}$ is ( $\hat{y}\rightarrow1$ ), the smaller Loss becomes.

Similarly, when $y=0$ , $L(\hat{y},y)=-\log{(1-\hat{y})}$ . We can see that the smaller $\hat{y}$ is ( $\hat{y}\rightarrow0$ ), the smaller Loss becomes.

(In other methods, variance may be used as the Loss function. It is not used here because the result obtained from variance is concave-convex, meaning that it has multiple local optima, which makes it inconvenient to use gradient descent later to find the global optimum. The Loss function above solves this problem.)

(4) Cost function

The cost function is the difference between the predicted values and actual values for all samples

$\textcolor{red}{J(w,b)=\frac{1}{m}\sum^m_{i=1}{L(\hat{y}^{(i)},y^{(i)})}=-\frac{1}{m}(y\log\hat{y}+(1-y)\log(1-\hat{y}))}$

2.3 Gradient Descent

The cost function measures prediction performance on the training set. We want to find suitable $w,b$ that make the cost function $J(w,b)$ as small as possible. This is where gradient descent is used.

Because the cost function $J(w,b)$ is a convex function, its shape is like the figure below. This is also why we defined the loss function in that form.

What we need to do is initialize values for $w,b$ , then move them in the direction of gradient descent until we find the point with the lowest gradient.

Using $w$ as an example, we repeatedly perform the following process: $w=w-\alpha\frac{dJ(w)}{dw}$ to update $w$ , where $\alpha$ is the learning-rate parameter. We can see that the larger the slope is, the larger the change in w at each iteration.

In practice, we use the following equations to solve for $w,b$ with gradient descent:

$w=w-\alpha\frac{\partial{J(w,b)}}{\partial{w}}$

$b=b-\alpha\frac{\partial{J(w,b)}}{\partial{b}}$

Gradient Descent

2.4 Computation Graphs

A computation graph performs calculations from left to right to calculate the cost function $J$ .

For a flowchart, it is easy to see the derivatives of $J$ . Backpropagation works backward from the final output to derive the derivatives of $J$ with respect to each intermediate variable and input.

In programs, the usual programming convention is to use dvar to represent the derivative of the final output variable with respect to the variable var.

The approximate code process is as follows:

# 初始化
J = 0
dw1 = 0
dw2 = 0
db = 0
# 遍历数据集
for i = 1 to m
	# 计算损失函数与成本函数
	zi = w1 * x1 + w2 * x2 +b
	ai = 1/(1+exp(-zi))
	J += -[yi * log(ai) + (1 - yi) * log(1-ai)]
	# 求导
	dzi = ai -yi
	dw1 += x1 * dzi
	dw2 += x2 * dzi
	db += dzi
# 解出各参数优化后的值
J/=m
dw1/=m
dw2/=m
db/=m
# 梯度下降
w1=w1-a*dw1
w2=w2-a*dw2
b=b-a*db

2.5 Vectorization

(1) What is vectorization?

Because both the input data $x$ and coefficients $w$ are column vectors, multiplying these two vectors with a non-vectorized method—that is, implementing it with a for loop—is very slow. A vectorized method, such as using z=np.dot(w,x) in numpy to calculate the dot product of the two vectors, is much faster.

import time

# 向量化
a = np.random.rand(1000000)
b = np.random.rand(1000000)
tic = time.time()
c = np.dot(a,b)
toc = time.time()
print("Vectorized:" + str(1000*(toc-tic))+"ms")
# 非向量化
a = np.random.rand(1000000)
b = np.random.rand(1000000)
tic = time.time()
for i in range(1000000):
	c += a[i]*b[i]
toc = time.time()
print("For Loop:" + str(1000*(toc-tic))+"ms")

# 结果向量化的方法用时1.5ms，for循环方法500ms

The difference is not yet obvious for this small algorithm. However, if training a neural network takes 300 hours with a non-vectorized method but only 1 hour with a vectorized method, the difference becomes very clear.

(2) Vectorizing forward propagation

The purpose of forward propagation is to calculate $z^{(i)}$ and $a^{(i)}$ .

$z^{(i)}=w^Tx^{(i)}+b$ $a^{(i)}=\sigma(z^{(i)})$

Vectorization can be implemented as follows: $Z=[z^{(1)}\ z^{(2)}\ \cdots\ z^{(m)}]=w^T\cdot{X}+[b\ b\ \cdots\ b]=[w^Tx^{(1)}+b\ w^Tx^{(2)}+b\ \cdots\ w^Tx^{(m)}+b]$ Here, $X$ is the input matrix. Each column contains all features of one sample, and there are m columns, meaning m samples in total. $w^T$ is the row vector composed of parameters $w^{(i)}\rightarrow w^{(nx)}$ .

The process above can be represented in a python program as follows:

Z=np.dot(w.T, X) + b

(3) Vectorizing gradient calculation

Gradient calculation means calculating $dz^{(i)}$ , $dw^{(i)}$ , and $db^{(i)}$ .

$dz^{(i)}=a^{(i)}-y^{(i)}$ $dw^{(i)}=x^{(i)}dz^{(i)}$ $db^{(i)}=dz^{(i)}$

It can be vectorized as follows:

Vectorize $dz^{(i)}$ , $dw^{(i)}$ , and $db^{(i)}$ :

$dw=[dw^{(1)}\ dw^{(2)}\ \cdots\ dw^{(m)}]$ $db=[db^{(1)}\ db^{(2)}\ \cdots\ db^{(m)}]$ $dz=[dz^{(1)}\ dz^{(2)}\ \cdots\ dz^{(m)}]$

Then $db=\frac{1}{m}\sum^{m}_{i=1}dz^{(i)}$ , $dw=\frac{1}{m}Xdz^T$

Expressed in python:

Z=np.dot(w.T,x)+b
A=sigmod(Z)
dZ=A-Y
db=np.sum(dZ)/m
dw=np.dot(X,dZ^T)/m
w=w-a1*dw
b=b-a2*db

(4) Summary To speed up code execution, we used vectorization, meaning that we use numpy to calculate datasets whenever possible instead of using for loops.

Input matrix $X.shape = (nx,m)$ : $m$ samples, each with $nx$ features

$X=[x^{(1)}\ x^{(2)}\ \cdots\ x^{(m)}]$ , where $x^{(i)}=[x^{(i)}_1\ x^{(i)}_2\ \cdots\ x^{(i)}_{nx}]^T$

Parameter matrix $W.shape=(nx,1)$ : one parameter $w$ for each feature

$W=[w_1\ w_2\ \cdots\ w_{nx}]^T$

Parameter $B.shape=(1,m)$ : one parameter $b$ for each feature

$B=[b\ b\ \cdots\ b]$

Linear value $Z.shape=(1,m)$ , equivalent to a simplified $\hat y$ : one calculated value is obtained for each sample

$Z=W^TX+b=[W^Tx^{(1)}+b\ \ \ \ W^Tx^{(2)}+b\ \ \cdots\ \ W^Tx^{(m)}+b]$

Predicted value $A.shape=(1,m)$ : one predicted value for each sample

$A=\hat Y=\sigma{(Z)}$

Loss function $L.shape=(1,m)$ : the distance between each sample’s predicted value and labeled value

$L=-Y\log{A}-(1-Y)\log{(1-A)}$

Cost function $J.spape=(1,1)$ : the average of all loss functions

$J=\frac{1}{m}L.sum()$

Derivative $dZ.shape=(1,m)$

$dZ=\frac{dJ}{dZ}=A-Y$

Derivative $dW.shape=(nx,1)$

$dW=\frac{1}{m}X\cdot dZ=[\frac{1}{m}dw^{(1)}\ \frac{1}{m}dw^{(2)}\ \cdots\ \frac{1}{m}dw^{(nx)}]^T$

Derivative $dB.shape=(1,1)$

$db=\frac{1}{m}dZ.sum()$

2.6 About numpy and Common Functions

It is best not to use rank 1 arrays such as a=np.random.randn(5). Instead, use an $(m,1)$ or $(1,n)$ matrix. In other words, define a row or column vector directly with a=np.random.randn(5,1), or use reshape to change its shape.

If you are unsure of a vector’s dimensions, you can use an assert statement to check them.

assert(a.shape==(5,1))

(1) sigmod function

def sigmoid(x):
    """
    参数:
    x -- 任意形状的numpy数组

    返回值:
    s -- sigmoid(x)
    """
    
    s=1/(1+np.exp(-x))
    
    return s

(2) Gradient calculation

def sigmoid_derivative(x):
    """
    参数:
    x -- numpy数组

    返回值:
    ds -- 计算的梯度值
    """
    
    s = 1/(1+np.exp(-x))
    ds = s*(1-s)
    
    return ds

(3) Reshaping an input image into [width x height x 3, 1]

def image2vector(image):
    """
    参数:
    image -- 特定形状的numpy数组 shape(length, height, depth)
    
    返回值:
    v -- 向量形式的数组 shape(length*height*depth, 1)
    """
    
    v = image.reshape(image.shape[0]*image.shape[1]*image.shape[2],1)
    
    return v

(4) Normalization

def normalizeRows(x):
    """
    参数:
    x -- 一个二维numpy数组 shape(n, m)
    
    返回值:
    x -- 按行归一化后的numpy矩阵
    """
    
    # 计算x数组的模
    x_norm = np.linalg.norm(x,ord=2,axis=1,keepdims=True)
    # 用x除以它的模
    x=x/x_norm

    return x

Parameter	Description	Calculation
ord=default	Euclidean norm: $l_2$	$\sqrt{x_1^2+x_2^2+\dots+x_n^2}$
ord=2	Euclidean norm: $l_2$	Same as above
ord=1	Taxicab norm: $l_1$	$\lvert x_1\rvert+\lvert x_2\rvert+\dots+\lvert x_n\rvert$
ord=np.inf	Infinity norm: $l_{\infty}$	$MAX\lvert x_i\rvert$
axis=1	Process by row vector
axis=0	Process by column vector
axis=None	Matrix norm

(5) softmax function

The softmax function is a normalization function used when an algorithm needs to classify two or more classes.