[Deep Learning Notes 03] Linear Regression, Softmax Regression, and Their Loss Functions

I. Linear Models

1.1 Mathematical Representation of the Model

• Input: $\textbf x=[x_1,x_2,\dots,x_n]^T$
• Weights: $\textbf w=[w_1, w_2, \dots, w_n]^T$, which determine each feature’s hero for the predicted value
• Bias: $b$, which determines what the prediction should be when all features are 0.
• The output is a weighted sum of the inputs: $y=w_1x_1+w_2x_2+\dots+w_nx_n+b$
• Vector form of the output: $y=<\textbf w,\textbf x>+b$

A linear model can be viewed as a single-layer neural network.

1.2 Measuring Prediction Quality

Measuring prediction quality means comparing the true value with the predicted value. Suppose $y$ is the true value and $\hat y$ is the estimate. There are many ways to measure the gap between them, such as the squared loss: $l(y,\hat y)=\frac{1}{2}(y-\hat y)^2$

1.3 Representation of Training Data

We usually collect data points that include training samples and their corresponding true values, denoted as:

$\textbf X=[\textbf x_1,\textbf x_2,\cdots,\textbf x_n]^T$

$\textbf y=[y_1,y_2,\cdots,y_n]^T$

Here $\textbf x_n$ is a column vector representing each sample. Thus each row of $\textbf X$ is one sample.

The predictions for all samples can then be written as:

$\hat{\textbf y}=\textbf X\textbf w+b$

1.4 Parameter Learning

Substitute the training loss into the squared loss formula mentioned in Section 1.2.

$l(\textbf w,b)=\frac{1}{2}(y-\hat y)^2=\frac{1}{2}\left(\mathbf{w}^\top \mathbf{x} + b - \hat y\right)$

$L(\mathbf{w}, b) =\frac{1}{n}\sum_{i=1}^n l^{(i)}(\mathbf{w}, b) =\frac{1}{n} \sum_{i=1}^n \frac{1}{2}\left(\mathbf{w}^\top \mathbf{x}^{(i)} + b - y^{(i)}\right)^2.$

By minimizing $L(\textbf w,b)$, we find the corresponding parameters $\textbf w$ and $b$.

1.5 Solving the Linear Model

For easier computation, let $\textbf X=[\textbf X, 1]$ and $\textbf w=[\textbf w, b]^T$: append a column of ones to the right of $\textbf X$, and below $\textbf w$ append the scalar $b$. Then we can compute $\textbf y = \textbf X \textbf w$ directly.

Substitute into the loss function:

$l(\textbf X,\textbf y,\textbf w)=\frac{1}{2n}||\textbf y-\textbf X \textbf w||^2$

$\frac{\partial}{\partial \textbf w}l(\textbf X,\textbf y，\textbf w)=\frac{1}{n}(\textbf y-\textbf X \textbf w)^T\textbf X$

Because the loss function is convex, at the optimum the gradient is 0, which gives

$\textbf w^*=(\textbf X^T\textbf X)^{-1}\textbf X\textbf y$

1.6 Linear Regression from Scratch

(1) Import packages

import random
import torch
from d2l import torch as d2l    # pip install d2l

(2) Build the dataset

Create a synthetic dataset with noise, using the true $w$ and $b$ to generate the dataset and labels.

def synthetic_data(w, b, num_examples):  
    """生成 y = Xw + b + 噪声。"""
    X = torch.normal(0, 1, (num_examples, len(w)))  # 均值为0，方差为1的随机数样本
    y = torch.matmul(X, w) + b
    y += torch.normal(0, 0.01, y.shape)  # 添加均值为0，方差为0.01的噪声
    return X, y.reshape((-1, 1))

true_w = torch.tensor([2, -3.4])
true_b = 4.2
features, labels = synthetic_data(true_w, true_b, 1000)

Each row of features contains a two-dimensional sample, and each row of labels contains one label value.

(3) Read batches

def data_iter(batch_size, features, labels):
    num_examples = len(features)  # 样本数量
    indices = list(range(num_examples))  # 生成每个样本的index
    random.shuffle(indices)  # 将index随机打乱
    for i in range(0, num_examples, batch_size):  # 步长为batch_size
	    # 获取i到i+batch_size的下标
        batch_indices = torch.tensor(indices[i:min(i + batch_size, num_examples)])
        # 每次yield返回一个值，下次调用从上次的yield开始
        yield features[batch_indices], labels[batch_indices]

batch_size = 10

for X, y in data_iter(batch_size, features, labels):
    print(X, '\n', y)
    break

(4) Initialize model parameters

w = torch.normal(0, 0.01, size=(2, 1), requires_grad=True)
b = torch.zeros(1, requires_grad=True)

(5) Define the model

def linreg(X, w, b):  
    """线性回归模型。"""
    return torch.matmul(X, w) + b

(6) Define the loss function

def squared_loss(y_hat, y):  
    """均方损失。"""
    return (y_hat - y.reshape(y_hat.shape))**2 / 2

(7) Define the optimization algorithm

def sgd(params, lr, batch_size):  
    """小批量随机梯度下降。"""
    with torch.no_grad():
        for param in params:
            param -= lr * param.grad / batch_size
            param.grad.zero_()

(8) Training loop

lr = 0.03
num_epochs = 3
net = linreg
loss = squared_loss

for epoch in range(num_epochs):
    for X, y in data_iter(batch_size, features, labels):
        l = loss(net(X, w, b), y)
        l.sum().backward()
        sgd([w, b], lr, batch_size)
    with torch.no_grad():
        train_l = loss(net(features, w, b), labels)
        print(f'epoch {epoch + 1}, loss {float(train_l.mean()):f}')

1.7 Concise Linear Regression Implementation

(1) Import packages

import numpy as np
import torch
from torch.utils import data
from d2l import torch as d2l

true_w = torch.tensor([2, -3.4])
true_b = 4.2
# 利用人工数据合成函数，生成样本和标签
features, labels = d2l.synthetic_data(true_w, true_b, 1000)

(2) Use existing framework APIs to read data

def load_array(data_arrays, batch_size, is_train=True):  
    """构造一个PyTorch数据迭代器。"""
    # 把输入的两类数据一一对应
    dataset = data.TensorDataset(*data_arrays)
    # 重新排序后返回
    return data.DataLoader(dataset, batch_size, shuffle=is_train)

batch_size = 10
data_iter = load_array((features, labels), batch_size)

# 下面代码用于在屏幕显示读取到的数据，打包成iter迭代器，然后依次读取
next(iter(data_iter))

(3) Define the model

from torch import nn

# 将线性层放到Sequential容器中
net = nn.Sequential(nn.Linear(2, 1))  # 指定输入维度2，输出维度1
net[0].weight.data.normal_(0, 0.01)  # 第0层线性层 -> 权重w -> 数据 -> 使用正态分布替换data
net[0].bias.data.fill_(0)  # 第0层线性层 -> 偏差b -> 数据 -> 填充0

(4) Mean squared error

loss = nn.MSELoss()

(5) Stochastic gradient descent

trainer = torch.optim.SGD(net.parameters(), lr=0.03)

(6) Training loop

num_epochs = 3
for epoch in range(num_epochs):
    for X, y in data_iter:
        l = loss(net(X), y)
        trainer.zero_grad()
        l.backward()
        trainer.step()
    l = loss(net(features), labels)
    print(f'epoch {epoch + 1}, loss {l:f}')

II. Basic Optimization — Gradient Descent

When a model has no closed-form solution, we usually solve it with this method.

1. Choose an initial value $w_0$
2. During training, repeatedly update the parameters: $w_t=w_{t-1}-\eta\frac{\partial l}{\partial w_{t-1}}$ ($\eta$: learning rate)

In other words, update $w$ along the gradient direction (the direction of steepest decrease) to find the optimal solution. This is like walking downhill along the steepest slope until you reach the foot of the mountain (the optimum).

(1) Hyperparameter 1: learning rate

The learning rate controls how large each parameter update is. If the learning rate is too small, parameter updates are slow; if it is too large, the model oscillates and fails to find the optimum.

(2) Hyperparameter 2: batch size

Computing the gradient over the entire training set is too time-consuming and may take minutes or even hours.

Therefore we usually randomly sample $b$ examples to approximate the training-set loss.

Batch size also cannot be too large or too small. If the batch is too small, it is hard to exploit parallel computation and GPU resources; if the batch is too large, memory usage increases and computation is wasted.

III. Softmax Regression

The regression problems mentioned above aim to estimate a continuous value. Classification problems predict a discrete category. Although Softmax regression is called regression, it is actually a classification problem.

Common classification problems include handwritten digit recognition, natural object classification, protein microscopy image classification, malware classification, and malicious comment classification.

Comparing regression and classification

Regression	Classification
Single continuous numeric output	Multiple outputs
Loss is the difference from the true value	Output i is the confidence of predicting i

1.1 How to Turn a Classification Problem into a Regression Problem

(1) Class encoding

For n classes, we can use one-dimensional effective encoding. The label is then a vector of length n, and the label vectors for each class are $\textbf y_1,\textbf y_2\dots \textbf y_n$.

The label for class i is $\textbf y_i=[0,0,1,0\dots 0]$, where only the i-th element is 1 and all other elements are 0.

(2) Choose the loss function

In Softmax, we can choose to train with the mean squared loss function.

(3) Make predictions

$\hat y=argmax\ o_i$

Here $o_i$ is the confidence for each class, and $argmax\ o_i$ is the index of the class with the highest confidence. This gives the class with the largest confidence.

The key to performing detection is to make the confidence for the correct class much larger than for the other classes. We usually need $o_y-o_i\ge\Delta(y,i)$

(4) Convert confidences to probabilities

We obtain the confidence for every class: $\textbf o=[o_1,o_2,\dots,o_n]$.

However, confidence measures how well the predicted object matches each class, not the probability of being recognized as that object. The conversion is as follows:

$\hat{\textbf y}=softmax(\textbf o)$

$\hat y_i=\frac{exp(o_i)}{\sum_k exp(o_k)}$

This makes every element of the confidence vector $\textbf o$ non-negative and sum to 1.

We then use the difference between the true probability $\textbf y$ and $\hat {\textbf y}$ as the loss.