[Statistical Learning Method Notes 01] Introduction to Statistical Learning

I. Statistical Learning

1.1 The Concept of Statistical Learning

（1）Concept

Statistical learning is the discipline in which computers build probabilistic and statistical models from data and use those models to predict and analyze data.

Statistical learning is machine learning in which computer systems improve system performance by applying data and statistical methods.

（2）Object

Data, such as numbers, text, images, video, audio, and so on.

（3）Purpose

Starting from data, extract features from the data, abstract models from the data, discover knowledge in the data, and ultimately return to data analysis and prediction.

1.2 Classification of Statistical Learning

1.2.1 Basic Categories

（1）Supervised learning

A machine learning problem of learning a predictive model from labeled data. That is, learning the statistical regularity of the mapping from input to output.

• Input space and output space: all possible values of the input and output.
  • When both input and output variables are continuous, it is a regression problem
  • When the output variable is discrete, it is a classification problem
  • When both input and output variables are variable sequences, it is a labeling problem
• Feature space: the space in which all feature vectors live when inputs are represented as feature vectors.
• Joint probability distribution: assume training and test data are generated independently and identically from a joint probability distribution P(X,Y).
• Hypothesis space: the set of mappings from input to output.

（2）Unsupervised learning

A machine learning problem of learning a predictive model from unlabeled data. That is, learning the statistical regularities or latent structure of the data.

• Input space and output space: the set of all possible values of the model’s input and output.
  • The input is an instance represented by a feature vector.
  • The output is an analysis result for the input, represented by the input’s category, transformation, or probability.
• Hypothesis space: the set of all models z=g(x), where X is the input space and Z is the latent structure space.

（3）Reinforcement learning

A machine learning problem in which an intelligent system learns an optimal behavioral policy through continuous interaction with the environment. The intelligent system observes a data sequence obtained from interaction with the environment; in essence, it learns optimal sequential decision making.

The interaction process between the intelligent system and the environment: at each step $t$, the intelligent system observes a state $s_t$ and reward $r_t$ from the environment and takes an action $a_t$. According to the intelligent system’s action, the environment determines the next step $t+1$ state $s_{t+1}$ and $r_{t+1}$. The goal is to maximize the system’s long-term cumulative reward.

A Markov decision process for reinforcement learning consists of the five-tuple <S, A, P, r, $\gamma$>:

• S: a finite set of states
• A: a finite set of actions
• P: a state transition probability function
• r: a reward function
• $\gamma$: a discount factor

Policy: a function of actions given states Value function: the mathematical expectation of the long-term cumulative reward of a policy starting from a given state Action-value function: the mathematical expectation of the long-term cumulative reward of a policy starting from a given state and action

（4）Semi-supervised learning and active learning

Semi-supervised learning: a machine learning problem of learning a predictive model using both labeled and unlabeled data.

Active learning: a machine learning problem in which the machine continually and actively presents instances for a teacher to label, then uses the labeled data to learn a predictive model.

1.2.2 Classification by Model

（1）Probabilistic models and non-probabilistic models

• Probabilistic model: takes the form of a conditional distribution P(y|x)
• Non-probabilistic model: takes the functional form y=f(x)

（2）Linear models and nonlinear models

• Linear models: perceptron, linear support vector machine, k-nearest neighbors, latent semantic analysis
• Nonlinear models: support vector machine, neural network

（3）Parametric models and nonparametric models

• Parametric model: assumes the dimension of the model parameters is fixed, and the model can be fully specified by a finite-dimensional parameter vector
• Nonparametric model: assumes the dimension of the model parameters is not fixed or is infinite, and grows as the amount of training data increases

1.2.2 Classification by Algorithm

（1）Online learning

Accept one sample at a time, make a prediction, then update the model, and repeat continuously.

（2）Batch learning

Accept all data at once, learn the model, and then make predictions.

1.2.3 Classification by Technique

（1）Bayesian learning

In the learning and inference of probabilistic models, use Bayes’ theorem to compute the conditional probability of a model given the data, and apply this principle to model estimation.

（2）Kernel methods

A machine learning method that uses kernel functions to represent and learn nonlinear models.

II. The Three Elements of Statistical Learning

2.1 Model

• Probabilistic model: the hypothesis space is the set of conditional probabilities. {f|Y=f(X)}
• Non-probabilistic model: the hypothesis space is the set of decision functions. {P|P(Y|X)}

2.2 Strategy

（1）Loss function

• 0-1 loss function: $L(Y,f(x))=0,1$
• Squared loss function: $L(Y,f(x))=(Y-f(X))^2$
• Absolute loss function: $L(Y,f(x))=|Y-f(X)|$
• Logarithmic loss function: $L(Y,f(x))=-logP(Y|X)$

（2）Risk function

The average of the loss function, also called expected loss.

When the sample size is large enough, empirical risk minimization helps ensure good learning performance. When the sample size is small, the learning performance may not be good, and overfitting may occur.

（3）Structural risk

Structural risk adds a regularization term representing model complexity on top of empirical risk.

$R(f)=\frac{1}{N}\sum^N_{i=1}L(y_i,f(x_i))+\lambda J(f)$

Here $J(f)$ is model complexity; the more complex the model, the larger its value. It is commonly taken as the L1 norm $||\omega||_1$ or the L2 norm $\frac{||\omega||^2}{2}$ $\lambda$ is a coefficient used to balance the risk function and model complexity.

The idea of structural risk is essentially Occam’s razor: among all possible model functions, choose the model that explains the known data well and is the simplest.

2.3 Algorithm

III. Generalization Ability

Generalization ability is the predictive ability of a learned model on unknown data.

V. Generative Models and Discriminative Models

Maximum likelihood estimation: given a set of data outputs, define the probability function of that data set, determine the data input by finding the maximum of the probability function.