Neural Network
Neural Network
Neuron
Following figure showing a neuron structure, where $\bs x = (\contia{x}{K})$ is input, $\bs w = (\contia{w}{K})$ is the weights, also is the training target of model, $f$ is the link function, and $\hat y$ is output.
A neuron will calculate the weighted inputs summation, where $x_0$ is an intercept term
$$ \begin{align*} u = \bs w^T \bs x = \sum_{i = 0}^{K} w_i x_i \end{align*} $$Convert $u$ to output via link function $f$
$$ \begin{align*} \hat y = f (u) \end{align*} $$At this point a neuron is like a generalized linear regression. If choose unit step function $f (u) = I \{ u > 0 \}$ as the link function, this neuron become a linear classifier; if choose logistic function as link function, it become Logistic Regression
Unit step function
The simplest link function is unit step function, this type of neuron called perceptron, usually set 0 as the threshold, only when the input is greater than the threshold can it be sensed.
$$ \begin{align*} f (u) = I \{ u > 0 \} = \begin{cases} 1 & \text{if } u > 0 \\ 0 & \text{if } u \leq 0 \end{cases} \end{align*} $$During gradient descent, iterate through the following methods
$$ \begin{align*} \bs w^{(\text{new})} = \bs w^{(\text{old})} - \eta (y - \hat y) \bs x \end{align*} $$Where $y$ is the true prediction value, and $\eta > 0$ is the learning rate, as the stride length in gradient descent. If choose a lower learning rate, it is less likely to miss the global optimum, but you will easily fall into the local optimal trap and require more iterations.
Using $u = 0$ as a classification criterion may be too hard, The unit step function also does not have good mathematical properties.
Logistic Function
Another link function choice is logistic function, see Logistic Regression
$$ \begin{align*} \hat y = \sigma (u) = \frac{1}{1 + e^{-u}} \end{align*} $$This function have some good mathematical properties, $y$ always in $(0, 1)$ interval, $\sigma (u)$ is differentiable, that will be useful in gradient descent.
$$ \begin{align*} \sigma (-u) & = 1 - \sigma(u) \\ \frac{d \sigma (u)}{du} & = \sigma (u) \sigma (-u) = \sigma (u) [ 1 - \sigma (u) ] \end{align*} $$i.e.
$$ \begin{align*} \frac{d \hat y}{du} = \hat y (1 - \hat y) \end{align*} $$In terms of the choice of loss function, if choose square error as loss function
$$ \begin{align*} E = \frac{1}{2} (y - \hat y)^2 \end{align*} $$The partial derivative of this function can be write as
$$ \begin{align*} \frac{\partial E}{\partial w_i} & = \frac{\partial E}{\partial \hat y} \frac{\partial \hat y}{\partial u} \frac{\partial u}{\partial w_i} \\ & = (y - \hat y) \hat y (1 - \hat y) x_i \end{align*} $$Then we can iterate along the gradient descent to find the optimal value
$$ \begin{align*} \bs w^{(\text{new})} = \bs w^{(\text{old})} - \eta (y - \hat y) \hat y (1 - \hat y) \bs x \end{align*} $$Backpropagation
Following figure showing a multi layer neural network structure, $\bs x = (\contia{x}{K})$ is input, $\bs h = (\contia{h}{N})$ is the hidden layer, $\hat {\bs y} = (\contia{\hat y}{M})$ is output. $\bs W_1 \in \bb R^{K \times N}$ and $\bs W_2 \in \bb R^{N \times M}$ are input-hidden layer and hidden-output layer weights
The calculate as following, where $\bs h = \sigma (\bs u_1)$ represent $h_1 = \sigma (u_{1, 1}), \cdots, h_N = \sigma (u_{1, N})$, so does $\hat {\bs y} = \sigma (\bs u_2)$
$$ \begin{align*} \bs u_1 & = \bs W_1^T \bs x, &&& \bs h & = \sigma (\bs u_1) \\ \bs u_2 & = \bs W_2^T \bs h, &&& \hat {\bs y} & = \sigma (\bs u_2) \\ \end{align*} $$Given $\bs y \in \bb R^M$ is the real prediction value. If squared error loss is still chosen as the loss function, the iteration is discussed in the following section
$$ \begin{align*} E = \frac{1}{2} \norm{\bs y - \hat {\bs y}}^2 = \frac{1}{2} (\bs y - \hat {\bs y})^T (\bs y - \hat {\bs y}) \end{align*} $$Gradient Descent
Start from the rightest weight $\bs W_2$, and then adjust the weight all the way to the left. If the model contains more layers, the gradient descent method is the same. For each layer, we will perform a partial derivative of the loss function, divided into 3 steps, to derive the input, output and weights.
First Gradient Descent
First, perform a partial differential on the output $\hat y$
$$ \begin{align*} \frac{\partial E}{\partial \hat {\bs y}} & = \bs y - \hat {\bs y} \end{align*} $$Next, perform a partial differential on the input $\bs u_2$, where $\circ$ is the Hadamard product, i.e. term-by-term multiplication, e.g. $(A \circ B)_i = A_i B_i$. Recorded with $\bs H_2$, It will still be used in the next gradient
$$ \begin{align*} \frac{\partial E}{\partial \bs u_2} & = \frac{\partial E}{\partial \bs {\hat y}} \frac{\partial \bs {\hat y}}{\partial \bs u_2} = (\bs y - \hat {\bs y}) \circ \hat y \circ (\bs 1 - \hat {\bs y}) = \bs H_2 \end{align*} $$Finally, perform a partial differential on the weights $\bs W_2$
$$ \begin{align*} \frac{\partial E}{\partial \bs W_2} & = \frac{\partial E}{\partial \bs u_2} \frac{\partial \bs u_2}{\partial \bs W_2} = \bs h \bs H_2^T = \bs G_2 \end{align*} $$Second Gradient Descent
After complete the rightest weight $\bs W_2$ gradient calclation, move left to the $\bs W_1$ gradient. Perform a partial differential on the output $\bs h$
$$ \begin{align*} \frac{\partial E}{\partial \bs h} & = \frac{\partial E}{\partial \bs u_2} \frac{\partial \bs u_2}{\partial \bs h} = \bs W_2 \bs H_2 \end{align*} $$Next, perform a partial differential on the input $\bs u_1$
$$ \begin{align*} \frac{\partial E}{\partial \bs u_1} & = \frac{\partial E}{\partial \bs h} \frac{\partial \bs h}{\partial \bs u_1} = \bs W_2 \bs H_2 \circ \bs h \circ (\bs 1 - \bs h) = \bs H_1 \end{align*} $$Finally, perform a partial differential on the weights $\bs W_1$
$$ \begin{align*} \frac{\partial E}{\partial \bs W_1} & = \frac{\partial E}{\partial \bs u_1} \frac{\partial \bs u_1}{\partial \bs W_1} = \bs x \bs H_1^T = \bs G_1 \end{align*} $$After the gradients of $\bs W_1$ and $\bs W_2$ are known, we can iterate in the following way
$$ \begin{align*} \bs W_1^{(\text{new})} & = \bs W_1^{(\text{old})} - \eta \bs G_1 \\ \bs W_2^{(\text{new})} & = \bs W_2^{(\text{old})} - \eta \bs G_2 \end{align*} $$In the above method, $\bs E_i$ will appear when performing partial differential on the output, which can be used in next layer partial differential to calcluate $\bs E_{i - 1}$. The $\bs W_2 \bs H_2$ seem as the “error” of hidden layer, the same is true if more hidden layers are added.