☰

Zhou's Website
🌐
☰

Feature Image

Principal Components Analysis (PCA)

Math, Data Science, Machine Learning, Unsupervised Learning
Principal component analysis (PCA) is a classic multivariate analysis method and a technique based on orthogonal transformation. Reduce the dimension while preserving the feature of the data as much as possible. Often used for data visualization and dealing with multicollinearity issues.
   Last Update:

Introduction

Principal component analysis (PCA) is a classic multivariate analysis method and a technique based on orthogonal transformation. Reduce the dimension while preserving the feature of the data as much as possible. Often used for

  1. dimensionality reduction
  2. data visualization
  3. multicollinearity

Preprocessing

Centerize

Since we will be using an orthogonal transformation, that means $F_q$ $(1 \leq q \leq p)$ will pass through the origin, we should center the data, i.e. centerize. In other words, if the data does not even go through origin, for example:

normalize

The original data (black dots) had some information, after projection, the projected data (red dots) overlapped into a dot, which means we lost all the information.

Normalize

Normalize has centerize and standardize. If each column may have different scales, for example:

Height (cm)Age (years)
18020
16022
17526
17018
……

The scale difference between height and age is about 10 times, so to maximize variance, PCA preference do height first, but the information of height may not more than age.

Linear Algerbra Theorem

The following proof will make extensive use of this theorem. If $A \in \mathbb R^{p \times p}$ is symmetric, then

$$ \begin{align*} \max_{u \in \mathbb R^p} u^T A u = \lambda_1 \geq \lambda_2 \geq \cdots \quad \lambda_p \geq \lambda_p = \min_{u \in \mathbb R^p} u^T A u, \end{align*} $$

where $\lambda_1, \lambda_2, \cdots, \lambda_p$ are the eigenvalues of $A$.

Principal Components

The goal of principal component analysis is to find the "direction" of the data. As shown in the red line in the figure below, if the data points are projected onto the red line, the projected points will have the largest variance.

First Principal Component

Find a line $F_1$ pass though origin, or equal to find a unit vector $u_1 \in \mathbb R^p$ ($\| u_1 \| = 1$), then projected points into $F_1$.

projection.png

Each point $x_i \in \mathbb R^p$ will be projected onto

$$ \begin{align*} z_i & = x_i^T u_1 \in \mathbb R, \\ p_{x_i} & = z_i u_1 = (x_i^T u_1) u_1 \in \mathbb R^p. \end{align*} $$

To maximize the variance of $z_i$,

$$ \begin{align*} u_1 & = \argmax_{u_1^T u_1 = 1} \text{Var} (X u_1) \\ & = \argmax_{u_1^T u_1 = 1} \, u_1^T \text{Var} (X) u_1 \end{align*} $$

Let $\text{Var} (X) = \Sigma \in \mathbb R^{p \times p}$, the solution of $u_1$ is the eigenvector corresponding to the largest eigenvalue $\lambda_1$ of $\text{Var} (X) = \Sigma$, i.e. $u_1$ satisfies

$$ \begin{align*} \Sigma u_1 = \lambda_1 u_1. \end{align*} $$

Do a diagonalization to $\Sigma = \Gamma \Lambda \Gamma^T$, we will call

$$ \begin{align*} C_1 = X \Gamma_1 \in \mathbb {R}^n \end{align*} $$

is the fitsr principle component.

Second Principal Component

Been fit data into $F_1$, now we can find $F_2$ that pass through origin and orthogonal to $F_1$, or equal to find a unit vector $u_2$ with $u_2^T u_1 = 0$

$$ \begin{align*} u_2 & = \argmin_{\substack{u_2^T u_2 = 1 \\ u_2^T u_1 = 0}} \, u_2^T \Sigma u_2. \end{align*} $$

The solution of $u_2$ is the eigenvector corresponding to the second largest eigenvalue $\lambda_2$ of $\Sigma$, and we will call

$$ \begin{align*} C_2 = X \Gamma_2 \in \mathbb {R}^n \end{align*} $$

is the second principle component.

Fitting Data Summary

The best fit of $u_q$ ($q \leq p$) is the eigenvector corresponding to the $q$-th largest eigenvalue $\lambda_q$ of $\Sigma$, and we will call

$$ \begin{align*} C_q = X \Gamma_q \in \mathbb {R}^n \end{align*} $$

is the $q$-th principle component. Or let

$$ \begin{align*} C = X \Gamma \in \mathbb {R}^{n \times p} \end{align*} $$

called PC transformation.

Quality of the Factorial

The total variance of $X$ is

$$ \begin{align*} \sum_{j = 1}^{p} \text{Var} \, (X_j) & = \sum_{j = 1}^{p} \text{Var} \, (C_j) \end{align*} $$

For each $q \leq p$,

$$ \begin{align*} \text{Var} \, (C_q) = \text{Var} \, (X u_q) = u_q^T \Sigma u_q = u_q^T (\lambda_q) u_q = \lambda_q. \end{align*} $$

Also $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0$, hence a measure of how well the first $q$ PC’s explain variation is given by

$$ \begin{align*} \tau_q = \frac{\sum_{i = 1}^{q} \lambda_j}{\sum_{i = 1}^{p} \lambda_j} = \frac{\sum_{i = 1}^{q} \text{Var} \, (C_j)}{\sum_{i = 1}^{p} \text{Var} \, (C_j)}. \end{align*} $$

So $0 \leq \tau_1 \leq \tau_2 \leq \cdots \leq \tau_p = 1$.

Example in R

PCA Demo

Use R programming language and iris data set to demonstrate. First, read the data

data = iris
y_colname = "Species"
x_colnames = setdiff(names(data), y_colname)

Where prcomp is short for PRincipal COMponent, center and scale. are used to normalize the data before doing PCA.

pca = prcomp(data[x_colnames], center = TRUE, scale. = TRUE)

print(pca)
summary(pca)

the output

Standard deviations (1, .., p=4):
[1] 1.7083611 0.9560494 0.3830886 0.1439265

Rotation (n x k) = (4 x 4):
                    PC1         PC2        PC3        PC4
Sepal.Length  0.5210659 -0.37741762  0.7195664  0.2612863 
Sepal.Width  -0.2693474 -0.92329566 -0.2443818 -0.1235096 
Petal.Length  0.5804131 -0.02449161 -0.1421264 -0.8014492 
Petal.Width   0.5648565 -0.06694199 -0.6342727  0.5235971 
Importance of components:
                          PC1    PC2     PC3     PC4
Standard deviation     1.7084 0.9560 0.38309 0.14393 
Proportion of Variance 0.7296 0.2285 0.03669 0.00518 
Cumulative Proportion  0.7296 0.9581 0.99482 1.00000

Explain

PC1PC2PC3PC4
Sepal.Length$u_{11}$$u_2$$u_3$$u_4$
Sepal.Width$u_{12}$
Petal.Length$u_{13}$
Petal.Width$u_{14}$
PC1PC2PC3PC4
Standard deviation$\sqrt{\lambda_1}$$\sqrt{\lambda_2}$$\sqrt{\lambda_3}$$\sqrt{\lambda_4}$
Proportion of Variance$\dps \frac{\lambda_1}{\sum_{j = 1}^{p} \lambda_j}$$\dps \frac{\lambda_2}{\sum_{j = 1}^{p} \lambda_j}$$\dps \frac{\lambda_3}{\sum_{j = 1}^{p} \lambda_j}$$\dps \frac{\lambda_4}{\sum_{j = 1}^{p} \lambda_j}$
Cumulative Proportion$\tau_1$$\tau_2$$\tau_3$$\tau_4$

To visualize the amount of explanation, the simple conclusion is "the higher the cumulative amount of explanation can be achieved with fewer PCs, the better."

pca.summary = summary(pca)

plot(pca.summary[['importance']]['Proportion of Variance', ], 
     type = "b",
     xlab = "PC index",
     ylab = "Proportion of Variance", 
     main = "Proportion of Variance",
     ylim = c(0, 1))

plot(pca.summary[['importance']]['Cumulative Proportion', ], 
     type = "b",
     xlab = "PC index",
     ylab = "Cumulative Proportion", 
     main = "Cumulative Proportion", 
     ylim = c(0, 1))

Data can be projected onto the PC axis for visualization.

pairs(pca$x, col = data$Species)

Data Explorer

Use the DataExplorer package to quickly obtain data properties including PCA.

library(DataExplorer)
# rotate the x-label 45 degrees
create_report(iris, 
              config = configure_report(plot_correlation_args = list(theme_config = list(axis.text.x = element_text(angle = 45)))))

References

Applied Multivariate Statistics Analysis, Hardle and Simar