Logistic Regression

2024/01/29    Last Update: 2024/07/02

Introduction

For predicting categorical variables, linear regression is not appropriate.

1. Not suitable for describing probability.
2. Not suitable for non-linear relation.

Binary Classification

Assume $Y_i \sim \ber (p_i)$, $p_i \in (0, 1)$, using given data $(x_{i1}, x_{i2}, \cdots, x_{ik})_{i = 1}^{n}$ to predict $p_i$

Logistic Function (Sigmoid Function)

Project $\bb R$ into $(0, 1)$

$$ \begin{align*} \sigma (x) = \frac{\exp (x)}{1 + \exp (x)} = \frac{1}{1 + \exp (-x)} \end{align*} $$

Write

$$ \begin{align*} \bs X_i = \begin{pmatrix} 1 \\ X_{i1} \\ \vdots \\ X_{ik} \end{pmatrix} \quad \text{and} \quad \bs \beta = \begin{pmatrix} \beta_0 \\ \beta_1 \\ \vdots \\ \beta_k \end{pmatrix} \end{align*} $$

Consider the linear combination of data ($\bs X^T \bs \beta$), then use a sigmoid function transform the value from $\bb R$ into $(0, 1)$. To predict $p_i$

$$ \begin{align*} \hat p_i = \sigma (\bs X_i^T \bs \beta) = \frac{\exp (\bs X_i^T \bs \beta)}{1 + \exp (\bs X_i^T \bs \beta)} \end{align*} $$

Loss Function

Our goal is to minimize the loss function

Square Error Loss (SEL)

Usually use in linear regression

$$ \begin{align*} SEL = (y - \hat p)^2 \end{align*} $$

Why not square error loss? Since $y \in \{ 0, 1 \}$ and $\hat y \in (0, 1)$, $\text{Range}(SEL) = (0, 1)$. Too weak!

Cross-Entropy Loss (CEL)

See natural link function in generalized linear model

$$ \begin{align*} CEL & = \begin{cases} -\ln ( \hat{p} ) & \text{ if } y = 1 \\ -\ln ( 1 - \hat{p} ) & \text{ if } y = 0 \\ \end{cases} \\ & = - y \ln (\hat p) - (1 - y) \ln (1 - \hat p) \end{align*} $$

$\text{Range}(CEL) = (0, \infty)$, suitable loss function of logistic regression

The total loss is

$$ \begin{align*} CEL & = \sum_{i = 1}^{n} - y_i \ln (\hat p_i) - (1 - y_i) \ln (1 - \hat p_i) \\ & = - \sum_{y_i = 1} \ln (\hat p_i) - \sum_{y_i = 0} \ln (1 - \hat p_i) \end{align*} $$

To minimize total loss ($\hat p$ is a function of $\bs \beta$)

$$ \begin{align*} \hat {\bs \beta} & = \argmin_{\bs \beta \in \bb R^p} \ CEL \\ & = \argmax_{\bs \beta \in \bb R^p} \ -CEL \\ & = \argmax_{\bs \beta \in \bb R^p} \ \sum_{y_i = 1} \ln (\hat p_i) + \sum_{y_i = 0} \ln (1 - \hat p_i) \end{align*} $$

which $\sum_{y_i = 1} \ln (\hat p_i) + \sum_{y_i = 0} \ln (1 - \hat p_i)$ is the log-likelihood of $\bs \beta$, to maximize it, if and only if maximize the likelihood function

$$ \begin{align*} \hat {\bs \beta} & = \argmax_{\bs \beta \in \bb R^p} \ L (\bs \beta) \\ & = \argmax_{\bs \beta \in \bb R^p} \ \prod_{y_i = 1} \hat p_i \prod_{y_i = 0} (1 - \hat p_i) \\ & = \argmax_{\bs \beta \in \bb R^p} \ \prod_{i = 1}^{n} \hat p_i^{y_i} (1 - \hat p_i)^{1 - y_i} \end{align*} $$

where $\prod_{i = 1}^{n} \hat p_i^{y_i} (1 - \hat p_i)^{1 - y_i}$ is the joint density of $y$, ($Y_i \sim \ber (p_i)$)

Estimate Coefficient

Need to maximize log-likelihood, but $\bs \beta$ has no close form

$$ \begin{align*} l (\bs \beta) = \sum_{i = 1}^{n} Y_i (\bs X_i^T \bs \beta) - \ln \left( 1 + \exp (\bs X_i^T \bs \beta) \right) \end{align*} $$

Use numerical approach to solve likelihood equations

$$ \begin{align*} \begin{cases} \dps 0 = \frac{\partial l}{\partial \beta_0} = \sum_{i = 1}^{n} (y_i - \hat p_i) \\ \dps 0 = \frac{\partial l}{\partial \beta_j} = \sum_{i = 1}^{n} (y_i - \hat p_i) x_{ij} & j = \conti{k} \end{cases} \end{align*} $$

Maximum Likelihood Estimation

We obtain

$$ \begin{align*} \hat p_i = \sigma (\bs X_i^T \bs \beta) = \frac{\exp (\bs X_i^T \bs \beta)}{1 + \exp (\bs X_i^T \bs \beta)} \end{align*} $$

Rewrite

$$ \begin{align*} \tilde p_i = \ln \left( \frac{\hat p_i}{1 - \hat p_i} \right) = \bs X_i^T \bs \beta \end{align*} $$

is called the fitted logit response function.

Example

Given data

1. Study time ($x$): numerical, in hour
2. Pass no not ($y$): binary, if pass, $y = 1$. otherwise, $y = 0$

Time	Pass
0.50	0
0.75	0
1.00	0
…	…
5.00	1
5.50	1

Solve the likelihood equations, get

$$ \begin{align*} \hat y = f (x) = \frac{\exp (\beta_0 + \beta_1 x)}{1 + \exp (\beta_0 + \beta_1 x)} \end{align*} $$

with $\beta_0 = -4.1$ and $\beta_1 = 1.5$

Plot $f (x) = 0.5$, predict $\hat y = I (f (x) \geq 0.5)$, green is right predict, red is false predict

Interpretation of Parameter

In simple logistic regression

$$ \begin{align*} \hat p (x) = \frac{\exp (\beta_0 + \beta_1 x)}{1 + \exp (\beta_0 + \beta_1 x)} \end{align*} $$

as

$$ \begin{align*} \tilde p (x) = \ln \left( \frac{\hat p (x)}{1 - \hat p (x)} \right) = \beta_0 + \beta_1 x \end{align*} $$

where $\hat p / (1 - \hat p)$ is the called odd and $\tilde p$ called log odd, so

$$ \begin{align*} \beta_1 = \tilde p (x + 1) - \tilde p (x) = \ln \left( \frac{\hat p(x + 1)}{1 - \hat p(x + 1)} / \frac{\hat p (x)}{1 - \hat p (x)} \right) \end{align*} $$

The estimate odds ratio is

$$ \begin{align*} \hat {OR} = \frac{\hat p(x + 1)}{1 - \hat p(x + 1)} / \frac{\hat p (x)}{1 - \hat p (x)} = \exp (\beta_1) \end{align*} $$

In last example, $\beta_1 = 1.5$, that is “the odds of pass the test increasing by $\exp (1.5) - 1 \approx 350\%$ with each additional hour”

Multinomial Classification

Now, $Y$ has $J$ level, i.e. $\text{Range} (Y) = \{\conti J\}$, use one-hot encoding for each $Y_i$

$$ \begin{align*} Y_{ij} & = \begin{cases} 1 & \text{if } Y_i = j \\ 0 & \text{if } Y_i \ne j \end{cases} \end{align*} $$

Assume $Y_i \sim \text{Multinomial} (1; p_{i1}, p_{i2}, \cdots, p_{iJ})$ with $\sum_{j = 1}^{J} p_{ij} = 1$ i.e.

$$ \begin{align*} P (Y_{ij} = 1) = p_{ij} \end{align*} $$

Baseline Category Logits

Take level $J$ be the baseline level, turning $J$ classification problem into $J - 1$ comparison $2$ classification problem, each model consider “is level $j$ or $J$”, let

$$ \begin{align*} \tilde p_{ij} = \ln \left( \frac{p_{ij}}{p_{iJ}} \right) = \bs X_i^T \bs \beta_j \end{align*} $$

To compare level $k$ and $l$

$$ \begin{align*} \ln \left( \frac{p_{ik}}{p_{il}} \right) & = \ln \left( \frac{p_{ik}}{p_{iJ}} \right) - \ln \left( \frac{p_{il}}{p_{iJ}} \right) \\ & = \bs X_i^T (\bs \beta_k - \bs \beta_l) \end{align*} $$

Compare each other level

$$ \begin{align*} p_{ij} = \begin{cases} \dps \frac{\exp (\bs X_i^T \bs \beta_j)}{1 + \sum_{k = 1}^{J - 1} (\bs X_i^T \bs \beta_k) } & j = \conti{J - 1} \\ \dps \frac{1}{1 + \sum_{k = 1}^{J - 1} (\bs X_i^T \bs \beta_k) } & j = J \\ \end{cases} \end{align*} $$

Maximum Likelihood Estimation

The joint density of $\ut Y$ is

$$ \begin{align*} f (\ut y) = \prod_{i = 1}^{n} f (y_i) = \prod_{i = 1}^{n} \prod_{j = 1}^{J} p_{ij}^{Y_{ij}} \end{align*} $$

Goal is to maximize log-likelihood:

$$ \begin{align*} l (\contia{\bs \beta}{J - 1}) = \sum_{i = 1}^{n} \sum_{j = 1}^{J - 1} {\Bigr[ } Y_{ij} (\bs X_i^T \bs \beta_j) - \ln \left[ 1 + \exp (\bs X_i^T \bs \beta_j) \right] {\Bigr] } \end{align*} $$

Inference of Parameter

Asymptotic MLE

Assume $\bs b$ is the MLE of $\bs \beta$, as $n \to \infty$

$$ \begin{align*} \bs b - \bs \beta \approx N (0, s^2 (\bs b)) \end{align*} $$

is an asymptotic normal, where $s^2 (\bs b)$ is asymptotic variance and covariance matrix, defined by

$$ \begin{align*} s^2 (\bs b) = [- \triangledown^2 l (\bs b)]^{-1} \end{align*} $$

where $- \triangledown^2 l (\bs b)$ is called observed Fisher information, write as

$$ \begin{align*} - \triangledown^2 l (\bs b)_{lm} & = \left. - \frac{\partial^2}{\partial \beta_l \partial \beta_m} l (\bs \beta) \right|_{\bs \beta = \bs b} \\ & = \sum_{i = 1}^{n} X_{il} X_{im} \sigma' (\bs X_i^T \bs b) \end{align*} $$

The log-likelihood function of $\bs \beta$ is

$$ \begin{align*} l (\bs \beta) & = \sum_{i = 1}^{n} Y_i (\bs X_i^T \bs \beta) - \ln \left( 1 + \exp (\bs X_i^T \bs \beta) \right) \\ & = \sum_{i = 1}^{n} Y_i \left( \sum_{j = 0}^{k} X_{ij} \beta_j \right) - \ln \left( 1 + \exp \left( \sum_{j = 0}^{k} X_{ij} \beta_j \right) \right) \end{align*} $$

First partial derivative

$$ \begin{align*} \frac{\partial}{\partial \beta_l} l (\bs \beta) & = \sum_{i = 1}^{n} Y_i X_{il} - X_{il} \left( \frac{\exp \left( \sum_{j = 0}^{k} X_{ij} \beta_j \right)}{1 + \exp \left( \sum_{j = 0}^{k} X_{ij} \beta_j \right) } \right) \\ & = \sum_{i = 1}^{n} Y_i X_{il} - X_{il} \sigma \left( \sum_{j = 0}^{k} X_{ij} \beta_j \right) \end{align*} $$

Second partial derivative

$$ \begin{align*} \frac{\partial^2}{\partial \beta_l \partial \beta_m} l (\bs \beta) & = - \sum_{i = 1}^{n} X_{il} X_{im} \sigma' \left( \sum_{j = 0}^{k} X_{ij} \beta_j \right) \\ & = - \sum_{i = 1}^{n} X_{il} X_{im} \sigma' \left( \bs X_i^T \bs \beta \right) \end{align*} $$

Finally, multiply both sides $-1$, apply $\bs \beta = \bs b$

$$ \begin{align*} - \left. \frac{\partial^2}{\partial \beta_l \partial \beta_m} l (\bs \beta) \right|_{\bs \beta = \bs b} & = \sum_{i = 1}^{n} X_{il} X_{im} \sigma' (\bs X_i^T \bs b) \end{align*} $$

For one parameter

$$ \begin{align*} \frac{b_j - \beta_j}{s (b_j)} \xrightarrow{D} N (0, 1) \end{align*} $$

where $s (b_j) = [s (\bs b)]_{jj}$, for $j = \contio{k}$

Single Variable Hypothesis Test (Wald Test)

To test

$$ \begin{align*} H : \beta_j = c \quad \text{against} \quad A : \beta_j \ne c \end{align*} $$

the approximate test statistic is

$$ \begin{align*} z^* = \frac{b_j - c}{s (b_j)} \end{align*} $$

In significance level $\alpha \in (0, 1)$

• If $z^* \leq z_{\alpha / 2}$ , conclude $H$
• If $t^* > z_{\alpha / 2}$ , conclude $A$

Confidence Interval

A $1 - \alpha$ confidence interval of $\beta_j$ is

$$ \begin{align*} b_j \pm s (b_j) z_{\alpha / 2} \end{align*} $$

So the confidence interval of odd ratio $\exp (\beta_j)$ is

$$ \begin{align*} \exp \left( b_j \pm s (b_j) z_{\alpha / 2} \right) \end{align*} $$

Likelihood Ratio Test

To test

$$ \begin{align*} & H : \beta_q = \beta_{q + 1} = \cdots = \beta_{p - 1} = 0 && (\text{Reduce model}) \\ & A : \text{exists } \beta_j = 0, \text{ for } j = q, q + 1, \cdots, p - 1 && (\text{Full model}) \end{align*} $$

the approximate test statistic is

$$ \begin{align*} G^2 = -2 \ln \left( \frac{L_r}{L_f} \right) = -2 [l_r - l_f] \end{align*} $$

where $L_r$ and $l_r$ is the likelihood and log-likelihood of reduce model.

In significance level $\alpha \in (0, 1)$

• If $G^2 \leq \chi_{p - q}^2 (1 - \alpha)$ , conclude $H$
• If $G^2 > \chi_{p - q}^2 (1 - \alpha)$ , conclude $A$

Example in R

Result:

References

Applied linear statistical models. NETER, John, et al. 1996.