Simple Linear Regression
Introduction
Given 2 random variable $X$ and $Y$, for example, $X$ express as study hours, and $Y$ express as score, we want to find a function $f$
$$ Y = f (X) $$called deterministic model, such that we can determine $Y$ after given $X$, for example $Y = f (X) = 10 + 15 X$, but deterministic model usually can not explain real problems, for example, even the same study time, but the score not always the same. There exists some uncontrolable random phenomenon, that effect $Y$, hence we will add a random variable to explain the model
$$ Y = f (X) + \varepsilon $$After given number of $n$ data, $(X_1, Y_1), (X_2, Y_2), \cdots, (X_n, Y_n)$, we want to find a linear model, called regression function, $\ut \beta = (\beta_0, \beta_1)$ called regression coefficient, and $\hat Y$ called fitted value
$$ \hat Y_i = f (X_i) = \beta_0 + \beta_1 X_i $$where $\ut \beta$ is determined by the data, $\beta_1$ is called slope and $\beta_0$ is called intercept. Every data is generated in randomess, write as
$$ Y_i = f (X_i) = \beta_0 + \beta_1 X_i + \varepsilon_i $$We assume
- $E (\varepsilon_i) = 0$
- $\sigma^2 (\varepsilon_i) = \sigma^2$
- $\sigma (\varepsilon_i, \varepsilon_j) = 0$
Therefore
- $E (Y_i) = \beta_0 + \beta_1 X_i$
- $\sigma^2 (Y_i) = \var (\beta_0 + \beta_1 X_i + \varepsilon_i) = \sigma^2$
- $\sigma (Y_i, Y_j) = \cov (\varepsilon_i, \varepsilon_j) = 0$
Least Squares Estimate (LSE)
For every data, we defined residual ($e_i$) is the different between real value and fitted value,
$$ e_i = Y_i - \hat Y_i = Y_i - \beta_0 - \beta_1 X_i $$then define residual sum of squares (SSE), this number can be use to measure how well of the fit
$$ SSE = \sum_{i = 1}^{n} e_i^2 = \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i)^2 $$and the goal least squares estimate is to minimize SSE, i.e.
$$ (b_0, b_1) = \argmin_{(\beta_0, \beta_1) \in \bb R^2} \ SSE $$Get
$$ \begin{align*} b_1 & = \frac{\sum_{i = 1}^{n} (X_i - \ol X) (Y_i - \ol Y)}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \\ b_0 & = \frac{1}{n} \sum_{i = 1}^{n} Y_i - b_1 \sum_{i = 1}^{n} X_i = \ol Y - b_1 \ol X \end{align*} $$where $\ol X$ and $\ol Y$ are mean of $X$ and $Y$, respectively
$$ \begin{align*} \ol X = \frac{1}{n} \sum_{i = 1}^{n} X_i \quad \ol Y = \frac{1}{n} \sum_{i = 1}^{n} Y_i \end{align*} $$Equations
Introducing two equations, use in following proof
$$ \dps \sum_{i = 1}^{n} (X_i - \ol X)^2 = \sum_{i = 1}^{n} X_i (X_i - \ol X) $$and
$$ \sum_{i = 1}^{n} (X_i - \ol X) (Y_i - \ol Y) = \sum_{i = 1}^{n} (X_i - \ol X) Y_i = \sum_{i = 1}^{n} X_i (Y_i - \ol Y) $$LSE Proof
To minimize SSE, set the first partial derivated to $0$
$$ \begin{align*} & 0 = \frac{\partial}{\partial \beta_0} SSE = - 2 \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i) \\ & 0 = \frac{\partial}{\partial \beta_1} SSE = - 2 \sum_{i = 1}^{n} X_i (Y_i - \beta_0 - \beta_1 X_i) \\ \end{align*} $$solve
$$ \begin{align*} 0 & = \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i) \\ 0 & = \sum_{i = 1}^{n} X_i (Y_i - \beta_0 - \beta_1 X_i) \\ \end{align*} $$Set $(b_0, b_1)$ is the solution of thoes equation, called normal equation
$$ \begin{align*} \sum_{i = 1}^{n} Y_i & = b_0 n + b_1 \sum_{i = 1}^{n} X_i \\ \sum_{i = 1}^{n} X_i Y_i & = b_0 \sum_{i = 1}^{n} X_i + b_1 \sum_{i = 1}^{n} X_i^2 \end{align*} $$Solving first equation, get
$$ \begin{align*} b_0 = \frac{1}{n} \left( \sum_{i = 1}^{n} Y_i - b_1 \sum_{i = 1}^{n} X_i \right) = \ol Y - b_1 \ol X \end{align*} $$Applying $b_0$ to the second equation
$$ \begin{align*} \sum_{i = 1}^{n} X_i Y_i & = b_0 \sum_{i = 1}^{n} X_i + b_1 \sum_{i = 1}^{n} X_i^2 \\ & = (\ol Y - b_1 \ol X ) \sum_{i = 1}^{n} X_i + b_1 \sum_{i = 1}^{n} X_i^2 \end{align*} $$get
$$ \begin{align*} b_1 & = \frac{\sum_{i = 1}^{n} X_i Y_i - \sum_{i = 1}^{n} X_i \ol Y}{\sum_{i = 1}^{n} X_i^2 - \ol X \sum_{i = 1}^{n} X_i} \\ & = \frac{\sum_{i = 1}^{n} X_i (Y_i - \ol Y) }{\sum_{i = 1}^{n} X_i ( X_i - \ol X)} \\ & = \frac{\sum_{i = 1}^{n} (X_i - \ol X) (Y_i - \ol Y) }{\sum_{i = 1}^{n} (X_i - \ol X)^2} \end{align*} $$Ξr write as matrix form, this usually using in multi-variable linear regression
$$ \begin{align*} \begin{pmatrix} \dps \sum_{i = 1}^{n} Y_i \\ \dps \sum_{i = 1}^{n} X_i Y_i \end{pmatrix} = \begin{pmatrix} n & \dps \sum_{i = 1}^{n} X_i \\ \dps \sum_{i = 1}^{n} X_i & \dps \sum_{i = 1}^{n} X_i^2 \end{pmatrix} \begin{pmatrix} \beta_0 \\ \beta_1 \end{pmatrix} \end{align*} $$Also can solve
$$ \begin{align*} \begin{pmatrix} \beta_0 \\ \beta_1 \end{pmatrix} & = \begin{pmatrix} n & \dps \sum_{i = 1}^{n} X_i \\ \dps \sum_{i = 1}^{n} X_i & \dps \sum_{i = 1}^{n} X_i^2 \end{pmatrix}^{-1} \begin{pmatrix} \dps \sum_{i = 1}^{n} Y_i \\ \dps \sum_{i = 1}^{n} X_i Y_i \end{pmatrix} \end{align*} $$Expression of $\beta$
Accroding to the previous result of, $b_1$ has serval expression
$$ \begin{align*} b_1 & = \frac{\sum_{i = 1}^{n} (X_i - \ol X) (Y_i - \ol Y)}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \\ & = \frac{\sum_{i = 1}^{n} (X_i - \ol X) Y_i}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \\ & = \frac{\sum_{i = 1}^{n} X_i (Y_i - \ol Y)}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \end{align*} $$Unbiased
$b_0$ and $b_1$ generated from LSE are all unbiased, i.e. $E (b_0) = \beta_0$ and $E (b_1) = \beta_1$. Since $E (Y_i) = \beta_0 + \beta_1 X_i$, hence
$$ \begin{align*} E (b_1) & = E \left( \frac{\sum_{i = 1}^{n} (X_i - \ol X) Y_i}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) \\ & = \frac{\sum_{i = 1}^{n} (X_i - \ol X) (\beta_0 + \beta_1 X_i)}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \\ & = \beta_0 \left( \frac{\sum_{i = 1}^{n} (X_i - \ol X) }{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) + \beta_1 \left( \frac{\sum_{i = 1}^{n} (X_i - \ol X) X_i }{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) \\ & = \beta_0 \cdot 0 + \beta_1 \cdot 1 \\ & = \beta_1 \end{align*} $$Also
$$ \begin{align*} E (\ol Y) & = E \left( \frac{1}{n} \sum_{i = 1}^{n} Y_i \right) \\ & = \frac{1}{n} \sum_{i = 1}^{n} (\beta_0 + \beta_1 X_i) \\ & = \beta_0 + \beta_1 \ol X \end{align*} $$hence
$$ \begin{align*} E (b_0) & = E (\ol Y - b_1 \ol X) \\ & = (\beta_0 + \beta_1 \ol X) - \beta_1 \ol X \\ & = \beta_0 \end{align*} $$Regression Line Properties
For given $n$ data, using LSE to get $b_0$ and $b_1$, we can estimate the regression line*
$$ \begin{align*} \hat Y = f (X) = \beta_0 + \beta_1 X \end{align*} $$called $\hat Y_i$ is the fitted value of $i$-th data
$$ \begin{align*} \hat Y_i = f (X_i) = \beta_0 + \beta_1 X_i \end{align*} $$The LSE have the following properties
Sum of $Y$ is equal to sum of $\hat Y$
$$ \sum_{i = 1}^{n} Y_i = \sum_{i = 1}^{n} \hat Y_i $$Fitted line pass to the the center of data $(\ol X, \ol Y)$, i.e.
$$ f (\ol X) = b_0 + b_1 \ol X = (\ol Y - b_1 \ol X) + b_1 \ol X = \ol Y $$The sum of residual is 0
$$ \sum_{i = 1}^{n} e_i = \sum_{i = 1}^{n} e_i X_i = \sum_{i = 1}^{n} e_i \hat Y_i = 0 $$
Estimation of Variance $\sigma^2$
Go directly to the conclusion and prove that it will be supplemented in subsequent chapters
$$ \begin{align*} E (SSE) = E \left( \sum_{i = 1}^{n} e_i^2 \right) = (n - 2) \sigma^2 \end{align*} $$Define the mean square error (MSE) is
$$ \begin{align*} MSE = \frac{SSE}{n - 2} \end{align*} $$Can get MSE is an unbiased estimate of $\sigma^2$
$$ \begin{align*} E (MSE) = E \left( \frac{SSE}{n - 2} \right) = \sigma^2 \end{align*} $$So that, we will use MSE to estimate $\sigma^2$, and use $\sqrt{MSE}$ to estimate $\sigma$.
Normal Assumption
Assume the distribution of random phenomenon $\varepsilon_i$ is normal distribution, i.e.
$$ Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i $$and $\varepsilon_i \iid N (0, \sigma^2)$ satisfies the three condition
- $E (\varepsilon_i) = 0$
- $\sigma^2 (\varepsilon_i) = \sigma^2$
- $\sigma (\varepsilon_i, \varepsilon_j) = 0$
Therefore, all the previous inferences are still applicable in the subsequent content, and it can be naturally inferred that
$$ \begin{align*} Y_i \sim N (\beta_0 + \beta_1 X_i, \sigma^2) \end{align*} $$also
- $E (Y_i) = \beta_0 + \beta_1 X_i$
- $\sigma^2 (Y_i) = \sigma^2$
- $\sigma (Y_i, Y_j) = 0$
Maximum Likelihood Estimation (MLE)
For given $X_i$, the probability density function (pdf) of $Y_i$ is
$$ \begin{align*} f (Y_i) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left[ - \frac{1}{2} \left( \frac{Y_i - \beta_0 - \beta_1 X_i}{\sigma} \right)^2 \right] \end{align*} $$So that, under given all data $X$, the likelihood function of parameters is
$$ \begin{align*} L (\beta_0, \beta_1, \sigma^2) & = \prod_{i = 1}^{n} \frac{1}{\sqrt{2 \pi} \sigma} \exp \left[ - \frac{1}{2} \left( \frac{Y_i - \beta_0 - \beta_1 X_i}{\sigma} \right)^2 \right] \\ & = (2 \pi \sigma^2)^{- n / 2} \exp \left[ - \frac{1}{2 \sigma^2} \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i)^2 \right] \end{align*} $$taking log, get log-likelihood function
$$ \begin{align*} l (\beta_0, \beta_1, \sigma^2) & = \ln L (\beta_0, \beta_1, \sigma^2) \\ & = - \frac{n}{2} \ln (2 \pi) - \frac{n}{2} \ln (\sigma^2) - \frac{1}{2 \sigma^2} \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i)^2 \end{align*} $$To maximize $l$, set the derivate derivative of $l$ to 0, i.e.
$$ \begin{align*} 0 & = \frac{\partial}{\partial \beta_0} l = \frac{1}{\sigma^2} \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i) \\ 0 & = \frac{\partial}{\partial \beta_1} l = \frac{1}{\sigma^2} \sum_{i = 1}^{n} X_i (Y_i - \beta_0 - \beta_1 X_i) \\ 0 & = \frac{\partial}{\partial \sigma^2} l = - \frac{n}{2 \sigma^2} + \frac{1}{\sigma^4} \sum_{i = 1}^{n} (Y_i - \beta_0 - \beta_1 X_i) \\ \end{align*} $$Set the solution is $(\hat \beta_0, \hat \beta_1, \hat \sigma^2)$ which satisfies
$$ \begin{align*} 0 & = \sum_{i = 1}^{n} (Y_i - \hat \beta_0 - \hat \beta_1 X_i) \\ 0 & = \sum_{i = 1}^{n} X_i (Y_i - \hat \beta_0 - \hat \beta_1 X_i) \\ \hat \sigma^2 & = \frac{\sum_{i = 1}^{n} (Y_i - \hat \beta_0 - \hat \beta_1 X_i)}{n} = \frac{SSE}{n} \end{align*} $$The solution of $(\hat \beta_0, \hat \beta_1)$ of MLE is equal to the solution LSE, and the solution of $\hat \sigma^2$ of MLE can express as
$$ \begin{align*} \hat \sigma^2 = \frac{n - 2}{n} MSE \end{align*} $$The estiamte of $\hat \sigma^2$ of MLE has slightly biased, as $n \to \infty$, $\hat \sigma^2$ is asymptotic unbiased.
$$ \begin{align*} E (\hat \sigma^2) = \frac{n - 2}{n} \sigma^2 \end{align*} $$but the estiamte of $\hat \sigma^2$ of LSE is unbiased, usually we will use the solution of $\sigma^2$ of LSE.
Inference of $\beta_1$
Before normal assumption, we proved LSE is the unbiased estimate of $b_1$, which means
add an unit of $X$, $Y$ will add $\beta_1$
write as
$$ \begin{align*} f (X + 1) - f (X) = [\beta_0 + \beta_1 (X + 1)] - [\beta_0 + \beta_1 X] = \beta_1 \end{align*} $$Distribution of $b_1$
Under normal assumption $Y_i \sim N (0, \sigma^2)$, we will get
$$ \begin{align*} b_1 & = \frac{\sum_{i = 1}^{n} (X_i - \ol X) Y_i}{\sum_{i = 1}^{n} (X_i - \ol X)^2 } \\ & \sim N \left( \beta_1, \frac{\sum_{i = 1}^{n} (X_i - \ol X)^2 \sigma^2 }{[\sum_{i = 1}^{n} (X_i - \ol X)^2]^2 } \right) \\ & \sim N \left( \beta_1, \frac{\sigma^2 }{\sum_{i = 1}^{n} (X_i - \ol X)^2 } \right) \end{align*} $$But $\sigma^2$ is unknown, we will use MSE to estimate $\sigma^2$, get sample variance
$$ \begin{align*} s^2 (b_1) = \frac{MSE}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \end{align*} $$Distribution of $(b_1 - \beta_1) / s (b_1)$
Under normal assumption
$$ \begin{align*} \frac{SSE}{\sigma^2} \sim \chi_{n - 2}^2 \end{align*} $$Which is independent both of $b_0$ and $b_1$, consider the result after normalization
$$ \begin{align*} \frac{b_1 - \beta_1}{s (b_1)} & = \frac{b_1 - \beta_1}{\sigma (b_1)} / \sqrt{ \frac{s^2 (b_1)}{\sigma^2 (b_1)} } \end{align*} $$where $(b_1 - \beta_1) / \sigma (b_1) \sim Z$ ($Z$ mean stand normal distribution), on the other hand
$$ \begin{align*} \frac{s^2 (b_1)}{\sigma^2 (b_1)} = \frac{MSE}{\sigma^2} = \frac{SSE}{\sigma^2 (n - 2)} \sim \frac{\chi_{n - 2}^2}{n - 2} \end{align*} $$hence
$$ \begin{align*} \frac{b_1 - \beta_1}{s (b_1)} \sim Z / \sqrt{ \frac{\chi_{n - 2}^2}{n - 2} } \sim t_{n - 2} \end{align*} $$Confidence interval of $\beta_1$
Given $q \in (0, 1)$, define $t_{n - 2} (q)$ as
$$ \begin{align*} P (t_{n - 2} \leq t_{n - 2} (1 - q)) = q \end{align*} $$Because of $t$-distribution is symmetrical, so that
$$ \begin{align*} t_{n - 2} (q) = - t_{n - 2} (1 - q) \end{align*} $$For a given level of significance level $\alpha \in (0, 1)$,
$$ \begin{align*} 1 - \alpha & = P \left( t_{n - 2} (\alpha / 2) \leq t_{n - 2} \leq t_{n - 2} (1 - \alpha / 2) \right) \\ & = P \left( t_{n - 2} \in \left( \pm t_{n - 2} (1 - \alpha / 2) \right) \right) \\ & = P \left( \frac{b_1 - \beta_1}{s (b_1)} \in \left( \pm t_{n - 2} (1 - \alpha / 2) \right) \right) \\ & = P \left( \beta_1 \in b_1 \pm s (b_1) t_{n - 2} (1 - \alpha / 2) \right) \end{align*} $$Hecne a confidence interval of $\beta_1$ under level $1 - \alpha$ is
$$ \begin{align*} b_1 \pm s (b_1) t_{n - 2} (1 - \alpha / 2) \end{align*} $$Hypothesis Test of $\beta_1$
Two-sided Test
For a given constant, and an hypothesis test
$$ H : \beta_1 = \beta_{10} \quad \text{vs} \quad A : \beta_1 \ne \beta_{10} $$the test quantity is
$$ t^* = \frac{b_1 - \beta_{10}}{s (b_1)} $$under level $\alpha$
- If $|t^*| \leq t_{n - 2} (1 - \alpha / 2)$ inference $H$
- If $|t^*| > t_{n - 2} (1 - \alpha / 2)$ inference $A$
One-sided Test
For a given constant, and an hypothesis test
$$ H : \beta_1 \leq \beta_{10} \quad \text{vs} \quad A : \beta_1 > \beta_{10} $$the test quantity is
$$ t^* = \frac{b_1 - \beta_{10}}{s (b_1)} $$under level $\alpha$
- If $|t^*| \leq t_{n - 2} (1 - \alpha)$ inference $H$
- If $|t^*| > t_{n - 2} (1 - \alpha)$ inference $A$
Inference of $\beta_0$
The esitmate of $b_1$ of LSE write as
$$ b_0 = \ol Y - b_1 \ol X $$where $\ol Y \sim N (0, \sigma^2 / n)$, on the other hand
$$ b_1 \ol X \sim \ol X \cdot N \left( 0, \frac{\sigma^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) \\ \sim N \left( 0, \frac{\sigma^2 \ol X^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) $$hence
$$ b_0 \sim N \left( 0, \sigma^2 \left[ \frac{1}{n} + \frac{\ol X^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right] \right) $$Use MSE replace of unknown $\sigma^2$, get
$$ s^2 (b_0) = MSE \left( \frac{1}{n} + \frac{\ol X^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) $$consider the result after normalization
$$ \frac{b_0 - \beta_0}{s (b_0)} \sim t_{n - 2} $$Its confidence interval construction and hypothesis testing methods are basically the same as $b_1$ and will not be described again.
Interval Estimate
For a given $X_h$, the point estimate of $Y_h$ is
$$ \hat Y_h = f (X_h) = b_0 + b_1 X_h $$For interval estimate of $Y_h$, we will consider two indervals
Confidence interval: used for parameter estimation, or estimation of old data, which consider the randomness of data $(\contia{\varepsilon}{n})$, i.e.
$$ \begin{align*} \hat Y_h & = b_0 + b_1 X_h \end{align*} $$Prediction interval: used for estimation of new data, not only consider the randomness of existing data, but also consider the randomness of new data, which consider
$$ \begin{align*} \hat Y_h & = b_0 + b_1 X_h + \varepsilon_h \end{align*} $$
Confidence Interval
For any exists point $X_h$, the distribution of $\hat Y_h$ is
$$ \begin{align*} \hat Y_h & = b_0 + b_1 X_h \\ & \sim N \left( \beta_0 + \beta_1 X_h, \sigma^2 \left[ \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right] \right) \\ & \sim N \left( E (Y_h), \sigma^2 \left[ \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right] \right) \end{align*} $$where
$$ \begin{align*} \sigma^2 (\hat Y_h) = \sigma^2 \left[ \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right] \end{align*} $$use MSE to estimate unknown $\sigma^2$, get
$$ \begin{align*} s^2 (\hat Y_h) = MSE \left[ \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right] \end{align*} $$Prove $\sigma^2 (\hat Y_h)$
Although there are separate distributions for $b_0$ and $b_1$, but need to consider $\sigma (b_0, b_1)$, whose covariance is generally not $0$, so another way is used
$$ \begin{align*} \hat Y_h = \ol Y + b_1 (X_h - \ol X) \end{align*} $$the variance is
$$ \begin{align*} \sigma^2 (\hat Y_h) & = \sigma^2 (\ol Y) + \sigma^2 (b_1 (X_h - \ol X)) + 2 \sigma (\ol Y, b_1 (X_h - \ol X)) \\ & = \frac{\sigma^2}{n} + \frac{\sigma^2 (X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} + 2 (X_h - \ol X)^2 \sigma (\ol Y, b_1) \end{align*} $$note that $\sigma (Y_i, Y_i) = \sigma^2 (Y_i) = \sigma^2$ and $\sigma (Y_i, Y_j) = 0$ for $i \ne j$, therefore
$$ \begin{align*} \sigma (\ol Y, b_1) & = \sigma \left( \frac{1}{n} \sum_{i = 1}^{n} Y_i, \frac{\sum_{i = 1}^{n} (X_i - \ol X) Y_i}{\sum_{j = 1}^{n} (X_j - \ol X)^2} \right) \\ & = \frac{\sigma^2}{n} \frac{\sum_{i = 1}^{n} (X_i - \ol X)}{\sum_{j = 1}^{n} (X_j - \ol X)^2} \\ & = 0 \end{align*} $$hence
$$ \begin{align*} \sigma^2 (\hat Y_h) = \sigma^2 \left[ \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right] \end{align*} $$Consider the result after normalization
$$ \begin{align*} \frac{\hat Y_h - E (Y_h)}{s (\hat Y_h)} \sim t_{n - 2} \end{align*} $$The confidence of $E (Y_h) = \beta_0 + \beta_1 X_h$ under level $1 - \alpha$ is
$$ \begin{align*} \hat Y_h \pm s (\hat Y_h) t_{n - 2} (1 - \alpha / 2) \end{align*} $$Prediction Interval
For a given new data $X_{h (new)}$, predict
$$ \begin{align*} \hat Y_{h (new)} = b_0 + b_1 X_{h (new)} \end{align*} $$Assume the real
$$ \begin{align*} Y_{h (new)} = \beta_0 + \beta_1 X_{h (new)} + \varepsilon_{h (new)} \end{align*} $$where $\varepsilon_{h (new)}$ still satisfies noemal assumption
- $E (\varepsilon_{h (new)}) = 0$
- $\sigma^2 (\varepsilon_{h (new)}) = \sigma$
- $\sigma (\varepsilon_{h (new)}, \varepsilon_i) = 0$ for all $i = \conti{n}$
Hence $Y_{h (new)}$ independent the old data $(\contia{Y}{n})$, need consider the randomness of new data $\varepsilon_{h (new)}$, define
$$ \begin{align*} \sigma^2 (pred) & = \sigma^2 (\hat Y_{h (new)} - Y_{h (new)}) \\ & = \sigma^2 (\hat Y_{h (new)}) + \sigma^2 (Y_{h (new)}) \\ & = \sigma^2 (\hat Y_{h (new)}) + \sigma^2 \end{align*} $$use MSE esitmate $\sigma^2$, get
$$ \begin{align*} s^2 (pred) & = s^2 (\hat Y_{h (new)}) + MSE \\ & = MSE \left( 1 + \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) \end{align*} $$after normalization
$$ \begin{align*} \frac{\hat Y_{h (new)} - Y_{h (new)}}{s (pred)} \sim t_{n - 2} \end{align*} $$use this to construct prediction Interval of $\hat Y_{h (new)}$
$$ \begin{align*} \hat Y_{h (new)} \pm s (pred) t_{n - 2} (1 - \alpha / 2) \end{align*} $$Confidence Intervals and Prediction Intervals
Whether it is a confidence interval or a prediction interval, you can see $(X_h - \ol X)^2$ in the numerator part of the variation, which is reflected as “the farther the data point $X_h$ is from the center point $\ol X$, the greater the variation”; in other words
The farther the data $X_h$ to be predicted is from the center point $\ol X$, the lower the prediction efficiency will be.
This phenomenon does not only appear in linear regression, it is a fundamental limitation of all predictive models
The prediction only performs interpolation but not extrapolation. For example, the value range of $X$ in the original data is only $(0, 1)$, and it is obviously not a wise move to predict the new data $X_h = 10$.
Interval Size
The center points of the prediction intervals all fall on $b_0 + b_1 X$. The sample variation between the confidence interval (using $s^2 (\hat Y_h)$) and the prediction interval (using $s^2 (pred)$) depends on “whether the randomness of the new data is considered”. There is a difference of one $MSE$ is the randomness derived from new data
$$ \begin{align*} s^2 (\hat Y_h) & = MSE \left( \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) \\ s^2 (pred) & = MSE \left( 1 + \frac{1}{n} + \frac{(X_h - \ol X)^2}{\sum_{i = 1}^{n} (X_i - \ol X)^2} \right) \end{align*} $$At a given confidence level, the confidence interval will be at the center of the prediction interval, and the confidence interval will be smaller than the prediction interval.
References
Applied linear statistical models. NETER, John, et al. 1996.