> For the complete documentation index, see [llms.txt](https://personal-94.gitbook.io/note-for-econometrics-1/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://personal-94.gitbook.io/note-for-econometrics-1/linear-regression/interpretations-of-linear-regression.md).

# Interpretations of Linear Regression

There are three ways to interpret Linear Regression:

1. Linear Conditional Expectation
2. Best Linear Approximation
3. Causal Model

Now, we talk about each way in detail.

## Linear Conditional Expectation

Suppose that:

$$
E\[Y \mid X]=X^{\prime} \beta
$$

and define: $$U=Y-E\[Y \mid X]$$

Recall the Conditional Expectation Function (CEF): $$\mathbb{E}\[Y \mid X=x]=m(x)$$, which is the best estimator of $$Y$$.

For $$\mathbb{E}\[Y \mid X]$$, we have $$\mathbb{E}\[Y \mid X] = \mathbb{E}\left\[Y \mid X\_1, X\_2, \ldots, X\_k\right] = m(X\_1, X\_2, \ldots, X\_k)$$.But in practice, the form of this function $$m(\cdot)$$ is unknown to us.

This Linear Conditional Expectation has several implications:

1. $$\mathbb{E}(U)=0$$.  

   $$\mathbb{E}\[U]=\mathbb{E}\_{X}\[\mathbb{E}\[U \mid X]]$$

             $$=\mathbb{E}\_{X}\[\mathbb{E}\[Y - X^{\prime}\beta \mid X]] =\mathbb{E}\_X\[\mathbb{E}\[Y| X]-X^{\prime} \beta]=\mathbb{E}\_X\[X^{\prime} \beta - X^{\prime} \beta] = 0$$
2. $$\mathbb{E}\[X U]=0$$

   $$\mathbb{E}\[X U]=\mathbb{E}\left\[X \cdot\left(Y-X^{\prime} \beta\right)\right]=\mathbb{E}\[X Y]-\mathbb{E}\left\[X X^{\prime} \beta\right]$$

                 $$=\mathbb{E}\[\mathbb{E}\[X Y \mid X]]-\mathbb{E}\left\[XX^{\prime} \beta\right]$$

   take $$E\[Y \mid X]=X^{\prime} \beta$$ into it we have:

                 $$= \mathbb{E}\[X \mathbb{E}\[Y \mid X]]-\mathbb{E}\left\[XX^{\prime} \beta\right]=0$$
3. $$\operatorname{Cov}(X, U)=0$$

For $$\beta$$ we got by this define method, it does not necessarily have a Causal Effect. Since one $$X\_i$$ change, others might also change.

Therefore, we also **cannot** interpret $$\beta\_j$$ as the **ceteris paribus** (i.e., holding $$X\_{-j}$$ and $$U$$ constant) effect of a one unit change in $$X\_i$$ on $$Y$$. We need more information to check if we can do that.

* We are only holding $$X\_j, j \neq i$$ constant, but we didn't hold $$U$$ constant in this structure.
* As $$U=Y-\mathbb{E}\[Y \mid X]$$, we have $$Y=\mathbb{E}\[Y \mid X]+U = m(X) + U$$
* So, $$\frac{\partial Y}{\partial X\_i}=\frac{\partial m(X)}{\partial X\_i}+\frac{\partial U}{\partial X\_i}$$
* $$\frac{\partial m(X)}{\partial X\_i} = \beta\_i$$ only if $$m(X)$$ is a linear function, e.g. $$m(X)=\beta\_0+\beta\_1 X\_1+\cdots+\beta\_k X\_k$$

## "Best" Linear Approximation

In general, the conditional expectation is probably NOT linear. We are just using a linear function to approximate this.

Suppose that: $$E\left\[Y^2\right]<\infty$$ and $$E\left\[X X^{\prime}\right]<\infty$$ (or, $$E\left\[X\_j^2\right]<\infty$$ for $$\left.1 \leq j \leq k\right)$$

Under these assumptions, one may consider what is the **"best" linear approximation** (i.e., function of the form $$X^{\prime} b$$ for some choice of $$b \in \mathbf{R}^{k+1}$$, $$b$$ as the best linear predictor for $$\beta$$) to the conditional expectation.&#x20;

To this end, consider the minimization problem for approximation/prediction error:

$$
\min \_{b \in \mathbf{R}^{k+1}} E\left\[\left(E\[Y \mid X]-X^{\prime} b\right)^2\right]
$$

Minimize over $$b$$ and denote $$\beta$$ as the solution to this minimization problem. Then,

* $$\beta$$ is called the best linear predictor, and it is the linear projection coefficient.
* $$b$$ is the generic coefficient vector.

But we still **cannot** interpret $$\beta\_j$$ as the ceteris paribus (i.e., holding $$X\_{-j}$$ and $$U$$ constant) effect of a one unit change in $$X\_j$$ on $$Y$$. Because we still do not know the information about the error term. Partial derivative only hold for other $$X\_i$$s.

### Best Linear Predictor (BLP)

Best linear predictor (BLP) can also be defined as:

$$
\beta \in \underset{b \in \mathbf{R}^{k+1}}{\operatorname{argmin}} E\left\[\left(Y-X^{\prime} b\right)^2\right]
$$

This $$\beta$$ is also a convenient way of summarizing the “best” linear predictor of $$Y$$ given $$X$$.

**Proof:**

$$\mathbb{E}\left\[\left(Y-X^{\prime} b\right)^2\right] =\mathbb{E}\left\[\left(Y-\mathbb{E}\[Y \mid X]+\mathbb{E}\[Y \mid X]-X^{\prime} b\right)^2\right]$$

$$= \mathbb{E}\left\[ (Y - \mathbb{E}\[Y|X])^2 \right] + \mathbb{E}\left\[ (\mathbb{E}\[Y \mid X] - X^{\prime} b)^2 \right]+2 \mathbb{E}\left\[(Y-\mathbb{E}\[Y \mid X])\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right)\right]$$

As for $$\mathbb{E}\left\[(Y-\mathbb{E}\[Y \mid X])\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right)\right]$$, we have that it is equal to,

$$\mathbb{E}\left\[\mathbb{E}\left\[(Y-\mathbb{E}\[Y \mid X])\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right) \mid X\right]\right)$$, and $$\mathbb{E}\[Y \mid X]-X^{\prime} b$$ only depend on $$X$$, we have:

$$\mathbb{E}\left\[(Y-\mathbb{E}\[Y \mid X])\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right)\right] = \mathbb{E}\[(\mathbb{E}\[Y \mid X]-X^{\prime} b)(\mathbb{E}\[Y-\mathbb{E}\[Y \mid X]\mid X])]$$

$$=\mathbb{E}\left\[\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right)(\mathbb{E}\[\mathbb{E}\[Y \mid X] -\mathbb{E}\[Y \mid X]])\right] = 0$$

So, we got:

$$\mathbb{E}\left\[\left(Y-X^{\prime} b\right)^2\right]=\mathbb{E}\left\[(Y-\mathbb{E}\[Y \mid X])^2\right]+\mathbb{E}\left\[\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right)^2\right]$$

However, only the part $$\mathbb{E}\left\[\left(\mathbb{E}\[Y \mid X]-X^{\prime} b\right)^2\right]$$ is depended on $$b$$, so optimize the whole function over $$b$$ is same to optimize over this part. Therefore, this BLP definition is same with the previous one.

### Summary: Two in One

Two interpretations from equivalent optimization problems:

$$
\beta \in \underset{b \in \mathbf{R}^{k+1}}{\operatorname{argmin}} E\left\[\left(E\[Y \mid X]-X^{\prime} b\right)^2\right] \text { and } \beta \in \underset{b \in \mathbb{R}^{k+1}}{\operatorname{argmin}} E\left\[\left(Y-X^{\prime} b\right)^2\right]
$$

**Note** $$E\left\[\left(Y-X^{\prime} b\right)^2\right]$$ **is a convex** (as a function of $$b$$ ) and this has the following implications.

* We can take the derivative on it and use FOC to solve $$b$$ and get $$\beta$$
* **However, in order to do this, we need to make more assumptions.** And I will introduce them in detail later in the estimation of $$\beta$$

Therefore, we can have that:

$$\begin{aligned} \frac{\partial \mathbb{E}\left\[\left(Y-X^{\prime} b\right)^2\right]}{\partial b} & =\mathbb{E}\left\[\frac{\partial\left(Y-X^{\prime} b\right)^2}{\partial b}\right]=\mathbb{E}\left\[\left(Y-X^{\prime} b\right) \cdot X\right] \ & \left.=\mathbb{E}\[Y X]-\mathbb{E}\left\[X^{\prime} b X\right]=\mathbb{E}\[Y X]-\mathbb{E}\left\[X^{\prime} X b\right]\right]=0\end{aligned}$$

We can solve $$b \rightarrow \beta$$. This is the **First Order Condition (FOC)**, note that in above equations:

1. We can change the order between the expectation and partial derivative because of the **dominated convergence theorem**.

   **Lemma:**

   Let $$X \in \mathcal{X}$$ be a random variable $$g: \mathbb{R} \times \mathcal{X} \rightarrow \mathbb{R}$$ a function such that $$g(t, X)$$ is integrable for all $$t$$ and $$g$$ is continuously differentiable w\.r.t. $$t$$. Assume that there is a random variable $$Z$$ such that $$\left|\frac{\partial}{\partial t} g(t, X)\right| \leq Z$$ a.s. for all $$t$$ and $$\mathbb{E}(Z)<\infty$$. Then

   $$
   \frac{\partial}{\partial t} \mathbb{E}(g(t, X))=\mathbb{E}\left(\frac{\partial}{\partial t} g(t, X)\right) .
   $$

   **Proof:**

   We have

   $$
   \begin{aligned} \frac{\partial}{\partial t} \mathbb{E}(g(t, X)) & =\lim \_{h \rightarrow 0} \frac{1}{h}(\mathbb{E}(g(t+h, X))-\mathbb{E}(g(t, X))) \ & =\lim \_{h \rightarrow 0} \mathbb{E}\left(\frac{g(t+h, X)-g(t, X)}{h}\right) \ & =\lim \_{h \rightarrow 0} \mathbb{E}\left(\frac{\partial}{\partial t} g(\tau(h), X)\right), \end{aligned}
   $$

   where $$\tau(h) \in(t, t+h)$$ exists by the mean value theorem. By assumption we have

   $$
   \left|\frac{\partial}{\partial t} g(\tau(h), X)\right| \leq Z
   $$

   and thus we can use the dominated convergence theorem to conclude

   $$
   \frac{\partial}{\partial t} \mathbb{E}(g(t, X))=\mathbb{E}\left(\lim \_{h \rightarrow 0} \frac{\partial}{\partial t} g(\tau(h), X)\right)=\mathbb{E}\left(\frac{\partial}{\partial t} g(t, X)\right) .
   $$

   This completes the proof.
2. We can change the order between the $$X$$ and $$b$$ because they are scalers.

## Casual Model

In order to analysis the Casual Effect using the model, suppose that: $$Y=g(X, U)$$, where $$X$$ are the observed determinants of $$Y$$ and $$U$$ are the unobserved determinants of $$Y$$.

Such a relationship is a model of how $$Y$$ is determined and may come from physics, economics, etc. The effect of $$X\_j$$ on $$Y$$ holding $$X\_{-j}$$ and $$U$$ constant (i.e., ceteris paribus) is determined by this function $$g$$.

If $$g$$ is differentiable. then it is given by

$$
D\_{x\_j} g(X, U) \text {. }
$$

If we assume further that $$g$$ is linear, such that,

$$
g(X, U)=X^{\prime} \beta+U,
$$

then the ceteris paribus effect of $$X\_j$$ on $$Y$$ is simply $$\beta\_j$$. We may normalize $$U$$ so that $$E\[U]=0$$, we can achieve this by replacing $$U$$ with $$U-E\[U]$$ and $$\beta\_0$$ with $$\beta\_0+E\[U]$$ if $$E\[U]=0$$ is not the original case.

On the other hand, $$E\[U \mid X], E\left\[U \mid X\_j\right]$$ and $$E\left\[U X\_j\right]$$ for $$1 \leq j \leq k$$ may or may not equal zero. This aspect is crucial in causal inference as it relates to the independence or correlation of the unobserved determinants with the observed variables. So, $$\beta\_j$$ might be biased for the ceteris paribus effect. To get the ceteris paribus effect, we need more assumptions.

### Potential Outcomes

Potential outcomes are an easy way to think about causal relationships.

Illustration: randomized controlled experiment where individuals are randomly assigned to a treatment (a drug) that is intended to improve their health status.

Notation: Let $$Y$$ denote the observed health status and $$X \in {0,1}$$ denote whether the individual takes the drug or not.

The **causal relationship** between $$X$$ and $$Y$$ can be described using the so-called **potential outcomes**:

* $$Y(0)$$ potential outcome in the absence of treatment
* $$Y(1)$$ potential outcome in the presence of treatment

Under health status, we have two health status variables: $$(Y(0), Y(1))$$, where

* $$Y(0)$$ is the value of the outcome that would have been observed if (possibly counter-to-fact) $$X$$ were 0.
* $$Y(1)$$ is the value of the outcome that would have been observed if (possibly counter-to-fact) $$X$$ were 1.

### Treatment Effects

* The difference $$Y(1)-Y(0)$$ is called the **treatment effect (TE)**.
* The quantity $$E\[Y(1)-Y(0)]$$ is usually referred to as the **average treatment effect (ATE)**.

Using this notation, we may rewrite the observed outcome as:

$$
Y=\beta\_0+\beta\_1 X+U \text { with } \beta\_1=Y(1)-Y(0) .
$$

Note that this is not quite "the" linear model: since the ccoefficient $$\beta\_1$$ is random. **For** $$\beta\_1$$ **to be constant. we need to assume that** $$Y(1)-Y(0)$$ **is constant across individuals.**

Under all these assumptions: we end up with a linear constant effect causal model with $$U$$ independent with $$X$$ (from the nature of the randomized experiment), $$E\[U]=0$$, and so $$E\[X U]=0$$.

Without assuming constant treatment effects, it can be shown that **a regression of** $$Y$$ **on** $$X$$ **identifies the average treatment effect**,

$$
\beta=\frac{\operatorname{Cov}\[Y, X]}{\operatorname{Var}\[X]}=E\[Y(1)-Y(0)]
$$

which is often called a **causal parameter** given that it is an average of causal effects.