Introduction to Linear Regression Analysis. Douglas C. Montgomery

Introduction to Linear Regression Analysis

href="#ulink_475f5fba-be26-5d8e-8924-25e5341535b1">Example 2.1. The upper portion of the table contains the fitted regression model. Notice that before rounding the regression coefficients agree with those we calculated manually. Table 2.3 also contains other information about the regression model. We return to this output and explain these quantities in subsequent sections.

2.2.2 Properties of the Least-Squares Estimators and the Fitted Regression Model

The least-squares estimators in18-1 and in18-2 have several important properties. First, note from Eqs. (2.6) and (2.7) that in18-3 and in18-4 are linear combinations of the observations yi. For example,

where in18-5 for i = 1, 2, …, n.

The least-squares estimators in18-6 and in18-7 are unbiased estimators of the model parameters β₀ and β₁. To show this for in18-8 , consider

since E(εi) = 0 by assumption. Now we can show directly that in19-1 and in19-2 , so

That is, if we assume that the model is correct [E(yi) = β₀ + β₁xi], then in19-3 is an unbiased estimator of β₁. Similarly we may show that of in19-4 is an unbiased estimator of β₀, or

The variance of in19-5 is found as

(2.13)

because the observations yi are uncorrelated, and so the variance of the sum is just the sum of the variances. The variance of each term in the sum is in19-6 , and we have assumed that Var(yi) = σ²; consequently,

(2.14)

The variance of in19-7 is

Now the variance of in19-8 is just in19-9 , and the covariance between in19-10 and in19-11 can be shown to be zero (see Problem 2.25). Thus,

(2.15)

Another important result concerning the quality of the least-squares estimators in19-12 and in19-13 is the Gauss-Markov theorem, which states that for the regression model (2.1) with the assumptions E(ε) = 0, Var(ε) = σ², and uncorrelated errors, the least-squares estimators are unbiased and have minimum variance when compared with all other unbiased estimators that are linear combinations of the yi. We often say that the least-squares estimators are best linear unbiased estimators, where “best” implies minimum variance. Appendix C.4 proves the Gauss-Markov theorem for the more general multiple linear regression situation, of which simple linear regression is a special case.

There are several other useful properties of the least-squares fit:

1 The sum of the residuals in any regression model that contains an intercept β0 is always zero, that is,This property follows directly from the first normal equation in Eqs. (2.5) and is demonstrated in Table 2.2 for the residuals from Example 2.1. Rounding errors may affect the sum.

2 The sum of the observed values yi equals the sum of the fitted values , orTable 2.2 demonstrates this result for Example 2.1.

3 The least-squares regression line always passes through the centroid [the point ] of the data.

4 The sum of the residuals weighted by the corresponding value of the regressor variable always equals zero, that is,

5 The sum of the residuals weighted by the corresponding fitted value always equals zero, that is,

2.2.3 Estimation of σ²

In addition to estimating β₀ and β₁, an estimate of σ² is required to test hypotheses and construct interval estimates pertinent to the regression model. Ideally we would like this estimate not to depend on the adequacy of the fitted model. This is only possible when there are several observations on y for at least one value of x (see Section 4.5) or when prior information concerning σ² is available. When this approach cannot be used, the estimate of σ² is obtained from the residual or error sum of squares,

(2.16)

A convenient computing formula for SS_Res may be found by substituting in20-3 into Eq. (2.16) and simplifying, yielding

Скачать книгу

Introduction to Linear Regression Analysis. Douglas C. Montgomery