HP 15c User Manual
Page 111
Section 4: Using Matrix Operations
111
Least-Squares Using Normal Equations
The unconstrained least-squares problem is known in statistical literature as multiple linear
regression. It uses the linear model
p
j
j
j
r
x
b
y
1
Here, b
1,
…, b
p
are the unknown parameters, x
l
, ..., x
p
are the independent (or explanatory)
variables, y is the dependent (or response) variable, and r is the random error having
expected value E(r) = 0, variance σ
2
.
After making n observations of y and x
1
, x
2
, ..., x
p
, this problem can be expressed as
y = Xb + r
where y is an n-vector, X is an n × p matrix, and r is an n-vector consisting of the unknown
random errors satisfying E(r) = 0 and Cov(r) = E(rr
T
) = σ
2
I
n
.
If the model is correct and X
T
X has an inverse, then the calculated least-squares solution
y
X
X
X
b
T
T
1
)
(
ˆ
has the following properties:
E(
bˆ
) = b, so that bˆ is an unbiased estimator of b.
Cov(
bˆ
) = E((
bˆ
− b)
T
(
bˆ
− b)) = σ
2
(X
T
X)
–l
, the covariance matrix of the estimator
bˆ
.
E(
rˆ
) = 0, where
rˆ
= y − X
bˆ
is the vector of residuals.
2
2
)
(
)
||
ˆ
(||
E
p
n
F
b
X
y
, so that
)
/(
||
ˆ
||
ˆ
2
2
p
n
F
r
is an unbiased estimator for
σ
2
. You can estimate Cov(
bˆ
) by replacing σ
2
by
2
ˆ
.
The total sum of squares
2
||
||
F
y
can be partitioned according to
2
||
||
F
y
= y
T
y
= (y − X
bˆ
+ X
bˆ
)
T
(y − X
bˆ
+ X
bˆ
)
= (y − X
bˆ
)
T
(y − X
bˆ
) - 2
bˆ
T
X
T
(y − X
bˆ
) + (X
bˆ
)
T
(X
bˆ
)
=
2
2
||
ˆ
||
||
ˆ
||
F
F
b
X
b
X
y
=
Squares
of
Sum
Regression
Squares
of
Sum
Residual
.
When the model is correct,
2
2
2
2
||
||
||
ˆ
||
E
p
p
F
F
Xb
b
X