Science Fair Project Encyclopedia

Science Fair Project Encyclopedia Contents Page

Covariance matrix

In statistics, the covariance matrix is a matrix of covariances between elements of a vector. If $x$ is an n-element column vector where $μ k$ is the expected value of the k^th element of $x$ , or $μ k = E(x k)$ , then the covariance matrix is defined as:

$\Sigma = \mathrm{cov}(\textbf{x})$

$= \mathrm{E} \left[ \left( \textbf{x} - \mathrm{E}[\textbf{x}] \right) \left( \textbf{x} - \mathrm{E}[\textbf{x}] \right)^\top \right]$

$= \mathrm{E} \begin{bmatrix} (x_1 - \mu_1)(x_1 - \mu_1) & (x_1 - \mu_1)(x_2 - \mu_2) & \cdots & (x_1 - \mu_1)(x_n - \mu_n) \\ (x_2 - \mu_2)(x_1 - \mu_1) & (x_2 - \mu_2)(x_2 - \mu_2) & \cdots & (x_2 - \mu_2)(x_n - \mu_n) \\ \vdots & \vdots & \ddots & \vdots \\ (x_n - \mu_n)(x_1 - \mu_1) & (x_n - \mu_n)(x_2 - \mu_2) & \cdots & (x_n - \mu_n)(x_n - \mu_n) \end{bmatrix}$

$= \begin{bmatrix} \mathrm{E}[(x_1 - \mu_1)(x_1 - \mu_1)] & \mathrm{E}[(x_1 - \mu_1)(x_2 - \mu_2)] & \cdots & \mathrm{E}[(x_1 - \mu_1)(x_n - \mu_n)] \\ \mathrm{E}[(x_2 - \mu_2)(x_1 - \mu_1)] & \mathrm{E}[(x_2 - \mu_2)(x_2 - \mu_2)] & \cdots & \mathrm{E}[(x_2 - \mu_2)(x_n - \mu_n)] \\ \vdots & \vdots & \ddots & \vdots \\ \mathrm{E}[(x_n - \mu_n)(x_1 - \mu_1)] & \mathrm{E}[(x_n - \mu_n)(x_2 - \mu_2)] & \cdots & \mathrm{E}[(x_n - \mu_n)(x_n - \mu_n)] \end{bmatrix}$

Note that the diagonal elements are the variances of the elements of $x$ . The $(i, i)$ element of $Σ$ is

E[(x i - μ i)(x i - μ i)] = E[(x i - μ i) 2]

which is the definition of the variance of $x i$ .

Contents

1 Scalar covariance

2 Covariance of a vector

3 Properties

4 Signal processing notation

5 Estimation

6 External references

Scalar covariance

In statistics, the covariance matrix generalizes the concept of variance from one to n dimensions, or in other words, from scalar-valued random variables to vector-valued random variables (tuples of scalar random variables). If x is a scalar-valued random variable with expected value μ then its variance is

σ 2 = cov(x) = E[(x - μ) 2]

Covariance of a vector

If $x$ is an $n \times 1$ column vector-valued random variable, then its variance is the $n \times n$ positive semidefinite matrix

$\Sigma = \mathrm{cov}(\textbf{x}) = \mathrm{E} \left[ \left( \textbf{x} - \mathrm{E}[\textbf{x}] \right) \left( \textbf{x} - \mathrm{E}[\textbf{x}] \right)^\top \right]$

The entries in this matrix are the covariances between all the $n$ different scalar components of $x$ . Since the covariance between a scalar-valued random variable and itself is its variance, it follows that, in particular, the entries on the diagonal of this matrix are the variances of the scalar components of $x$ . This may appear to be a property of this matrix that depends on which coordinate system is chosen for the space in which the random vector $x$ resides. However, it is true generally that if $u$ is any unit vector, then the variance of the projection of $x$ on $u$ is $u\top \Sigma u$ . (This point is expanded upon somewhat at Talk:Covariance matrix. It is a consequence of an identity that appears below.)

Nomenclatures differ. Some statisticians, following the probabilist William Feller, call this the variance of the random vector $x$ , because it is the natural generalization to higher dimensions of the 1-dimensional variance. Others call it the covariance matrix, because it is the matrix of covariances between the scalar components of the vector $x$ .

Properties

With scalar-valued random variables $x$ , we have the identity

cov(a x) = a 2 cov(x)

if $a$ is constant, i.e., not random. If $x$ is an $n \times 1$ column vector-valued random variable and $A$ is an $m \times n$ constant (i.e., non-random) matrix, then $A x$ is an $m \times 1$ column vector-valued random variable, whose variance must therefore be an $m \times m$ matrix. It is

$\mathrm{cov} (A \textbf{x}) = A \Sigma A^\top$

This covariance matrix (though very simple) is a very useful tool in many very different areas. From it a transformation matrix can be derived that allows one to completely decorrelate the data or, from a different point of view, to find an optimal basis for representing the data in a compact way. This is called principal components analysis (PCA) in statistics and Karhunen-Loève transform (KL-transform) in image processing.

Signal processing notation

In signal processing, the correlation matrix of a vector $x$ is denoted

$\textbf{R}_x = \mathrm{E} \left[ \textbf{x} \textbf{x}^\top \right]$ ,

and the covariance matrix is denoted

$\textbf{C}_x = \mathrm{E} \left[ \left( \textbf{x} - \mathrm{E}[\textbf{x}] \right) \left( \textbf{\textbf{x}} - \mathrm{E}[\textbf{x}] \right)^\top \right]$

for real signals. Complex signals involving imaginary numbers requires a slight adjustment to the formula:

$\textbf{C}_x = \mathrm{E} \left[ \left( \textbf{x} - \mathrm{E}[\textbf{x}] \right) \left( \textbf{\textbf{x}} - \mathrm{E}[\textbf{x}] \right)^{* \top} \right]$

where $* \top$ indicates the conjugate transpose (or the complex conjugate of the transpose).

The relationship between the correlation and covariance matrices can be expressed as

$\textbf{R}_x = \textbf{C}_x + \mathrm{E} \left[ \textbf{x} \right] \mathrm{E} \left[\textbf{x}^\top \right]$ .

Estimation

The derivation of the maximum-likelihood estimator of the covariance matrix of a multivariate normal distribution is perhaps surprisingly subtle. It involves the spectral theorem and the reason why it can be better to view a scalar as the trace of a $1 \times 1$ matrix than as a mere scalar. See estimation of covariance matrices.

External references

Covariance Matrix at Mathworld

Categories: Statistics

Science Fair Project Encyclopedia Contents Page

10-26-2009 08:16:03
The contents of this article is licensed from www.wikipedia.org under the GNU Free Documentation License. Click here to see the transparent copy and copyright details

All Science Fair Projects