In mathematics, an invariant subspace of a linear mapping over some vector space V is a subspace W of V such that T(W) is contained in W. An invariant subspace of T is said to be T invariant. The restriction of T to the invariant subspace is denoted

T | _W

Certainly V itself, and the subspace {0}, are trivially invariant subspaces for every linear operator V → V. There may be no non-trivial invariant subspace of V, such as a rotation of a two-dimensional real vector space.

Another example: let v be an eigenvector of T, i.e. Tv = λv. Then W = span {v} is T invariant.

Over finite dimensional vector spaces

Over a finite dimensional vector space every linear transformation can be represented via a matrix.

Suppose now W = span { v₁, ..., v_k} is a T invariant subspace. We shall complete v_j into a basis B of V. Then the matrix of T with respect to the basis B will be as follows:

where the upper-left block express the fact that each image of vector of W is in W itself since it is a linear combination of vectors in W.

The general case

The invariant subspace problem concerns the case where V is a Hilbert space over the complex numbers, of dimension > 1, and T is a bounded operator. It asks whether T always has a non-trivial closed invariant subspace. This problem is unsolved (2005). In case V is only assumed to be a Banach space, it was shown in 1984 by Charles Read that there are counterexamples.

More generally, invariant subspaces are defined for sets of operators (operator algebras, group representations) as subspaces invariant for each operator in the set.