In mathematics, a unitary representation of a group G is a linear representation π of G on a complex Hilbert space V such that π(g) is a unitary operator for every g ∈ G. The general theory is well-developed in case G is a locally compact (Hausdorff) topological group and the representations are strongly continuous .

The theory of unitary representations of groups is closely connected with harmonic analysis. In the case of an abelian group G, a fairly complete picture of the representation theory of G is given by Pontryagin duality. In general, the unitary equivalence classes of irreducible unitary representations of G makes up its unitary dual. This set can be identified to the spectrum of the C*-algebra associated to G by the group C*-algebra construction. This is a topological space.

The general form of the Plancherel theorem tries to describe the regular representation of G on L²(G) by means of a measure on the unitary dual. For G abelian this is given by the Pontryain duality theory. For G compact, this is done by the Peter-Weyl theorem; in that case the unitary dual is a discrete space, and the measure attaches an atom to each point of mass equal to its degree.

One of the pioneers in constructing a general theory of unitary representations was George Mackey.

The theory has been widely applied in quantum mechanics since the 1920s.

Formal definitions

Let G be a topological group. A strongly continuous unitary representation of G on a Hilbert space H is a group homomorphism from G into the unitary group of H,

such that g → π(g) ξ is a norm continuous function for every ξ ∈ H.

Note that if G is a Lie group, this representation is necessarily smooth (respectively real analytic ) with respect to the differentiable structure (respectively real analytic structure) of the Lie group.

Complete reducibility

A unitary representation is completely reducible , in the sense that for any closed invariant subspace, the orthogonal complement is again a closed invariant subspace. This is at the level of an observation, but is a fundamental property. For example, it implies that finite dimensional unitary representations are always a direct sum of irreducible representations, in the algebraic sense.

Since unitary representations are much easier to handle than the general case, it is natural to consider unitarizable representations, those that become unitary on the introduction of a suitable complex Hilbert space structure. This works very well for finite groups, and more generally for compact groups, by an averaging argument applied to an arbitrary hermitian structure. For example, a natural proof of Maschke's theorem is by this route.

In general, for non-compact groups, it is a more serious question which representations are unitarizable.