In mathematics, a maximal compact subgroup K of a topological group G is a subgroup K that is a compact space, in the subspace topology, and maximal amongst such subgroups. It is therefore a maximal subgroup amongst compact subgroups, rather than being (alternate possible reading) a maximal subgroup that happens to be compact; which would probably be called a compact maximal subgroup, but in any case is not the intended meaning.

An example would be the subgroup O(2), the orthogonal group, inside the general linear group GL₂(R). A related example is the circle group SO(2) inside SL₂(R). Evidently SO(2) inside GL₂(R) is compact and not maximal.

Maximal compact subgroups may not exist, in a given G. In the theory of semisimple Lie groups, maximal compact groups are shown to exist, and play a basic role in the representation theory when G is not compact. In that case a maximal compact subgroup K must be a compact Lie group, for which the theory is easier. Restricting representations from G to K, and inducing representations from K to G, are basic operations, and quite well understood; their theory includes that of spherical functions .

The algebraic topology of the semisimple groups is also largely carried by a maximal compact subgroup K. This can be expressed by means of the Gram-Schmidt process, concretely, or more abstractly by the Iwasawa decomposition of G, in which K occurs in a product with a contractible subgroup.