A massively parallel is a distributed memory computer system which consists of many individual nodes, each of which is essentially an independent computer in itself, and in turn consists of at least one processor, its own memory, and a link to the network that connects all the nodes together.

Nodes comunicate by passing messages, using standards such as MPI. Nearly all supercomputers as of 2005 are massively parallel, with the largest having tens of thousands of CPUs. All these CPUs make for an impressive peak FLOPS (FLoating point Operations Per Second) numbers , but actually acheiving anywhere near these numbers in real-world applications is very difficult and efficiencies of five to twenty percent are quite normal. These numbers is so low due to the diffculty of keeping the CPUs busy. In MP computers they actually spend a large fraction of their time waiting for data.