Multiprocessing is traditionally known as the use of multiple concurrent processes in a system as opposed to a single process at any one instant. Like multitasking which allows multiple processes to share a single CPU, multiple CPUs may be used to execute multiple threads within a single process.

Multiprocessing for general tasks is often fairly difficult to achieve due to various programs holding internal data, known as state (or context). Essentially the programs are typically written in such a fashion that they assume their data is incorruptible. However if another copy of the program is running on another processor, the two copies can interfere with each other by both attempting to read and write their state at the same time. A variety of programming techniques are used to avoid this problem, including semaphores and other checks and blocks which allow only one copy of the program to change such values at a time.

Another problem is that processors often use a speed-increasing technique known as caching in which small pools of very fast memory are associated with each processor in order to allow them to work with temporary values very quickly. This can lead to a situation in which each processor is working in a separate cache, rather than in the shared memory; changes to a processor's local cache will not be communicated to other processors until the contents of the cache are written to shared memory. This cannot be helped via programming techniques because it is invisible to the programs themselves. In this case the problem requires additional hardware in order to make sure that all caches on the various processors are up to date, and synchronized with one another.

There are a number of ways to solve this latter problem, of varying complexity. The most common for smaller systems is symmetric multiprocessing (SMP) while larger systems use non-uniform memory access (NUMA) multiprocessing. Multiprocessors may be thought as subgenre of distributed shared memory system, namely hardware one.

Multiprocessing systems fall into one of two general classes:

Tightly coupled multiprocessor systems contain multiple CPUs that are connected at the bus level. These CPUs may have access to a central shared memory (SMP), or may participate in a memory hierarchy with both local and shared memory (NUMA). The IBM p690 Regatta is an example of a high end SMP system.

Loosely coupled multiprocessor systems (often referred to as clusters) are based on multiple standalone single or dual processor commodity computers interconnected via a high speed communication system (gigabit Ethernet is common). A Linux Beowulf cluster is an example of a loosely coupled system.

Historically, tightly coupled systems perform better and are physically smaller than loosely coupled systems (try to visualize 500 commodity computers in racks), but they are much more expensive up front, and don't retain their value as well - nodes in a loosely coupled system are, after all, commodity computers and can live a second life as desktops upon retirement from the cluster.

As technology advances, the differences in performance between loosely coupled and tightly coupled multiprocessor systems will continue to shrink. When 10gb Ethernet becomes commonplace, performance differences will become a non-issue and physical size may become the only justification for tightly coupled systems.

See also: parallel computing, computer multitasking