Lessons in programming often start with a definition of the term algorithm. Webster's dictionary defines algorithm as “a step-by-step procedure for solving a problem.” Not only does this definition lend itself naturally to an imperative programming style, but it often also leads to a focus on sequential programming. However, the truth is that program execution is hardly ever in a step-by-step fashion, even if it may sometimes appear to be so. This nonsequentiality can be due to multiple instructions being in flight simultaneously, that is, the instructions are in various stages of their executions at the same time. This is true even when a program is presented as a linear sequence of instructions, and its correctness depends on their execution in that exact sequence. This is also true when the program is “parallel” instead, that is, the order among instructions is not necessarily specified.

In this book, we focus on this parallel programming, where instructions are neither specified nor expected to be in a single sequence. Further, the execution of these programs is also in a parallel context, where potentially several thousand instructions, or even more, execute at any given time.

Concurrency and Parallelism

Sometimes the terms “concurrent” and “parallel” are informally used interchangeably, but it is important to recognize the distinction. Parallelism may be defined as performing two activities at the same time. These activities may be related in some manner or not. Usually, these activities are not instantaneous: each takes a finite time. Two related activities are said to be concurrent if there is no predetermined order between them – they may or may not overlap in time when they do occur. We will see that in certain situations, concurrency is not desirable, and a relative order is imposed. When such an order is enforced on two activities, they clearly cannot be executed in parallel.

Although our focus in this book is on parallel programming, concurrency must often be managed in a parallel program, and we discuss practical aspects of concurrency as well.

Why Study Parallel Programming

Natural processes are inherently parallel, whether they be molecular and nuclear behavior, weather and geological phenomena, or biological and genetic manifestation. By no means does that imply that their simulation and computation must be parallel.