In Chapter 4 we learned that the measurements we make of a computer system are subject to error and, thus, are said to be “noisy.” We also learned that, if the errors in the measurements can reasonably be modeled with a Gaussian distribution, we can use confidence intervals to quantify the precision of the measurements. Alternatively, if the errors in our measurements are not Gaussian, we can normalize the set of measurements by computing the averages of groups of four or more randomly selected measurements. We can then use confidence intervals to characterize the variability of these average values.

While these confidence intervals tell us something about how much noise there is in our measurements, we ultimately want to use these measurements to make a decision about some aspect of the performance of one or more computer systems. For instance, we may want to compare the performance of some component of two different systems, or we may want to determine how some change to a system, such as installing a new operating system or enhancing a communications protocol, affects its performance. Since there is noise in any of the measurements we make, however, we need a technique for determining whether any changes we see are due to random fluctuations in the measurements or whether they are actually significant in a statistical sense.

At this point, many statistics textbooks would introduce the concept of hypothesis testing, which is a statistical technique for making decisions. With this technique, mutually exclusive hypotheses are proposed as statements or assumptions about the population or process that is being measured.