Sampling judges

How many judges shall we employ in a judgment study in which our primary interest is in the encoders rather than the judges, and who should they be? The major factors determining the answers to these questions are (1) the average reliability coefficient (r) between pairs of judges chosen at random from a prescribed population and (2) the nature of the population of judges to which we want to generalize our results.

Effective reliability

Suppose our goal were to establish the definition of the encoder's state (A) or of some encoder nonverbal behavior (B). We might decide to employ judges' ratings for our definition. As we shall see shortly, if the reliability coefficient (any product moment correlation such as r, point biserial r, or phi) were very low we would require more judges than if the reliability coefficient were very high. Just how many judges to employ is a question for which some useful guidelines can be presented (Rosenthal, 1973; Uno, Koivumaki, & Rosenthal, 1972).

If we had a sample of teachers whose nonverbal warmth we wanted to establish we might begin by having two judges rate each teacher's warmth based on the videotaped behavior of each teacher. The correlation coefficient reflecting the reliability of the two judges' ratings would be computed to give us our best (and only) estimate of the correlation likely to be obtained between any two judges drawn from the same population of judges.