Book contents
- Frontmatter
- Contents
- Preface
- PART I THE DESIGN OF JUDGMENT STUDIES
- 1 The nature of judgment studies
- 2 Sampling judges and encoders
- 3 Stimulus selection and presentation
- 4 Judges' responses
- PART II THE ANALYSIS OF JUDGMENT STUDIES
- PART III THE META-ANALYSIS OF JUDGMENT STUDIES
- Appendix Statistical tables
- References
- Name index
- Subject index
2 - Sampling judges and encoders
Published online by Cambridge University Press: 06 November 2009
- Frontmatter
- Contents
- Preface
- PART I THE DESIGN OF JUDGMENT STUDIES
- 1 The nature of judgment studies
- 2 Sampling judges and encoders
- 3 Stimulus selection and presentation
- 4 Judges' responses
- PART II THE ANALYSIS OF JUDGMENT STUDIES
- PART III THE META-ANALYSIS OF JUDGMENT STUDIES
- Appendix Statistical tables
- References
- Name index
- Subject index
Summary
Sampling judges
How many judges shall we employ in a judgment study in which our primary interest is in the encoders rather than the judges, and who should they be? The major factors determining the answers to these questions are (1) the average reliability coefficient (r) between pairs of judges chosen at random from a prescribed population and (2) the nature of the population of judges to which we want to generalize our results.
Effective reliability
Suppose our goal were to establish the definition of the encoder's state (A) or of some encoder nonverbal behavior (B). We might decide to employ judges' ratings for our definition. As we shall see shortly, if the reliability coefficient (any product moment correlation such as r, point biserial r, or phi) were very low we would require more judges than if the reliability coefficient were very high. Just how many judges to employ is a question for which some useful guidelines can be presented (Rosenthal, 1973; Uno, Koivumaki, & Rosenthal, 1972).
If we had a sample of teachers whose nonverbal warmth we wanted to establish we might begin by having two judges rate each teacher's warmth based on the videotaped behavior of each teacher. The correlation coefficient reflecting the reliability of the two judges' ratings would be computed to give us our best (and only) estimate of the correlation likely to be obtained between any two judges drawn from the same population of judges.
- Type
- Chapter
- Information
- Judgment StudiesDesign, Analysis, and Meta-Analysis, pp. 9 - 37Publisher: Cambridge University PressPrint publication year: 1987
- 4
- Cited by