We can make better decisions when we have a better understanding of the different sources of variance that impact job performance ratings. A failure to do so cannot only lead to inaccurate conclusions when interpreting job performance ratings, but often misguided efforts aimed at improving our ability to explain and predict them. In this paper, we outline six recommendations relating to the interpretation of predictive validity coefficients and efforts aimed at predicting job performance ratings. The first three focus on the need to evaluate the effectiveness of selection instruments and systems based only on the variance they can possibly account for. When doing so, we find that it is not only possible to account for the majority of the variance in job performance ratings that most select systems can possibly predict, but that we’ve been able to account for this variance for years. Our last three recommendations focus on the need to incorporate components related to additional sources of variance in our predictive models. We conclude with a discussion of their implications for both research and practice.

Anti-work philosophy holds that work, in and of itself, tends to be harmful for most people. Some anti-work theorists even advocate for the abolition of paid employment altogether. We argue that, while endorsement of the radical ideology of anti-work is in no way necessary for I/O psychologists, considering the thinking behind these ideas can be beneficial. In fact, reviewing the tenets of anti-work may prompt some to a broad reconsideration of the nature and purpose of the I/O field and its role, nested as it is in potentially problematic power dynamics both within organizations and in broader society. In this article, after describing anti-work’s core tenets, we outline a number of research directions and practical applications inspired by this perspective. While in some cases these may involve the creation of new theory, constructs, and interventions, they often simply entail the repurposing or refocusing of existing ones that are more attuned to the problematic nature of work. Possibilities for research include, but are not limited to, the examination of the prevalence and nature of “managerialism,” how we might better understand the psychological character of organized labor and its outcomes, and how to encourage healthier manifestations of employee engagement. In terms of practice, we bring to the reader’s attention how anti-work might inspire extensions or adjustments in how we recruit and onboard, train managers, improve job characteristics, measure performance and work with unions and other political advocates. Ultimately, consideration of anti-work’s assertion of the inevitable authoritarian character of employment, combined with I/O psychology’s emphases on objectivity and the translation of science into practice, can spark inquiry and innovation.

Although assessment centers (ACs) are usually designed to measure stable competencies (i.e., dimensions), doubt about whether or not they reliably do so has endured for 70 years. Addressing this issue in a novel way, several published Generalizability (G) theory studies have sought to isolate the multiple sources of variance in AC ratings, including variance specifically concerned with competencies. Unlike previous research, these studies can provide a definitive answer to the AC construct validity issue. In this article, the historical context for the construct validity debate is set out, and the results of four large-scale G-theory studies of ACs are reviewed. It is concluded that these studies demonstrate, beyond reasonable doubt, that ACs do not reliably measure stable competencies, but instead measure general, and exercise-related, performance. The possibility that ACs measure unstable competencies is considered, and it is suggested that evidence that they do so may reflect an artefact of typical AC design rather than a “real” effect. For ethical, individual, and organizational reasons, it is argued that the use of ACs to measure competencies can no longer be justified and should be halted.

As organizations are becoming increasingly aware of the importance of selecting inclusive leaders, this paper proposes measuring inclusive leadership proficiency as an assessment center (AC) dimension. We propose that ACs present a novel way to effectively assess inclusive leadership using interpersonal behavioral simulation exercises, such as role plays. It is argued that AC-measured inclusive leadership can provide incremental prediction of leader performance beyond commonly assessed AC dimensions; and that it positively predicts follower performance and follower demographic diversity. We conclude by suggesting ways future research might empirically investigate the validity and reliability of AC-measured inclusive leadership in organizational settings.

This study generates new insights on the role of initial impressions in assessment centers. Drawing from the “thin slices” of behavior paradigm in personality and social psychology, we investigate to what extent initial impressions of assessees—based on different slices of assessment center exercises (i.e., two minutes at the beginning, middle, and end of AC exercises)—are consistent across and within AC exercises, and are relevant for predicting assessment center performance and job performance. Employed individuals (N = 223) participated in three interactive assessment center exercises, while being observed and evaluated by trained assessors. Based upon video-recordings of all assessment center exercises, a different, untrained group of raters subsequently provided ratings of their general initial impressions of assessees for the beginning, middle, and end of each exercise. As criterion measure, supervisors rated assessees’ job performance. Results show that initial impressions in assessment centers are (a) relatively stable, (b) consistently predict assessment center performance across different slices of behavior (i.e., across the three time points and exercises), and (c) mostly relate to job performance.

Evidence from previous research suggests that frame-of-reference (FOR) training is effective at improving assessor ratings in many organizational settings. Yet no research has presented a thorough examination of systematic sources of variance (assessor-related effects, evaluation settings, and measurement design features) that might influence training effectiveness. Using a factorial ANOVA and variance components analyses on a database of four studies of frame-of-reference assessor training, we found that (a) training is most effective at identifying low levels of performance and (b) the setting of the training makes little difference with respect to training effectiveness. We also show evidence of the importance of rater training as a key determinant of the quality of performance ratings in general. Implications for FOR training theory and practice are discussed.

Preliminary research has demonstrated that not all assessment center (AC) exercises are viewed as equally just or motivating. The current research builds upon this research and investigates the relationships between six AC exercises and perceptions of self-efficacy, motivation, assessor bias, and fairness. Using a 2 × 2 × 2 experimental design (two informational justice interventions and one rating timing intervention), 286 working adults completed a survey designed to investigate differences between AC exercises and to investigate interventions designed to influence AC exercise perceptions. The results show not only significant perceptual differences between assessor-rated exercises and an ability test but also differences among the rated exercises. The results suggest that an ability test can be perceived as both among the most just and motivating exercises. Lastly, even though the experimental interventions did not have their anticipated effects, the results suggest benefits to having assessors rate recorded participant behaviors versus rating “live” behaviors, benefits that to a certain extent depend on whether participants had previously attended an assessment center.

Ethical leaders are those who exemplify moral behavior personally, as well as those who facilitate follower ethical behavior. Although recent attention has been given to the ethical leadership construct, there remains a lack of innovation regarding the assessment and development of ethical leaders in organizations. To address these issues, a pilot study was conducted to examine the convergent validity of an ethical leadership assessment center, as well as the efficacy of using assessment center feedback to foster ethical leadership. Assessees completed a battery of pre tests, a virtual business simulation with a novel exercise, and a set of post tests. Half of the assessees were randomly assigned to a feedback condition, whereas the other half did not receive feedback until after the post tests were completed. Due to low statistical power, quantitative results were inconclusive. Nevertheless, qualitative insights were gained that point to implications for validating assessment center methodologies when assessing and developing ethical leadership.