Political science journals have, for good reason, faced increased scrutiny because of the potential for biases in the editorial process. The representation of women lags behind their distribution in the discipline. Given the importance of publication in hiring, tenure, and promotion, if there are biases in the editorial process, it is vital to the discipline that we determine where in the process these occur and do what is necessary to eliminate them.
Political Behavior uses a double-blind review process. When manuscripts are submitted, the editor determines their fit for the journal in terms of both substance and quality to decide if it is going to be sent out for peer review. At this stage, the editor knows the identity of the author(s). This initial screen results in more than one quarter (30% by August 2017) of all submissions being rejected without external review. Obviously, this is one potential location of any potential bias in the process.
If the manuscript is determined to fit the journal and, in the editor’s view, has the potential to be recommended for publication by the reviewers, it is sent out for blind review. At this stage, the reviewers should not know the identity of the author(s). Of course, the review process is less than ideal and there are certainly instances when the reviewers know the identity of the author(s). It is certainly plausible that the reviewer recommendations might also be a source of any bias in the process.
To try to empirically evaluate this, an undergraduate research assistant coded the data for 851 submissions to Political Behavior from January 2015 until August 2017. For each of these manuscripts, she coded the gender of the author(s), the rank of the senior author, and the initial decision.Footnote 1 For manuscripts that were submitted for external review, the research assistant coded the gender of the reviewer and the categorical rating he or she gave. Other editors have coded the methodological approach of the manuscript. For Political Behavior, this is not a meaningful distinction. All but a handful of the submissions are quantitative or formal.
Following the model used by Ansell and Samuels, this report proceeds as follows. The next section reports the descriptive statistics. I then move to a series of statistical tests to determine if there are any statistically significant differences in the outcomes of the review process based on the gender of the authors. Finally, I examine how the gender of the reviewers results in any differences in either the recommendations of the reviewers or the editorial decision. I find no evidence that the gender of the authors influences the outcome of the review process at Political Behavior.
DESCRIPTIVE STATISTICS
This section provides some simple descriptive statistics about manuscripts submitted to Political Behavior. Table 1 presents a simple summary of the distribution of authors based both on submissions and manuscripts that are granted a revise and resubmit. Again, the coding of the authorship is based on the number of authors (one versus more than one) and the gender of the author(s). Solo-authored work is divided into male or female authors. Collaborative work is divided into three groups: all male, all female, or a team that contains at least one woman and at least one man. The majority of manuscripts submitted to Political Behavior have multiple authors (45.7% are solo) and male authors are much more common than female authors. As seems to be the case with other journals, manuscripts with more than one author appear to have a better chance in the review process than solo authored work, regardless of the gender of the authors.
As seems to be the case with other journals, manuscripts with more than one author appear to have a better chance in the review process than solo authored work, regardless of the gender of the authors.
Table 1 Manuscripts by Gender of Authors, Political Behavior, 2015–2017
Table 2 presents the results of editorial outcomes of manuscripts based on the gender of the authors. To begin, an χ2 test indicates that there is not a significant pattern in the table (p >0.06), but only if one takes a strong stance that the p of 0.05 is the ideal level. The clear difference in the table is from manuscripts with a solo female author. That work is less likely to be desk rejected, but also less likely to be granted a revise and resubmit. Additionally, work written by a team of authors remains more likely to be accepted than work written by single authors. I will return to these results when I explore the multivariate models.
Table 2 Manuscript Outcome by Gender of Authors, Political Behavior, 2015–2017
MULTIVARIATE ANALYSES
While these preliminary analyses are illuminating, there are several other factors that could readily influence the outcome of an editorial process and these may be correlated with authors’ gender. To test this, I have created two dichotomous variables that will capture the result of the editorial process. The first is simply an indicator of being sent out for review (the alternative being a desk rejection). The second is an indicator of receiving a revise and resubmit instead of a rejection after undergoing peer review. For this second variable, all of the manuscripts that were desk rejected (zeros in the first dependent variable) are coded as missing. The first variable captures the initial stage in the editorial process—are there differences in the editor’s decision to send manuscripts out for review based on the gender of the author(s)? The second captures the result of the peer review process. Given that a manuscript is sent out for review, are there differences in the editor’s decision based on the characteristics of the author(s)?
Predictors and Controls
The two predictors of interest are the categories from table 1. The authors’ genders are captured through four indicators, with a solo male author serving as the reference for comparison. The four indicators are: (1) Solo Author Female; (2) All Male Team; (3) All Female Team; and (4) Mixed Gender Team. I also control for three potential confounders in the model. The first of these is the rank of the most senior author. This variable takes on four values: (1) Nonacademic; (2) Graduate Student; (3) Untenured Faculty Member (regardless of specific title); and (4) Tenured Faculty Member. This variable needs to be treated differently for the second (reject with review versus revise and resubmit) variable because there are no manuscripts where the most senior author is a nonacademic and the manuscript was given a revise and resubmit. As a result, the reference category in that model is a tenured faculty member.
In the second model, the one for manuscripts that underwent peer review, there are two additional variables that capture aspects of peer review. The first is the number of reviews the manuscript received. As editor, I am likely to reject a manuscript that receives two negative reviews without waiting for a third review. This variable, then, is expected to be positively related to the outcome of the review process. The second variable is the number of female reviewers. This tests for possible gender differences in the reviews of men and women.
Results
The results of the model predicting whether a manuscript is desk rejected or sent out for review are presented in table 3. This is a simple logit with the independent variables listed in the previous section. The dependent variable is coded so that a positive coefficient means that manuscripts are more likely to undergo peer review, instead of a reject without review.
Table 3 Multivariate Model of Decisions from Political Behavior
Note: *p < 0.05
The results in table 3 show that there are no systematic differences in the outcome of the review process based on gender. In the first column, the model predicting being sent out for review, none of the coefficients capturing the gender of the authors are significant. Post hoc tests also indicate that there are no significant differences between any of the other indicators of the gender composition of the authors (χ2 = 1.29, p > 0.70).
There are effects based on the rank of the author. Only one of the indicators of rank is statistically significant. Perhaps surprisingly, it is the indicator of having a graduate student as the highest rank. It is important to note, however, that the effect of being a graduate student is not statistically different from the effect of having an untenured or tenured faculty member as the senior author (χ2 = 1.79, p > 0.40).
The second column of table 3 presents the model predicting the likelihood that a manuscript will be given a revise and resubmit, given that the manuscript was sent out for review. Again, the reference category for author rank shifted to tenured faculty.
There are few systematic patterns in the likelihood that a manuscript will be given a revise and resubmit instead of rejected after review. None of the indicators of the gender composition of the authors are statistically significant. All of the different codings of the gender of the authors are statistically insignificant.Footnote 2 There are also no significant differences based on the rank of the author.
Only one predictor of the outcome of the review process is significant. The more reviews a manuscript receives, the more likely it is to be accepted. This, however, is endogenous to the process. I chose to end the review process early for manuscripts that have two negative reviews. The number of women who reviewed the manuscript does not predict the outcome of the review process.
GENDER AND REVIEWER RECOMMENDATIONS
In this section, I examine if there are gender differences in the qualitative recommendation that the reviewers give to a manuscript. The editorial system that Political Behavior uses allows reviewers to choose from four options: accept, minor revisions, major revisions, or reject. These have been recoded into a dichotomous variable where accept and minor revisions are coded as 1 while the other recommendations (or no recommendation) is coded as 0. This serves as the dependent variable in table 4.
Table 4 Multivariate Model of Reviewer Ratings from Political Behavior
Note: *p < 0.05
The main new independent variables are the same as in table 3. I have also added a dichotomous variable coding the gender of the reviewer. This variable takes the value 0 for male reviewers and 1 for female reviewers. I also include all of the variables from the previous model.Footnote 3
Only one predictor of the outcome of the review process is significant. The more reviews a manuscript receives the more likely it is to be accepted. This, however, is endogenous to the process.
Results
Table 4 shows the results of a logit model predicting the reviewer’s recommendation. The standard errors are clustered by the manuscript. The first column pools all reviewers and includes an indicator for reviewer gender. The second and third columns separate the data based on the gender of the reviewer, presenting the model for male reviewers (column 2) and female reviewers (column 3) separately.
In the full model, the only significant effects are the comparison between solo-authored and coauthored work. Both work written by all male and all female teams are more likely to get positive reviews than solo-authored work. Reviews of manuscripts written by mixed gender teams are not significantly different than reviews of manuscripts written by individual men or individual women. At the same time, the reviews of manuscripts written by mixed gender teams are not significantly different from reviews of manuscripts either written by all male or all female teams. Still, the general conclusion is that manuscripts written by multiple authors get higher reviews than manuscripts written by single authors.
Male reviewers are significantly more likely to score all male teams more positively than manuscripts written by the reference category. Female reviewers are significantly more likely to score manuscripts written by all female teams more positively than manuscripts written by the reference category.
There are no other significant predictors of the review score. In column 1, the results of the gender of the reviewer indicate that women are less likely to give positive reviews to manuscripts, but that this is not a statistically significant difference (p > 0.10). The rank of the senior author also do not predict the score the reviewer gives.
Column 1 captures any mean differences between male and female reviewers. It does not, however, capture if male and female reviewers respond differently to features of the manuscripts. The second and third columns split the sample by the gender of the reviewer and reports the model for each. For most of the independent variables, there remains no effect on the reviewer score. Where there is a difference, however, is how male and female reviewers score coauthored work by teams of the same gender as the reviewer. Male reviewers are significantly more likely to score all male teams more positively than manuscripts written by the reference category. Female reviewers are significantly more likely to score manuscripts written by all female teams more positively than manuscripts written by the reference category. In neither case, though, are the coefficients between the male and female author teams different from one another.
The main finding in these models is that there is no evidence of bias based on the gender of authors. Female authors are definitely underrepresented in the pages of Political Behavior but this does not seem to be the result of the editorial practices at the journal.
CONCLUSIONS
The main finding in these models is that there is no evidence of bias based on the gender of authors. Female authors are definitely underrepresented in the pages of Political Behavior but this does not seem to be the result of the editorial practices at the journal. There is also no evidence that the rank of the authors (outside of nonacademics) is related to the editorial decision. There also do not appear to be any differences in the evaluations offered by reviewers based on their gender.
The skew in the publications at Political Behavior seem to be the result of differences in the pool of submissions we receive. This is still disappointing. When I became editor, one of my goals was to solicit more manuscripts from women. It appears that I have not been successful at these efforts.
