Based on the results Teele and Thelen (2017) report, CPS appears to do fairly “well” in terms of gender balance, at least compared to other journals in political science. Nonetheless, their findings still raise the question of whether and to what extent CPS’ editorial processes are biased.
CPS employs a quasi-triple blind review process. Submissions are blind-reviewed internally and then editors make the decision to send the paper out for blind peer review or not. Editors learn the author’s name, rank, and affiliation after the initial evaluation.
We collected data on the 2,134 submissions received from January 2013 to September 2016. For each manuscript, we have authors’ gender, institutional affiliation, and rank. We recorded each paper’s primary methodological approach: quantitative, qualitative, or mixed. We also have information on the outcome of each manuscript—whether it was desk rejected or sent out for external review, and, if the latter, whether it was accepted for publication or not.Footnote 1
We first provide descriptive statistics on the pool of submissions and then assess whether author gender impacts the likelihood that a paper is sent out for review or not and (conditional on having been sent out for review) whether it is accepted.
We find that gender does not systematically predict a manuscript’s success at any stage of the editorial process at CPS. More important factors include rank, coauthorship, and methodological approach. Specifically, we find that at the internal review stage solo-authored papers are less likely to be sent out than coauthored papers, regardless of authors’ gender combination, and qualitative papers are less likely to be sent out than quantitative and mixed-methods papers. Conditional on having been sent out, at the decision stage authors’ rank proves to be a key predictor of success.
The absence of evidence that gender predicts editorial outcomes at CPS suggests that the reason women remain under-represented in CPS’ table of contents relative to their numbers in the subfield has to do with rates of submission to the journal by gender. Women simply submit relatively fewer papers, whether on their own or in collaboration with other scholars, male or female.
DESCRIPTIVE STATISTICS
Table 1 provides information on outcomes based on one of five categories of authors. The first two categories indicate whether a manuscript has a single author and the author’s gender. The other three categories indicate that a manuscript was coauthored, and by what gender combination of authors.Footnote 2
Table 1 Manuscripts by Gender of Authors, 2013–2016
Table 1 shows that most submissions to CPS have one author, and that the modal manuscript is written by a solo male scholar. At the other end of the spectrum, manuscripts submitted by a team of women are rare, comprising about 4% of all submissions.
The table also provides preliminary evidence that collaborative teams are more likely to find success, relative to papers written by either a male or female solo author. The proportion of accepted papers is lower for solo authors than the proportion of submitted papers, while the reverse is true for all three categories of coauthored papers.
Table 2 provides more detail, exploring decisions based on authorship type. Manuscripts written by one man seem more likely to be desk-rejected than those written by one woman. However, this difference largely disappears at the second stage of the editorial process, where manuscripts written by one female enjoy only a slightly higher acceptance rate (6.6% vs. 6.4%). Given the 5/2 ratio of solo male/solo female authors at the submission stage, this table helps explain why twice as many papers authored by one man appear in CPS for every paper authored by one woman.
Table 2 Manuscript Outcome Based on Gender, 2013–2016
Note that coauthored manuscripts appear to be sent out for review at higher rates than solo-authored manuscripts—and coauthored manuscripts appear to have higher rates of R&R/acceptance than all solo-authored manuscripts.
Note that coauthored manuscripts appear to be sent out for review at higher rates than solo-authored manuscripts—and coauthored manuscripts appear to have higher rates of R&R/acceptance than all solo-authored manuscripts.
The next two tables switch the focus to a potential confounding variable: methodological approach. Table 3 provides descriptive statistics regarding submissions to CPS. About 2/3 of all papers CPS receives are quantitative, while only 1/4 are purely qualitative and fewer than 10% employ mixed methods.Footnote 3
Table 3 Submissions by Manuscript’s Methodology
Table 4 focuses on outcomes by methodological approach. Qualitative papers fare the worst at the internal review stage: only 25% of qualitative submissions are sent out for review. Of these, 20% (26/128) obtain an R&R and are eventually accepted. The low numbers of qualitative submissions, and their lack of success at the internal review stage, explains why only 13% of all papers that appear in CPS employ qualitative methods.
Table 4 Outcome of 2013–2016 Submissions by Manuscript Methodology
Table 4 also indicates that quantitative papers are sent out for review at more than twice the rate of qualitative papers—about 55%. However, quantitative papers are given an R&R/acceptance rate similar to that of qualitative papers—about 20% (153/763). Mixed methods papers are sent out at approximately the same rate as quantitative papers, and are R&Red/accepted at approximately the same rate (about 22%).
The characteristics of each manuscript vary across other important dimensions. Table 5 breaks down a manuscripts’ methodological approach by author gender. Solo authors are more likely to submit qualitative papers, while collaborations are far more likely to be quantitative. Almost 3/4 of all group submissions are quantitative.
Table 5 Manuscript Type by Gender
The results across these tables suggest that simple comparisons of categories cannot reliably answer the question of whether gender bias is present or not. To gain confidence, we turn to multiple regression analysis.
GENDER AND EDITORIAL OUTCOMES: A MULTIPLE REGRESSION APPROACH
Which variables are relatively more important for editorial outcomes—gender, method, or something else? We run tests on two dichotomous dependent variables: 1) Review Required, and 2) Accept. The first takes the value 0 if a manuscript is desk rejected and a 1 if it is sent out for review. The second takes the value 1 if a paper is accepted for publication and a 0 otherwise. Because both variables are dichotomous, we estimate a series of logit models. When regressing on Review Required we make use of the entire sample of manuscripts, but when regressing on Accept we narrow the sample to manuscripts sent out for review.
Predictors and Controls
We use as main predictors the five categories from table 1, collapsed into a single variable along the following lines:
• Solo Male Author (solomale) for manuscripts with one male author.
• Solo Female Author (solofemale) for manuscripts with one female author .
• Multi-Author Male (multimale) for coauthored manuscripts written by a team of men.
• Multi-Author Female (multifemale) for coauthored manuscripts written by a team of women.
• Multi-Author Mixed (multimixed) for coauthored manuscripts written by a team including at least 1 woman and 1 man.
We use solo male as the reference category. In a model including only the remaining categories, the intercept would be equivalent to the coefficient of solomale, and the coefficients on the remaining categories would indicate the likelihood that manuscripts in that category differ from the baseline.
We also include a number of potential confounding or control variables:
• A manuscript’s methodological approach, via dummies: either quant for quantiative, qual for qualitative, or mixed for multi-method pieces.Footnote 4
• Highest academic rank (highestRank) tests for whether authors or teams of authors with greater professional experience are more likely to have papers sent out and/or accepted. This takes on values from 1–4 based on the author with the highest academic rank, as follows: 1 for graduate student, 2 for post-doctoral researcher, 3 for assistant professor or the equivalent, and 4 for tenured professor or the equivalent.
Results
We present our simplest logit models in table 6. The coefficients have been exponentiated so that we can interpret them as Odds-Ratios. Recall that the baseline category for comparison is solomale. For reference purposes and for later use (see below) we also provide estimates for identically specified models that use multimale as a baseline, in table 7.
Table 6 Logit Estimates in Odds Rations, Solo Male Baseline
*<0.1; **p<.05; ***p<.01
Table 7 Logit Estimates in Odds Ratios, Multi-Author Male Baseline
*p<0.1; **p<0.05; ***p<0.01
In table 6, in terms of whether a paper is sent out for review, model 1 suggests that the odds a paper authored by one woman is sent out for review is about 31% higher than a paper written by a man. This result is statistically significant at conventional levels. Likewise, coauthored papers (whether they include a woman or not) are more likely to move on to review than papers authored by one man. Results in model 2 suggest that the rank of the solo author or the most experienced author on a collaborative team does not change the results.
In table 6, in terms of whether a paper is sent out for review, model 1 suggests that the odds a paper authored by one woman is sent out for review is about 31% higher than a paper written by a man. This result is statistically significant at conventional levels. Likewise, coauthored papers (whether they include a woman or not) are more likely to move on to review than papers authored by one man.
These results suggest that in terms of whether a paper is sent out for review or not, the authors that are most disadvantaged are solo male authors. This result appears again in table 7, where solo-authored male manuscripts are the least likely to move forward relative to the baseline.
In terms of whether a paper is accepted or not, model 3 in table 6 suggests that conditional on being sent out, the odds of a coauthored paper with mixed genders gaining acceptance are about 67% higher than a paper with one male author. No other differences appear and even this difference disappears entirely in model 4, when we control for highest rank.Footnote 5
In short, although gender and/or collaboration seem to offer some advantage over solo male authors at the initial stage of the editorial process, author rank swamps the effects of gender at the final decision stage. We say “seems” and “appears” because we still need to control for a paper’s methodological approach.Footnote 6
To discover whether gender matters controlling for method and rank, in table 8 we provide predicted probabilities of the likelihood a paper is sent out for review and (conditional on being sent out) accepted. We simulated these probabilities from the posterior distribution of our base model using Zelig.Footnote 7 We hold rank constant at “Tenured.” Since we know that rank is an important and significant predictor after peer review, we expect that the “accept” probabilities will be relatively high. Finally, because we are describing posterior distributions we provide means and standard deviations to assess the probability a paper moves forward or is accepted.
Table 8 Predicted Probabilities by Methodology
Echoing findings in table 6, in terms of whether a paper is sent out or not, the aggregate results (the “overall’ results at the top of table 8) imply that solo female authors have about a five point advantage over solo male authors, regardless of methodological approach. However, it’s also clear that all coauthored papers are sent out at higher rates than all solo-authored papers.
Results in the rest of the table confirm that methodological approach matters a great deal for whether a paper is sent out. When all-female teams produce quantitative papers, these tend to have the highest probability of moving forward, almost 74%, about 16 points higher than the probability for mixed-author pieces. On the other hand, the probability of a qualitative paper by all-female teams being sent out for review is only 26.2%, 13.4 percentage points lower than the probability an equivalent all-male piece moving forward. Finally, solo authored female manuscripts seem to have the overall advantage when writing mixed methodology pieces, even when compared to collaborations.
The effects of gender also show up at the acceptance stage. Notice that the overall acceptance rates all hover around 26%, which makes sense given that we have assumed the authors of these manuscripts are tenured—the model simulates probabilities for papers most likely to be accepted because of rank. In the aggregate, solo male authors have less than a one-point advantage over solo female authors at gaining acceptance, conditional on the paper being sent out.
However, again we see that success also varies depending on methodological approach. Solo male authors have slightly greater success with quantitative and mixed methods papers (by 3 and 13 percentage points respectively), while solo female authors have substantially greater success with qualitative papers (by almost 9 percentage points).
Turning to collaborations, papers with a mixed gender team have an advantage over all-male and all-female teams (by 4.3 and 6.8 percentage points respectively).
In sum, controlling for authors’ highest rank and methodological approach, gender does not exert a consistent effect on editorial decisions. For the modal submission—a quantitative paper by a tenured scholar—collaboration matters much more than gender at the initial stage, but does not consistently matter at the acceptance stage. Meanwhile, gender matters only for all-male quantitative papers, which are less likely to be published than papers by all-female or mixed-gender teams. For qualitative papers, across stages solo male authors fare the worst, but collaborations fare the best by far. All-male collaborations have some advantage at the internal review stage, but mixed-gender collaborations fare better than all other groups at the acceptance stage.
CONCLUSIONS
The fact that gender does not systematically predict whether a paper is sent out and/or accepted suggests that there is no systematic bias against women in the editorial process at CPS. In turn, this suggests that the journal’s quasi-triple blind internal review process appears to accomplish what it was set up to do, which was to remove potential unconscious biases against author rank and gender.Footnote 8 The factors that matter most are whether a paper is a collaboration and its methodological approach. These results raise important questions about the nature of CPS’ pool of submissions, a subject that is beyond the scope of this investigation.
ACKNOWLEDGMENTS
We thank our editorial assistants Thomas Vargas and Katrina Heimark for help compiling this report.
