Ending gender-based harassment in peer review

A few days ago Fiona Ingleby, a postdoctoral fellow at the University of Sussex (she’s an evolutionary biologist who works on sex-specific behavior and other phenotypes in Drosophila) sent out a series of Tweets reporting on a horrifically sexist review she had received after submitting a paper to PLOS ONE. 

There is so much horrible and wrong here, it’s hard to know where to begin. It is completely reprehensible that anyone would think this, let alone write it; that someone would think it was OK to submit a formal review of a paper that said “get a male co-author”; that they would chastise someone for supposed biases without seeing their own glaring ones; that the editor asleep on the job and didn’t look at the review before sending it out or, worse, read the review and thought it wasn’t problematic; that the editor was willing to reject a paper based on an obviously biased review; that the editor didn’t realize that one of their most important roles is to make sure that reviews like this never get sent out or factored into publishing decisions; that PLOS not only allowed this happen but didn’t respond to the authors’ complaint until they took to Twitter several weeks later.

(Let me just disclose for anyone reading this who doesn’t know – I am a founder of PLOS and am on its Board of Directors.  I’ll probably get chastised for commenting publicly on this, but I think it’s important to not just subject PLOS to the same scrutiny and criticism I would bring to the table if it were some other publisher, but to hold PLOS to an even higher standard. This should not have happened, and PLOS needs to not only learn from this, but fix things so that it never happens again. Also, I should add that I have no inside information about this case – I know nothing about it except what has been written about publicly.)

I wish I could say that this review was shocking. But sadly it’s not. As anyone who is paying even the slightest bit of attention should know, science has a serious sexism problem. These kinds of attitudes remain commonplace, and impact women at all stages of their careers in myriad ways. And so it defies credulity to think this is an isolated incident in publishing – if one review like this got through, one has to assume many more like it have been and will be written (indeed they have been) and so we not only have to respond to this event, but we have to do whatever it takes to stop it from ever happening again.

Furthermore, I’ve seen all manner of profanity applied to the review and reviewer – all deserved – for their awful sexist attitudes and acts. But it’s critical that we not dismiss this as just an asshole being an asshole. This happened in a professional setting and clearly targeted the gender of the authors in a way that was not only inappropriate, but which would have had a negative effect on their careers by denying them publication and appropriate credit for their work. So let’s call this what it is – an unambiguous case of harassment.

So what do we do about this? Obviously gender-based harassment happens all over the place. But this particular case happened in the context of science publishing, and PLOS in particular, and I am writing this to ask for help in thinking about what PLOS should do to prevent this from happening (and just to be clear – I don’t run PLOS – but I will do everything I can to make sure all good ideas get implemented).

How do we respond to this reviewer and any future reviewer who engages in harassment in their review?

Once the case became public, PLOS quickly removed the reviewer from its reviewer database, and presumably they will never be asked to review for PLOS again. (I’m still not 100% sure exactly what this means – it seems like we need to do more than remove them from the database – they need to be blacklisted in some manner so that they are never asked to review for PLOS again).

This is obviously a necessary response. But it is also insufficient. First of all, it’s  pretty light punishment – it’s not like people are clamoring to review for PLOS (or any other publisher for that matter). But more importantly, PLOS is but one of many publishers, and accounts for only a few percent of all published papers. This reviewer is still in a position to review for the thousands of other publishers on the planet, so not very much has been accomplished with this action. One can hope the reviewer has learned something from the public discussion of their review, but we certainly can not count on that. So something else needs to be done.

Which bring us to a sticky issue. To do anything more than PLOS has already done would require revealing the reviewers identity either publicly, or at least to the publishers of other journals for which they are likely to review – and reviewers agree to review with the clear expectation that their identity will be kept secret unless they choose to reveal it. While publishers clearly have a duty to protect the anonymity of their reviewers, they also have a responsibility to protect people from harassment. And in this case these two are in conflict. My first instinct is to say, “You do something like this, you lose the right to hide behind the veil of anonymity”, but it’s not as clearcut as I’d like it to be.

It’s no secret to people who read this blog that I have long been against anonymous peer review. But I do recognize that it has a real value, especially to people who are at vulnerable stages in their careers and would not feel comfortable giving their honest opinions if they had to attach their identity to it. In the long run I think we can change the culture of science so they wouldn’t feel that way, but that’s a separate issue. The fact is that right now reviewer anonymity is the norm, and I think it would make a lot of people nervous if publishers granted themselves the right to reveal reviewer identities.

But surely, publishers would reveal reviewer identities in some situations – say if a reviewer physically threatened an author or engaged in some other frankly illegal activity in their review. So clearly anonymity is not inviolable, and the question is whether sexist and harassing reviews raise to the level where the publisher’s interest in protecting others from abuse trumps its interest in preserving reviewer anonymity. I think it does, and furthermore feel it’s a cop out on the part of publishers to hide behind review anonymity here. Engaging in harassing behavior in peer review should void your guarantee of anonymity, full stop.

Obviously, one superficial way to resolve this conflict is to intercept all harassing reviews and make sure they never are seen by the authors – a sort of “no harm, no foul” response. But while this protects the authors from the proximal harms of a biased and sexist review, it doesn’t deal with the harasser. The responsibility of the journal to prevent others from being harassed shouldn’t change because their behavior was caught early.

There are serious challenges in implementing something like this – for example, who would make the decision that something is harassment? – but I am confident we can figure them out. One thing that all publishers can do is to spell out very clearly the kinds of behavior that are unacceptable and what the consequences are for engaging in them. It seems like you shouldn’t have to say “don’t harass people”, but clearly you do. And having very clear policies would likely both help prevent harassment and make it easier to deal with harassers. When this case first came to my attention, I looked around to see if PLOS has some kind of “code of conduct” policy for reviewers, but I couldn’t find one. Maybe I missed it, but if so, then it’s likely not being seen by reviewers. I thought I might find them at the Committee on Publication Ethics, but their code of conduct policy doesn’t seem to deal with this either. Does anyone know of such a policy? I was at a meeting last month sponsored by India Bioscience – the program guide has a great “Code of Conduct” for meeting attendees – this would be a good place to start. [UPDATE: A comment from Irene Hames pointed me to this “Ethical Guidelines for Peer Reviewers” from COPE].

I’m very curious what other people think about this, especially because I’m a bit concerned that my overall feeling that anonymous peer review is bad is coloring my judgment here. But seriously, what could be more important for a publisher to do than protect their authors from harassment? If they’re not willing to do whatever that takes, they should just close up shop.

The role of editors in preventing harassment

It’s hard to fathom how a review as blatantly sexist and harassing as this one was not only sent back to the authors, but used as the sole basis for a negative publication decision on the submission. There are really only two possibilities – neither of them good: the academic editor handling the manuscript failed to fully read the review, or they read it and didn’t find its contents objectionable. So either the editor doesn’t take their job seriously or they are complicit to harassment. Whatever the answer, they shouldn’t be handling manuscripts, and PLOS has asked them to resign their position (and, presumably, will not send them any more manuscripts even if they don’t formally resign).

This editor (again, I don’t know their identity, or anything about their past performance for PLOS) was one of approximately 7,000 academic editors who handle manuscripts for PLOS ONE. The vast majority of the people who edit and review for PLOS take their work seriously and are constructive in their reviews. However, with that many editors it’s inevitable that some are going to do their job poorly. But we can’t just write this off as a bad editor. PLOS has intentionally (and for good reasons) devolved a lot of autonomy to its editors. But in doing so it has magnified the effect that a bad or negligent editor can have, and this increases the need for PLOS to train its editors well, to oversee their work carefully, and to respond rapidly when problems arise – all of which PLOS failed on here.

One issue has to do with the way that editors conceive of their job. It’s always seemed to me that many academic editors think that their primary responsibility is to identify reviewers and then to render decisions on papers after reviews are in. They recognize that they sometimes have to adjudicate between reviewers with different opinions – making them a kind of super reviewer. But I seldomly hear academic editors talk about another – arguably more important – aspect of their job, which is to protect authors from lazy, capricious or hostile reviewers. In my experience most editors almost always pass on reviews to authors even if they disagree with them or think they were inadequate – it’s somehow felt to be bad form to have asked for a review to then turn around and not use it. This needs to change. I would argue that protecting authors from reviewer malfeasance or malignancy is the most important role for editors in our current publishing system. Maybe PLOS and other journals already do this, but every academic editor should be trained to recognize and deal with the various types of harassment and other bad reviewer behaviors that we know exist.

But training can only go so far, and we have to assume that there is going to be considerable variance in the manner in which editors work and that some fraction of papers will be handled poorly, especially for a journal like PLOS ONE where a large number of the editors are young and relatively inexperienced. PLOS knows this, of course, and has long wrestled both with how to get more consistent behavior out of its editors and to deal with problems when they arise. There are two general possibilities: there could be a second layers of more experienced editors or staffers who review every decision letter for its adherence to PLOS’s editorial standards and code of conduct before it goes out, or PLOS could assume that most decisions are good and rely on feedback from authors (aka complaints) to identify problems.

You can understand why PLOS ONE and most other journals that already rely heavily on academic editors generally choose the later solution – it’s hard enough to find people to handle manuscripts – adding a second layer of review would slow things down even further and make them more expensive. But if you’re going to use this strategy, then it seems like it’s imperative that you respond to issues – especially serious ones – quickly. And PLOS failed to do this – the authors say they had been waiting for almost a month for PLOS to respond to their complaint about how their manuscript was handled.

PLOS really has to fix this. But I also think they should consider what it would take to have every decision letter screened before sending it out to authors. This would not only go a long way towards preventing harassment in the review process, but also ensuring that the whole process is more fair (I’ve fielded a fair number of complaints about the failure of editors to properly implement PLOS ONE’s editorial policies – one decision letter I saw described a paper as “technically sound, but not of sufficient interest to merit publication in PLOS ONE” – a clear contradiction of PLOS ONE‘s standards for inclusion).

How much would this cost? Seems like you could hire someone who looks at 2-3 decision letters an hour, so lets say 20 a day, or 5,000 a year. Even if you pay this person a very good salary, you’re only talking $20-$25/article to make sure people aren’t being harassed and are otherwise being treated fairly. Considering that we spend around $6,500/published article on average across the industry, this seems like a pittance.

Protecting authors in an open review/post-publication review world

I’ve written a lot about why I think the whole system of pre-publication peer review that dominated science publishing needs to be replaced with a system where papers are published whenever authors feel they are ready, and peer review happens post-publication and is not limited to 2 or 3 handpicked reviewers. I’m not going to rehash why I think this system is better – you can read my arguments here and here. PLOS will begin the first stages of this transition soon. More open peer review will discourage some of the bad behavior that takes place when reviewers are anonymous. Taking away the power individual reviewers currently have to influence the fate of a paper and thus the careers of its authors should make review more fair. However, protections the formal structure of peer review affords authors from bad reviewer behavior could easily be undermined if we try to rely too heavily on the wisdom of the crowd to police peer review.The sexist attitudes that reared their ugly head in this case are not going to go away because we change the way peer review works. So it’s very important that, in trying to fix other aspects of science publishing, we don’t end up increasing authors exposure to abuse. In this world I think the things discussed above – very clear codes of conduct for reviewers, and proactive policing of reviews – become even more important. And while I’ve been convinced that it’s important to allow reviewers to be unnamed to authors and readers, it’s imperative that they not be truly anonymous – somebody (publisher, scientific society, etc…) has to know who reviewers are so that harassment and other abusive behaviors can be discouraged and dealt with appropriately when they occur.

Please let me know what you think about these issues. I’m sure others have better ideas than I do about how to prevent and deal with harassment in science publishing today and in the future.


 

UPDATE: Several people on Twitter have noted that the term “sexual harassment” is specific to cases involving unwanted sexual advances. The terms “sexist” and “gender bias” were suggested by some, but I don’t think that captures the egregiousness of the offense, so I changed the title and text to “gender-based harassment”, which I think is more appropriate.

This entry was posted in science and tagged , , , . Bookmark the permalink. Both comments and trackbacks are currently closed.

44 Comments

  1. WT
    Posted May 3, 2015 at 8:23 pm | Permalink

    I’d be interested in your response to this blog post: http://www.ljzigerell.com/?p=3104

    In particular, this paragraph:

    “Consider this hypothetical. The main professional organization in biology decides to conduct research and draft a statement on gender bias in biology. The team selected to perform this task includes only men. The peer reviewer from this episode suggests that including women on the team would help “serve as a possible check against interpretations that may sometimes be drifting too far away from empirical evidence into ideologically based assumptions.” Is that sexism, too? If not, why not? If so, then when ‒ if ever ‒ is it not sexist to suggest that gender diversity might be beneficial?”

    • JJ, Ph.D
      Posted May 4, 2015 at 10:53 am | Permalink

      The problem is not with the crux of the recommendation but, rather, the justification and proposed methodology for that recommendation. In the PLOS case, the assumption is that the male colleagues would be the ones who possess the “correct” ways of thinking/interpreting data. In the example you’ve quoted, the assumption is that the men would automatically be biased and that it is the women who would possess the “correct” ways of thinking/interpreting data. You can soundly justify diversity without communicating sexist ideology (in case it isn’t clear, both examples–the PLOS case and the ones you cite–would, indeed, reflect sexist communication). You can do so by considering the relationship between diversity and innovation, productivity, and satisfaction with outcomes, among other things. Or hell, you could even do it because you want the makeup of this fictitious gender diversity committee to represent the population that you’d be generalizing to.

      The reviewer from the PLOS case could have simply told them to check to make sure that they aren’t making inferential leaps WITH specific, statistical examples and WITHOUT telling them that a man should help them. This recommendation is made all the time (sans gender comments) in peer review; academics can’t get it right all the time.

      But to what I suspect your point really is: Why should discussion/treatment of gender issues encourage increased participation from women in male-dominated contexts and not vice versa? I think both are needed–male nurses deserve a voice just like female doctors do. I don’t think the author of this piece would strongly disagree with that, either. You are either a general fan of intellectual gymnastics, or you are clinging to a Straw Man.

      Denial is not the same thing as progress: Women tend to be more of a focus on these conversations because women still, in general, face additional barriers from all angles that make it difficult for them to achieve the same status and power on society that men do. You can argue that men and women ought to be equal, but they still aren’t–even today we don’t live in comparable realities. Until we do, those who make the argument that the “benefits” women receive aren’t fairly being extended to men (or that women are “stealing everything”) is being made in incredibly bad taste.

      • Posted May 4, 2015 at 6:25 pm | Permalink

        Hi JJ, Ph.D,

        I’m not sure what the Straw Man refers to in this case, but I think that “fan of intellectual gymnastics” might be an appropriate label, depending on what that means. I don’t think that I improve my thinking by agreeing with what I perceive to be the conventional wisdom. In this case, I perceived a tension between believing that gender diversity has benefits and believing that it is sexist to suggest that gender diversity would have benefits in a specific situation, so I was interested in thinking more clearly about the topic. You have helped with that, so I thank you.

        You addressed the hypothetical consistently and noted that gender diversity has benefits beyond helping improve the interpretation of data on gender bias. But I think that considering the relationship between diversity and the benefits of diversity requires assumptions about the mechanisms by which these benefits are produced.

        You write, alluding to this mechanism: “In the example you’ve quoted, the assumption is that the men would automatically be biased and that it is the women who would possess the ‘correct’ ways of thinking/interpreting data.”

        But I don’t think that is the assumption in the hypothetical. I think that the assumption in the suggestion that the team include men and women is this: 1. men and women have different experiences and thus different perspectives on the topic of gender bias such that 2. members of both sexes should be expected to have biases that tend to lead them to over- or underestimate or not fully appreciate the size and nature of gender bias, but 3. through communication with each other these biases can be reduced so that 4. the end interpretation is closer to a correct appreciation of the size and nature of gender bias.

  2. amygdala
    Posted May 3, 2015 at 9:06 pm | Permalink

    Thank you for this post. I would certainly hope that there’s no blowback at you for opening up this discussion. The issues are important.

    Open peer review would likely help, at least for reviewers and editors aware of potential biases and motivated to address them. I’m less optimistic that it would ameliorate implicit bias, which is an issue for all of us. Blinded reviews would address that more directly. Whether there is an additional back-end review, or the current editorial process is maintained, perhaps a brief checklist for those editors would be useful.

    Again, thank you for the considerable thought you have put into this. After 20+ years, I am leaving academic medicine because the misogyny I had hoped would diminish with time gets worse with seniority. It is reassuring that major institutions, such as PLoS, are willing to look for solutions.

  3. Dorothy Bishop
    Posted May 3, 2015 at 10:53 pm | Permalink

    Re having editors block inappropriate reviews, you say
    “Obviously, one superficial way to resolve this conflict is to intercept all harassing reviews and make sure they never are seen by the authors – a sort of “no harm, no foul” response. But while this protects the authors from the proximal harms of a biased and sexist review, it doesn’t deal with the harasser.”
    I think it could work if the editor also wrote to the reviewer explaining what they had done and why. As an editor, I once told a reviewer that I had taken offensive (not sexist, just rude) language out of their review before relaying it to the author, and I got a very contrite and apologetic response.
    Not saying everyone would react the same, but, as you know, anonymity does bring out the worst in people, and sometimes they just need reminding that someone who knows their identity is evaluating how they are behaving.
    Overall, though, I think non-anonymous review is the answer – with reviews published with the articles. The latter, one hopes, would offer some counter to the possible harms of being revealed as a negative reviewer – in that people other than the authors might see that the reviewer is someone who is good at critical evaluation (assuming they are).

  4. Onuralp
    Posted May 3, 2015 at 11:33 pm | Permalink

    A possible remedy to implement might be to set up a “fair conduct” board that would welcome appeals from authors who think that their submission is mishandled on non-scholar grounds, and to make the communication public in an instructive and exemplary way as to prevent the misconduct, if any, from happening again.

    In principle, this board would attend to issues that are not, or cannot easily be, addressed during the traditional peer-review process. Such issues would include the unfortunate case subject to this piece as well as practices like bullying that may easily go unnoticed or be considered okay unless directly called upon.

    Such a board can also help the community assess more rigorously the type and frequency of various malpractices in peer-review publication process in order to better appreciate the pitfalls of the current publication system in general.

    Clearly, the structure of this board should be discussed thoroughly by all parties involved to make sure that it is sufficiently versatile to address all possible issues of concern while it is not so vague to allow exploitation.

  5. Posted May 4, 2015 at 3:41 am | Permalink

    Lots of good sense here, needless to say. Just wanted to pick up on this:

    “Engaging in harassing behavior in peer review should void your guarantee of anonymity, full stop.”

    It makes me very nervous to read such things, because the boundary between what is and isn’t “harassing behaviour” is not a clear one. We’re bound to run into situation where an editor feels a line has been cross, and a reviewer in good faith feels that it hasn’t. To break promised anonymity in such cases would be very unfortunate. Since the whole purpose of anonymity in review is to allow reviewers to speak uncomfortable truths, I think that (if we retain anonymity at all) we need to be extremely careful about compromising it.

    For avoidance of doubt: I am NOT in any way defending that reviewer in the current incident. I am trying to ensure that we don’t take a hard case, and make a bad law from it.

  6. Posted May 4, 2015 at 3:59 am | Permalink

    Also, just to confirm this:

    “One issue has to do with the way that editors conceive of their job. It’s always seemed to me that many academic editors think that their primary responsibility is to identify reviewers and then to render decisions on papers after reviews are in. They recognize that they sometimes have to adjudicate between reviewers with different opinions – making them a kind of super reviewer.”

    For what it’s worth, that is exactly what I thought an academic editor’s job was. Which helps to show how widespread the misapprehension you’re describing is.

    “But I seldomly hear academic editors talk about another – arguably more important – aspect of their job, which is to protect authors from lazy, capricious or hostile reviewers.”

    Having been on the receiving end of several lazy and capricious reviews myself, and one hostile one, it hadn’t occurred to me that an editor was supposed to protect me from them. I assumed I, as the author, was supposed to see all the reviews, respond to them, and make a case for the editor to accept my (revised) manuscript.

    Might this be something that varies between different fields of scholarship?

  7. Manfred Zorn
    Posted May 4, 2015 at 7:36 am | Permalink

    Most universities now have a compliance and ethics department that operates a complaint process for such sexual harassment. By filing a complaint against the reviewer with their institution, it would achieve 2 things: there is a standard process in place that investigates the harassment and provides a venue for the accused to present his/her side of the story; and secondly, the institution is alerted to this behavior and can no longer claim it as isolated incidents. A person that crafts such reviews most likely exhibits similar behavior in “real” life with students and colleagues. If we can stop that behavior at the institution, it would affect all publishers this person will ever come in contact with.

  8. Fred Mast
    Posted May 4, 2015 at 7:43 am | Permalink

    This egregious example of gender harassment (I don’t think you can call it sexual harassment even though the comments were sexist) opens a view to a larger problem within scientific culture in general. The reviewer in this case has clear biases against the gender of the authors, but also their age. A more pervasive form of bias is one based on pedigree. While it could be argued that the pedigree of an author does weigh on the quality of a manuscript it is (should be) clear that the age or sex of an author is irrelevant. In most cases, comments (whether positive or negative) based on these forms of bias are unnecessary, and in the case of gender or age bias may even be criminal depending on jurisdiction.
    Our goal should be one of education and to strive towards more objectivity in scientific publishing. I think your position on moving away from prepublication reviews is motivated by wanting more objectivity in the reviewer process.
    Blacklisting the sexist reviewer may be justified but I would rather like to see an opportunity for the reviewer to learn from this mistake and change their ways.
    In the interim, the quickest and simplest solution to this problem would be to more clearly lay out the expectations of what should and should not be included in a review of a manuscript (Objective criticism good; subjective criticism bad). The job of the editor should be to ensure the review meets the qualifications of an objective review and identify comments which come from either positive or negative bias. If the review fails the objectivity test the remedy could take several forms, e.g. if the issue is minor the review could be sent back for edits. In more serious cases the review could be itself rejected and another review commissioned in its place. In this manner, reviewers that are serially unable to offer objective reviews would be separated from those that do and could be avoided in the future. For the authors of a manuscript, these guidelines would give them a mechanism for launching an appeal should their manuscript be rejected because of their gender, age, or pedigree.

  9. Morgan Price
    Posted May 4, 2015 at 7:51 am | Permalink

    I like the idea of clear guidelines for reviews. And if you encourage authors and editors to request that another editor read the decision letter if they feel that the guidelines might have been violated, then I don’t think that screening them all would be necessary.

  10. Comradde PhysioProff
    Posted May 4, 2015 at 7:58 am | Permalink

    This is horrible for all the reasons you explained, and is egregious blatant gender bias, but it isn’t sexual harassment. And you do a disservice to the fight against both sexual harassment and gender bias to unnecessarily conflate these terms.

    • Posted May 4, 2015 at 9:19 am | Permalink

      Sorry. I didn’t mean to misuse the term. I just wanted to convey that this is actually worse than just bias because the review contains actual harassing behavior. I don’t think bias adequately captures it – telling people to get male co-authors is harassment. I’ve updated this to call it “gender-based harassment” which I think is the appropriate term. Do you agree?

      • Posted May 4, 2015 at 9:30 am | Permalink

        The review stated: “It would probably also be beneficial to find one or two male biologists to work with (or at least obtain internal peer review from, but better yet as active co-authors)…”

        To suggest that X “would probably be beneficial” is not telling people to do X. I am not endorsing the review, but, for the purposes of discussing the review, I would like the review described accurately.

        Your original post states: “This happened in a professional setting and clearly targeted the gender of the authors in a way that was not only inappropriate, but which would have had a negative effect on their careers by denying them publication and appropriate credit for their work.”

        Is this a claim that the reviewer rejected the manuscript because of the sex of the authors? If so, what is the basis for the belief that the reviewer would have recommended accepting the manuscript had it been written by men?

      • Posted May 4, 2015 at 1:57 pm | Permalink

        That’s somewhat better, but it’s not helpful to call this “harassment” at all. It’s egregious gender bias, but harassment has a specific legal meaning that really isn’t implicated here, and it is, at best, confusing to use it. What is wrong with calling this a really egregious instance of gender bias?

        • Posted May 4, 2015 at 2:34 pm | Permalink

          I’m not sufficiently expert in the legal issues to know whether this meets a technical definition of harassment, but I think it’s important to distinguish bias from harassment, which to me, in one of its forms, means that you do or say things whose purpose is to damage the career or reputation of the recipients of your deeds/words, which I think this case qualifies as.

          • Comradde PhysioProff
            Posted May 4, 2015 at 3:01 pm | Permalink

            There is more to harassment than that. And as far as this particular instance, we don’t know *anything* about this reviewer’s purpose. Anyway, this is semantics, but you are using the term “harassment” in an idiosyncratic fashion for no good reason. I get that you want to emphasize how bad this incident is, but terrible egregious bias captures it just fine.

          • Posted May 4, 2015 at 3:14 pm | Permalink

            I wasn’t trying to speculate about the reviewer’s intentions, I just don’t think the reviewers intentions matter one way or another. But anyway, I’m sorry for using the term idiosyncratically.

  11. Posted May 4, 2015 at 8:28 am | Permalink

    Hi Michael,

    It’s commendable to see you addressing this in public and seeking advice on ways to prevent sexism on PLOS ONE in the future. My colleagues, Dr Buddhini Samarasinghe and Professor Rajini Rao, and I have covered the issue of sexism in academic publishing on STEM Women. While the context was different, the broad suggestions we made have relevance here. We suggested that publishers needed to set clear gender equity and diversity guidelines that are communicated to editors, authors and their readers. We see that better training on sexism should be provided. We also argue there needs to be a system of accountability, which you have also alluded to.

    Academic and research institutions need to implement safeguards and provide support to prevent sexism and harassment. All too often in STEM, we find ourselves being reactionary to public crises and media outcry. It’s time for institutions to become proactive.

    There are career repercussions for sexual harassment and abuse in other aspects of academic life; publishing should be no different. There needs to be policies in place that define behaviour that is unacceptable, and a plan to address these issues swiftly. Transparency is one central way to achieve gender equity. Greater collaboration between publishers and academic institutions would help to better regulate and train researchers, reviewers, editors and others.

    Here are some general suggestions:
    * Publishers should be clear about their gender equity policies, including what they are doing to improve women and under-represented minorities’ participation and representation in scientific publishing
    * Set out the complaints procedure and consequences for sexism (and other forms of discrimination and exclusion). Optimise this process so that sexism, racism, homophobia, transphobia, ableism and other forms of abuse and discrimination are handled swiftly. The aim should be to reduce damage to the women and minorities who launch a complaint, and to reach an equitable resolution for these under-represented authors
    * Provide mandatory training materials to editors. Training materials can be hosted online and you might even consider setting up a short online training “course” (1 hour for example) covering basic principles, such as awareness about unconscious bias as well as PLOS ONE’s policies and commitment to gender bias
    * You’re right in noting that it’s already very tough to get reviewers given these duties are unpaid, but reviewers should nonetheless be directed to gender policy guidelines about gender inclusion and the consequences for abuse, harassment or bias. Ultimately, however, editors should be acting as gatekeepers for abuse of power and discrimination, and they should be advocates for gender diversity
    * Those who manage the complaints should also be given mandatory training on gender and diversity

    You are wise to presume this is unlikely to be an isolated incident. If sexist abuse escaped one PLOS editor it is safe to assume there may be other examples, as you pointed out. Even if most reviewers and editors are professional and fair, incidents such as this have broader implications, given the so-called “leaky pipeline” in STEM. For the benefit of other readers: this term describes how women are already disadvantaged at every step of their education and careers, and institutional barriers and everyday experiences at every stage increase the likelihood that women will drop out of academia. As a mid-to-long term plan, you might consider addressing gender bias as an ongoing commitment. This may include hiring independent evaluators to assess how PLOS’s current structure and review process might be improved to increase gender diversity. There are many useful models being used around the world that have practical outcomes.

    This particular incident may present an opportunity for PLOS to move towards becoming a publishing leader on gender diversity. Why not leverage off your publishing networks to see how PLOS and other science publishers might collaborate on gender equity and inclusion? Addressing industry standards is one way. Collaborating in partnership with other institutions in higher education, medical research and so on would also go a long way to pre-empting future problems that alienate women in STEM from submitting their papers to science publishers.

    Happy to talk more on this Michael. We’re linked on Twitter if you want to reach out.

    Links to STEM Women’s work on reducing gender bias in science publishing: http://www.stemwomen.net/recognising-sexism/ and a video discussion https://www.youtube.com/watch?v=8cYdUQY66Po

    • Posted May 4, 2015 at 9:09 am | Permalink

      Hi Zuleyka,

      “This may include hiring independent evaluators to assess how PLOS’s current structure and review process might be improved to increase gender diversity.”

      At a certain level of abstraction, the reviewer in this episode suggested that a manuscript on gender bias might benefit from gender diversity in interpreting its data: a charitable interpretation of the review is that the manuscript authors were all members of one sex, so the reviewer suggested that a perspective from a member of the other sex might help, given that the manuscript was about gender bias.

      Maybe the reviewer would not have been consistent in suggesting gender diversity, but I am not aware of any evidence of that counterfactual and, in any event, that inconsistency would not change the status of the review itself.

      I’m trying to reconcile the beliefs that (1) gender diversity is a good thing, and (2) it is sexist to assume that men and women would have different perspectives. If we can’t assume that men and women have different perspectives, then why is gender diversity important?

      WT mentioned my original hypothetical above. I would be interested in responses from anyone to that hypothetical or to this hypothetical:

      PLOS ONE decides to conduct a review to check for sexism in its peer review process and to make changes to help prevent sexism from occurring in its peer review process. The managing editor randomly selects three editors for this review team, but all three editors happen to be men. Is it sexist — or sexual harassment — to suggest to the managing editor that the review team include a female editor to guard against interpretations that might disadvantage female researchers?

      • Posted May 7, 2015 at 7:36 am | Permalink

        Hi L.J.,
        Gender diversity is a term that has a specific meaning in gender studies – it comes out of intersectional feminist writing that demonstrates how cis-gender men, especially White men, are given special privileges by society and that the views, experiences and interests of women and minorities should be better represented. The concept of gender diversity is intended to show how cis-gender White women, minority people of various backgrounds, transgender people (especially transgender women of colour), and Indigenous people are discriminated against through institutions and everyday practices. Gender diversity studies provide frameworks for unpacking the ways in which this happens and how to make society more inclusive of different genders.

        It is sexist to ask that a study about gender differences written by two women needs a male author – this implies that women cannot be objective about gender. This is because men, as the default category in sexist thinking, are considered the universal human experience. In STEM, men’s gender is never questioned, but women’s gender is constantly under scrutiny, and used to undermine women’s advancement.

        Men are more likely to author papers as a single author; their scientific papers are given greater prestige; men are over-represented in meetings, panel chairs and as keynote speakers; men are hired, promoted and rewarded in STEM careers more than women. In other words, men’s gender goes unexamined yet it provides various benefits and protections that women and gender minorities do not have access. All of this scientific evidence is covered in a plethora of scientific studies. I discuss this in my research and in my other writing. Start with this: why men are more likely to disbelieve that gender bias exists in STEM: http://othersociologist.com/2015/01/19/sociology-gender-bias-science/

        • Posted May 8, 2015 at 7:42 am | Permalink

          Hi Zuleyka,

          Thank you for your response. It seems that I have taken the term “gender diversity” too literally, so thank you for the correction.

          I appreciate the idea of intersectionality, but I think that it makes sense to discuss when to stop the intersecting. Straight-white-cisgender-male is a large heterogeneous category that contains persons with a lot of privilege and persons with little privilege. I’m not sure that we need to put a thumb on the scale for, say, the daughter of a billionaire because of the privileges extended to the son of a poor coal miner. It’s possible that, within STEM, the coal miner’s son would have more advantages than the billionaire’s daughter, but I would not bet on that possibility, especially if we extend our considerations to aggregate privileges in life.

          If there is a need to discuss the privileges that one group receives, then let’s discuss the instances in which members of other groups receive privileges and the instances in which the first group is disadvantaged. To not have this heterogeneity in privilege acknowledged is itself a form of privilege.

          As for the peer review:

          You write: “It is sexist to ask that a study about gender differences written by two women needs a male author – this implies that women cannot be objective about gender. This is because men, as the default category in sexist thinking, are considered the universal human experience.”

          The conclusion in the first statement in that passage — “it is sexist” — seems to rely on the premise that men are “the default category in sexist thinking.” Thus, it appears that the conclusion (the peer review is sexist) is assumed by the premise (the reviewer was using sexist thinking, in which men are the default category). Sure, if we assume that the reviewer was using sexist thinking, it makes sense to conclude that the review thought in a sexist way.

          I have no problem with the interpretation of the review as sexist, but I think that it relies on at least one assumption. Sure, maybe it is correct that the reviewer thought that women *in general* could not be objective about gender; but it might also be that the reviewer thought only that *these particular reviewers* were not objective about their topic. I’m not arguing that this is definitely the case, but it is at least a possibility, especially given the large number of caveats the reviewer placed in the passage about sex differences (e.g., “perhaps”, “on average”, “a bit faster”, “As unappealing as this may be to consider”, “marginally better”).

          Thanks for the links to your site. I have already read a bit there, but I plan to read further.

        • Posted May 8, 2015 at 7:48 am | Permalink

          Ugh. Should be *these particular researchers*.

  12. Posted May 4, 2015 at 9:07 am | Permalink

    “In this world I think the things discussed above – very clear codes of conduct for reviewers, and proactive policing of reviews – become even more important.”

    I totally agree. You might be interested to know that COPE has a relevant resource – the COPE Ethical Guidelines for Peer Reviewers.
    pdf here http://publicationethics.org/files/u7140/Peer%20review%20guidelines.pdf
    or access via the Resources page http://publicationethics.org/resources/guidelines

    We produced these when I was on COPE Council, partly in response to the increasing number of problem cases involving peer review and peer reviewers we were seeing, and partly to help early career researchers in particular gain awareness of the ethical obligations of peer reviewers at every stage of the peer-review process. Very few researchers get any formal training in peer review.

    The guidelines say peer reviewers should ‘not allow their reviews to be influenced by the origins of a manuscript, by the nationality, religious or political beliefs, gender or other characteristics of the authors, or by commercial considerations’. Adding something that more specifically addresses the issues raised by this case would be a good addition to the next version.

  13. Posted May 4, 2015 at 10:09 am | Permalink

    “There could be a second layers of more experienced editors or staffers who review every decision letter for its adherence to PLOS’s editorial standards and code of conduct before it goes out”

    Lots of smaller journals do this already, in that all decisions are approved by either the Editor in Chief or a Managing Editor. It’s their job to push for consistency in editorial decisions across the journal as a whole, and to catch and correct egregious reviews or editorial decisions.

    We occasionally come across reviews where the language has veered from objective criticism of the paper into ad hominem attack, and in these cases we rewrite the review in more moderate language and ask the reviewer to approve the new version. They typically recognize that what they wrote was not appropriate and hopefully take that on board for the future.

  14. Posted May 4, 2015 at 10:19 am | Permalink

    “There could be a second layers of more experienced editors or staffers who review every decision letter for its adherence to PLOS’s editorial standards and code of conduct before it goes out”

    Lots of smaller journals do this already, in that all decisions are approved by either the Editor in Chief or a Managing Editor. It’s their job to push for consistency in editorial decisions across the journal as a whole, and to catch and correct egregious reviews or editorial decisions.

    We occasionally come across reviews where the language has veered from objective criticism of the paper into ad hominem attack, and in these cases we rewrite the review in more moderate language and ask the reviewer to approve the new version. They typically recognize that what they wrote was not appropriate and hopefully take that on board for the future.

    You’re absolutely right that an additional layer of oversight adds to the cost of peer review. However, the main financial burden (your $6500 figure) stems from employing editorial office staff to handle the review process of papers that then get rejected – this contributes about (1/acceptance rate-1)*$300 of unnecessary expense for each paper that gets accepted. I wrote a blog post about this here: http://www.molecularecologist.com/2015/02/why-is-science-publishing-so-damn-expensive/

  15. Posted May 4, 2015 at 10:27 am | Permalink

    “There could be a second layers of more experienced editors or staffers who review every decision letter for its adherence to PLOS’s editorial standards and code of conduct before it goes out”

    Lots of smaller journals do this already, in that all decisions are approved by either the Editor in Chief or a Managing Editor. It’s their job to push for consistency in editorial decisions across the journal as a whole, and to catch egregious reviews or editorial decisions.

    We occasionally come across reviews where the language has veered from objective criticism of the paper into ad hominem attack, and in these cases we rewrite the review in more moderate language and ask the reviewer to approve the new version. They typically recognize that what they wrote was not appropriate and hopefully take that on board for the future.

    You’re absolutely right that an additional layer of oversight adds to the cost of peer review. However, the main financial burden (your $6500 figure) stems from employing editorial office staff to handle the review process of papers that then get rejected – this contributes about (1/acceptance rate-1)*$300 of unnecessary expense for each paper that gets accepted. I wrote a blog post about this here: http://www.molecularecologist.com/2015/02/why-is-science-publishing-so-damn-expensive/

    [my solution to this expense crisis is independent peer review, which acts as a broker between authors and journals and thus raises acceptance rates, but I appreciate that you’ve got a quite different vision]

  16. Bob O'H
    Posted May 4, 2015 at 1:33 pm | Permalink

    My thoughts, for what they’re worth:

    1. How to respond to the reviewer
    You ask “But seriously, what could be more important for a publisher to do than protect their authors from harassment?”, but I think you’ve already answered that: don’t send the review to the authors. What you’re asking is, I think, “should a publisher protect other people from this harasser?”. I think this is a much more difficult problem, because you’re stepping outside the remit of the publisher, in a context where your remit gives you obligations to the harasser.

    Overall, I think you have to respect the anonymous side of peer review if that’s what you’re running. This is partly because the alternative is to have specific rules for when you’ll break anonymity, and if someone wants to be an arsehole they’ll be able to do it without breaking the rules. So I’m not sure you gain as much as you’d like (except perhaps a whole set of byzantine rules about how to behave). The alternative is to have vaguer guidelines, but that’s going to create other problems of inconsistency (whether real or perceived).

    I agree with Tim Vines that contacting the reviewer is an important step, and may help. Some of the comments that we’ve seen may ave just been naivety on behalf of the reviewer (others, though definitely were not), so it might be enough to point out the errors.

    Ultimately this is a balancing act, and to some extent I’m playing devil’s advocate. Partly I’m worried that you’re trying to solve a very rare but visible problem, and by doing so you may cause problems that will be more common but less obvious.

    The good news is that whatever you decide to do, someone will think it’s wrong.

    2. Should the editorial system be changed?
    It sounds like PLOS ONE might have a problem, with (and this is an awful straw-man: it’s not accurate bit it’ll make the point) inexperienced AEs who do not have a lot of training or personal support. Training and having a system of senior editors would help, but I agree that the cost issue is important, and you’re so big that some automation could pay dividends (plus you’re meant to be pushing the bounds!).

    Most decisions will be fine, so you don’t really want to see them. You really want to filter the reviews so that you only see the ones that need attention. One approach might thus be to have a semi-automated system of editorial review. The computer can flag decisions that need to be reviewed (e.g. if there is only one review, or if the reviewers give divergent recommendations, or if the AE is either new in the role or asks for help). This would need a bit of thought, particularly to do it in a way that isn’t seen as threatening to the AEs, and which also catches most of the submissions that are problematic.

  17. FS
    Posted May 5, 2015 at 4:30 am | Permalink

    would double-blind review not be a far neater solution than the various tortuous and expensive solutions suggested otherwise?

    • Bob O'H
      Posted May 7, 2015 at 10:26 am | Permalink

      In my opinion, no. Partly because it doesn’t work (in one study 43% of reviewers worked out who the authors were), and also because I don’t see how it would solve problems at the AE level. We’ll still get poor reviews, even with double-blinding, so solutions have to lie elsewhere.

  18. Daniel Falush
    Posted May 5, 2015 at 4:39 am | Permalink

    Yes, please do not call this sexual harassment or for that matter harassment. Its really not helpful to this discussion. Its my experience and I suspect many others that egregious reviews (whether offensive, ridiculous or both) are ultimately less problematic to receive as an author than the cunning ones which communicate an equivalent message more politely. Particularly Science and Nature often seem to reject papers based on absurd reviews, sometimes fully expecting you to dispute the decision. Still, this example is in a different category from anything I have received..

  19. Cynthia Wolberger
    Posted May 5, 2015 at 6:50 am | Permalink

    While this review is a particularly egregious example of sexism and bias, I would not classify it as harassment, which would involve intimidation, pressure or creation of a hostile environment. As troubling as it is that a scientist (I presume it was a scientist) would write such a mind-boggling review, the more troubling aspect is that the review was passed along to the authors without comment. I can’t decide which possibility is worse – that an editor read the review and thought it was acceptable, or that the editor simply didn’t bother to look at it and just sent it along. The fact that the author had to tweet about it to get PLOS One to act on her complaint is further evidence of an editorial process there needs urgent attention.

    I do not think that reviewer training and longer reviewer guidelines are the answer – I doubt even current guidelines are read very carefully and including sensitivity training won’t get them any more attention. I also do not favor signed reviews, as I can think of too many situations in which a reviewer would not want to put their name to a well-reasoned critical review of another PI who has the power to harm the reviewer’s career in some way.

    A better way to stop a review like this from being disseminated or even written in the first place is to have reviewers share and comment on one another’s reviews prior to sending them out to the authors. In this system, which is used by Elife, the reviewers’ identities are known to one another. This process puts brakes on all types of negative reviewer behavior in two ways: the reviewer knows that his/her peers will see and comment on their review, and it provides the other reviewers an opportunity to weigh in on one another’s statements. Even if the reviewer in question felt that his comments were legitimate, the chance that one or two other reviewers would agree is (I would think) quite low. Another benefit to the commenting process is that it forces the editor to read all the reviews and moderate the discussion (which is done on-line). While this system does not guarantee an unbiased process, it reduces the chance that bias or just plain erroneous comments will be allowed to stand and get passed along to the author.

    A more insidious source of bias is the “confidential comments to the editor” section that most journals allow. This is where authors can share biases cloaked as concerned comments, without the author ever finding out about them. I don’t know what proportion of papers are rejected based on these comments, but I do know of some egregious examples where reviewers have taken advantage of this back door to share negative comments about a lab and the perceived reliability of their work. This door needs to be closed and the reasons for paper rejection made more transparent.

    • Posted May 5, 2015 at 7:18 am | Permalink

      I agree with all those suggestions. The eLife model generally works well, but it has a negative side – which is that it is slow and I’m not sure it would work at scale.

      • Cynthia Wolberger
        Posted May 5, 2015 at 1:43 pm | Permalink

        Agreed, the Elife model is more work. But even a scaled-down version in which reviewers get to see the other reviews, with names, and have a chance to revise before sending the reviews to authors would help. It would make people think twice about being unprofessional or sloppy and would allow the other reviewers to alert the editor to problems. Not sure already overloaded editors could handle this, but I think it would improve the review process.

  20. Posted May 5, 2015 at 7:27 am | Permalink

    I’d like to take exception to several peoples’ objection to the use of the term “harassment” here. I understand that one of the reasons is that people want to reserve harassment as a legal term, so I’ve been reading up on the legal definition. There isn’t any single one, but I think this captures the essence: “the act of systematic and/or continued unwanted and annoying actions of one party or a group, including threats and demands”. And, while it might be hard to make the label of harassment stick to any particular egregiously sexist review, my point was that this is not an isolated incident, and that there is – whether coordinated or not – an ongoing and consistent effort on the part of many reviewers to use the peer review process to make the act of publishing more unpleasant and difficult for women, and that the net effect of these, and related acts outside of publishing, has a negative impact on women’s careers and induces many to leave the field. So I think the term harassment does apply here and we should use it.

    • Posted May 5, 2015 at 7:59 am | Permalink

      I think that the statements made in the review slamming the abilities woman scientists in particular, knowing this review would go to the female authors, certainly crosses the line to harassment.

      • Posted May 5, 2015 at 12:04 pm | Permalink

        Hi Sean,

        Which part of the review are you characterizing as “slamming the abilities [of] woman scientists in particular” in a way that constitutes harassment?

        The public parts of the review excerpted in the post above raised the *possibility* of sex differences in terms such as “as unappealing as this may be to consider” and “might be”; the review used the term “probably” when characterizing “on average” sex differences in running speed, used the word “perhaps” *twice* when discussing the possibility that men might work longer hours “on average” than women, and raised the possibility that sex differences in first authorship in better journals might be due to “bias at the journal.”

    • Posted May 5, 2015 at 8:57 am | Permalink

      I completely agree with this (though I also agree with the change from ‘sexual harassment’ to ‘gender-based harassment’ earlier–the former I think has different connotations). In my workplace, the key words are “hostile work environment”–if somebody brings those words to HR or management it gets taken very seriously, regardless of whether the source of the hostility meets some legal definition of “harassment”. It still means that somebody is being made to feel uncomfortable in their work environment, and the cause of that needs to be addressed, even if it’s not intentional on the part of the source of hostility. That’s fine for one workplace, but the scientific community in general does not have a single HR department to go to, say, and this kind of hostility does wear people down–especially women and minorities–emotionally and professionally, and it’s persistent. So I think it definitely constitutes harassment, and handwringing over that or turning to legal definitions is unhelpful.

    • Posted May 5, 2015 at 9:01 am | Permalink

      It seems to me that the discussion of whether to apply particular labels to the review is a method by which to shortcut thought about the review. Sexual harassment is bad, and this review is sexual harassment, therefore this review is bad. QED.

      If it is sexual harassment to suggest that members of one sex could benefit from the perspective of members of the other sex in understanding and interpreting data on gender bias, then please say so. If it is not correct to generalize the review in that way because the reviewer would not have made the same review for the same manuscript written by an all-male team, then let’s discuss the evidence for that counterfactual. If there’s another, better way to think about the review, then let’s discuss that.

    • Daniel Falush
      Posted May 5, 2015 at 9:54 am | Permalink

      I think the terminological issue is by saying it is harassment you are implying specifically that the email is part of a coordinated set of actions, e.g. ” a course of action”, in English law. If English law is not good enough for you, look at wikipedia, which says it is characteristically repetitive. I think the point is that actions are quite correctly considered worse – all else being equal – in terms of the malicious intent behind them and the overall effect they can have if they are coordinated rather than independent and that is why coordinated malicious actions need a specific label, in law and in everyday life and that label is harassment. I know much more than I want to about this, alas.

      • Posted May 5, 2015 at 11:13 am | Permalink

        I’m not a lawyer and don’t pretend to be interpreting this in a strictly legal sense. But the definition on Wikipedia does seem instructive:

        It is commonly understood as behavior intended to disturb or upset, and it is characteristically repetitive. In the legal sense, it is intentional behavior which is found threatening or disturbing.

        We’re stuck, as we often are in such cases, with the issue of intent. Here we don’t know if the reviewer intended to be disturbing, or upsetting, but we certainly know that it was found to be threatening or disturbing. Also, I think it’s fair to say this is repetitive behavior – even if this particular reviewer hasn’t engaged in this kind of review before, I don’t think there is ample evidence that this is a repetitive behavior within peer review writ large.

        • Daniel Falush
          Posted May 5, 2015 at 12:32 pm | Permalink

          My experience is that the legal system is as blind as everyone else – if not more so – in deciding on questions of intent and like everyone else, they in practice rely on rules of thumb. We would certainly be in a better situation to decide on the reviewer’s intent if we did have another example of his to go on and that is one reason why laws on this topic typically require repeated action. Another is the magnification of effects. So the way I would like to phrase your position is that you are saying society as a whole – in which sexist bullying is frequent – provides the context of repetitive behaviour (spelled in the English way ;)) within which to decide on the motivation underlying and effects of this review. My view is that this kind of reasoning does have its place in some kinds of decisions but if e.g. harassment regulations were drafted based on this it would be BAD.

5 Trackbacks