The widely held notion that high-impact publications determine who gets academic jobs, grants and tenure is wrong. Stop using it as an excuse.

In response to my previous post on boycotting non-OA journals, my friend Gavin Sherlock made the following comment:

I laud what you are doing, and you have changed the world of publishing forever for the better. However, I was specifically told by my chair that I need a Nature or Science paper to make my tenure packet bulletproof, so you shouldn’t underestimate the tenure argument.

This comment pretty much sums up why closed access publishing still dominates. Like most scientists, Gavin agrees that the system we have is bad, and that progress towards open access is a good thing. But, in the face of advice that he needs a glamour mag paper for his tenure package his pre-tenure facebook feed was filled with “Just submitted a paper to Nature” and “Just submitted a paper to Science“.

I am not here to criticize Gavin. These few indiscretions not withstanding, he has a long and exemplary history of open access publication. Rather I use him as an example of just how powerful and toxic the glamour mag culture in science has become. If it can get to him, it can get to anyone.

I am also not here to dwell on how crude a measure of impact the impact factor is [1], or how the tyranny of the impact factor is destroying science. Peter Lawrence (see also this list put together by Sean Eddy) has written extensively and eloquently on the subject.

Rather I want to challenge the key assumption – made by nearly everyone – that choosing not to publish your work in the highest impact factor journal you can convince to accept it is tantamount to career suicide. It is ubiquitously repeated by everyone from the most successful senior scientists to first year graduate students. And, judging by their publishing practices, most of them must believe it to be true. But I don’t think it is.

Before I explain, I should note that my comments will deal exclusively with science in the United States. We have, mercifully, not followed the incredibly misguided policies used in many European and Asian countries which use formula that explicitly include impact factor to allocate jobs and money. The underlying attitude may be as strong here, but at least it is not hard-coded.

I can not deny that there is a very strong correlation between the impact factor of the journals in which someone has published and their success in landing jobs, grants and tenure. The evidence is all around me: 11 of the 15 assistant professors in my department at Berkeley had published at least one Science, Nature or Cell paper as a graduate student or postdoctoral fellow (we’ll return to the other four later). And more systematic studies have found a similar correlation [2].

But, as we know, correlation does not imply causation. Even if hiring, grant review and tenure committees completely ignored journal titles and focused exclusively on the quality of the science (as they should), we would still expect there to be a strong correlation between success and impact factor. Scientists are so conditioned to believe that impact factors matter that most design their experiments, write their papers and jostle with editors to get their work into the “best” journal possible. Since the peer reviewers who ultimately make (or at least strongly influence) the publishing decisions are drawn from the same pool of scientists who make hiring, funding and tenure decisions, it is no surprise that the same work is valued in all of these venues. Thus, the idea that impact factors are paramount would be a self-fulfilling prophesy even if it were completely untrue!

Of course it is not completely untrue. I have seen too many colleagues lazily use the presence or absence of SNC publications as the primary factor in screening job applicants, as a reason to or not to fund a grant application, and as a proxy for whether someone should or should not be tenured. It is also undoubtedly true that, all other things being equal, high impact publications can make a difference. However, glamour mag publications are neither necessary (see the 4 assistant professors discussed above), nor sufficient (we routinely pass on candidates with SNC publications).

Encouraging the people we train to focus so exclusively on journal titles as the determinant of their success downplays the many other factors that play into these decisions: letters of recommendation, how effectively they communicate in person, and, most importantly, the inherent quality of their science. Sure, reviewers sometimes take shortcuts, but the quality of the underlying science and candidate matter a lot – and in most cases are paramount.

My own lab provides several examples that demonstrate this reality. My graduate students have gone on to great postdocs and many have landed prestigious fellowships “despite” having only published in open access journals. More curiously, I have had four postdoctoral fellows go out onto the academic job market, who  all got great jobs: at Wash U., Wisconsin, Idaho and Harvard Medical School. Not only did none of them have glamour mag publications from my lab. None of them had yet published the work on the basis of which they were hired! They got their interviews on the basis of my letters and their research statements, and got the jobs because they are great scientists who had done outstanding, as of yet unpublished, work. If anything demonstrates the fallacy of the glamour mag or bust mentality this is it.

So, when I suggest that we all refuse to publish in non-open access journals, I am not being cavalier about the career prospects of the next generation. I don’t suggest we abandon them to the winds of fate. Rather I believe we can simultaneous do right by science, by the public AND by our trainees by explaining to them what is at stake, pointing out the holes in the prevailing wisdom they hear from all sides, and then explaining and defending their actions to the hilt when we write letters on their behalf.

Scientific publishing is broken, and it’s dragging down the field. We can either sit by and do nothing, allowing another generation to be captured by the allure of high impact publications. Or we can show some courage, shake off this silly dogma, and lead the next generation to a place that will be better for all concerned. You know what I choose. Please join me.

UPDATE: I want to reemphasize my central point. Getting jobs, grants and tenure is a competitive process in which the quality of an individual scientists previous work and future plans are evaluated. Getting a paper published a competitive process in which the quality of a piece of work and its potential impact is evaluated. It is no surprise that the results of these two processes are correlated. But it is a logical fallacy of the highest degree to conclude from this correlation that it is the journal in which your work gets published, rather than its inherent merits, that plays the dominant role in determining your success in science.

In spite of this perfectly reasonable (and I believe correct) alternative hypothesis, the scientific establishment, and most scientists, continue to reinforce the idea that one must always grope for the highest impact factor journal. Given that this leads to so many negative consequences for science – encouraging glamour over rigor, slowing scientific progress by delaying publication while papers bounce from journal to journal, and massively inhibiting the much-needed transformation from subscription-based to open access publication – it is absolutely essential that we not only fail to act on its precepts, but that we challenge its underlying assumptions, highlight empirical evidence that counters it, and otherwise do whatever we can to eradicate this deeply cynical and highly destructive mentality from our field.

This entry was posted in open access, PLoS, publishing. Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.

43 Comments

  1. Posted February 4, 2012 at 8:59 pm | Permalink

    So you’re telling your trainees not to mess around with PLoS Biology or PLoS Genetics and their star-chamber pre-peer-review assessments of “impact” and submitting everything to PLoS ONE?

  2. Michael Eisen
    Posted February 4, 2012 at 9:07 pm | Permalink

    I have been for years. They’re mostly weaned of PLoS Biology. PLoS Genetics is still popular. The transition to PLoS ONE would be complete if post-publication (and along with publication) assessment were working well – which it will be soon.

  3. Posted February 4, 2012 at 11:44 pm | Permalink

    Great post Michael, and I want to believe it’s true. But you’ve got quite a name and reputation that will get your students and trainees far. What about for people who don’t come from such prestigious backgrounds?

  4. Posted February 5, 2012 at 2:06 am | Permalink

    Michael, I totally respect your crusade in favor of OA. You are doing great and important work for the community and I truly wish the argument you make here is correct. Unfortunately, I am afraid the reality is different. It is wonderful that your grad students have gone on to great fellowships and jobs without glam mag papers. But you said it yourself “too many colleagues lazily use the presence or absence of SNC publications… all other things being equal…”. So it is fair to assume that your students would have had an easier job landing that fellowship or job. Perhaps they would have landed an even better job. I suspect they would have an easier time getting their first grants, papers etc.

    The fact is it does make a difference because too many colleagues are convinced that SNC papers are a proxy for high quality. The argument that some get hired without such papers doesn’t mean that SNC papers do not increase the chances of early stage scientists to get their dream job.

    I encourage my students/postdocs to publish in PLoS One and OA journals. But how can I oppose them when they want to aim “higher” knowing that it will undoubtedly boost their career prospects?

  5. Joanna
    Posted February 5, 2012 at 5:06 am | Permalink

    Hi there, I’m a first year PhD student in the UK and just getting into the whole publishing thing (one paper out and 2 about to be submitted from my pre-PhD RA post). This post made me feel a lot better about all those other things I’ve been reading the last few weeks on this subject. However, I could do with some advice (having also just read your previous post http://www.michaeleisen.org/blog/?p=890). My PI has suggested sending one of the papers we’re finalising to an Elsevier journal. It’s not huge impact as far as I’m aware but having signed the boycott I’m now feeling like a hypocrite unless I say something to her. I don’t know how aware she is about the recent online debate. I have a great professional relationship with her but I’m not entirely sure how I should go about bringing the subject up. Should I just email her with my concerns and maybe add in a link regarding the debate? I feel like I want to contribute to changing things as the current situation is so absurd and frustratingly entrenched but I also feel a little lacking in control… Thoughts welcome!

  6. Posted February 5, 2012 at 6:22 am | Permalink

    More curiously, I have had four postdoctoral fellows go out onto the academic job market, who all got great jobs: at Wash U., Wisconsin, Idaho and Harvard Medical School. Not only did none of them have glamour mag publications from my lab. None of them had yet published the work on the basis of which they were hired! They got their interviews on the basis of my letters and their research statements, and got the jobs because they are great scientists who had done outstanding, as of yet unpublished, work.

    I’m an evolutionary biologist at UC Berkeley and an Investigator of the Howard Hughes Medical Institute.

    How many post-docs do you think are gonna get jobs like that under circumstances like that who don’t come from institutions like yours and–even more importantly–whose mentors aren’t Hughes investigators?

    When one of my post-docs has her paper triaged without peer review by the PLoS Biology star chamber, and then triaged without peer review by the PLoS Genetics star chamber, how can I in good conscience tell her to send it to PLoS ONE–which I know to be viewed as of right now as a “dump” journal by most people whose opinions matter–instead of to another relevant journal that is viewed as of right now as a “prestigious” journal by most people whose opinions matter?

    Don’t get me wrong. It is great that you and your post-docs are in the privileged position of being able to tell the glamour journals–including those that are open access–to fucke offe. If more people in that privileged position do the same, it will definitely move the needle in the direction we all agree is desirable.

    But not everyone is privileged like that, and cannot be expected to do the same. I was a post-doc in the just-started lab of an assistant professor with no real reputation and definitively not plugged into the Hughes network. If I didn’t publish the most important part of my post-doctoral work in Nature, I wouldn’t have ever got job interviews at the kind of institution where I am now employed.

    I know of people who got excellent assistant professor jobs at prestigious institutions contemporaneously with me on the basis of being plugged into the right networks–yes, including the Hughes network–who never got any substantial funding, who never published any research from their labs (presumably thus destroying the careers of any post-docs who were in their labs), who accordingly got shitcanned by their institutions, and who then–again on the basis of being plugged into the right networks–got *second* jobs at equally prestigious institutions to try again (we’ll see how many post-docs’ careers they destroy this time around).

    I don’t know if you know the blogger Driftglass, but he has a very apt saying that he applies to the Beltway political-media Village:

    There are two Rules for left-wing bloggers:

    (1) There is a Club.

    (2) You are not in it.

    There is a biomedical research Club. Different rules apply to those who are in it and those who are not.

    • NeuroProf
      Posted July 2, 2013 at 1:46 am | Permalink

      Very well said. I fully agree with PhysioProf. I am in the Neurology-Neuroscience field, and here, too, there are people in the “Club” and people not in it. Different rules apply to the two groups.

  7. Posted February 5, 2012 at 7:05 am | Permalink

    A very balanced and encouraging post, yet confirming that you do indeed lower your already small chances of getting a job by not publishing in GlamMagz. The way I read your post, your text contradicts the title. The text says: there are a few fortunate souls who actually manage to get a job without CNS papers, but for the rest it most likely won’t work that way. Of all graduate students, probably about 10% end up getting a job in academia, if that many. Out of those, there have been rumors that there have been candidates who worked 9-5 and never published in CNS and also got a job. This may be the case, but are you going to bet your rent, food and clothing on that you are going to be one of those fortunate few?

    For most, the job market is already stacked against them. I’m not going to tell my students to worsen their already ridiculously low odds even further by not publishing in CNS, even though I’d prefer to publish everything in P1.

    Honestly, I don’t get this boycott thing at all. Elsevier writes on their website that they have 970,000 volunteer authors/reviewers/board members.
    http://www.elsevier.com/wps/find/intro.cws_home/ataglance
    Unless you get up to a significant percentage of that, they’ll just do business as usual. However, if you get just a bunch of large enough libraries to cut subscriptions, you have immediate, hard-hitting, financial impact.

  8. Posted February 5, 2012 at 8:37 am | Permalink

    Disclaimer: I work in bioinformatics and genomics, was trained at a better institution than Berkeley :) , and have some rather high-falutin’ collaborators, so maybe this doesn’t apply to everyone else… but it’s my current perspective, as an asst professor coming up for tenure in under two years.

    I am putting a lot of effort into not only doing the best science I can (aren’t we all?) but also into making our tools and approaches open and reusable (open source, open development) and *publicizing* them. Our next paper submission will be posted to arXiv, just like our last; the source code and data will be available in a very public way, including on Amazon EC2 in runnable form, and on github as well; and the submission will be accompanied by a veritable blizzard of blog entries. Admittedly, I think our next paper is pretty super awesome and very broadly impactful, so I will be doing a full court press on it; but I expect to blog & publicize in this way for many or most of my papers, even the ones that are of more limited or directed interest.

    I will also probably be submitting it to PLoS One, because it offers the best change for getting the paper out quickly and thus realizing its potential impact more quickly.

    The basic philosophy behind my approach is this: I prefer to bet on having an *actual* impact on the field, and doing so quickly, rather than watching my work be delayed and rejected for impact “considerations”. What about peer review and the benefits thereof? Well, I’ve published over 25 papers so far, in three or four different fields, and very rarely has a peer reviewer actually found something *wrong* with a paper of ours; they contribute in other ways, but addressing things like comments and text updates can still be done with a PLoS One pub.

    For evaluation, promotion and tenure, and granting purposes, I’m now putting my Google Scholar profile link on my CV and adding specific citation numbers for each paper as well. Hopefully this will be useful in convincing evaluators that I am, in fact, useful. (Wish me luck :)

    Base line, the purpose of research is to advance science, *not* to give me a sinecure job. A job of some sort is necessary but by no means sufficient to advance science. On the other hand, if I *do* actually advance science, then *someone* is going to hire me, and that’s good enough for me.

    I think there are parallels with the educational situtuation — we recently had a discussion here at MSU about the future of undergraduate education. My take was that it’s increasingly looking like the US model for higher ed will implode in the next 10-15 years, because it is not clear that we are delivering educational value commensurate with the cost of the undergrad experience. We need to become more accountable in terms of producing trained & educated students, not just students who have a degree. I think the same applies to research: we need to argue for funding on our relevance to society, which involves accelerating the rate at which we actually make advances; glamor pubs seem to be at best loosely correlated with this.

    I think we should strive to go forth and *be* impactful, not just *look* impactful according to a rather arbitrary Rube Goldberg-esque model of publication. And that’s what the bigger discussion should be about. If someone can convince me and others that Nature, Science, etc. actually *add value* — by showing, for example, that higher impact publications actually have more of an impact, or are more likely to be correct, or yada yada — then I would be very interested in seeing that and might change my tune. Until then, it’s all hand waving and I feel like the Open Access model is far more obviously connected to the things we *should* be caring about.

    Just my 2 cents. And I might be out of a job in a few years ;) .

  9. Posted February 5, 2012 at 9:42 am | Permalink

    With some caveats, I have to admit to a similar sentiment to “Comrade PhysioProf.” While participating on faculty candidate searches (both as a graduate student representative at one institution, and as faculty at my current institution), I have not only seen the Science/Nature/Cell (and PLoS Biology) effect, but also the “lab of origin” and “institution of origin” effect. That is, the lab (and institution) where you did your PhD and Post-Doctoral work had a substantial effect on how some members of the committee perceived the candidate. While the quality and “substantive” nature of their scientific work was also important, both the journals they published in and where they did the work seemed to (at least sometimes) override other considerations.

    If these observations are in fact representative of hiring committees (and grant panels, promotion and tenure committees, etc…) then it does put people coming from Michael’s lab at an advantage (Under the assumption of a “high” reputation for UC Berkeley, and the Eisen lab). It would seem to me that such considerations are as problematic as the title of the journal in which scientific work is published in.

  10. fairscientist
    Posted February 5, 2012 at 10:28 am | Permalink

    another great post.. but i see this more as a wishful thinking than reflecting the reality…

    may be a repetition of what others have already mentioned above…how do you think a post-doc/junior faculty aspirant will be viewed who comes not from, a hhmi lab, a named professor’s lab who can write great reference letters and who has not published in snc even if he/she has a great research proposal? i can tell you that his/her chance is v v low in comparison to someone who has published in snc journals and/or come from a hhmi lab with a named investigator and/or who agrees to write a great reference letter on his/her behalf .

    the other day, our chair/head told me that no matter what, aim to get your papers published in snc…i want the reputation of my department to go up…..now, what should i do!!!! i can pack my bag, go and join your department (unfortunately, i am not going to get a job there anyway, so that chance is gone!!!), fight it out (the chance of success is zero), publish in snc (provided the manuscript passes the politics of publication in those journals) or rebel & publish in oa journals (and invite wrath of my chair). all the options are nasty…..pl suggest what should i do?

    thank you

  11. realitycheck
    Posted February 5, 2012 at 10:37 am | Permalink

    Having read the post and comments, I have to disagree with the title of your posting although I do read that you state you exclusively consider science in the US. However, I don’t believe policies are so greatly different across the Atlantic.
    I am at a European Institution where our contracts state that we have to publish data or review articles with impact factors higher than a certain level in order to gain tenure. Failure to do so within the tenure-track period will lead to non-appointment. This was a major factor (but not the only one I must add) in a colleague recently being asked to leave i.e. had their employment terminated.
    I can certainly believe that not every single scientist who receives tenure or a prestigious grant has a high impact paper of the journals you mention. However, in my experience in different institutions, it is one of the major deciding factors in gaining tenure and I have now seen it written in black-and-white.

  12. Posted February 5, 2012 at 10:39 am | Permalink

    Here’s the message I took home: You don’t have to have a SNC publication to get a TT-job or tenured. Okay, here’s my question: How many of those who did not have a SNC publication have Harvard, Yale, or MIT on their educational CV for graduate school or a postdoc or even undergrad? Because the other feeling I get is that unless you have that or the glamour mag publication or even both, you aren’t going to get a TT-position, let alone think about tenure.

  13. Posted February 5, 2012 at 10:51 am | Permalink

    The obvious way is not always the best way.

    I went to grad school at Oregon Health and Science University, not (say) Boston because I already lived in Portland and I loved living there. Also, it was an inexpensive place to live — a very real consideration when you’re about to spend 5 years on a graduate stipend. In my Department’s class of 6 students, 5 got TT or equivalent positions. Meanwhile, of several friends from college who went to Harvard/MIT for grad school, only one ended up with a faculty-equivalent position (he’s at NASA-JPL).

    While the journals you publish in and the labs you train in certainly matter, tenure committees DO look at the “whole package.” And HHMI or no, National Academy or no, the choice of postdoc advisor is key. Does S/he have a good reputation for rigor and creativity (vs. raw competitiveness)? Does the PI have a reputation for directly competing with and crushing former trainees, or is S/he known for giving recent fledglings the space (and reagents, and general support) to make their way in the world?

    In my postdoc lab, most people got TT jobs. Of the seven most recent, five had never had a S/N/C paper, and six did not have one during their postdocs. And I know, and have interviewed, a ton of people who had S/N/C papers and never got a TT position. Publications in glamour journals can help get a hiring committee’s attention, but they are neither necessary nor sufficient to land a good academic job.

    Now, anecdotes are not statistics. But it’s rather important to note that most of the things that “everyone knows” about hiring and advancement are *also* anecdotes and, again, not statistics. There are many ways to succeed, and many ways to fail. The crucial point — a point that one learns early and sometimes painfully in mountaineering — is that the “obvious” way is not always a safe or passable route.

  14. Posted February 5, 2012 at 12:28 pm | Permalink

    I’m being serially argumentative but I think you are totally missing the forest for the trees.

    I love OA, and use it when I can, but to single this one issue out as a moral test of character of your colleagues is utterly insane. There is so much else wrong with science too:

    -the patronage system of “it’s who you know” (which you actually cite as a plus, saying its a reason why SNC pubs are overrated/unnecessary, when in fact SNC could be a counterbalance to this massive systemic injustice)

    -the many flaws of peer review in publication & funding (which are surely compounded by narrowing the choice of publishers)

    -the bogus paper structure imposed by many journals: http://heathen-hub.com/showthread.php?t=1033
    (another factor that might influence choice of journal)

    Add to this the numerous reasons which might legitimately lead an author to think their paper will have more impact long-term in a journal with more readers (if access is the ONLY important factor, why not just stick everything on the web?), and I find myself increasingly uncomfortable with this (even part-satirical) demonization of people who use non-OA journals. I try to use OA when I can but I’m very uneasy with this type of ideological, black-and-white polemic. It’s a distraction, singling out this one issue. It sounds increasingly like a General Assembly at Occupy Oakland ;-) And let’s face it- you are not a disinterested party in this discussion.

    OA is great. RWA sucks. Let’s not go overboard and attempt to dictate the publication choices of our colleagues. It should be about making better options available – not removing options you think are suboptimal. I just don’t buy that current OA models are the final word, and until the final word arrives, shaming users of other publishers seems counterproductive.

  15. Posted February 5, 2012 at 12:35 pm | Permalink

    Just to be clear, I agree with you on all the substantive advantages of OA, but still find plenty of room for individual discretion over where to publish. I do think funding agencies have the right to require OA, but I don’t agree that individuals have a moral duty to go OA in the meantime. (I do agree that it is laudable when individuals do this, I just can’t accept that it’s reprehensible when they don’t. There are too many genuinely bad practices out there to get really lathered up about this)

  16. Mark Johnston
    Posted February 5, 2012 at 12:49 pm | Permalink

    I wish you were right, Mike, but, like all evangelists, you oversimplify (as the comments to your post amply illustrate).

    The other point that’s missed here, and one that I submit is at least as important as OA, is: who is it at the magazines that’s deciding our fate (and the course of Science)? As you know, I have stated my case on this at Genetics 181: 355–356.

  17. Michael Eisen
    Posted February 5, 2012 at 1:01 pm | Permalink

    Ian Holmes – I was specifically not uniquely tying this to OA.

    I made it clear I was not demonizing Gavin for his choices. And many OA journals are as guilty of encouraging the selective glamour game as anyone – my own PLoS Biology being a case in point.

    The point I was trying to make here is that I believe people are mistakenly being driven to believe that the most important thing they can do is to publish in the highest impact journal they can. And, as a result, they are making bad and unnecessary choices – of which shunning some OA is but one of many.

    I would also point out that this mentality – and the conservative it induces in peoples’ publishing behaviors – is the biggest reason there hasn’t been more effective reform in all the other areas you highlight. So this is not a single-issue crusade.

  18. Michael Eisen
    Posted February 5, 2012 at 1:03 pm | Permalink

    Mark Johnston – Obviously, the world of science is a complex place, and it’s not possible to capture every nuance in a single post. But don’t you think the prevailing mentality that impact is everything is a much much greater oversimplification?

  19. Posted February 5, 2012 at 1:26 pm | Permalink

    Comrade PP,
    Why shoot the messenger? Eisen may be fortunate enough to have the blessing of HHMI and UCB, but he is also trying to break up the ‘club’. As has been rightly pointed out, the worst part of the club is anonymous peer review, and Eisen is taking aim at that. The reality is complex and multi-faceted. Overturning the status quo is not any easier in science than in politics and finance – and look how well it’s going in the latter realms… We are in for a long struggle. Not for perfection, but for the greatest degree of fairness and transparency we can achieve. Decisions based on rational analysis, not perceptions of who has the ‘right’ connections, publishes in the ‘right’ journals.

    For years I worked in relative obscurity. (My record is public on google scholar [publications] and google plus [positions, etc.].) The only times I’ve published in Science and Nature were as a minor coauthor on genome papers with a cast of dozens. (Well, OK, I published a couple of commentaries in Science; but these were not primary research articles, which is the main topic here.) The one time I published in Cell was in 1980 when Cell was not yet a glam mag. I’ve never been part of the ‘club’s one of cool kids, never wanted to be, and often actively sought to avoid it. Being part of the club tends to mean thinking a certain way, which I felt was counterproductive. I was a postdoc for 5 years, in the biotech industry for 7 years then on ‘soft money’ in academia for 8 years before somewhat reluctantly accepting my first faculty position (U of Arizona). At the time, I needed a job… From 1990 till at least the late 90′s I was considered a somewhat strange, fringe player in plant genetics, and almost completely unknown outside plant biology. Most would now say I’ve been very successful. But I’m the same person now that I was 15 years ago.

    I don’t think one should measure one’s success by what most would say about you, or by whether you have a faculty position in a major research institution or by whether HHMI has anointed you. I prefer to measure success primarily by what I think I have and haven’t accomplished relative to my own standards and expectations, tempered by the opinions of those few scientists I know well and respect deeply.

    I truly do not believe it matters very much in the long run where one publishes. The most important thing is to publish. It is equally important to get the word out – if you don’t ‘sell’ yourself in today’s world, it’s not likely anyone else will. The approach I am embracing today is to try to weave together standard publication and social media – blogs, tweets, email, etc. I send my colleagues links to papers I publish (mostly in Frontiers lately). Now I tweet about them, and I tweet my blogs. I also tweet about others’ work. I do what I can in any situation to promote young scientists who seem to me to have potential. Helping each other is a good thing. You don’t have to be part of any club to do that or to benefit from it. But you do need to interact with others, in whatever ways you can and choose to.

    A final word: don’t whine, don’t get angry, get to work, be creative, do the best science you can, be an active participant, be open, try to help others as much as you help yourself. There are no answers, only possibilities, and creating possibilities is largely one’s own responsibility.

  20. Posted February 5, 2012 at 1:59 pm | Permalink

    Why shoot the messenger?

    Dude, no one is trying to shoot the messenger. As I have already said, it’s great for people with sufficient institutional and reputational status to tell the glamour mags to fucke offe. But it is wrong to vilify those in weaker positions to go along to get along.

  21. AB
    Posted February 5, 2012 at 2:59 pm | Permalink

    My general impression is that most commenters are saying: “OA is all great, but I don’t want to be the first one to do it, I’ll join when it’s going to be common practice and won’t hurt my personal chances of getting a job”.

    This is a valid point of view, but I don’t share it. I always try to do what I think is right, and if that’s going to drive away a few people who don’t understand my reasons, oh well, what are you going to do. I know that by doing great science, I will never be lost, I will always have a job (at least, some job) and I’ll be happy with myself.

    The other point I wanted to make is: yes, reputation in science is important (like everywhere else), so it does matter who you are, who you know, where you are from etc. Most of us started from 0. We didn’t know any big scientists when we went to college (maybe a tiny biology department at a medium size european university, like I did?), and then slowly we distinguished ourselves in class, we got good grades, we joined a good prof at our university for summer practice, the prof recommended us to a good PhD program, we got accepted (because of the grades and letters from prof), we did a great job there, went to conferences, met more and more people, built a reputation, got a post-doc offer etc. How else is it supposed to work? So I don’t see how the fact that I am now in a relatively famous lab in a big university means that I’m more privileged than anybody else. It didn’t just happen to me, I wasn’t just born this way…

  22. Posted February 5, 2012 at 3:05 pm | Permalink

    Mike- I guess I agree with that. The times I’ve let the science come first and the CV-points come second, have been the most rewarding & successful projects I’ve worked on. However, it’s a lot easier to say that now I’ve got job security. It is probably always a wise move to forget the resume & focus on the science, but it takes enormous courage for a young researcher to do that, and I think we should be heaping praise on the few that do it, rather than lamenting the judgment of the many who don’t. But in any case, I now see where you’re coming from & essentially agree.

  23. Mark Johnston
    Posted February 5, 2012 at 9:10 pm | Permalink

    Mike Eisen wrote:
    “Obviously, the world of science is a complex place, and it’s not possible to capture every nuance in a single post. But don’t you think the prevailing mentality that impact is everything is a much much greater oversimplification?”

    Indeed I do (though the people who hold that ‘impact is everything’ oversimplify for different reasons than you do)! Please don’t get me wrong: I’m on your side. Like you, I think we should shun the magazines, but for a different reason: the undue influence of the professional (i.e., non-peer) editors is destructive to Science.

    • atcgphd
      Posted February 11, 2014 at 11:20 am | Permalink

      I think we’ve come to a place where pretty much everyone agrees the Glam Mags are having a corrosive effect on the practice of academic science (whether they openly admit that opinion or not). But I don’t think it’s fair to point the finger at the editors of those journals. For one, it has a little bit of the smell of anti-alt-ac about it, at a time when 15% or less of those who are granted Ph.D.s will go on to a tt position. I mean, we made these people, I don’t think we should begrudge them the opportunity to feed themselves while still participating in a field of endeavor they likely spent 10+ years training for. For another, it abdicates responsibility – if it weren’t for the deans, P&T committees, and hiring committees who substitute their judgment for more rigorous evaluation of their own, it wouldn’t be such a problem. Let me repeat that. Deans, chairs, study sections, hiring committees, and P&T committees who rely on (or even give special consideration to) CNS publishing are at least an equal part of the problem as the “professional editor” class.

      I will also note that I have observed many professional editors are women; at least, the gender balance is much more equal amongst editors than it is amongst post-tenureed professors, who still tend to skew (rather heavily) male. Given that observation, take the criticism of the professional editor class (which I have heard widely, and not just here; Mark, I was there in Vancouver for Jasper’s performance) and make of it what you will.

      Finally, a bit of a thought experiment. Let’s imagine that we could wave a wand and eliminate Elsevier. Abolish Cell Press. No more Nature. No more journals with professional editors at all. What would happen? It seems to me that new “prestige” journals, with academic editors and reviewers, would simply take their place. e Life, PLoS Genetics, PLoS Biology. P&T decisions, funding decisions, would now be about whether or not you have an eLife paper. And maybe that would be better. Definitely, in some ways, it would be!! To have all publishing be OA wold be a huge win. But a lot of the inequality in science, such as the ability to *publish* in those new “top journals” would still persist. It’s not going to fundamentally change the game, just shift it slightly to one side.

      I would argue that even more than the CNS phenomenon, the fundamental problem in science right now is a large mismatch between supply and demand – the amount of grant funds available, relative to the number of mouths at the trough. When paylines are so low, and we’re all clawing each others’ eyes out to survive (in a maximally collegial way, of course!) some “halo” will emerge that can be used to justify the choice of hiring/funding one over another.

      I think if we really want to solve the problem we all agree exists, which is fundamentally the failure of meritocracy (due to our inability to distinguish the 5th percentile from the 15th?), we need to look a lot deeper than the existence of the professional editor.

  24. fairscientist
    Posted February 5, 2012 at 9:16 pm | Permalink

    can we all agree to…at least who are active/supporting oa publishing..

    1. make every student, post-doc, scientist and collaborator (before you take a student/postdoc and before one gets into a collaboration) understand that we have a firm policy in getting all results out in an oa journal (does not matter which oa journal, as long as the results are freely available)
    2. publish the manuscript in its entirety in arXiv.org the same time you submit the paper to a oa journal
    3. insist that the oa journal that you are sending your results to make the names of reviewers known (the journal can choose which reviewer get to review the paper but must let authors know their names). the anonymous reviewer is one of the most bogus practice by journals, oa ones included
    4. speak to his/her departmental chair/head to promote oa publishing (yes, this is the litmus test. having said, that i don’t have an answer how to cope with a situation when the chair develops animosity towards you)
    5 speak up publicly against the goons, the for-profit publishers, when you get a chance, be it in departmental committees, public lectures or scientific conferences

    i think what michael eisen is doing is noble. can we all help in our little way to achieve that!!!

    thank you

  25. Posted February 6, 2012 at 1:12 pm | Permalink

    I admire your post and what you’re trying to do in general, but I think your call to boycott all non-OA journals is doing a disservice to to the OA movement. It shifts the focus from “should I publish my next paper in an OA journal?” to “should I boycott SNC”. The former is crucial to OA, the latter (taken literally) is purely “academic” for most and inconsequential to the OA movement in practical terms.

    Example: Bjorn Brems writes in his comment “I’m not going to tell my students to worsen their already ridiculously low odds even further by not publishing in CNS, even though I’d prefer to publish everything in P1.” The real question is what made Bjorn publish his last paper listed on his publication list in Behavioral Processes (impact factor 1.5) and not PLoS ONE (impact factor 4.4)?

    Similarly: Ian Dworkin writes “I have not only seen the Science/Nature/Cell (and PLoS Biology) effect, but also the “lab of origin” and “institution of origin” effect.” That doesn’t sound implausible, but the real question is what made Ian publish his last paper listed on his publication list in Evolution (impact factor 5.7) and not PLoS ONE (impact factor 4.4)? Is this difference in prestige (as measured by IF) arguably significant for Ian’s career?

  26. Posted February 6, 2012 at 5:04 pm | Permalink

    I strongly support the goal of reducing the importance of impact factors (or H factors) in the assessment of career progression and success. A point not yet stressed is that these are very field dependent. In imaging, it is hard to get above 4 and journals like science and cell are out of the subject area. Also, aiming for high impact journals ensures a high rejection rate and, on average, increased publication times. With OA and journals that use pubmed, we should focus on getting the information out asap to increase efficiency and decrease costs.

    I don’t assess applicants to my lab based on impact factor (just a comment for the applicants). I encourage others in review committees to support the quality of the package.

    Good luck with this.
    Jeff

  27. Posted February 6, 2012 at 10:48 pm | Permalink

    t is absolutely essential that we not only fail to act on its precepts, but that we challenge its underlying assumptions, highlight empirical evidence that counters it, and otherwise do whatever we can to eradicate this deeply cynical and highly destructive mentality from our field.

    No argument from me there. I’m currently writing (with one-co-Author so far) a manuscript reviewing the peer-reviewed literature on ho well journal rank (as established by IF) predicts various other metrics of quality, such as citations, retractions or effect size estimations. It turns out, so far, there is some, but weak evidence that journal rank is predicting ‘good’ science, while there is stronger evidence that IF predicts ‘bad’ science.

  28. Posted February 6, 2012 at 10:57 pm | Permalink

    The real question is what made Bjorn publish his last paper listed on his publication list in Behavioral Processes (impact factor 1.5) and not PLoS ONE (impact factor 4.4)?

    This paper was an invited review in a special issue covering the plenary lectures of a conference :-)

    But as for any other papers: my rule until tis year has been that if the journal where the work would get published doesn’t raise an eyebrow, it goes to P1. That cut down the list of possible journals to 5ish or so: some GlamMagz and then P1. If everybody would do that, thousands of journals would feel a pinch very quickly.

    I’m optimistic that I’ll sign a contract this year that doesn’t necessitate it any more to pay attention to jornal rank in order to put food on the table. After that time-point, I’ll either negotiate the same rule with my co-authors or go for P1 right away.

  29. Posted February 7, 2012 at 12:07 am | Permalink

    “This paper was an invited review in a special issue covering the plenary lectures of a conference.” Yes, I’ve done that myself, and now I regret most of those decisions. These papers could have gone to better and more accessible journals, would have been published much faster, and no-one cares or notices anymore that they sit in a thematic issue.

    “But as for any other papers: my rule until tis year has been that if the journal where the work would get published doesn’t raise an eyebrow, it goes to P1. That cut down the list of possible journals to 5ish or so: some GlamMagz and then P1. If everybody would do that, thousands of journals would feel a pinch very quickly.”

    I couldn’t agree more. I think that’s the way to go for now: http://proteinsandwavefunctions.blogspot.com/2012/02/where-am-i-sending-my-next-paper-and.html

  30. Posted February 7, 2012 at 2:52 am | Permalink

    These papers could have gone to better and more accessible journals

    Every single one of my papers is accessible for free from my website both in PDF and HTML. Posters are all accessible in PDF. Some papers are also on Nature Precedings. This means all my papers are accessible in Google Scholar (not in PubMed, though).

    No argument on your other points, though. I’ll be giving more thought about whether or not to accept these invitations now, especially given that, e.g., Behavioral Processes is an Elsevier journal.

  31. Michael Brandeis
    Posted February 16, 2012 at 3:39 pm | Permalink

    Excellent post and remarkable work for promoting open access journals.
    Our institute has for many years mostly recruited SNC publishers with disastrous results. Not only has none of them EVER republished in a journal even close to SNC but they have done on average much worse than those that were recruited on the basis of less prestigious publications.

    The most disturbing point in using SNC publications as a major criterion is that a handful of professional editors are yielding an unprecedented power over the entire academia. These editors might do a professional decent job and were trained as scientists but they are not top scientists themselves. But even if they were all Nobel laureates it would be wrong to give so few the power over so many. True these manuscripts all undergo peer review but in the end (and even more in the beginning) it is the decision of the editor if something will get reviewed and accepted (most recently editors completely ignored my reviews and accepted manuscripts I strongly rejected).
    This brings us to the second most disturbing issue of these journals. Whether your manuscript will be accepted depends in many cases more then anything else on the status of your mentor. One side of this issue is that junior scientists have a much lower chance to publish a paper in SNC than established scientists when their papers are of equal quality. The worse aspect is that powerful mentors will get often manuscripts into SNC that are flawed and should not have been published at all.

  32. Gavin Sherlock
    Posted February 28, 2012 at 3:01 pm | Permalink

    One slightly off topic comment, but not totally irrelevant, is that I think journal editors should be blinded to the authors of papers they are considering to send out to review. To my mind, different authors are treated differently in that process – I see a lot of crap published by “high profile investigators” in the glamor mags, and you know that if you (as in me) were the senior author that it would have been rejected without review. Possibly reviewers should be blinded to the authors too, though I could probably work it out some non-trivial fraction of the time. The existence of the “club” is greatly facilitated and perpetuated by the fact that many decisions are based on who the senior author is, rather than the content of the manuscript itself.

    • atcg52003
      Posted February 11, 2014 at 8:51 am | Permalink

      Gavin, I agree with you. I know some people say blinding the authors is a waste of time, because eds/reviewers will “know anyway” based on the subject, and/or because authors will just find a way to identify themselves unambiguously in the text (presumably by heaping excessive praise on their last publication, or some such?).

      But as I understand it, our colleagues in the humanities have started blinding submissions to one of the premier journals in their field, and have found that it does have an impact on the profile of those who are accepted for publication – work from a broader segment of the academy is accepted, instead of the round-up of the usual institutional and endowed-chair suspects.

      I expect folks will criticize this suggestion by saying that english is not biochemistry, but IMHO to hang an argument against blinding on the fact that, yes, science & the humanities cover different subject matter – is missing the point. The experiment should be tried.

  33. Posted March 1, 2012 at 3:51 pm | Permalink

    You speak of the necessity of separating correlation from causation… then present anecdotes. Not even anecdotes with meaningful numbers. You have the complete set of numbers at your disposal for your lab; why are they missing from this post? “Four out of N” is not any more meaningful than “X out of N”.

    More importantly, your argument is that some number of people from your lab have gotten good jobs without high-impact publications. Yet you’ve already acknowledged that studies have shown a high correlation between high-impact pubs and success; and made the argument that this is correlation, not causation; and in fact that this correlation is inevitable. You then argue your point by providing anecdotes that do not show this correlation!

    You have already admitted the correlation. Providing anecdotes that do not share this correlation only proves your anecdotes are irrelevant to the larger trends that you are talking about.

  34. Posted March 7, 2012 at 2:10 am | Permalink

    it is good to see “market failures” dominating science as they dominate wall st and silicon valley. whenever there’s a monopoly of power, inferiority reigns. wall st bought washington for a decade or two (i.e., gave massive amounts to BOTH parties in every election cycle), and ended up blowing up our banking system. microsoft, once sufficiently large, effectively dictated that manufacturers, starting with recently emasculated IBM, ship hardware with only the world’s worst (and only non-multitasking) OS installed.
    as unpopular as markets are these days, these examples all share a common attribute: the system failed when there *ceased* to be a market because a small group of interests became powerful enough to dictate terms to all participants. great job taking on the journals! i gave up Windows for Linux years ago, i recently switch to a credit union from a bank, and after reading your post, i’m cancelling my subscription to Nature….

  35. Agatha C.
    Posted April 8, 2012 at 10:48 pm | Permalink

    What about private industry? I have contacts at Genentech who say the minimum qualifications for laboratory technicians or research assistants include a PhD from a top-level school and at least two SNC publications. For a lab tech job cleaning glassware!

    Also, when my biology club friends visited Stanford, the admissions officers told us that graduate admissions were so competitive that PhD applicants–not post-docs, not faculty candidates–need to have a SNC publication from their undergrad or internship research. However, my thesis advisor thought that sounded fishy because publication schedules and application timelines alone would make it difficult for an undergrad to have something published in time to submit in an application package.

    Does anyone here know anything about these situations?

  36. BugDoc
    Posted May 3, 2012 at 11:27 am | Permalink

    @AgathaC: the part about Stanford requiring a C/N/S paper for graduate admissions is not true. In our recent recruiting season, we lost one candidate to Stanford. Admittedly, this person was very good, but certainly no C/N/S paper, just a middle author paper in a solid but not high end journal.

  37. Posted September 18, 2012 at 12:18 am | Permalink

    Wonderful post. I think in principle everyone agrees with you, but the pressure
    /power of suggestion from senior scientists and powers-that-be with control over grant money cannot always be ignored. Hence, its difficult to foresee any loss of SNC power in future. However, a lot of departments (in the US, not so in Asia) are looking more at numbers of citations and H-index numbers. Can you comment on that too in the context of journal impact factors.

  38. Archie
    Posted January 28, 2013 at 12:17 am | Permalink

    Slightly off-topic- Here is a problem that applies to a lot of journals, especially the professional ones. With a limited pool of “editors”, they do not have the expertise to make decisions on a lot of manuscript that fall outside their comfort zone aka expertise. As a membrane protein crystallographer, I have had at least 3 papers turned down over the last 2 years because they landed at the desk of an RNA biologist who was unable to understand the importance of membrane proteins “in general” (the same editor at the same journal, 2 times). Maybe, things would not be that bad if, along with OA, a larger pool of scientists offered to work as editors, even on a art-time basis. With SNC, they tend to publish “hot-topics”, as defined by their own standards. The quality of the work is often not as important as the current degree of “hotness” and the flavor of the time when the paper is sent to SNC.

  39. farlan
    Posted August 9, 2013 at 8:05 pm | Permalink

    who cares about this crap outside of the world of science

    no one, that’s who

    what a waste of words and emotion

    we publish our work as droplets into a sea. the “impact” of each droplet is too complex and in fact impossible to characterise. therefore any conversation about it is BORING.

    Just get on with the ideas you think are worth putting some effort into. you can do no more than that…the end

    • Terry Pride, USAapdt
      Posted January 17, 2014 at 3:50 pm | Permalink

      Farlan,
      how delightful to have a guest come by & declare the entire article, & all the comments thereto, superfluous.

      I’m not a scientist & have not published; I am, however, a supporter of good science, & I agree that professional editors have too much power over peer-reviewed journals & the review / publication process.
      I also strongly support the immediacy & accessibility of open-access. The adamant hoarding of useful or potent information behind impenetrable walls, & the delay of publication so that it’s brought out in a GlamMag, slow science immeasurably.

      Science is a collaborative process. Information-hoarding & backlogs of yet-to-be-published journal articles are hamstringing progress. Cross-pollination of concepts, insights, & data is an intrinsic part of incubating good science. We need to share more, & share sooner.

19 Trackbacks

Post a Comment

Your email is never published nor shared. Required fields are marked *

*
*

You may use these HTML tags and attributes <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>