A common response to posting peer reviews (e.g. eLife, EMBO J) is it's wasted effort because no-one will read them. Well, breaking news: not many people read the paper either...
EMBO estimate ~10% of readers look at the peer reviews. Given a peer review takes ~5 hours to generate and one would certainly conclude a paper takes >50 hours to generate, then a readers/hour-worked calculation surely argues in favor of posting peer reviews - no?
@richardsever @jkpritch It would be better to assess the impact of posting reviews. It’s a good thing for transparency, but I’ve not seen a comment on social media, in a talk, or in a paper stating “this paper claims X, but the reviewers were not convinced” or had a colleague tell me that they were going to cite a paper but didn’t based on posted reviews. Could change, but seems more likely that the stamp of publication (yes, even at eLife) will dominate.
@jkpritch @richardsever I’m excepting those papers for which a problem was identified post-publication and therefore there was a lot of interest in the reviews re: the question “how did this get published?”
Other than that, I just haven’t seen posted reviews affect the course of science. I’d love to see counterexamples, as I want to believe it’s useful.
@StearnsLab @jkpritch @richardsever Isn't this exclusion a little bit like saying, well, why have the black box in the jetliner, since they almost never crash? (This also excludes a lot of other arguments in favor of posting peer reviews—for example, I rarely look at them as a way to judge a paper, but I often look at them as an insight into editorial process...)
@marcdionne @StearnsLab @jkpritch @richardsever I personally regularly read reviewer comments and author responses for papers that are bang on my research interests at eLife, EMBO J, Nat Comm etc. There's a lot to learn there, scientifically and rhetorically, and I wish I had access to this when I was a PhD student and postdoc to understand the ins and outs of peer review.
@christlet @marcdionne @StearnsLab @jkpritch @richardsever Sometimes reviewers put forward great hypotheses or propose experiments that are rightfully considered by the editor & authors outside the scope of the paper, but are great scientific discussions. They remind me of a vanishing genre: transcripts of workshop discussions. I first learned #immunology from the volumes of Basel Tolerance workshop - they are incredible
cc @mucida
https://books.google.pt/books/about/The_Tolerance_Workshop.html?id=3VMezQEACAAJ&redir_esc=y
@christlet @marcdionne @richardsever @jkpritch @cyrilpedia @mucida Yes, I wish that could even be part of regular submissions - an appendix to a paper, for example - which would also have the potential benefit of shortening the time of review.
@christlet @cyrilpedia @marcdionne @StearnsLab @jkpritch @richardsever @mucida if there a citation format to refer to? Something like [see review report in Xxx. et al.]? Should they have a separate DOI?
@robine @cyrilpedia @marcdionne @StearnsLab @jkpritch @richardsever @mucida Good question. What about reviews published next to preprints via @ReviewCommons - are they citable via a distinct DOI?
@cyrilpedia @christlet @StearnsLab @jkpritch @richardsever @mucida This is a nice point—there are reviews that are thoughtful and interesting enough to be worth reading. (Unfortunately, "experiments that are... considered by the editor... outside the scope of the paper" seem also to be a vanishing genre, but I digress...)
@cyrilpedia @christlet @StearnsLab @jkpritch @richardsever @mucida Another important issue is whether publishing reviews changes the behaviour of reviewers or editors. I once had high hopes for this.
@marcdionne @mucida @christlet @cyrilpedia @jkpritch @richardsever I’ve seen no evidence of this. However, the group discussion part of current eLife review practice does change reviewer behavior and should be more widely adopted.
@richardsever @cyrilpedia @marcdionne @jkpritch @christlet @mucida 💯 Transcribed question and answer sessions in symposium volumes was a great feature of long ago.
@cyrilpedia @christlet @marcdionne @StearnsLab @jkpritch @richardsever @mucida I think this is it- a transparent narrative of a manuscript's evolution under #PeerReview- it can capture great insight, offer perspective of what next step could be & flag warning signs for particular approaches/limitations you can onboard for your own studies. You don't need to look at all of them, but they are priceless for some studies which really align/push with your research. How to be a better reviewer too...
@cyrilpedia @christlet @marcdionne @StearnsLab @jkpritch @richardsever @mucida Yes, this! I agree that education and transparency are great reasons to make reviews accessible, but it’s more than that. It’s also about sharing how we as scientists arrive at consensus. Having access to the reviews shows the designated experts don’t always agree. They often have different ideas, find different experiments flawed or compelling.
@cyrilpedia @christlet @marcdionne @StearnsLab @jkpritch @richardsever @mucida And those differences of opinion, that process of discussion, inherently undermine the horrifyingly common misconception that peer review certifies a paper as “correct” (fundamentally an argument from authority and one that never fails to get on my last nerve).
@kristine_willis @cyrilpedia @christlet @StearnsLab @jkpritch @richardsever @mucida I prefer to say that the peer review process usually helps establish that the paper is not *obviously* wrong (but some obviously wrong papers get through—arsenic life, STAP cells....)
@marcdionne @cyrilpedia @christlet @StearnsLab @jkpritch @richardsever @mucida I think that’s fair. But I would add, what’s not obviously wrong now may be very obviously wrong in the future.
@kristine_willis @cyrilpedia @christlet @StearnsLab @jkpritch @richardsever @mucida Yes—I carefully chose the ones that seemed at the time to be likely to be wrong—papers that were only believed at all because they had been "validated by peer review".
@cyrilpedia @marcdionne @StearnsLab @jkpritch @richardsever @mucida Another example is when authors provide data or figures just for reviewers to show that they tried X or Y and it didn't work or wouldn't be a useful addition to the paper