I'm a bit disappointed that the article doesn't talk about the problem in the humanities as well, but A: the author's background probably isn't in it and B: you can just about scrub the word "science" out and scribble in "humanities" and it reads just about right.
There are some obvious differences, mostly in data collection -- not in empirical evaluation, as both are equally empirical (say something happened in the book that didn't and you're just as wrong as if you fudge numbers in a lab study). Data in the humanities comes in many forms, from the novel you're discussing to receipts found in a grocer's, but one thing that's true is that you're supposed to know all the stuff that's already been written on the topic.
That's obviously fairly untenable. But more importantly, it's evaluated just the same way. The article uses an example about studying people watching dog videos. A reader can say "this only makes sense if cat videos also have the same effect" and suddenly you're faced with either fudging your numbers or running a totally separate study that will take as long as the first one did. This happens all the time in the humanities.
I think nearly every paper I've published, and a few I haven't, did this. The last one, the one that's sort of got me suffering burn out from academic writing at all, was on chaos magic, Lovecraft, Moorcock, and fandom. I was made by the editors to talk about topics that were totally off topic, but to source them I had to use the editors' own papers, since they were the only people to write about them.
I was once told that my paper on Gibson's sprawl trilogy needed to discuss all his other books. When I submitted that version, a new reader said the paper felt unfocused and should talk mostly about the sprawl trilogy. I was once accused, in yet another paper, of thinking something referred to the occult because I didn't understand gothic fiction -- as some of you may know, my literal degree is in gothic fiction, and also the line in question was "Farewell to all occult adventure; the farce is played." Definitely not related to the occult right, particularly when you consider the speaker is a member of a secret society of thieves and occultists who do bizarre rituals of an evening.
One of the problems with peer review generally, from what I have been able to see, is that the readers are not engaged in the process -- they may be, but they are not required to be. So they are not obligated to help one make one's research any better.
Imagine if you accepted a student's paper and said it was wrong because it didn't look at enough research -- but you didn't say what the research should be. And yes, I know some people do that. I don't, though. If the missing research is in finances, or budgets, or psych studies, I say that. My very first academic submission was turned down because I hadn't read a particular article about the novel, but the reader did not say what that article was. Given that I'd read everything I could possibly find on that novel, this comment was not going to make the article miraculously appear.
This is sort of why I do things like reread gothic novels on cohost. I could be spending that time polishing the three complete academic papers I have saved to my hard drive, including one that was sent back with a revise and resubmit request. But what's the point? I'm out of the job market hell, through an accident of fate (my job is still shit and underpays, but my family unit finances are ok now, and that's another post, how a significant amount of college education is subsidized by a partner who makes more money). No one will ever comment on the articles, if anyone even reads them. Why not do pretty similar work that reaches people? Maybe I'm not innovating in my field, but I'm reaching interested parties who can go on to a greater appreciation of the material.
In other words, if I don't suffer the slings and arrows of the faulty peer review system, I can actually teach
I have seen peer review prove useful, but—universally, I think—in smaller journals with good editors. Perhaps this is a local phenomenon, I don't know, but my home sub-field emphasises the editor's role in review, and some editors will send you a decision that says something along the lines of 'X and Y parts of Reviewer 2's comments might improve the piece; Z part offers nothing but vindictive cobblers, so you should ignore it'. Or even, in one case, 'Reviewer 2 was wrong about everything, so we're finding a Reviewer 3 and won't be using 2 again'.
I once saved an article from rejection at the hands of a reviewer who'd fundamentally misunderstood it by sending the editors a 3,500-word rebuttal of the review; writing the rebuttal helped me grasp how to make the original article clearer and more logical in its pursuit of its argument, so that other readers wouldn't fall into the same errors. Did peer review work to improve the article in that case? Maybe. But if so then, again, it happened because the journal had good editors willing to judge competing cases rather than simply enact reports.
(I wrote the rebuttal immediately, in a fit of pique, then left it for a week while I calmed down, and then stripped all the sarcasm back out before I sent it.)
All journals in my corner of the humanities count as small in the grand scheme of things. I once co-authored a piece that got published in Science, which served up an eye-opener. The idea that a journal might be big enough to have its own press officer seemed and seems hilarious to me. For what it's worth, I didn't think the peer review process on that work helped improve it.
Two other stray thoughts across the hums/sciences divide. First, I'm not sure the history of peer review is as brief and simple as Mastroianni's account of it, though the broad point that its /universal/ use started recently does hold. Second, I perceive a difference between the articles I write that argue a case to the point of (term of art:) moral certainty, and those that argue a case to some looser level of probability. I wonder whether the two sorts demand two different sorts of assessment.
A person only has so much time to distribute across teaching (within or outside universities), research, writing, paperwork &c, and @calliope's closing point about teaching hints at how peer review's vices interact with other problems.
