Posts Tagged ‘Peer review’

Funding Research by Lottery?: How Lucky Do You Feel After Submitting a Grant

Reporter: Stephen J. Williams, Ph.D.

A recent article in Nature: “Science Funders Gamble on Grant Lotteries” discusses an odd twist to the anxiety most researchers feel after submitting grants to an agency.  Now, along with the hours of fretting over details and verbiage in a grant application, it appears that not only great science, but the luck of the draw may be necessary to get your work funded.  The article, by David Adam, discusses the funding strategy of the Health Research Council of New Zealand, which since 2015, has implemented a strategy of awarding grants through random selection.  Although limited in scope and size (mainly these grants are on very highly speculative and potentially transformative research and awards are usually less that $150,000 NZD) was meant to promote the applicants in submitting more risky ideas that are usually submitted in traditional peer reviewed grants.

Random chance will create more openness to ideas that are not in the mainstream

–  Margit Osterloh, economist at University of Zurich

Margit also mentions that many mid-ranking applications which are never funded could benefit from such a lottery system.

The Swiss National Science Foundation (SSFS) is also experimenting with this idea of random selection.  The Health Research Council states the process in not entirely random.  A computer selects the projects at random based on a random number generator.  A panel then decides if they are a reasonably good and well written application.

Some researchers have felt this random process could help eliminate much bias that can be baked into the traditional peer review process.  However there are many who feel the current process of peer review panels are a necessary and rigorous step in the granting process, analyzing applications which would most likely have the best chances to succeed based on the rigor of the proposed science.

However Osterloh feels that the lottery idea produces a humbling effect. As Margit said

If you know you have got a grant or a publication which is selected partly randomly, then you will know very well you are not the king of the Universe

Humility in science: a refreshing idea.  However the lottery idea will not mean that scientists need not prepare a careful and well written application.  Applications that are ranked very low would not be in the lottery.  However, if one feels lucky, maybe the obscene hours of worrying about each sentence written, or that figures for preliminary data should be altered at the 11th hour before submission might be a thing of the past.

Of course if you are a lucky person.



Read Full Post »

Peer Review and Health Care Issues

Larry H. Bernstein, MD, FCAP, Reporter


(Medscape – Dec 1, 2014)

Peer-reviewed journals retracted 110 papers over the last 2 years. Nature reports the grim details in “Publishing: the peer review scam”.

When a handful of authors were caught reviewing their own

papers, it exposed weaknesses in modern publishing systems.

Editors are trying to plug the holes.


The Hill reports that the FDA may lift its ban on blood donations from gay men. The American Red Cross has voiced its support for lifting of the ban.

Advisers for the Food and Drug Administration (FDA) will meet this week to decide whether gay men should be allowed to donate blood, the agency’s biggest step yet toward changing the 30-year-old policy.

If the FDA accepts the recommendation, it would roll back a policy that has been under strong pressure from LGBT advocates and some members of Congress for more than four years.

“We’ve got the ball rolling. I feel like this is a tide-turning vote,” said Ryan James Yezak, an LGBT activist who founded the National Gay Blood Drive and will speak at the meeting. “There’s been a lot of feet dragging and I think they’re realizing it now.”

Groups such as the American Red Cross and America’s Blood Centers also voiced support of the policy change this month, calling the ban “medically and scientifically unwarranted.”

The FDA will use the group’s recommendation to decide whether to change the policy.

“Following deliberations taking into consideration the available evidence, the FDA will issue revised guidance, if appropriate,” FDA spokeswoman Jennifer Rodriguez wrote in a statement.

This reporter has more than 20 years of Blood Bank experience.  The factor in favor of the recommendation is that the HIV 1/2 and other testing is accurate enough to leave the question of donor lifestyle irrelevant.  However, it remains to be seen whether the testing turnaround time is sufficient to prevent the release of units that may be contaminated prior to transfusion, which is problematic for platelets, that have short expirations. In all cases of donor infection, regardless of whether units are released, a finding leads to not releasing the product or to recall.


Democrats made a strategic mistake by passing the Affordable Care Act, Sen. Charles Schumer (N.Y.), the third-ranking member of the Senate Democratic leadership, said Tuesday.

Schumer says Democrats “blew the opportunity the American people gave them” in the 2008 elections, a Democratic landslide, by focusing on healthcare reform instead of legislation to boost the middle class.

“After passing the stimulus, Democrats should have continued to propose middle class-oriented programs and built on the partial success of the stimulus,” he said in a speech at the National Press Club.

He said the plight of uninsured Americans caused by “unfair insurance company practices” needed to be addressed, but it wasn’t the change that people wanted when they elected Barack Obama as president.

“Americans were crying out for an end to the recession, for better wages and more jobs; not for changes in their healthcare,” he said.

This reader finds the observation by Senator Schumer very perceptive, regardless of whether the observation in hindsight might have had a different political outcome.  It has been noted that President Obama had a lot on his plate.  Moreover, we have not seen such a poor record of legislation in my lifetime.  There are underlying issues of worldview of elected officials that also contribute to the events.




N AT U R E |  2 7 N O V  2 0 1 4; VO L 5 1 5 : 480-82.

Most journal editors know how much effort it takes to persuade busy researchers to review a paper. That is why the editor of The Journal of Enzyme Inhibition and Medicinal Chemistry was puzzled by the reviews for manuscripts by one author — Hyung-In Moon, a medicinal-plant researcher then at Dongguk University in Gyeongju, South Korea.

The reviews themselves were not remarkable: mostly favourable, with some suggestions about how to improve the papers. What was unusual was how quickly they were completed — often within 24 hours. The turnaround was a little too fast, and Claudiu Supuran, the journal’s editor-in-chief, started to become suspicious.

In 2012, he confronted Moon, who readily admitted that the reviews had come in so quickly because he had written many of them himself. The deception had not been hard to set up. Supuran’s journal and several others published by Informa Healthcare in London
invite authors to suggest potential reviewers for their papers. So Moon provided names, sometimes of real scientists and sometimes pseudonyms, often with bogus e-mail addresses that would go directly to him or his colleagues. His confession led to the retraction of 28 papers by several Informa journals, and the resignation of an editor.

Moon’s was not an isolated case. In the past 2 years, journals have been forced to retract more than 110 papers in at least 6 instances of peer-review.

Moon’s case is by no means the most spectacular instance of peer-review rigging in recent years. That honour goes to a case that came to light in May 2013, when Ali Nayfeh, then editor-in-chief of the Journal of Vibration and Control, received some troubling news. An author who had submitted a paper to the journal told Nayfeh that he had received e-mails about it from two people claiming to be reviewers. Reviewers do not normally have direct contact with authors, and — strangely — the e-mails came from generic-looking Gmail accounts rather than from the professional institutional accounts that many academics use (see ‘Red flags in review’).
Nayfeh alerted SAGE, the company in Thousand Oaks, California, that publishes the journal. The editors there e-mailed both the Gmail addresses provided by the tipster, and the institutional addresses of the authors whose names had been used, asking for proof of identity and a list of their publications.ew rigging. What all these cases had in common was that researchers exploited vulnerabilities in the publishers’ computerized systems to dupe editors into accepting manuscripts, often by doing their own reviews. The cases involved publishing behemoths Elsevier, Springer, Taylor & Francis, SAGE and Wiley, as well as Informa, at least one of the systems — could make researchers vulnerable to even more serious identity theft. “For a piece of software that’s used by hundreds of thousands of academics worldwide, it really is appalling,” says Mark Dingemanse, a linguist at the Max Planck Institute for Psycholinguistics in Nijmegen, the Netherlands, who has used some of these programs to publish and review papers.

A 14-month investigation that came to involve about 20 people from SAGE’s editorial, legal and production departments. It showed that the Gmail addresses were each linked to accounts with Thomson Reuters’ ScholarOne, a publication-management system used by SAGE and several other publishers, including Informa. Editors were able to track every paper that the person or people behind these accounts had allegedly written or reviewed, says SAGE spokesperson Camille Gamboa. They also checked the wording of reviews, the details of author-nominated reviewers, reference lists and the turnaround time for reviews (in some cases, only a few minutes). This helped the investigators to ferret out further suspicious-looking accounts; they eventually found 130.

SAGE investigators came to realize that authors were both reviewing and citing each other at an anomalous rate. Eventually, 60 articles were found to have evidence of peer-review tampering, involvement in the citation ring or both. “Due to the serious nature of the findings, we wanted to ensure we had researched all avenues as carefully as possible before contacting any of the authors and reviewers,” says Gamboa. When the dust had settled, it turned out that there was one author in the centre of the ring: Peter Chen, an engineer then at the National Pingtung University of Education (NPUE) in Taiwan, who was a co-author on practically all of the papers in question.

Moon and Chen both exploited a feature of ScholarOne’s automated processes. When a reviewer is invited to read a paper, he or she is sent an e-mail with login information. If that communication goes to a fake e-mail account, the recipient can sign into the system under whatever name was initially submitted, with no additional identity verification. Jasper Simons, vice-president of product and market strategy for Thomson Reuters in Charlottesville, Virginia, says that ScholarOne is a respected peer-review system and that it is the responsibility of journals and their editorial teams to invite properly qualified reviewers for their papers.

ScholarOne is not the only publishing system with vulnerabilities. Editorial Manager, built by Aries Systems in North Andover, Massachusetts, is used by many societies and publishers, including Springer and PLOS. The American Association for the Advancement of Science in Washington DC uses a system developed in-house for its journals Science, Science Translational Medicine and Science Signaling, but its open-access offering, Science Advances, uses Editorial Manager. Elsevier, based in Amsterdam, uses a branded version of the same product, called the Elsevier Editorial System.

Usually, editors in the United States and Europe know the scientific community in those regions well enough to catch potential conflicts of interest between authors and reviewers. But Lindsay says that Western editors can find this harder with authors from Asia — “where often none of us knows the suggested reviewers”. In these cases, the journal insists on at least one independent reviewer, identified and invited by the editors.

Read Full Post »

Amazon.com and their Top 1000 Reviewers

Reporter: Aviva Lev-Ari, PhD, RN


There Are No Free Lunches When it Comes to Reviewing

Filip Kesler, a Silicon Valley Entrepreneur, and Trevor Pinch of Cornell University’s Department of Science and Technology Studies, completed a systematic study of Amazon.com and their top 1000 reviewers. Their research focuses on a baffling notion that Amazon, essentially, has people working for them for free.  How is it that Amazon, in its immensity, could be receiving a “free lunch” from all of the individuals that write reviews for the website? We’ve all been told countless times, “There’s no such thing as a free lunch!” So, what’s the catch? Pinch and Kesler found that reviewers are indeed receiving something in return, although it might not be what you expect.

Amazon.com: Humble Beginnings

What started in a garage in 1994, as a email store for retailing books, has grown into one of the largest online retailers in the world of online commerce.  Amazon now offers consumers much more than books, evolving from exclusively being an online literary-style book seller to the largest electronic retailer on the internet. Their goals these days is to sell everything and encourage product reviews of, literally, everything. Initially, Amazon hired editors to write features and reviews but discovered that people would review for free. After trying to fix glitches in the review system, such as total reviewer anonymity, flaws still exist in a model that many businesses view as the Gold Standard. Customer reviews are being plagiarized, fake reviews are popping up everywhere. According to the study, anyone who is a registered customer can write a review without any obligation from Amazon to have bought the item that’s under review.

Do Good, Feel Good

A copious amount of time is being spent behind a keyboard writing reviews, Pinch and Kesler report. The average time spent reviewing is 9 years. One Amazon reviewer reported having reviewed consistently for 13 years. With this, specific methodologies have been formulated in regards to review writing. Reviews must be honest and fair, the respondents report. A reviewers job “is to help people make a decision, not make it for them” through justifying an opinion with facts instead of quickly stating a personal one.

In sum, 166 of the top reviewers responded to the study, each taking great pride in their reviews. Surprisingly, their responses lent a different return upon investment than expected in the “free lunch” conversation. “Self expression” and “enjoyment” were ranked highly when respondents were asked why they chose to write reviews. “Altruism” and developing a “sense of community” followed close behind. Along with the sense of community comes an overwhelming care about their rank and reputation as a reviewer. Admittedly, in the “real world” reviewer recognition “is not a big deal, but it is an important deal in a community.” Features such as Amazon Friends have brought reviewers closer and have begun to influence their reasons and techniques for reviewing.
A sense of self accomplishment is given to some reviewers. “I like the fact that I rank high, I don’t rank much higher anywhere else in life.” said one respondent. Another admitted, “I have often said that Amazon ‘saved me’ because it gave me back my passion for writing. In all honesty my Amazon rank has become the foundation of my self worth. I work at a library, but I consider myself a reviewer first and foremost- and that badge tells me that I am accomplishing something.”

Someone’s Getting Free Merchandise?

In addition to self-fulfillment, respondents also reported being offered free merchandise in exchange for reviews. According to the study, 85% of respondents report being sent free

books or other products to review by publishers, authors and the like. A large number of reviews by higher-ranked reviewers are indeed for books they have been given free. “The fact is that most readers of these reviews do not know that there is any relationship at all, never mind a direct one, between the producer of the product and the reviewer” the study states.

Overwhelmingly, Amazon.com reviewers “really care about books (and music and movies) and reviewing them. It’s a form of work that can on occasions become frustrating and even addictive. It’s a form of work that can bring rewards beyond the activity itself in terms of utilitarian benefits that can accumulate to some top reviewers.”

Our Take on the Conversation

The truth of the matter is this: putting so much emphasis on being a reviewer is scary. Instead of being a tool for businesses, the review is morphed into a payback for a free lunch, and sometimes the need for a massive chocolate cake for dessert. Too much emphasis on identity and self-fulfillment breaks the fundamental reason for reviewing: having both informed consumers and informed businesses.

The practice of sending reviewers free merchandise to review, although not against Amazon’s online review guidelines, is questionable. According to the study reviewers, after having read a book or tried a product, usually don’t post negative reviews, or they give authors the option of what they want done. Overwhelmingly, authors are choosing to refuse to have the negative review out in public. The study found that more than 80 percent of the reviews on the site were positive all because 85 percent of these reviewers receive free product to review. Many
reviewers thus are only posting positive reviews, even when negative ones should be transparent. It’s indisputably true that online reviewers are indeed being influenced by outside factors. Without diving to the bottom of the ethics pool and opening a whole new can of worms, in essence, this can’t be a good thing.

Whether the reward is tangible or intangible, the age old adage holds true. There really is no such thing as a free lunch. Amazon.com and its reviewers are living proof.

To view Trevor Pinch’s delivery of this study at Cornell’s ILR Conference Center in Midtown Manhattan click here.



Read Full Post »

%d bloggers like this: