Posts Tagged ‘Bias measurement tools’

Larry H Bernstein, MD, FCAP, Curator benefit of anthocyanins from apples and berries noted for men

After significant studies have been completed, particularly on a relationship between anthocyanins consumption and decreasd risk of Parkinson’s Disease in men, it is unclear why a comparable effect is not seen in women.  This would lead one to ask questions about predominant time course of development in relationship to androgen activity.  Pre- and postmenopausal status would seem to make no difference. It is reported that the anthocyanins cross the blood brain barrier.  There are other questions that need to be raised.  There is a decline in the production of transthyretin by the choroid plexus in the elderly – not sex related – with an elevation of homocysteine that is reciprocal to decline in transthyretin-RBP complex, that is related to AD.  This is mediated by cystathionine-beta synthase, and involves matrix metalloproteinases.  A mechanism for Parkinson’s Disease has been postulated to be related to Parkin gene expression, but how does this work, and why do we see the sex assymetry?

Eating flavonoids protects men against Parkinson’s disease

General DietMissed – Medical Breakthroughs • Tags: AnthocyaninFlavonoidHarvard University,HealthNeurologyParkinsonParkinson DiseaseUniversity of East Anglia       07 Apr 2012

Men who eat flavonoid-rich foods such as berries, tea, apples and red wine significantly reduce their risk of developing Parkinson’s disease, according to new research by Harvard University and the University of East Anglia (UEA).

Published today in the journal Neurology ®, the findings add to the growing body of evidence that regular consumption of some flavonoids can have a marked effect on human health. Recent studies have shown that these compounds can offer protection against a wide range of diseases including heart disease, hypertension, some cancers and dementia.

This latest study is the first study in humans to show that flavonoids can protect neurons against diseases of the brain such as Parkinson’s.

Around 130,000 men and women took part in the research. More than 800 had developed Parkinson’s disease within 20 years of follow-up. After a detailed analysis of their diets and adjusting for age and lifestyle, male participants who ate the most flavonoids were shown to be 40 per cent less likely to develop the disease than those who ate the least. No similar link was found for total flavonoid intake in women.

The research was led by Dr Xiang Gao of Harvard School of Public Health in collaboration with Prof Aedin Cassidy of the Department of Nutrition, Norwich Medical School at UEA.

“These exciting findings provide further confirmation that regular consumption of flavonoids can have potential health benefits,” said Prof Cassidy.

“This is the first study in humans to look at the associations between the range of flavonoids in the diet and the risk of developing Parkinson’s disease and our findings suggest that a sub-class of flavonoids called anthocyanins may have neuroprotective effects.”

Prof Gao said: “Interestingly, anthocyanins and berry fruits, which are rich in anthocyanins, seem to be associated with a lower risk of Parkinson’s disease in pooled analyses. Participants who consumed one or more portions of berry fruits each week were around 25 per cent less likely to develop Parkinson’s disease, relative to those who did not eat berry fruits. Given the other potential health effects of berry fruits, such as lowering risk of hypertension as reported in our previous studies, it is good to regularly add these fruits to your diet.”

Flavonoids are a group of naturally occurring, bioactive compunds found in many plant-based foods and drinks. In this study the main protective effect was from higher intake of anthocyanins, which are present in berries and other fruits and vegetables including aubergines, blackcurrants and blackberries. Those who consumed the most anthocyanins had a 24 per cent reduction in risk of developing Parkinson’s disease and strawberries and blueberries were the top two sources in the US diet.

The findings must now be confirmed by other large epidemiological studies and clinical trials.

Parkinson’s disease is a progresssive neurological condition affecting one in 500 people, which equates to 127,000 people in the UK. There are few effective drug therapies available.  Dr Kieran Breen, director of research at Parkinson’s UK said: “This study raises lots of interesting questions about how diet may influence our risk of Parkinson’s…   there are still a lot of questions to answer and much more research to do before we really know how important diet might be for people with Parkinson’s.”


Eating berries may lower risk of Parkinson’s

Missed – Medical Breakthroughs • Tags: BerryDoctor of PhilosophyFlavonoidParkinson,Parkinson DiseaseXiang Gao    Public release date: 13-Feb-2011

ST. PAUL, Minn. –New research shows men and women who regularly eat berries may have a lower risk of developing Parkinson’s disease, while men may also further lower their risk by regularly eating apples, oranges and other sources rich in dietary components called flavonoids. The study was released today and will be presented at the American Academy of Neurology’s 63rd Annual Meeting in Honolulu April 9 to April 16, 2011.

Flavonoids are found in plants and fruits and are also known collectively as vitamin P and citrin. They can also be found in berry fruits, chocolate, and citrus fruits such as grapefruit.

The study involved 49,281 men and 80,336 women. Researchers gave participants questionnaires and used a database to calculate intake amount of flavonoids. They then analyzed the association between flavonoid intakes and risk of developing Parkinson’s disease. They also analyzed consumption of five major sources of foods rich in flavonoids: tea, berries, apples, red wine and oranges or orange juice. The participants were followed for 20 to 22 years.

During that time, 805 people developed Parkinson’s disease. In men, the top 20 percent who consumed the most flavonoids were about 40 percent less likely to develop Parkinson’s disease than the bottom 20 percent of male participants who consumed the least amount of flavonoids. In women, there was no relationship between overall flavonoid consumption and developing Parkinson’s disease. However, when sub-classes of flavonoids were examined, regular consumption of anthocyanins, which are mainly obtained from berries, were found to be associated with a lower risk of Parkinson’s disease in both men and women.

“This is the first study in humans to examine the association between flavonoids and risk of developing Parkinson’s disease,” said study author Xiang Gao, MD, PhD, with the Harvard School of Public Health in Boston. “Our findings suggest that flavonoids, specifically a group called anthocyanins, may have neuroprotective effects. If confirmed, flavonoids may be a natural and healthy way to reduce your risk of developing Parkinson’s disease.”
May 10, 2013

Could eating peppers prevent Parkinson’s?

Missed – Medical Breakthroughs • Tags: American Neurological AssociationAnnals of Neurology,Group Health CooperativeNicotineParkinsonParkinson’s diseaseSolanaceaeUniversity of Washington

Contact: Dawn Peters 781-388-8408 Wiley

Dietary nicotine may hold protective key

New research reveals that Solanaceae—a flowering plant family with some species producing foods that are edible sources of nicotine—may provide a protective effect against Parkinson’s disease. The study appearing today inAnnals of Neurology, a journal of the American Neurological Association and Child Neurology Society, suggests that eating foods that contain even a small amount of nicotine, such as peppers and tomatoes, may reduce risk of developing Parkinson’s.

Parkinson’s disease is a movement disorder caused by a loss of brain cells that produce dopamine. Symptoms include facial, hand, arm, and leg tremors, stiffness in the limbs, loss of balance, and slower overall movement. Nearly one million Americans have Parkinson’s, with 60,000 new cases diagnosed in the U.S. each year, and up to ten million individuals worldwide live with this disease according to the Parkinson’s Disease Foundation. Currently, there is no cure for Parkinson’s, but symptoms are treated with medications and procedures such as deep brain stimulation.

Previous studies have found that cigarette smoking and other forms of tobacco, also a Solanaceae plant, reduced relative risk of Parkinson’s disease. However, experts have not confirmed if nicotine or other components in tobacco provide a protective effect, or if people who develop Parkinson’s disease are simply less apt to use tobacco because of differences in the brain that occur early in the disease process, long before diagnosis.

For the present population-based study Dr. Susan Searles Nielsen and colleagues from the University of Washington in Seattle recruited 490 patients newly diagnosed with Parkinson’s disease at the university’s Neurology Clinic or a regional health maintenance organization, Group Health Cooperative. Another 644 unrelated individuals without neurological conditions were used as controls. Questionnaires were used to assess participants’ lifetime diets and tobacco use, which researchers defined as ever smoking more than 100 cigarettes or regularly using cigars, pipes or smokeless tobacco.

Vegetable consumption in general did not affect Parkinson’s disease risk, but as consumption of edible Solanaceae increased, Parkinson’s disease risk decreased, with peppers displaying the strongest association. Researchers noted that the apparent protection from Parkinson’s occurred mainly in men and women with little or no prior use of tobacco, which contains much more nicotine than the foods studied.

“Our study is the first to investigate dietary nicotine and risk of developing Parkinson’s disease,” said Dr. Searles Nielsen. “Similar to the many studies that indicate tobacco use might reduce risk of Parkinson’s, our findings also suggest a protective effect from nicotine, or perhaps a similar but less toxic chemical in peppers and tobacco.” The authors recommend further studies to confirm and extend their findings, which could lead to possible interventions that prevent Parkinson’s disease.


This study is published in Annals of Neurology. Media wishing to receive a PDF of this article may contact

Full citation: “Nicotine from Edible Solanaceae and Risk of Parkinson Disease.” Susan Searles Nielsen, Gary M. Franklin, W.T. Longstreth Jr, Phillip D. Swanson and Harvey Checkoway. Annals of Neurology; Published May 9, 2013 (DOI:10.1002/ana.23884).

URL Upon Publication:

Author Contact: To arrange an interview with Dr. Susan Searles Nielsen, please contact Leila Gray with the University of Washington Health Sciences News Office at +1 206-685-0381 or at

About the Journal

Annals of Neurology, the official journal of the American Neurological Association and the Child Neurology Society, publishes articles of broad interest with potential for high impact in understanding the mechanisms and treatment of diseases of the human nervous system. All areas of clinical and basic neuroscience, including new technologies, cellular and molecular neurobiology, population sciences, and studies of behavior, addiction, and psychiatric diseases are of interest to the journal. The journal is published by Wiley on behalf of the
American Neurological Association and Child Neurology Society. For more information, please visit

Flavonoids from berries shown to protect men against Parkinson’s disease

December 19, 2013 · by MrT

by: John Phillip, John is a Certified Nutritional Consultant and Health Researcher

(NaturalNews) Past research bodies have confirmed the health-protective effect of a natural diet rich in flavonoids to protect against a wide range of diseases including heart disease, hypertension, some cancers, and dementia. Researchers from Harvard University and the University of East Anglia have published the result of a study in the journalNeurology that demonstrates how these plant-based phytonutrients can significantly lower the risk of developing Parkinson’s disease, especially in men.

Flavonoids from healthy foods such as berries, tea, apples, and red wine cross the delicate blood-brain barrier to protect neurons against neurologic diseases such as Parkinson’s. This large scale study included more than 130,000 men and women participants that were followed for a period of twenty years. During this time, more than 800 individuals developed Parkinson’s disease.

A diet high in flavonoids from berries lowers Parkinson’s disease risk by forty percent

After a detailed analysis of their diets and adjusting for age and lifestyle, male participants who ate the most flavonoids were shown to be forty percent  less likely to develop the disease than those who ate the least. No similar link was found for total flavonoid intake in women.

This was the first study to examine the connection between flavonoid consumption and the development of Parkinson’s disease. The findings suggest that a sub-class of flavonoids called anthocyanins may exhibit neuroprotective effects. Participants consuming one or more portions of berry fruits each week were around twenty-five percent less likely to develop Parkinson’s disease, relative to those who did not eat berry fruits.

Flavonoids are the bioactive, naturally occurring chemical compounds found in many plant-based foods and drinks.

This study demonstrated the main protective effect was from the consumption of anthocyanins, which are present in berries and other fruits and vegetables including aubergines, blackcurrants, and blackberries. Strawberries and blueberries are the two most common sources of flavonoids in the US diet, contributing to a twenty-four percent lowered risk in this research.

Parkinson’s disease is among a group of chronic diseases presently affecting one in 500 people, with new cases on the rise. Drug therapies are ineffective and bear significant side effects.

Nutrition experts recommend adding a minimum of three to five servings of flavonoids to your diet each week. Include all varieties of berries, apples, and green tea to guard against Parkinson’s disease and other neurodegenerative illnesses.



Read Full Post »

Risk of Bias in Translational Science

Author: Larry H. Bernstein, MD, FCAP


Curator: Aviva Lev-Ari, PhD, RN


Assessment of risk of bias in translational science

Andre Barkhordarian1, Peter Pellionisz2, Mona Dousti1, Vivian Lam1,Lauren Gleason1, Mahsa Dousti1, Josemar Moura3 and Francesco Chiappelli14*  

1Oral Biology & Medicine, School of Dentistry, UCLA, Evidence-Based Decisions Practice-Based Research Network, Los Angeles, USA

2Pre-medical program, UCLA, Los Angeles, CA

3School of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil

4Evidence-Based Decisions Practice-Based Research Network, UCLA School of Dentistry, Los Angeles, CA

Journal of Translational Medicine 2013, 11:184

This is an Open Access article distributed under the terms of the Creative Commons Attribution License


Risk of bias in translational medicine may take one of three forms:

  1. a systematic error of methodology as it pertains to measurement or sampling (e.g., selection bias),
  2. a systematic defect of design that leads to estimates of experimental and control groups, and of effect sizes that substantially deviate from true values (e.g., information bias), and
  3. a systematic distortion of the analytical process, which results in a misrepresentation of the data with consequential errors of inference (e.g., inferential bias).

Risk of bias can seriously adulterate the internal and the external validity of a clinical study, and, unless it is identified and systematically evaluated, can seriously hamper the process of comparative effectiveness and efficacy research and analysis for practice. The Cochrane Group and the Agency for Healthcare Research and Quality have independently developed instruments for assessing the meta-construct of risk of bias. The present article begins to discuss this dialectic.


As recently discussed in this journal [1], translational medicine is a rapidly evolving field. In its most recent conceptualization, it consists of two primary domains:

  • translational research proper and
  • translational effectiveness.

This distinction arises from a cogent articulation of the fundamental construct of translational medicine in particular, and of translational health care in general.

The Institute of Medicine’s Clinical Research Roundtable conceptualized the field as being composed by two fundamental “blocks”:

  • one translational “block” (T1) was defined as “…the transfer of new understandings of disease mechanisms gained in the laboratory into the development of new methods for diagnosis, therapy, and prevention and their first testing in humans…”, and
  • the second translational “block” (T2) was described as “…the translation of results from clinical studies into everyday clinical practice and health decision making…” [2].

These are clearly two distinct facets of one meta-construct, as outlined in Figure 1. As signaled by others, “…Referring to T1 and T2 by the same name—translational research—has become a source of some confusion. The 2 spheres are alike in name only. Their goals, settings, study designs, and investigators differ…” [3].

1479-5876-11-184-1  Fig 1. TM construct

Figure 1. Schematic representation of the meta-construct of translational health carein general, and translational medicine in particular, which consists of two fundamental constructs: the T1 “block” (as per Institute of Medicine’s Clinical Research Roundtable nomenclature), which represents the transfer of new understandings of disease mechanisms gained in the laboratory into the development of new methods for diagnosis, therapy, and prevention as well as their first testing in humans, and the T2 “block”, which pertains to translation of results from clinical studies into everyday clinical practice and health decision making [[3]]. The two “blocks” are inextricably intertwined because they jointly strive toward patient-centered research outcomes (PCOR) through the process of comparative effectiveness and efficacy research/review and analysis for clinical practice (CEERAP). The domain of each construct is distinct, since the “block” T1 is set in the context of a laboratory infrastructure within a nurturing academic institution, whereas the setting of “block” T2 is typically community-based (e.g., patient-centered medical/dental home/neighborhoods [4]; “communities of practice” [5]).

For the last five years at least, the Federal responsibilities for “block” T1 and T2 have been clearly delineated. The National Institutes of Health (NIH) predominantly concerns itself with translational research proper – the bench-to-bedside enterprise (T1); the Agency for Healthcare Research Quality (AHRQ) focuses on the result-translation enterprise (T2). Specifically: “…the ultimate goal [of AHRQ] is research translation—that is, making sure that findings from AHRQ research are widely disseminated and ready to be used in everyday health care decision-making…” [6]. The terminology of translational effectiveness has emerged as a means of distinguishing the T2 block from T1.

Therefore, the bench-to-bedside enterprise pertains to translational research, and the result-translation enterprise describes translational effectiveness. The meta-construct of translational health care (viz., translational medicine) thus consists of these two fundamental constructs:

  • translational research and
  • translational effectiveness,

which have distinct purposes, protocols and products, while both converging on the same goal of new and improved means of

  • individualized patient-centered diagnostic and prognostic care.

It is important to note that the U.S. Patient Protection and Affordable Care Act (PPACA, 23 March 2010) has created an environment that facilitates the pursuit of translational health care because it emphasizes patient-centered outcomes research (PCOR). That is to say, it fosters the transaction between translational research (i.e., “block” T1)(TR) and translational effectiveness (i.e., “block” T2)(TE), and favors the establishment of communities of practice-research interaction. The latter, now recognized as practice-based research networks, incorporate three or more clinical practices in the community into

  • a community of practices network coordinated by an academic center of research.

Practice-based research networks may be a third “block” (T3)(PBTN) in translational health care and they could be conceptualized as a stepping-stone, a go-between bench-to-bedside translational research and result-translation translational effectiveness [7]. Alternatively, practice-based research networks represent the practical entities where the transaction between

  • translational research and translational effectiveness can most optimally be undertaken.

It is within the context of the practice-based research network that the process of bench-to-bedside can best seamlessly proceed, and it is within the framework of the practice-based research network that

  • the best evidence of results can be most efficiently translated into practice and
  • be utilized in evidence-based clinical decision-making, viz. translational effectiveness.

Translational effectiveness

As noted, translational effectiveness represents the translation of the best available evidence in the clinical practice to ensure its utilization in clinical decisions. Translational effectiveness fosters evidence-based revisions of clinical practice guidelines. It also encourages

  • effectiveness-focused,
  • patient-centered and
  • evidence-based clinical decision-making.

Translational effectiveness rests not only on the expertise of the clinical staff and the empowerment of patients, caregivers and stakeholders, but also, and

  • most importantly on the best available evidence [8].

The pursuit of the best available evidence is the foundation of

  • translational effectiveness and more generally of
  • translational medicine in evidence-based health care.

The best available evidence is obtained through a systematic process driven by

  • a research question/hypothesis that is articulated about clearly stated criteria that pertain to the
  • patient (P), the interventions (I) under consideration (C), for the sought clinical outcome (O), within a given timeline (T) and clinical setting (S).

PICOTS is tested on the appropriate bibliometric sample, with tools of measurements designed to establish the level (e.g., CONSORT) and the quality of the evidence. Statistical and meta-analytical inferences, often enhanced by analyses of clinical relevance [9], converge into the formulation of the consensus of the best available evidence. Its dissemination to all stakeholders is key to increase their health literacy in order to ensure their full participation

  • in the utilization of the best available evidence in clinical decisions, viz., translational effectiveness.

To be clear, translational effectiveness – and, in the perspective discussed above, translational health care – is anchored on obtaining the best available evidence,

  • which emerges from highest quality research.
  • which is obtained when errors are minimized.

In an early conceptualization [10], errors in research were presented as

  • those situations that threaten the internal and the external validity of a research study –

that is, conditions that impede either the study’s reproducibility, or its generalization. In point of fact, threats to internal and external validity [10] represent specific aspects of systematic errors (i.e., bias) in the

  • research design,
  • methodology and
  • data analysis.

Thence emerged a branch of science that seeks to

  • understand,
  • control and
  • reduce risk of bias in research.

Risk of bias and the best available evidence

It follows that the best available evidence comes from research with the fewest threats to internal and to external validity – that is to say, the fewest systematic errors: the lowest risk of bias. Quality of research, as defined in the field of research synthesis [11], has become synonymous with

  • low bias and contained risk of bias [1215].

Several years ago, the Cochrane group embarked on a new strategy for assessing the quality of research studies by examining potential sources of bias. Certain original areas of potential bias in research were identified, which pertain to

(a) the sampling and the sample allocation process, to measurement, and to other related sources of errors (reliability of testing),

(b) design issues, including blinding, selection and drop-out, and design-specific caveats, and

(c) analysis-related biases.

A Risk of Bias tool was created (Cochrane Risk of Bias), which covered six specific domains:

1. selection bias,

2. performance bias,

3. detection bias,

4. attrition bias,

5. reporting bias, and

6. other research protocol-related biases.

Assessments were made within each domain by one or more items specific for certain aspects of the domain. Each items was scored in two distinct steps:

1. the support for judgment was intended to provide a succinct free-text description of the domain being queried;

2. each item was scored high, low, or unclear risk of material bias (defined here as “…bias of sufficient magnitude to have a notable effect on the results or conclusions…” [16]).

It was advocated that assessments across items in the tool should be critically summarized for each outcome within each report. These critical summaries were to inform the investigator so that the primary meta-analysis could be performed either

  • only on studies at low risk of bias, or for
  • the studies stratified according to risk of bias [16].

This is a form of acceptable sampling analysis designed to yield increased homogeneity of meta-analytical outcomes [17]. Alternatively, the homogeneity of the meta-analysis can be further enhanced by means of the more direct quality-effects meta-analysis inferential model [18].

Clearly, one among the major drawbacks of the Cochrane Risk of Bias tool is

  • the subjective nature of its assessment protocol.

In an effort to correct for this inherent weakness of the instrument, the Cochrane group produced

  • detailed criteria for making judgments about the risk of bias from each individual item[16], and
  • that judgments be made independently by at least two people, with any discrepancies resolved by discussion [16].

This approach to increase the reliability of measurement in research synthesis protocols

  • is akin to that described by us [19,20] and by AHRQ [21].

In an effort to aid clinicians and patients in making effective health care related decisions, AHRQ developed an alternative Risk of Bias instrument for enabling systematical evaluation of evidence reporting [22]. The AHRQ Risk of Bias instrument was created to monitor four primary domains:

1. risk of bias: design, methodology, analysis scoring – low, medium, high

2. consistency: extent of similarity in effect sizes across studies within a bibliome scoring – consistent, inconsistent, unknown

3. directness: unidirectional link between the interventions of interest and the sought outcome, as opposed to multiple links in a casual chain scoring – direct, indirect

4. precision: extent of certainty for estimate of effect with respect to the outcome scoring – precise, imprecise In addition, four secondary domains were identified:

a. Dose response association: pattern of a larger effect with greater exposure (Present/Not Present/Not Applicable or Not Tested)

a. Confounders: consideration of confounding variables (Present/Absent)

a. Strength of association: likelihood that the observed effect is large enough that it cannot have occurred solely as a result of bias from potential confounding factors (Strong/Weak)

a. Publication bias

The AHRQ Risk of Bias instrument is also designed to yield an overall grade of the estimated risk of bias in quality reporting:

•Strength of Evidence Grades (scored as high – moderate – low – insufficient)

This global assessment, in addition to incorporating the assessments above, also rates:

–major benefit

–major harm

–jointly benefits and harms

–outcomes most relevant to patients, clinicians, and stakeholders

The AHRQ Risk of Bias instrument suffers from the same two major limitations as the Cochrane tool:

1. lack of formal psychometric validation as most other tools in the field [21], and

2. providing a subjective and not quantifiable assessment.

To begin the process of engaging in a systematic dialectic of the two instruments in terms of their respective construct and content validity, it is necessary

  • to validate each for reliability and validity either by means of the classic psychometric theory or generalizability (G) theory, which allows
  • the simultaneous estimation of multiple sources of measurement error variance (i.e., facets)
  • while generalizing the main findings across the different study facets.

G theory is particularly useful in clinical care analysis of this type, because it permits the assessment of the reliability of clinical assessment protocols.

  • the reliability and minimal detectable changes across varied combinations of these facets are then simply calculated [23], but
  • it is recommended that G theory determination follow classic theory psychometric assessment.

Therefore, we have commenced a process of revision the AHRQ Risk of Bias instrument by rendering questions in primary domains quantifiable (scaled 1–4),

  • which established the intra-rater reliability (r = 0.94, p < 0.05), and
  • the criterion validity (r = 0.96, p < 0.05) for this instrument (Figure 2).



Figure 2. Proportion of shared variance in criterion validity (A) and inter-rater reliability (B) in the AHRQ Risk of Bias instrument revised as described.
Two raters were trained and standardized 
[20] with the revised AHRQ Risk of Bias and with the R-Wong instrument, which has been previously validated[24]. Each rater independently produced ratings on a sample of research reports with both instruments on two separate occasions, 1–2 months apart. Pearson correlation coefficient was used to compute the respective associations. The figure shows Venn diagrams to illustrate the intersection between each two sets data used in the correlations. The overlap between the sets in each panel represents the proportion of shared variance for that correlation. The percent of unexplained variance is given in the insert of each panel.

A similar revision of the Cochrane Risk of Bias tool may also yield promising validation data. G theory validation of both tools will follow. Together, these results will enable a critical and systematic dialectical comparison of the Cochrane and the AHRQ Risk of Bias measures.


The critical evaluation of the best available evidence is critical to patient-centered care, because biased research findings are fundamentally invalid and potentially harmful to the patient. Depending upon the tool of measurement, the validity of an instrument in a study is obtained by means of criterion validity through correlation coefficients. Criterion validity refers to the extent to which one measures or predicts the value of another measure or quality based on a previously well-established criterion. There are other domains of validity such as: construct validity and content validity that are rather more descriptive than quantitative. Reliability however is used to describe the consistency of a measure, the extent to which a measurement is repeatable. It is commonly assessed quantitatively by correlation coefficients. Inter-rater reliability is rendered as a Pearson correlation coefficient between two independent readers, and establishes equivalence of ratings produced by independent observers or readers. Intra-rater reliability is determined by repeated measurement performed by the same subject (rater/reader) at two different points in time to assess the correlation or strength of association of the two sets of scores.

To establish the reliability of research quality assessment tools it is necessary, as we previously noted [20]:

•a) to train multiple readers in sharing a common view for the cognitive interpretation of each item. Readers must possess declarative knowledge a factual form of information known to be static in nature a certain depth of knowledge and understanding of the facts about which they are reviewing the literature. They must also have procedural knowledge known as imperative knowledge that can be directly applied to a task in this case a clear understanding of the fundamental concepts of research methodology, design, analysis and inference.

•b) to train the readers to read and evaluate the quality of a set of papers independently and blindly. They must also be trained to self-monitor and self-assess their skills for the purpose of insuring quality control.

•c) to refine the process until the inter-rater correlation coefficient and Cohen coefficient of agreement are about 0.9 (over 81% shared variance). This will establishes that the degree of attained agreement among well-trained readers is beyond chance.

•d) to obtain independent and blind reading assessments from readers on reports under study.

•e) to compute means and standard deviation of scores for each question across the reports, repeat process if the coefficient of variations are greater than 5% (i.e., less than 5% error among the readers across each questions).

The quantification provided by instruments validated in such a manner to assess the quality and the relative lack of bias in the research evidence allows for the analysis of the scores by means of the acceptable sampling protocol. Acceptance sampling is a statistical procedure that uses statistical sampling to determine whether a given lot, in this case evidence gathered from an identified set of published reports, should be accepted or rejected [12,25]. Acceptable sampling of the best available evidence can be obtained by:

•convention: accept the top 10 percentile of papers based on the score of the quality of the evidence (e.g., low Risk of Bias);

•confidence interval (CI95): accept the papers whose scores fall at of beyond the upper confidence limit at 95%, obtained with mean and variance of the scores of the entire bibliome;

•statistical analysis: accept the papers that sustain sequential repeated Friedman analysis.

To be clear, the Friedman test is a non-parametric equivalent of the analysis of variance for factorial designs. The process requires the 4-E process outlined below:

•establishing a significant Friedman outcome, which indicates significant differences in scores among the individual reports being tested for quality;

•examining marginal means and standard deviations to identify inconsistencies, and to identify the uniformly strong reports across all the domains tested by the quality instrument

•excluding those reports that show quality weakness or bias

•executing the Friedman analysis again, and repeating the 4-E process as many times as necessary, in a statistical process akin to hierarchical regression, to eliminate the evidence reports that exhibit egregious weakness, based on the analysis of the marginal values, and to retain only the group of report that harbor homogeneously strong evidence.

Taken together, and considering the domain and the structure of both tools, expectations are that these analyses will confirm that these instruments are two related entities, each measuring distinct aspects of bias. We anticipate that future research will establish that both tools assess complementary sub-constructs of one and the same archetype meta-construct of research quality.


  1. Jiang F, Zhang J, Wang X, Shen X: Important steps to improve translation from medical research to health policy.

    J Trans Med 2013, 11:33. BioMed Central Full Text OpenURL

  2. Sung NS, Crowley WF Jr, Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, Larson EL, Scheinberg D, Reece EA, Slavkin H, Dobs A, Grebb J, Martinez RA, Korn A, Rimoin D:Central challenges facing the national clinical research enterprise.

    JAMA 2003, 289:1278-1287. PubMed Abstract | Publisher Full Text OpenURL

  3. Woolf SH: The meaning of translational research and why it matters.

    JAMA 2008, 299(2):211-213. PubMed Abstract | Publisher Full Text OpenURL

  4. Chiappelli F: From translational research to translational effectiveness: the “patient-centered dental home” model.

    Dental Hypotheses 2011, 2:105-112. Publisher Full Text OpenURL

  5. Maida C: Building communities of practice in comparative effectiveness research. In Comparative effectiveness and efficacy research and analysis for practice (CEERAP): applications for treatment options in health care. Edited by Chiappelli F, Brant X, Cajulis C. Heidelberg: Springer–Verlag; 2012.

    Chapter 1


  6. Agency for Healthcare Research and Quality: Budget estimates for appropriations committees, fiscal year (FY) 2008: performance budget submission for congressional justification.

    Performance budget overview 2008. webcite. Accessed 11 May 2013


  7. Westfall JM, Mold J, Fagnan L: Practice-based research—“blue highways” on the NIH roadmap.

    JAMA 2007, 297:403-406. PubMed Abstract | Publisher Full Text OpenURL

  8. Chiappelli F, Brant X, Cajulis C: Comparative effectiveness and efficacy research and analysis for practice (CEERAP) applications for treatment options in health care. Heidelberg: Springer–Verlag; 2012. OpenURL

  9. Dousti M, Ramchandani MH, Chiappelli F: Evidence-based clinical significance in health care: toward an inferential analysis of clinical relevance.

    Dental Hypotheses 2011, 2:165-177. Publisher Full Text OpenURL

  10. Campbell D, Stanley J: Experimental and quasi-experimental designs for research. Chicago, IL: Rand-McNally; 1963. OpenURL

  11. Littell JH, Corcoran J, Pillai V: Research synthesis reports and meta-analysis. New York, NY: Oxford Univeristy Press; 2008. OpenURL

  12. Chiappelli F: The science of research synthesis: a manual of evidence-based research for the health sciences. Hauppauge NY: NovaScience Publisher, Inc; 2008. OpenURL

  13. Higgins JPT, Green S: Cochrane handbook for systematic reviews of interventions version 5.0.1. Chichester, West Sussex, UK: John Wiley & Sons. The Cochrane collaboration; 2008. OpenURL

  14. CRD: Systematic Reviews: CRD’s guidance for undertaking reviews in health care. National Institute for Health Research (NIHR). University of York, UK: Center for reviews and dissemination; 2009. PubMed Abstract| Publisher Full Text OpenURL

  15. McDonald KM, Chang C, Schultz E: Closing the quality Gap: revisiting the state of the science. Summary report. U.S. Department of Health & Human Services. AHRQ, Rockville, MD: Summary report. AHRQ publication No. 12(13)-E017; 2013. OpenURL

Read Full Post »