Posts Tagged ‘pneumonia’

Multiple Barriers Identified Which May Hamper Use of Artificial Intelligence in the Clinical Setting

Reporter: Stephen J. Williams, PhD.

From the Journal Science:Science  21 Jun 2019: Vol. 364, Issue 6446, pp. 1119-1120

By Jennifer Couzin-Frankel


In a commentary article from Jennifer Couzin-Frankel entitled “Medicine contends with how to use artificial intelligence  the barriers to the efficient and reliable adoption of artificial intelligence and machine learning in the hospital setting are discussed.   In summary these barriers result from lack of reproducibility across hospitals. For instance, a major concern among radiologists is the AI software being developed to read images in order to magnify small changes, such as with cardiac images, is developed within one hospital and may not reflect the equipment or standard practices used in other hospital systems.  To address this issue, lust recently, US scientists and government regulators issued guidance describing how to convert research-based AI into improved medical images and published these guidance in the Journal of the American College of Radiology.  The group suggested greater collaboration among relevant parties in developing of AI practices, including software engineers, scientists, clinicians, radiologists etc. 

As thousands of images are fed into AI algorithms, according to neurosurgeon Eric Oermann at Mount Sinai Hospital, the signals they recognize can have less to do with disease than with other patient characteristics, the brand of MRI machine, or even how a scanner is angled.  For example Oermann and Mount Sinai developed an AI algorithm to detect spots on a lung scan indicative of pneumonia and when tested in a group of new patients the algorithm could detect pneumonia with 93% accuracy.  

However when the group from Sinai tested their algorithm from tens of thousands of scans from other hospitals including NIH success rate fell to 73-80%, indicative of bias within the training set: in other words there was something unique about the way Mt. Sinai does their scans relative to other hospitals.  Indeed, many of the patients Mt. Sinai sees are too sick to get out of bed and radiologists would use portable scanners, which generate different images than stand alone scanners.  

The results were published in Plos Medicine as seen below:

PLoS Med. 2018 Nov 6;15(11):e1002683. doi: 10.1371/journal.pmed.1002683. eCollection 2018 Nov.

Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.

Zech JR1, Badgeley MA2, Liu M2, Costa AB3, Titano JJ4, Oermann EK3.



There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task.


A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855-0.866) on the joint MSH-NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU (P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set (i.e., new data from within the hospital system used for training data; P values both <0.001). The highest internal performance was achieved by combining training and test data from MSH and NIH (AUC 0.931, 95% CI 0.927-0.936), but this model demonstrated significantly lower external performance at IU (AUC 0.815, 95% CI 0.745-0.885, P = 0.001). To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH-NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model (10× MSH risk P < 0.001; 10× NIH P = 0.002), but this outperformance failed to generalize to IU (MSH 10× P < 0.001; NIH 10× P = 0.027). CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system-specific biases.


Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.

PMID: 30399157 PMCID: PMC6219764 DOI: 10.1371/journal.pmed.1002683

[Indexed for MEDLINE] Free PMC Article

Images from this publication.See all images (3)Free text



Surprisingly, not many researchers have begun to use data obtained from different hospitals.  The FDA has issued some guidance in the matter but considers “locked” AI software or unchanging software as a medical device.  However they just announced development of a framework for regulating more cutting edge software that continues to learn over time.

Still the key point is that collaboration over multiple health systems in various countries may be necessary for development of AI software which is used in multiple clinical settings.  Otherwise each hospital will need to develop their own software only used on their own system and would provide a regulatory headache for the FDA.


Other articles on Artificial Intelligence in Clinical Medicine on this Open Access Journal include:

Top 12 Artificial Intelligence Innovations Disrupting Healthcare by 2020

The launch of SCAI – Interview with Gérard Biau, director of the Sorbonne Center for Artificial Intelligence (SCAI).

Real Time Coverage @BIOConvention #BIO2019: Machine Learning and Artificial Intelligence #AI: Realizing Precision Medicine One Patient at a Time

50 Contemporary Artificial Intelligence Leading Experts and Researchers


Read Full Post »

Rapid diagnosis of septicemia

Larry H. Bernstein, MD, FCAP, Curator



Rapid Diagnosis of Infection in the Critically Ill, a Multicenter Study of Molecular Detection in Bloodstream Infections, Pneumonia, and Sterile Site Infections

Jean-Louis Vincent,  David Brealey, Nicolas Libert, Nour Elhouda Abidi, Michael O’Dwyer, Kai Zacharowski, Malgorzata Mikaszewska-Sokolewicz, et al.
Crit Care Med. 2015;43(11):2283-2291.     http://www.medscape.com/viewarticle/853105

Objective: Early identification of causative microorganism(s) in patients with severe infection is crucial to optimize antimicrobial use and patient survival. However, current culture-based pathogen identification is slow and unreliable such that broad-spectrum antibiotics are often used to insure coverage of all potential organisms, carrying risks of overtreatment, toxicity, and selection of multidrug-resistant bacteria. We compared the results obtained using a novel, culture-independent polymerase chain reaction/electrospray ionization-mass spectrometry technology with those obtained by standard microbiological testing and evaluated the potential clinical implications of this technique.

Design: Observational study.

Setting: Nine ICUs in six European countries.

Patients: Patients admitted between October 2013 and June 2014 with suspected or proven bloodstream infection, pneumonia, or sterile fluid and tissue infection were considered for inclusion.

Interventions: None.

Measurements and Main Results: We tested 616 bloodstream infection, 185 pneumonia, and 110 sterile fluid and tissue specimens from 529 patients. From the 616 bloodstream infection samples, polymerase chain reaction/electrospray ionization-mass spectrometry identified a pathogen in 228 cases (37%) and culture in just 68 (11%). Culture was positive and polymerase chain reaction/electrospray ionization-mass spectrometry negative in 13 cases, and both were negative in 384 cases, giving polymerase chain reaction/electrospray ionization-mass spectrometry a sensitivity of 81%, specificity of 69%, and negative predictive value of 97% at 6 hours from sample acquisition. The distribution of organisms was similar with both techniques. Similar observations were made for pneumonia and sterile fluid and tissue specimens. Independent clinical analysis of results suggested that polymerase chain reaction/electrospray ionization-mass spectrometry technology could potentially have resulted in altered treatment in up to 57% of patients.

Conclusions: Polymerase chain reaction/electrospray ionization-mass spectrometry provides rapid pathogen identification in critically ill patients. The ability to rule out infection within 6 hours has potential clinical and economic benefits.


The availability of rapid and reliable infectious disease diagnostics that can provide results directly from patient specimens represents a major unmet need in managing critically ill patients. Current sepsis guidelines recommend initiation of IV antibiotic therapy as early as possible, ideally within the first hour,[1] as any delay in effective antimicrobial therapy may result in decreased survival.[2] Effective therapy requires that the identity of causative pathogens and their resistance patterns are known. However, the current standard-of-care, which depends on blood culture-based initial diagnosis, often takes at least 48–72 hours to provide a result. Furthermore, cultures often remain negative even when bacterial or fungal infections are strongly suspected,[3] in part, related to concurrent antibiotic treatment.[4]

Molecular diagnostic techniques that do not depend on growth of organisms in culture may offer a distinct advantage over current methods. Most of the recently described molecular methods, however, rely on culture amplification as a precursor to diagnosis.[5–8] Although these techniques may accelerate diagnosis for positive cultures, they do not address the significant proportion of false-negative cultures observed in patients with sepsis. In addition, many of these methods also use targeted pathogen detection with limited pathogen coverage such that negative results are often not highly predictive.

Polymerase chain reaction followed by electrospray ionization-mass spectrometry (PCR/ESI-MS) can detect more than 800 bloodstream infection (BSI)-relevant pathogens in a single assay and in approximately 6 hours.[9–13] It can also identify three classes of antibiotic resistance markers associated with resistance to methicillin (mecA), vancomycin (vanA/vanB), and carbapenems (KPC). Using this technique, we recently demonstrated 83% sensitivity and 94% specificity compared with culture for direct detection of pathogens in whole-blood specimens from patients with suspected BSIs.[13]

Here, we describe findings from the multicenter observational Rapid Diagnosis of Infections in the Critically Ill (RADICAL) study. The primary objective was to compare results obtained using the novel culture-independent PCR/ESI-MS technology with those obtained from standard microbiological testing as a measure of clinical performance. Secondarily, to broadly address the clinical value of PCR/ESI-MS detections, a panel of independent clinical adjudicators was used to identify changes in patient management that may have occurred had the results from the PCR/ESI-MS technology been available for clinical use and assumed to be correct.

Patient Inclusion and Exclusion Criteria

Patients were considered for inclusion if they had 1) suspected or proven severe infection or sepsis and or 2) suspected or proven healthcare-associated pneumonia (HAP/HCAP), ventilator-associated pneumonia (VAP), or severe community-acquired pneumonia (sCAP). Because pneumonia is the most common precipitating cause of sepsis, there may be an overlap between these two populations, but patients were included in one of the two groups, not both. Pneumonia (HAP/HCAP, VAP, and sCAP) was diagnosed in patients with an endotracheal tube in situ and a new infiltrate on chest radiograph plus temperature more than 38°C or less than 35°C, increased sputum production, increased or decreased WBC count (> 12 or < 4 cells/mL3), or a clinical suspicion of pneumonia, and the treating clinician expected the patient to still be intubated the next day.

The following exclusion criteria were used: the treating clinician expected the patient to be discharged from the ICU on the day of evaluation or the following day, the treatment intent was palliative, the clinician was not committed to aggressive treatment, or death was deemed imminent and inevitable. Patients who had previously been included, but were readmitted to the ICU during the same hospitalization, were not included a second time.


Data Analysis

Results obtained with the PCR/ESI-MS technology for each specimen were compared with those obtained using conventional microbiology methods for the same sample. If multiple specimens were taken from a patient per standard-of-care protocols, each was independently analyzed in this study. Agreement and concordance were assessed using a McNemar test[16] and Cohen κ.[17] All percentages and CIs for proportions were calculated using the exact method and are rounded to the nearest percentage. Direct comparison of positive and negative results was conducted with organism identification for each method (conventional microbiology vs PCR/ESI-MS) for each specimen type. Coagulase-negative staphylococcus and other common skin contaminants were annotated as “potential contaminants” for both methods and excluded from the overall analysis, as previously described.[13]

Discrepant results between the PCR/ESI-MS and culture cannot be directly confirmed by an independent method, as previously described.[13] Two approaches were used to resolve discrepancies. In a subset of patients, multiple samples were collected per standard-of-care. This included two independent fresh venipunctures (left arm vs right arm) or one venipuncture plus one sample collected from an indwelling line. Paired analysis of PCR/ESI-MS testing results between these independently collected samples was conducted to indicate the likelihood of true infection. In addition, independent clinical adjudication (described below) was performed using all the clinical data collected as part of the study, including standard-of-care microbiology results and PCR/ESI-MS results.



Of 543 patients enrolled in the study, 14 did not have matching PCR/ESI-MS or standard-of-care microbiology results and were excluded from the final analysis. Table 1 shows the patient demographics, reflecting a typically heterogeneous ICU population: one third of the patients were admitted from the emergency department; 75% were exposed to one or more antibiotics prior to study enrolment. Overall mortality was 29%, with cardiac arrest, septic shock, multiple organ failure, and acute respiratory distress syndrome accounting for ~62% of deaths.

BSI Analysis

A total of 616 direct whole-blood specimens from the 529 patients were tested to assess the accuracy of organism identification. PCR/ESI-MS results from analysis of blood using the bacteria, antibiotic resistance, and Candida BSI assay were compared with results from standard clinical microbiology cultures. As shown in Table 2, there were 228 PCR/ESI-MS positive specimens (36.5%) for at least one pathogen compared with 68 positive specimens by culture (10.9%). The total number of positive tests for each method was statistically different (McNemar test statistic = 137.6; df = 1; p < 0.0001). There were 55 samples that were positive for the same organism with both techniques (Table 2), yielding an overall concordance of identification (calculated sensitivity) of 81% (95% CI, 70–89%) and a κ value of 0.25 (95% CI, 0.18–0.31). In 13 instances, culture identified an organism that was either negative by PCR/ESI-MS (6/13) or the identity of the organism reported by PCR/ESI-MS did not match the organism identified by microbiology testing (7/13) (Table S1, Supplemental Digital Content 1,http://links.lww.com/CCM/B418). In contrast, PCR/ESI-MS reported a BSI-relevant organism in 173 additional specimens that were culture negative, resulting in a calculated assay specificity of 69% (Table S2, Supplemental Digital Content 1, http://links.lww.com/CCM/B418). Finally, there were 384 concordant negative specimens, yielding a negative predictive value (NPV) of ~97% (95% CI, 94–98%).

The frequencies of organisms detected from BSI specimens are shown in Figure 1. The distributions of the top 10 species detected by microbiology and those detected by PCR/ESI-MS were similar. The largest single discrepancy between the two methods by sheer volume of detections was in the identification of Escherichia coli. Although culture and PCR/ESI-MS techniques both reported E. coli as the most abundant species, PCR/ESI-MS detection was 4-fold higher (89 vs 21). Other organisms in which blood culture performed less well included the enterococcus species, Enterococcus faecalis (1 vs 10) and Enterococcus faecium (2 vs 25), Candida albicans (2 vs 13), and Staphylococcus aureus (14 vs 31). In contrast, Pseudomonas aeruginosa detection was comparable between the two methods (6 vs 8). Additional analysis of the PCR/ESI-MS results showed that the levels (genome equivalent/mL) of organisms reported in the majority of these PCR/ESI-MS positive, but culture-negative, cases were similar to cases in which culture matched PCR/ESI-MS detections (data not shown).

Figure 1.


Bacteria and Candida detected in the Rapid Diagnosis of Infections in the Critically Ill (RADICAL) study. Distribution of organisms reported by polymerase chain reaction/electrospray ionization-mass spectrometry (PCR/ESI-MS) (blue bar) and culture (red bar) observed in the RADICAL study are shown, sorted by decreasing order of PCR/ESI-MS reported organisms. Both methods showed similar distribution for the top eight reportable organisms that were seen >5 times by PCR/ESI-MS, with some minor reshuffling of the order. PCR/ESI-MS showed a longer tail of reportable organisms that were infrequent (≤5 times). Normal skin flora are shown below the line were not included in further analysis by either method.


Nonbloodstream Infections

Heterogeneous samples from patients with suspected pneumonia or sterile site infections were also obtained in several cases. Overall, there were 185 LRT samples (88 BAL, 96 ETA, and 1 other) and 110 SF&T samples (36 intraperitoneal fluid, 14 pleural fluid, 11 CSF, 13 tissue, and 36 other fluid types). Results from the analysis of these specimens are shown in Table 3. LRT and SF&T specimens often had multiple detections reported by both methods in several samples. Only the primary detections by either method were included in the analysis. The overall sensitivities for concordance between standard-of-care and PCR/ESI-MS were 84% (95% CI, 74–91%) and 85% (95% CI, 72–93%), respectively. As for the bloodstream infection data, the McNemar test for both the LRT and the SF&T sample data showed that the total number of samples considered positive was significantly different for culture versus PCR/ESI-MS (McNemar test statistic = 20.9 for LRT and 15.2 for SF&T; p < 0.0001 in both cases). Also similar to the bloodstream infection data, there was more agreement in the contingency table comparing culture to PCR/ESI-MS than would be expected by chance (LRT κ = 0.35; 95% confidence limits, 0.23, 0.47 and SF&T κ = 0.27; 95% confidence limits, 0.11, 0.43). For LRT specimens, there was no statistically significant difference in sensitivity (p = 0.677) or specificity (p = 0.444) when testing the hypothesis that the BAL proportion – the ETA proportion was equal to zero.

In 151 patients, two or more specimen types (BSI plus LRT and/or SF&T) were obtained and analyzed. In 86 of these 151 cases (57%), the same organisms were reported by PCR/ESI-MS in all samples tested from an individual patient (data not shown). In comparison, culture concordance between the sample types was seen in only 19 cases (12%), driven largely by no detection reported in the BSI culture results.

Resistance Markers

There were no identified cases of Klebsiella-associated carbapenemase. There was a single report of vancomycin-resistant Enterococci, which was matched across the two methods. There were 23 reports of mecA+ staphylococcus organisms (seven in BSI samples, 13 in LRT samples, and three in SF&T samples), with the following agreement between PCR/ESI-MS and culture: for BSI samples, results were concordant in four cases, and PCR/ESI-MS was positive and culture negative in three; for LRT samples, results were concordant in three cases, PCR/ESI-MS was positive with culture negative in nine, and PCR/ESI-MS was negative with culture positive in one; and the three cases in the SF&T samples were concordant across PCR/ESI-MS and culture.




The important findings of the RADICAL study are that PCR/ESI-MS detected BSI pathogens with high overall sensitivity and NPV; PCR/ESI-MS was three times more likely to identify an organism than standard culture; and, if available, PCR/ESI-MS results may have altered the treatment regimen in as many as 57% of patients.

Sepsis affects a large proportion of the critically ill population. Despite improvements in recent years, morbidity and mortality rates remain high.[18] The importance of initiating treatment as soon as possible has been highlighted and shown to be associated with improved outcomes,[2] yet this finding needs to be balanced against the direct risks and stewardship issues arising from overzealous or inappropriate antibiotic use.

Rapid diagnosis of severe infection or sepsis is thus crucial not only to optimize a patient’s chances of survival but also to encourage responsible antibiotic use. However, diagnosing infection accurately in critically ill patients is challenging. Characteristic clinical and laboratory signs of severe infection, such as tachycardia, fever, and altered WBC count, are nonspecific and are often present in other acute conditions. Biomarkers, such as C-reactive protein and procalcitonin, are also nonspecific and are of more value in ruling out infection than in making a definite diagnosis.[19] Microbiological culture results are negative in many patients with sepsis, largely because prior antimicrobial therapy affects ex vivo growth in culture medium.[1] Certain microorganisms are also particularly difficult to culture, requiring specific growth media or a particular environment. As culture results often require several days to become available, patients with suspected severe infection are, therefore, often started on empiric broad-spectrum antibiotics to increase the likelihood that a pathogenic organism will be adequately covered. This approach, although valid in terms of preventing delays in starting treatment with currently available diagnostic techniques, has several negative aspects, including the potential for toxicity with multiple antibiotics, the high-associated costs, and the effects of antibiotic pressure on the development of antimicrobial resistance.[20]

Availability of a technique that could provide more rapid pathogen identification directly from patient samples could, therefore, represent a marked improvement in terms of enabling more rapid diagnosis and earlier initiation of appropriate antimicrobial therapy, with associated beneficial effects on outcomes, antimicrobial resistance, costs, and toxicity. Various methods have been suggested for this purpose, including single pathogen assays, which are of limited use in patients with suspected sepsis in whom multiple organisms may be involved; selected-pathogen assays, which use specific molecular targets to identify some 20–35 species;[21–23] and broad-range pathogen assays, which use universal or conserved targets to identify many hundreds of species, but for which earlier versions lacked sensitivity due to the small volumes of blood extracted for analysis.[24,25]


Importantly, in 41% of cases, the panel of independent experts would have recommended a change in management, including initiation of therapy, altered antimicrobial spectrum, and/or change in duration of therapy, based on the PCR/ESI-MS results. This percentage increased to 57 when PCR/ESI-MS tests were positive.

Second, the greater detection rate of E. coli, S. aureus, E. faecium, C. albicans, and Klebsiella pneumoniae by PCR/ESI-MS compared with routine culture was unanticipated, and the explanation is unclear. Prior to study inclusion, most patients were exposed to combinations of two or more antibiotics active against Gram-positive and Gram-negative organisms and were often receiving one or more antifungals in addition. As stated above, the bacterium/fungus may have been largely cleared with preexisting antibiotics, hence the negative culture results, but remaining DNA remnants in the circulation may have been sufficient to give a positive PCR/ESI-MS. The sensitivity of the technique increases the risk of identifying contaminants and commensals; however, the pathogens most frequently detected in the study are those associated with infection. Accepting the validity of these data, the PCR/ESI-MS test could be of importance to help target antimicrobial therapy in patients who have already started antimicrobials and have negative cultures (salvage microbiology).[29] Further, ideally interventional, studies are warranted to confirm and further explore these findings.

Read Full Post »