Advertisements
Feeds:
Posts
Comments

Posts Tagged ‘Artificial Intelligence & Clinical Trials’


Multiple Barriers Identified Which May Hamper Use of Artificial Intelligence in the Clinical Setting

Reporter: Stephen J. Williams, PhD.

From the Journal Science:Science  21 Jun 2019: Vol. 364, Issue 6446, pp. 1119-1120

By Jennifer Couzin-Frankel

 

In a commentary article from Jennifer Couzin-Frankel entitled “Medicine contends with how to use artificial intelligence  the barriers to the efficient and reliable adoption of artificial intelligence and machine learning in the hospital setting are discussed.   In summary these barriers result from lack of reproducibility across hospitals. For instance, a major concern among radiologists is the AI software being developed to read images in order to magnify small changes, such as with cardiac images, is developed within one hospital and may not reflect the equipment or standard practices used in other hospital systems.  To address this issue, lust recently, US scientists and government regulators issued guidance describing how to convert research-based AI into improved medical images and published these guidance in the Journal of the American College of Radiology.  The group suggested greater collaboration among relevant parties in developing of AI practices, including software engineers, scientists, clinicians, radiologists etc. 

As thousands of images are fed into AI algorithms, according to neurosurgeon Eric Oermann at Mount Sinai Hospital, the signals they recognize can have less to do with disease than with other patient characteristics, the brand of MRI machine, or even how a scanner is angled.  For example Oermann and Mount Sinai developed an AI algorithm to detect spots on a lung scan indicative of pneumonia and when tested in a group of new patients the algorithm could detect pneumonia with 93% accuracy.  

However when the group from Sinai tested their algorithm from tens of thousands of scans from other hospitals including NIH success rate fell to 73-80%, indicative of bias within the training set: in other words there was something unique about the way Mt. Sinai does their scans relative to other hospitals.  Indeed, many of the patients Mt. Sinai sees are too sick to get out of bed and radiologists would use portable scanners, which generate different images than stand alone scanners.  

The results were published in Plos Medicine as seen below:

PLoS Med. 2018 Nov 6;15(11):e1002683. doi: 10.1371/journal.pmed.1002683. eCollection 2018 Nov.

Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.

Zech JR1, Badgeley MA2, Liu M2, Costa AB3, Titano JJ4, Oermann EK3.

Abstract

BACKGROUND:

There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task.

METHODS AND FINDINGS:

A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855-0.866) on the joint MSH-NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU (P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set (i.e., new data from within the hospital system used for training data; P values both <0.001). The highest internal performance was achieved by combining training and test data from MSH and NIH (AUC 0.931, 95% CI 0.927-0.936), but this model demonstrated significantly lower external performance at IU (AUC 0.815, 95% CI 0.745-0.885, P = 0.001). To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH-NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model (10× MSH risk P < 0.001; 10× NIH P = 0.002), but this outperformance failed to generalize to IU (MSH 10× P < 0.001; NIH 10× P = 0.027). CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system-specific biases.

CONCLUSION:

Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.

PMID: 30399157 PMCID: PMC6219764 DOI: 10.1371/journal.pmed.1002683

[Indexed for MEDLINE] Free PMC Article

Images from this publication.See all images (3)Free text

 

 

Surprisingly, not many researchers have begun to use data obtained from different hospitals.  The FDA has issued some guidance in the matter but considers “locked” AI software or unchanging software as a medical device.  However they just announced development of a framework for regulating more cutting edge software that continues to learn over time.

Still the key point is that collaboration over multiple health systems in various countries may be necessary for development of AI software which is used in multiple clinical settings.  Otherwise each hospital will need to develop their own software only used on their own system and would provide a regulatory headache for the FDA.

 

Other articles on Artificial Intelligence in Clinical Medicine on this Open Access Journal include:

Top 12 Artificial Intelligence Innovations Disrupting Healthcare by 2020

The launch of SCAI – Interview with Gérard Biau, director of the Sorbonne Center for Artificial Intelligence (SCAI).

Real Time Coverage @BIOConvention #BIO2019: Machine Learning and Artificial Intelligence #AI: Realizing Precision Medicine One Patient at a Time

50 Contemporary Artificial Intelligence Leading Experts and Researchers

 

Advertisements

Read Full Post »


Real Time Coverage @BIOConvention #BIO2019: Precision Medicine Beyond Oncology June 5 Philadelphia PA

Reporter: Stephen J Williams PhD @StephenJWillia2

Precision Medicine has helped transform cancer care from one-size-fits-all chemotherapy to a new era, where patients’ tumors can be analyzed and therapy selected based on their genetic makeup. Until now, however, precision medicine’s impact has been far less in other therapeutic areas, many of which are ripe for transformation. Efforts are underway to bring the successes of precision medicine to neurology, immunology, ophthalmology, and other areas. This move raises key questions of how the lessons learned in oncology can be used to advance precision medicine in other fields, what types of data and tools will be important to personalizing treatment in these areas, and what sorts of partnerships and payer initiatives will be needed to support these approaches and their ultimate commercialization and use. The panel will also provide an in depth look at precision medicine approaches aimed at better understanding and improving patient care in highly complex disease areas like neurology.
Speaker panel:  The big issue now with precision medicine is there is so much data and hard to put experimental design and controls around randomly collected data.
  • The frontier is how to CURATE randomly collected data to make some sense of it
  • One speaker was at a cancer meeting and the oncologist had no idea what to make of genomic reports they were given.  Then there is a lack of action or worse a misdiagnosis.
  • So for e.g. with Artificial Intelligence algorithms to analyze image data you can see things you can’t see with naked eye but if data quality not good the algorithms are useless – if data not curated properly data is wasted
Data needs to be organized and curated. 
If relying of AI for big data analysis the big question still is: what are the rates of false negative and false positives?  Have to make sure so no misdiagnosis.

Please follow LIVE on TWITTER using the following @ handles and # hashtags:

@Handles

@pharma_BI

@AVIVA1950

@BIOConvention

# Hashtags

#BIO2019 (official meeting hashtag)

Read Full Post »


 

Live Coverage: MedCity Converge 2018 Philadelphia: AI in Cancer and Keynote Address

Reporter: Stephen J. Williams, PhD

8:30 AM -9:15

Practical Applications of AI in Cancer

We are far from machine learning dictating clinical decision making, but AI has important niche applications in oncology. Hear from a panel of innovative startups and established life science players about how machine learning and AI can transform different aspects in healthcare, be it in patient recruitment, data analysis, drug discovery or care delivery.

Moderator: Ayan Bhattacharya, Advanced Analytics Specialist Leader, Deloitte Consulting LLP
Speakers:
Wout Brusselaers, CEO and Co-Founder, Deep 6 AI @woutbrusselaers ‏
Tufia Haddad, M.D., Chair of Breast Medical Oncology and Department of Oncology Chair of IT, Mayo Clinic
Carla Leibowitz, Head of Corporate Development, Arterys @carlaleibowitz
John Quackenbush, Ph.D., Professor and Director of the Center for Cancer Computational Biology, Dana-Farber Cancer Institute

Ayan: working at IBM and Thompon Rueters with structured datasets and having gone through his own cancer battle, he is now working in healthcare AI which has an unstructured dataset(s)

Carla: collecting medical images over the world, mainly tumor and calculating tumor volumetrics

Tufia: drug resistant breast cancer clinician but interested in AI and healthcareIT at Mayo

John: taking large scale datasets but a machine learning skeptic

moderator: how has imaging evolved?

Carla: ten times images but not ten times radiologists so stressed field needs help with image analysis; they have seen measuring lung tumor volumetrics as a therapeutic diagnostic has worked

moderator: how has AI affected patient recruitment?

Tufia: majority of patients are receiving great care but AI can offer profiles and determine which patients can benefit from tertiary care;

John: 1980 paper on no free lunch theorem; great enthusiasm about optimization algortihisms fell short in application; can extract great information from e.g. images

moderator: how is AI for healthcare delivery working at mayo?

Tufia: for every hour with patient two hours of data mining. for care delivery hope to use the systems to leverage the cognitive systems to do the data mining

John: problem with irreproducible research which makes a poor dataset:  also these care packages are based on population data not personalized datasets; challenges to AI is moving correlation to causation

Carla: algorithisms from on healthcare network is not good enough, Google tried and it failed

John: curation very important; good annotation is needed; needed to go in and develop, with curators, a systematic way to curate medial records; need standardization and reproducibility; applications in radiometrics can be different based on different data collection machines; developed a machine learning model site where investigators can compare models on a hub; also need to communicate with patients on healthcare information and quality information

Ayan: Australia and Canada has done the most concerning AI and lifescience, healthcare space; AI in most cases is cognitive learning: really two types of companies 1) the Microsofts, Googles, and 2) the startups that may be more pure AI

 

Final Notes: We are at a point where collecting massive amounts of healthcare related data is simple, rapid, and shareable.  However challenges exist in quality of datasets, proper curation and annotation, need for collaboration across all healthcare stakeholders including patients, and dissemination of useful and accurate information

 

9:15 AM–9:45 AM

Opening Keynote: Dr. Joshua Brody, Medical Oncologist, Mount Sinai Health System

The Promise and Hype of Immunotherapy

Immunotherapy is revolutionizing oncology care across various types of cancers, but it is also necessary to sort the hype from the reality. In his keynote, Dr. Brody will delve into the history of this new therapy mode and how it has transformed the treatment of lymphoma and other diseases. He will address the hype surrounding it, why so many still don’t respond to the treatment regimen and chart the way forward—one that can lead to more elegant immunotherapy combination paths and better outcomes for patients.

Speaker:
Joshua Brody, M.D., Assistant Professor, Mount Sinai School of Medicine @joshuabrodyMD

Director Lymphoma therapy at Mt. Sinai

  • lymphoma a cancer with high PD-L1 expression
  • hodgkin’s lymphoma best responder to PD1 therapy (nivolumab) but hepatic adverse effects
  • CAR-T (chimeric BCR and TCR); a long process which includes apheresis, selection CD3/CD28 cells; viral transfection of the chimeric; purification
  • complete remissions of B cell lymphomas (NCI trial) and long term remissions past 18 months
  • side effects like cytokine release (has been controlled); encephalopathy (he uses a hand writing test to see progression of adverse effect)

Vaccines

  •  teaching the immune cells as PD1 inhibition exhausting T cells so a vaccine boost could be an adjuvant to PD1 or checkpoint therapy
  • using Flt3L primed in-situ vaccine (using a Toll like receptor agonist can recruit the dendritic cells to the tumor and then activation of T cell response);  therefore vaccine does not need to be produced ex vivo; months after the vaccine the tumor still in remission
  • versus rituximab, which can target many healthy B cells this in-situ vaccine strategy is very specific for the tumorigenic B cells
  • HoWEVER they did see resistant tumor cells which did not overexpress PD-L1 but they did discover a novel checkpoint (cannot be disclosed at this point)

 

 

 

 

 

 

 

 

 

Please follow on Twitter using the following #hashtags and @pharma_BI

#MCConverge

#AI

#cancertreatment

#immunotherapy

#healthIT

#innovation

#precisionmedicine

#healthcaremodels

#personalizedmedicine

#healthcaredata

And at the following handles:

@pharma_BI

@medcitynews

 

Please see related articles on Live Coverage of Previous Meetings on this Open Access Journal

LIVE – Real Time – 16th Annual Cancer Research Symposium, Koch Institute, Friday, June 16, 9AM – 5PM, Kresge Auditorium, MIT

Real Time Coverage and eProceedings of Presentations on 11/16 – 11/17, 2016, The 12th Annual Personalized Medicine Conference, HARVARD MEDICAL SCHOOL, Joseph B. Martin Conference Center, 77 Avenue Louis Pasteur, Boston

Tweets Impression Analytics, Re-Tweets, Tweets and Likes by @AVIVA1950 and @pharma_BI for 2018 BioIT, Boston, 5/15 – 5/17, 2018

BIO 2018! June 4-7, 2018 at Boston Convention & Exhibition Center

https://pharmaceuticalintelligence.com/press-coverage/

Read Full Post »


Tweets for AI and Machine Learning in Clinical Trials April 12th, 2018 hosted at Pfizer’s Innovation Research Lab in Cambridge, MA @AVIVA1950 @pharma_BI

 

 

 

Read Full Post »