Leaders in Pharmaceutical Business Intelligence (LPBI) Group

5/26/2023

Evaluation of ChatGPT for information for cancer patients: answering questions regarding misconceptions

Reporter: Stephen J. Williams, Ph.D.

The National Cancer Institute (NCI) maintains an excellent and comprehensive website for patient information, complete with common questions and answers that patients may have regarding their cancer type and therapy.  Their page, “Common Cancer Myths and Misconceptions” is meant to alleviate a cancer patient’s fears by providing correct information which answers any misconceptions or misinformation a patient might have concerning their treatment and prognosis.  The site is well visited but, in many instances, patients are overwhelmed by the load of information. Therefore, a chatbot may alleviate the patient’s trepidations in interacting with the site and extracting the correct information they need.  A paper in JNCI “Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information” used the information on this page as source of training for ChatGPT and tested the output for correct information given common queiries.   The results showed answers supplied by ChatGPT were accurate as analyzed by a team of cancer experts.  Therefore ChatGPT, trained on NCI information, may provide an excellent ChatBot experience for new cancer patients and alleviate fears and misconceptions concerning their tumor type.

Paper citation: Johnson SB, King AJ, Warner EL, Aneja S, Kann BH, Bylund CL. Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information. JNCI Cancer Spectr. 2023 Mar 1;7(2):pkad015. doi: 10.1093/jncics/pkad015. PMID: 36929393; PMCID: PMC10020140.

Abstract

Data about the quality of cancer information that chatbots and other artificial intelligence systems provide are limited. Here, we evaluate the accuracy of cancer information on ChatGPT compared with the National Cancer Institute’s (NCI’s) answers by using the questions on the “Common Cancer Myths and Misconceptions” web page. The NCI’s answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy (accurate: yes vs no). Ratings were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Additionally, word count and Flesch-Kincaid readability grade level for each individual response were evaluated. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs for questions 1 through 13 (ĸ = ‒0.03, standard error = 0.08). There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Overall, the results suggest that ChatGPT provides accurate information about common cancer myths and misconceptions.

The queries are given below as the Table 1 in thier paper (source: https://pubmed.ncbi.nlm.nih.gov/36929393/)

Table 1.

Cancer myths and misconceptions questions, answers, and descriptive information^a

Question	NCI answer	NCI WC	NCI FKG	ChatGPT answer	C WC	C FKG
Is cancer a death sentence?	In the United States, the likelihood of dying from cancer has dropped steadily since the 1990s. Five-year survival rates for some cancers, such as breast, prostate, and thyroid cancers, now are 90% or better. The 5-year survival rate for all cancers combined is currently about 68%. For more information, see the Annual Report to the Nation on the Status of Cancer. It is important to note, however, that these rates are based on data from large numbers of people. How long an individual cancer patient will live and whether he or she will die from the disease depend on many factors, including whether the cancer is slow or fast growing, how much the cancer has spread in the body, whether effective treatments are available, the person’s overall health, and more.	133	15.8	Cancer is not necessarily a death sentence. The prognosis for people with cancer depends on many factors, including the type and stage of the cancer, the person’s overall health, and the effectiveness of treatment. Some types of cancer can be cured, while others may be managed through treatments such as surgery, chemotherapy, radiation, or targeted therapies. In some cases, people with cancer may live many years with the disease, while in other cases, cancer may be more aggressive and may progress quickly. It is important to work with a healthcare team to determine the best treatment plan based on the specifics of your situation.	104	11.4
Will eating sugar make my cancer worse?	No. Although research has shown that cancer cells consume more sugar (glucose) than normal cells, no studies have shown that eating sugar will make your cancer worse or that, if you stop eating sugar, your cancer will shrink or disappear. However, a high-sugar diet may contribute to excess weight gain, and obesity is associated with an increased risk of developing several types of cancer. For more information, see the NCI fact sheet on Obesity and Cancer.	76	12.7	There is no scientific evidence to support the idea that eating sugar can directly cause cancer or make it worse. However, there is some evidence to suggest that high sugar intake may be associated with an increased risk of certain types of cancer, such as breast and colon cancer. This is because a diet high in sugar is often associated with being overweight or obese, and carrying excess weight has been linked to an increased risk of cancer. Additionally, high sugar intake may be linked to inflammation in the body, which has also been associated with an increased risk of cancer. However, it is important to note that the link between sugar intake and cancer risk is still being studied, and more research is needed to understand the relationship between these factors. It is generally recommended to limit your intake of added sugars, such as those found in sugary drinks, sweets, and baked goods, as part of a healthy diet. This is because a diet high in added sugars has been linked to an increased risk of several health problems, including tooth decay, type 2 diabetes, and heart disease. Instead of sugary foods, it is important to focus on eating a diet that is rich in a variety of vegetables, fruits, whole grains, and lean protein sources. This can help you maintain a healthy weight and reduce your risk of chronic diseases, including cancer.	234	13.7
Do artificial sweeteners cause cancer?	No. Researchers have conducted studies on the safety of the artificial sweeteners (sugar substitutes) saccharin (Sweet ‘N Low®, Sweet Twin®, NectaSweet®); cyclamate; aspartame (Equal®, NutraSweet®); acesulfame potassium (Sunett®, Sweet One®); sucralose (Splenda®); and neotame and found no evidence that they cause cancer in humans. All of these artificial sweeteners except for cyclamate have been approved by the Food and Drug Administration for sale in the United States. For more information, see the NCI fact sheet on Artificial Sweeteners and Cancer.	80	9.5	There is no clear evidence that artificial sweeteners cause cancer. Some studies have suggested a link between the use of artificial sweeteners and an increased risk of certain types of cancer, while other studies have not found such a link. The results of these studies have been mixed, and more research is needed to understand the relationship between artificial sweeteners and cancer. The U.S. Food and Drug Administration (FDA) has approved several artificial sweeteners as safe for consumption, including aspartame, saccharin, and sucralose. These sweeteners have been extensively studied and have been found to be safe for human consumption at the levels typically consumed. However, it is important to note that some people may be sensitive to certain artificial sweeteners and may experience side effects such as headaches, digestive issues, or allergic reactions. If you are concerned about the safety of artificial sweeteners, it is always a good idea to talk to your healthcare provider for guidance.	157	14.9
Is cancer contagious?	In general, no. Cancer is not a contagious disease that easily spreads from person to person. The only situation in which cancer can spread from one person to another is in the case of organ or tissue transplantation. A person who receives an organ or tissue from a donor who had cancer in the past may be at increased risk of developing a transplant-related cancer in the future. However, that risk is extremely low—about two cases of cancer per 10,000 organ transplants. Doctors avoid the use of organs or tissue from donors who have a history of cancer. In some people, cancers may be caused by certain viruses (some types of human papillomavirus, or HPV, for example) and bacteria (such as Helicobacter pylori). While a virus or bacterium can spread from person to person, the cancers they sometimes cause cannot spread from person to person. For more information about cancer-causing viruses and bacteria, see the NCI fact sheet on Helicobacter pylori and Cancer and our pages on HPV and Cancer and Infectious Agents.	174	13.9	Cancer is not a contagious disease, meaning it cannot be transmitted from person to person like a cold or the flu. Cancer is a disease in which abnormal cells divide and grow uncontrollably, forming tumors. There are many different types of cancer, and each type can have a different cause. Some types of cancer are caused by genetic factors, while others may be caused by exposure to certain substances, such as tobacco smoke or radiation. Some types of cancer may also be caused by infections, such as the human papillomavirus (HPV), which can cause certain types of cancer, including cervical cancer and some types of head and neck cancer. However, these infections are not transmitted through casual contact and are not contagious in the same way that infectious diseases are.	130	11.2
Does my attitude—positive or negative—determine my risk of getting or surviving cancer?	To date, there is no convincing scientific evidence that links a person’s “attitude” to their risk of developing or dying from cancer. If you have cancer, it’s normal to feel sad, angry, or discouraged sometimes and positive or upbeat at other times. People with a positive attitude may be more likely to maintain social connections and stay active, and physical activity and emotional support may help you cope with your cancer. For more information, see the NCI fact sheet on Stress and Cancer.	83	11.2	It is not clear whether having a positive or negative attitude directly affects the risk of developing cancer or the likelihood of surviving cancer. While it is important to maintain a positive outlook and to have a support system during cancer treatment, there is no scientific evidence to suggest that a positive attitude can prevent cancer or improve survival. Cancer is a complex disease that is influenced by a variety of factors, including genetics, environmental exposures, and lifestyle habits. While it is important to take care of your overall health and to adopt healthy habits that may reduce your risk of cancer, it is not possible to control all of the factors that may influence your risk of developing cancer. It is also important to recognize that cancer and its treatment can be physically and emotionally challenging. It is normal to experience a range of emotions during this time, and it is important to take care of your mental health and seek support if you need it.	167	20.2
Can cancer surgery or a tumor biopsy cause cancer to spread in the body?	The chance that surgery will cause cancer to spread to other parts of the body is extremely low. Following standard procedures, surgeons use special methods and take many steps to prevent cancer cells from spreading during biopsies or surgery to remove tumors. For example, if they must remove tissue from more than one area of the body, they use different surgical tools for each area. For information about how cancer spreads in the body, see our page on Metastatic Cancer.	80	11.0	In most cases, cancer surgery or a tumor biopsy will not cause cancer to spread in the body. In fact, these procedures are typically done to remove or diagnose cancer that is already present in the body. During surgery, the goal is to remove as much of the cancer as possible, including any visible tumors and any cancer cells that may have spread to nearby tissues. A biopsy is a procedure in which a small sample of tissue is removed and examined under a microscope to determine if cancer is present. In some cases, it is possible that cancer cells may be released into the body during surgery or a biopsy. This can occur when cancer cells are present in the fluids or tissues surrounding the tumor. However, it is important to note that these cells are usually present in very low numbers and are not likely to cause the cancer to spread. It is important to work with a healthcare team to determine the best treatment approach for your situation. This may include surgery, chemotherapy, radiation, or other treatments, depending on the type and stage of your cancer. Your healthcare team can provide more information about the potential risks and benefits of these treatments and help you make the best decision for your situation.	215	13.4
Will cancer get worse if exposed to air?	No. Exposure to air will not make tumors grow faster or cause cancer to spread to other parts of the body. For information about how cancer spreads in the body, see our page on Metastatic Cancer.	36	8.8	Exposing cancer to air will not cause it to get worse. Cancer is a disease in which abnormal cells grow and divide uncontrollably, forming tumors. These abnormal cells are present in the body, and they do not require air to grow or spread. It is important to note that cancer surgery may involve exposing cancerous tumors to air. During surgery, the goal is to remove as much of the cancer as possible, including any visible tumors and any cancer cells that may have spread to nearby tissues. In some cases, the cancer may be removed by cutting into the skin, which can expose the cancerous tissue to air. In other cases, the cancer may be removed through laparoscopic surgery, which involves making small incisions in the skin and inserting instruments and a camera through the incisions. It is important to work with a healthcare team to determine the best treatment approach for your situation. This may include surgery, chemotherapy, radiation, or other treatments, depending on the type and stage of your cancer. Your healthcare team can provide more information about the potential risks and benefits of these treatments and help you make the best decision for your situation.	198	12.8
Do cell phones cause cancer?	No, not according to the best studies completed so far. Cancer is caused by genetic mutations, and cell phones emit a type of low-frequency energy that does not damage genes. For more information, see the NCI fact sheet on Cell Phones and Cancer Risk.	44	7.5	There is no clear evidence that cell phones or other wireless devices cause cancer. Some studies have suggested a possible link between the use of cell phones and an increased risk of certain types of cancer, such as brain cancer. However, other studies have not found such a link. The World Health Organization (WHO) has classified radiofrequency (RF) fields, which are emitted by cell phones and other wireless devices, as possibly carcinogenic to humans. However, this classification is based on limited evidence from studies of high RF field exposure in animals, and it does not mean that these fields definitely cause cancer. The WHO has also noted that the levels of RF fields emitted by cell phones are typically much lower than those that have been associated with increased cancer risk in animal studies. More research is needed to understand the potential health effects of RF fields and the use of cell phones. In the meantime, it is generally recommended to take steps to reduce your RF field exposure, such as using hands-free devices or texting instead of talking on the phone, and keeping your phone away from your body when it is not in use.	196	15.3
Do power lines cause cancer?	No, not according to the best studies completed so far. Power lines emit both electric and magnetic energy. The electric energy emitted by power lines is easily shielded or weakened by walls and other objects. The magnetic energy emitted by power lines is a low-frequency form of radiation that does not damage genes. For more information, see the NCI fact sheet on Electromagnetic Fields and Cancer.	66	13.2	There is no clear evidence that living near power lines causes cancer. Some studies have suggested a possible link between living near power lines and an increased risk of certain types of cancer, such as leukemia and brain cancer. However, other studies have not found such a link. Power lines emit extremely low frequency (ELF) electromagnetic fields (EMFs), which are a type of non-ionizing radiation. Non-ionizing radiation is generally considered to be less harmful than ionizing radiation, which is known to cause DNA damage and increase the risk of cancer. The levels of ELF EMFs emitted by power lines are typically much lower than the levels that have been associated with increased cancer risk in animal studies. More research is needed to understand the potential health effects of ELF EMFs and the risk of cancer. In the meantime, it is generally recommended to take steps to reduce your exposure to all types of EMFs, including those emitted by power lines. This may include using shielding materials to block EMFs, and keeping a safe distance from sources of EMFs when possible.	180	13.7
Are there herbal products that can cure cancer?	No. Although some studies suggest that alternative or complementary therapies, including some herbs, may help patients cope with the side effects of cancer treatment, no herbal products have been shown to be effective for treating cancer. In fact, some herbal products may be harmful when taken during chemotherapy or radiation therapy because they may interfere with how these treatments work. Cancer patients should talk with their doctor about any complementary and alternative medicine products—including vitamins and herbal supplements—they may be using. For more information, see Complementary and Alternative Medicine.	91	15.0	There are no herbal products that have been proven to cure cancer. While some herbs and dietary supplements may have potential health benefits, there is limited scientific evidence to support the use of these products as a treatment for cancer. It is important to note that the use of unproven or untested treatments, including herbal products, can be dangerous and may cause serious harm. Some unproven treatments may contain harmful ingredients, may interfere with proven cancer treatments, or may delay the use of effective treatments. If you are considering using an herbal product or other alternative treatment for cancer, it is important to speak with your healthcare provider first. Your healthcare provider can help you understand the potential risks and benefits of these treatments and can help you make an informed decision about your care. It is always a good idea to seek guidance from a qualified healthcare provider before starting any new treatment.	154	15.5
If someone in my family has cancer, am I likely to get cancer, too?	Not necessarily. Cancer is caused by harmful changes (mutations) in genes. Only about 5 to 10% of cancers are caused by harmful mutations that are inherited from a person’s parents. In families with an inherited cancer-causing mutation, multiple family members will often develop the same type of cancer. These cancers are called “familial” or “hereditary” cancers. The remaining 90 to 95% of cancers are caused by mutations that happen during a person’s lifetime as a natural result of aging and exposure to environmental factors, such as tobacco smoke and radiation. These cancers are called “non-hereditary” or “spontaneous” cancers. For more information about the risk of getting cancer, see the NCI fact sheet on Genetic Testing for Inherited Cancer Susceptibility Syndromes and Cancer Causes and Risk Factors.	128	13.5	Having a family member with cancer may increase your risk of developing cancer, but it does not guarantee that you will develop the disease. The risk of developing cancer can be influenced by a variety of factors, including genetics, environmental exposures, and lifestyle habits. If you have a family member with cancer, it is important to be aware of your personal risk factors and to take steps to reduce your risk of cancer. This may include adopting healthy habits such as not smoking, maintaining a healthy weight, and exercising regularly. It may also be helpful to speak with your healthcare provider about your family medical history and any recommended screening tests or preventive measures. It is also important to remember that cancer is a complex disease, and not all cases are caused by genetics. Many people who develop cancer do not have a family history of the disease, and conversely, some people who have a family history of cancer may never develop the disease.	164	17.4
If no one in my family has had cancer, does that mean I’m risk-free?	No. Based on the most recent data, about 38% of men and women will be diagnosed with cancer at some point during their lives. Most cancers are caused by genetic changes that occur throughout a person’s lifetime as a natural result of aging and exposure to environmental factors, such as tobacco smoke and radiation. Other factors, such as what kind of food you eat, how much you eat, and whether you exercise, may also influence your risk of developing cancer. For more information, see Cancer Causes and Risk Factors.	90	11.8	Having a family history of cancer does not guarantee that you will develop the disease, and not having a family history does not mean that you are risk-free. While genetics can play a role in the development of cancer, there are many other factors that can influence your risk of the disease, including environmental exposures, lifestyle habits, and other medical conditions. It is important to be aware of your personal risk factors for cancer and to take steps to reduce your risk. This may include adopting healthy habits such as not smoking, maintaining a healthy weight, and exercising regularly. It is also important to follow recommended screening guidelines for cancer, which can help detect the disease at an early stage when it is most treatable. If you have concerns about your risk of cancer or have any symptoms that you are concerned about, it is always a good idea to speak with your healthcare provider for guidance.	157	18.7
Do antiperspirants or deodorants cause breast cancer?	No. The best studies so far have found no evidence linking the chemicals typically found in antiperspirants and deodorants with changes in breast tissue. For more information, see the NCI fact sheet on Antiperspirants/Deodorants and Breast Cancer.	38	12.6	There is no clear evidence that the use of antiperspirants or deodorants causes breast cancer. Some studies have suggested a possible link between the use of these products and an increased risk of breast cancer, but the results of these studies have been mixed and more research is needed to understand the relationship between antiperspirants and deodorants and breast cancer. The National Cancer Institute (NCI) states that there is currently no strong evidence to support a link between the use of antiperspirants or deodorants and breast cancer. The NCI notes that while some studies have suggested a possible link between the use of these products and an increased risk of breast cancer, the results of these studies have been inconclusive and more research is needed to understand the relationship between these products and breast cancer. If you are concerned about the safety of antiperspirants or deodorants, it is always a good idea to speak with your healthcare provider for guidance. Your healthcare provider can provide more information about the potential risks and benefits of these products and can help you make an informed decision about your personal care.	188

 

 

5/20/2023

Synchronous Bilateral Breast Cancer: A Case Report Piloting and Evaluating the Implementation of the AI-Powered Large Language Model (LLM) ChatGPT

Reporter: Stephen J. Williams, Ph.D.

A report by Naik et al. 2023 compares  a case reports of a rare type of breast cancer, synchronous bilateral breast cancer generated either by only by ChatGPT, or a combination of ChatGPT and human curators, in order to increase accuracy and speed of case report generation.  Clinically, synchronous bilateral breast cancer is defined as the precence of invasive or in situ breast cancer in both breasts diagnosed within 6 months of each other.  The diagosis is difficult al s-BBC presents at different stages and histopathological characteristics, including prognostic factors. The authors present the case report of s-BBC along with the treatment, prognosis, and clinical management.  Details of clinical MRI images as well as molecular porfiles of the cancer are given.  To generate the report the authors supplied ChatGPT with ten PubMed references for purposes of a query to assist in generating an AI-authored case report.

As credit is given in the acknowledgements:

Acknowledgments

The manuscript includes input from the AI-powered large language model (LLM) ChatGPT (OpenAI, San Francisco, CA), although a significant (<50%) component of the text is manually (human) generated. Text generated by ChatGPT versus human input, as well as text analysis via GPTzero (OpenAI, San Francisco, CA) of the three queries to ChatGPT and one query via domain-trained LLM (Bloom), as well as author impressions of the writing process, are available in the appendix.

 

Abstract

Primary breast carcinoma is the most common cancer type in women, and although bilateral synchronous breast cancers (s-BBC) remain quite rare, the reported incidence may increase with the adoption of more sensitive imaging modalities. Here, we present a case of histomorphological and clinically distinct s-BBC, together with a discussion of clinical management decisions, prognosis, and treatment standards and how these relate to outcomes vis-à-vis more established standards in unifocal breast carcinoma. The case report also constitutes a pilot and formal evaluation of a large language model (LLM) of ChatGPT as a tool to aid in generating a single patient case report.

 

You can read the full report here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10183235/

Below is Dr. Gurda’s comments which are found at the conclusion of the paper:

Author impressions (Dr. Gurda) 

Unfortunately, there is no such thing as a “free lunch” and not everything is as easy as it seems. The discussion sections generated by my queries to ChatGPT seemed to make sense, and superficially, the text generated appeared sensible, but the numbers and the citations produced were entirely fictitious. The authors and the journals existed; the article titles and numbers made general sense but were apparently pulled out of “thin air”, or rather generated by a confluence of text in the training set. As an example, ChatGPT text reads:

“In a retrospective study of 166 patients with s-BBC who underwent BCS, the 10-year overall survival rate was 77%, and the disease-free survival rate was 67%.

The study found that the presence of lobular carcinoma in situ, an extensive intraductal component, and lymph node involvement were significant predictors of worse outcomes (1). (1) Sun J, Huo L, Xie C, et al. The prognosis of synchronous bilateral breast cancer: a single-center experience. PLoS One. 2015;10(6):e0128343.”

It sounds credible, but this study does not exist. The authors exist, the journal exists, but the article and, most importantly, the data cited do not. Overall, for the purposes of generating a discussion section, ChatGPT at this point in its development (early 2023) appears to mostly have the characteristics of a ‘confidence man’ — superficially accurate and strong, but the substance rings hollow. Nonetheless, as stated elsewhere, I found ChatGPT and other LLMs to be useful in generating a skeletonized outline, organizing my thoughts about the subject in a logical manner (even if I had to gather the evidence for the arguments being made in the discussion ‘manually’ later), and in general as a motivation to get started and/or overcome writer’s block. Lastly, the full text of the article. What was generated by ChatGPT (verbatim) is highlighted, and manual human entry and edits are in standard text (Figures ​(Figures4,4, ​,55):

 

 

Note that the authors had to manually add in references.  ChatGPT, as discussed in last week’s article, does not return its source of information.

 

 

 

 

Also note that ChatGPT could not determine which figure in a paper, or which data should be referred to in the AI generated case report.  This also had to be manually added in by the authors.

 

And last to note:

The authors (not AI) had obtained consent or consent waivers from all study participants.  Note how human involvement is needed in order for proper regulatory compliance.  It is probable that an AI generated report would have not included such a statement or consent acquisition, therefore underlies the initial question:

Is AI and ChatGPT, used in clinical reporting, HIPAA compliant?

Naik HR, Prather AD, Gurda GT. Synchronous Bilateral Breast Cancer: A Case Report Piloting and Evaluating the Implementation of the AI-Powered Large Language Model (LLM) ChatGPT. Cureus. 2023 Apr 14;15(4):e37587. doi: 10.7759/cureus.37587. PMID: 37193434; PMCID: PMC10183235.

 

 

 

 

5/06/2023

Can ChatGPT Revolutionize How Cancer Patients Retrieve Relevant Information? A Critique of a head-to-head Comparison with Google Search

Reporter: Stephen J. Williams, Ph.D.

A recent paper in JNCI Cancer Spectrum “Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift” discusses the utility of ChatGPT versus Google search as a suitable source for cancer patient information on their disease and treatment [1].  The authors compared Google Search results with ChaptGPT results upon a series of queries related to cancer and cancer risk and outcome, specifically questions which would be comparable to frequently asked questions by patients.  However, contrary to the paradigm-shift as suggested by the title, a more nuanced analysis of these results suggest that either format has its pros and cons as far as reliable information for patients, and that ChatGPT is in its infancy with respect to its ability to supply simple, yet comprehensive information.

 

Both search engines were asked questions such as

• What are the most common cancer types?
• What exactly is cancer?

The authors reported the main response from Google and top 5 responses from ChatGPT.   Google returned “snippets” or one to two sentence answers for example that breast, prostate, and lung cancer are most common cancers.  These snippets from Google would always include the source, usually as a referenced web link.  ChatGPT answers were more nuanced with provided contextualization according to the patients characteristics (for example if female or male or age).  When asked “What is cancer” ChatGPT, over 5 returned responses, showed variability in supplemental information such as causes, treatments, symptoms.  However ChatGPT did not return weblinks unlike Google Search.

This unreferenced material would be a hindrance to the professional, who requires a hyperlink to a source in order to investigate sources and verify additional related information.  The authors suggest the disruptive nature of ChatGPT is its ability to formulate responses and advice to complex questions, rather than simple queries.

This divergence between utility for a patient versus professional can be seen when analyzing the responses to more specific questions related to discrete medical terminology.

The authors asked GPT and Google these three questions:

• Is coughing a sign of lung cancer
• What is the prognosis of metastatic pancreatic cancer
• Does pembrolizumab cause fever, and should I go to the hospital

 

Let’s look at the differences in response:

The first notable difference is in supplemental information and sources.  While ChatGPT does not give you their source of information Google returns informative and authoritative weblinks:

Is coughing a sign of lung cancer è 7,510,000 weblinks from Google

Google response: “The main symptoms of lung cancer include a cough that does not go away after 3 weeks; a long -standing cough that gets worse; chest infections that keep coming back”

Note: this is short but informative enough for a patient however not as much scientific information however the link to the NHS is sufficient.

ChatGPT responses: Among its many responses include the statement “Coughing can sometimes be a sign of lung cancer, but…” and then describes other diseases which show sign of coughing as symptom.  None of the responses give a duration of symptoms like the Google response.

What is the prognosis of metastatic potential è 29.500.000 weblinks from Google

This was more telling as to the difference between the clinical relevance between Google and ChatGPT.

Google response snippet: “Stage IV pancreatic cancer has a five-year survival rate of 1%.  The average patient diagnosed with late-stage pancreatic cancer will live for about 1 year after diagnosis” and weblink from JohnsHopkins.  The information is clinically relevant as uses an authoritative source of medical information which informs metastatic is referred to as stage IV.  In addition Google, using the source from JohnsHopkins, gives correct and up to date information on PROGNOSTIC values.  However look at ChatGPT response:

ChatGPT responses:

Response 1: “Metastatic pancreatic cancer is a very serious and difficult to treat form of the disease. The prognosis for people with metastatic pancreatic cancer is generally poor, with most people surviving less than six months after being diagnosed. However, it is important to remember that each person is different, and some people may be able to live longer with the disease. Treatment options, such as chemotherapy and radiation, can help to extend life expectancy and improve quality of life. It is important to work with a healthcare team to develop a treatment plan that is tailored to your individual needs.”

Response 2: “The prognosis for metastatic pancreatic cancer is generally poor. This type of cancer has a tendency to spread quickly and is often difficult to treat. In most cases, the cancer has already spread to other parts of the body by the time it is diagnosed, making it difficult to completely remove the cancer. The average life expectancy for people with this type of cancer is about six months, although some people may live longer with treatment. However, treatment options for metastatic pancreatic cancer are limited and may not be effective in all cases. It is important to talk with a doctor about the specific prognosis for an individual’s case.”

Response 3: “The prognosis of metastatic pancreatic cancer, also known as stage 4 pancreatic cancer, is generally not very good. This is because by the time pancreatic cancer has reached this advanced stage, it has typically spread to other parts of the body and is more difficult to treat. Most people with stage 4 pancreatic cancer survive for less than six months after being diagnosed, and only a small percentage of people live for more than one year. However, it is important to remember that every person is different, and some people may respond better to treatment than others. It is always best to discuss your specific situation with your doctor.”

Note that ChatGPT takes a more conversational approach and tries to make the language in a less technical reading level (useful for patients) however it takes about five tries before ChatGPT give prognostic values (putting prognosis at 6 months; no five year survival which most clinicians report) yet talks more about treatment, what most clinicians would refer to as more predictive information.

Does pembrolizumab cause fever, and should I go to the hospital è 119,000 from Google

Note the a Google search returns the manufacturers website on Keytruda side effects at https://www.keytruda.com/side-effects/  however ChatGPT first response is rather disappointing:

“I’m sorry, but I am not able to browse the internet and therefore do not have access to information about the effects of pembrolizumab. I would strongly recommend speaking with a healthcare professional if you are experiencing fever and are concerned about your health. They will be able to provide you with personalized medical advice based on your specific situation.”

For a patient ChatBot this would be very frustrating.  Other answers say that pembolizumab is a type of cancer treatment called an immune checkpoint inhibitor and then says fever is a side effect in which should be reported to your doctor but this is the third possible response from GhatGPT.

“Pembrolizumab is a type of cancer treatment known as an immune checkpoint inhibitor. It can cause side effects such as fever, but it is important to discuss any concerns with your doctor. If you are experiencing fever or other severe symptoms, you should contact your doctor immediately and follow their advice. They will be able to provide you with personalized recommendations based on your individual situation. In some cases, it may be necessary to go to the hospital for further evaluation and treatment.”

The fourth response lists pembrolizumab as a type of cancer therapy called immunotherapy, which is a more understandable term for the patient.

In summary, while Google returns very informative weblinks, ChatGPT does have the capacity to relate information in a more conversational tone and less technical verbiage, more suitable for patients.  However it could be frustrating to patients to have to go through multiple iterations to obtain a suitable answer, and no references are given nor source information.  For the clinician ChatGPT seems to have more learning and training to do to reach a level which would be suitable for the clinical level.  ChatGPT ability to relate to patients though could be a benefit.

Link for Source article: https://academic.oup.com/jncics/article/7/2/pkad010/7049531 

Reference

1. Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ: Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectrum 2023, 7(2).

 

4/23/2023

ChatGPT and Potential Healthcare Implications of Large Language Models – a talk from Dr. George Shih, M.D. FACR

at the March NCI Imaging and Informatics Community Webinar (IICW)! During the event, Weill Cornell Medicine’s Dr. George Shih will talk about ChatGPT and large language models.

ChatGPT is an AI chatbot, built on top of large language models, with the potential to impact many aspects of healthcare. Specifically, Dr. Shih will discuss:

examples of ChatGPT in healthcare, including current limitations.

potential implications of ChatGPT and large language models for researchers, providers, patients, and more.

how ChatGPT and AI’s evolution over time might impact users.

Both the Center for Biomedical Informatics and Information Technology and the Cancer Imaging Program organize the monthly NCI IICW. During the first Monday of every month, this event features scientific presentations and project updates. To receive updates on future topics and to access previous presentations and recordings, visit the IICW webpage.

Dr.Shih focuses his talk on the use of ChatGPT on analyzing DICOM images for radiology.  The ChapGPT did a reasonable job at report generation and even coding in Python, however many radiologists are still hesitant to use it as some medical terminology ChatGPT has problems with.  However Dr. Shih believes that will be addressed over time.  One interesting example was a case study given to ChatGPT with five choices on further evaluations to suggest.  ChatGPT, much like multiple radiologists, suggested an answer not give in the list of choices given to ChatGPT, and surprisingly ChatGPT did not use any of the choices given to it but went out to literature to find best suggestions.

However he does state that ChatGPT is not good at giving citations or referencing the literature correctly.  Note a very good explanation on how GhatGPT3 works (OpenAI build on ChatGPT then with human instructors became ChatGPT4.  the basis for ChatGPT is the algorithm is predicting the next word, like your phone does when you start typing a word

Source: https://datascience.cancer.gov/news-events/events/chatgpt-and-potential-healthcare-implications-large-language-models#event-post-event-stuff

Watch the video of this talk

https://wiki.nci.nih.gov/display/NICC?preview=/362972750/523567855/NCI%20Imaging%20Community%20Call-20230306%201801-1.mp4#NCIImagingandInformaticsCommunityWebinar-2023 

 

4/13/2023

The utility of ChatGPT for cancer treatment information

ABSTRACT

The use of large language models (LLMs) such as ChatGPT for medical question-answering is becoming increasingly popular. However, there are concerns that these models may generate and amplify medical misinformation. Because cancer patients frequently seek to educate themselves through online resources, some individuals will likely use ChatGPT to obtain cancer treatment information. This study evaluated the performance and robustness of ChatGPT in providing breast, prostate, and lung cancer treatment recommendations that align with National Comprehensive Cancer Network (NCCN) guidelines. Four prompt templates were created to explore how differences in how the query is posed impacts response. ChatGPT output was scored by 3 oncologists and a 4th oncologist adjudicated in cases of disagreement. ChatGPT provided at least one NCCN-concordant recommendation for 102/104 (98%) prompts. However, 35/102 (34.3%) of these also included a recommendation that was at least partially non-concordant with NCCN guidelines. Responses varied based on prompt type. In conclusion, ChatGPT did not perform well at reliably and robustly providing cancer treatment recommendations. Patients and clinicians should be aware of the limitations of ChatGPT and similar technologies for self-education.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

The authors thank the Woods Foundation for their generous support of this work.