Kullback-Liebler Distance | Leaders in Pharmaceutical Business Intelligence (LPBI) Group

Posts Tagged ‘Kullback-Liebler Distance’

Amplifying Information Using S-Clustering and Relationship to Kullback-Liebler Distance: An Application to Myocardial Infarction

Posted in Bio Instrumentation in Experimental Life Sciences Research, Biomarkers & Medical Diagnostics, Chemical Biology and its relations to Metabolic Disease, Computational Biology/Systems and Bioinformatics, Ecosystems & Industrial Concentration in the Medical Device Sector, FDA Regulatory Affairs, Health Economics and Outcomes Research, Health Law & Patient Safety, HealthCare IT, International Global Work in Pharmaceutical, Medical Devices R&D Investment, Personalized and Precision Medicine & Genomic Research, Pharmaceutical Analytics, Population Health Management, Genetics & Pharmaceutical, Regulated Clinical Trials: Design, Methods, Components and IRB related issues, Scientist: Career considerations, Statistical Methods for Research Evaluation, tagged Anomaly identification, Bernoulli trial, clustering methods, combinatorial analysis, EHR, IT validation, Kullback-Liebler Distance, learning algorithms, multivariate classification, S-clustering, Shannon-Weaver Information theory on September 22, 2012| 3 Comments »

typical changes in CK-MB and cardiac troponin in Acute Myocardial Infarction (Photo credit: Wikipedia)

Reporter and curator:

Larry H Bernstein, MD, FCAP

This posting is a followup on two previous posts covering the design and handling of HIT to improve healthcare outcomes as well as lower costs from better workflow and diagnostics, which is self-correcting over time.

The first example is a non technology method designed by Lee Goldman (Goldman Algorithm) that was later implemented at Cook County Hospital in Chicago with great success. It has been known that there is over triage of patients to intensive care beds, adding to the costs of medical care. If the differentiation between acute myocardial infarction and other causes of chest pain could be made more accurate, the quantity of scare resources used on unnecessary admissions could be reduced. The Goldman algorithm was introduced in 1982 during a training phase at Yale-New Haven Hospital based on 482 patients, and later validated at the BWH (in Boston) on 468 patients.They demonstrated improvement in sensitivity as well as specificity (67% to 77%), and positive predictive value (34% to 42%). They modified the computer derived algorithm in 1988 to achieve better results in triage of patients to the ICU of patients with chest pain based on a study group of 1379 patients. The process was tested prospectively on 4770 patients at two university and 4 community hospitals. The specificity improved by 74% vs 71% in recognizing absence of AMI by the algorithm vs physician judgement. The sensitivity was not different for admission (88%). Decisions based solely on the protocol would have decreased admissions of patients without AMI by 11.5% without adverse effects. The study was repeated by Qamar et al. with equal success.

Pain in acute myocardial infarction (front) (Photo credit: Wikipedia)

An ECG showing pardee waves indicating acute myocardial infarction in the inferior leads II, III and aVF with reciprocal changes in the anterolateral leads. (Photo credit: Wikipedia)

Acute myocardial infarction with coagulative necrosis (4) (Photo credit: Wikipedia)

Goldman L, Cook EF, Brand DA, Lee TH, Rouan GW, Weisberg MC, et al. A computer protocol to predict myocardial infarction in emergency department patients with chest pain. N Engl J Med. 1988;318:797-803.

A Qamar, C McPherson, J Babb, L Bernstein, M Werdmann, D Yasick, S Zarich. The Goldman algorithm revisited: prospective evaluation of a computer-derived algorithm versus unaided physician judgment in suspected acute myocardial infarction. Am Heart J 1999; 138(4 Pt 1):705-709. ICID: 825629

The usual accepted method for determining the decision value of a predictive variable is the Receiver Operator Characteristic Curve, which requires a mapping of each value of the variable against the percent with disease on the Y-axis. This requires a review of every case entered into the study. The ROC curve is done to validate a study to classify data on leukemia markers for research purposes as shown by Jay Magidson in his demonstation of Correlated Component Regression (2012)(Statistical Innovations, Inc.) The test for the contribution of each predictor is measured by Akaike Information Criteria and Bayes Information Criteria, which have proved to be critically essential tests over the last 20 years.

I go back 20 years and revisit the application of these principles in clinical diagnostics, but the ROC was introduced to medicine in radiology earlier. A full rendering of this matter can be found in the following:
R A Rudolph, L H Bernstein, J Babb. Information induction for predicting acute myocardial infarction.Clin Chem 1988; 34(10):2031-2038. ICID: 825568.

Rypka EW. Methods to evaluate and develop the decision process in the selection of tests. Clin Lab Med 1992; 12:355

Rypka EW. Syndromic Classification: A process for amplifying information using S-Clustering. Nutrition 1996;12(11/12):827-9.

Christianson R. Foundations of inductive reasoning. 1964. Entropy Publications. Lincoln, MA.

Inability to classify information is a major problem in deriving and validating hypotheses from PRIMARY data sets necessary to establish a measure of outcome effectiveness. When using quantitative data, decision limits have to be determined that best distinguish the populations investigated. We are concerned with accurate assignment into uniquely verifiable groups by information in test relationships. Uncertainty in assigning to a supervisory classification can only be relieved by providing suffiuciuent data.

A method for examining the endogenous information in the data is used to determine decision points. The reference or null set is defined as a class having no information. When information is present in the data, the entropy (uncertainty in the data set) is reduced by the amount of information provided. This is measureable and may be referred to as the Kullback-Liebler distance, which was extended by Akaike to include statistical theory. An approach is devised using EW Rypka’s S-Clustering has been created to find optimal decision values that separate the groups being classified. Further, it is possible to obtain PRIMARY data on-line and continually creating primary classifications (learning matrices). From the primary classifications test-minimized sets of features are determined with optimal useful and sufficient information for accurately distinguishing elements (patients). Primary classifications can be continually created from PRIMARY data. More recent and complex work in classifying hematology data using a 30,000 patient data set and 16 variables to identify the anemias, moderate SIRS, sepsis, lymphocytic and platelet disorders has been published and recently presented. Another classification for malnutrition and stress hypermetabolism is now validated and in press in the journal Nutrition (2012), Elsevier.
G David, LH Bernstein, RR Coifman. Generating Evidence Based Interpretation of Hematology Screens via Anomaly Characterization. Open Clinical Chemistry Journal 2011; 4 (1):10-16. 1874-2416/11 2011 Bentham Open. ICID: 939928

G David; LH Bernstein; RR Coifman. The Automated Malnutrition Assessment. Accepted 29 April 2012.
http://www.nutritionjrnl.com. Nutrition (2012), doi:10.1016/j.nut.2012.04.017.

Keywords: Network Algorithm; unsupervised classification; malnutrition screening; protein energy malnutrition (PEM); malnutrition risk; characteristic metric; characteristic profile; data characterization; non-linear differential diagnosis

Summary: We propose an automated nutritional assessment (ANA) algorithm that provides a method for malnutrition risk prediction with high accuracy and reliability. The problem of rapidly identifying risk and severity of malnutrition is crucial for minimizing medical and surgical complications. We characterized for each patient a unique profile and mapped similar patients into a classification. We also found that the laboratory parameters were sufficient for the automated risk prediction.
We here propose a simple, workable algorithm that provides assistance for interpreting any set of data from the screen of a blood analysis with high accuracy, reliability, and inter-operability with an electronic medical record. This has been made possible at least recently as a result of advances in mathematics, low computational costs, and rapid transmission of the necessary data for computation. In this example, acute myocardial infarction (AMI) is classified using isoenzyme CKMB activity, total LD, and isoenzyme LD-1, and repeated studies have shown the high power of laboratory features for diagnosis of AMI, especially with NSTEMI. A later study includes the scale values for chest pain and for ECG changes to create the model.

LH Bernstein, A Qamar, C McPherson, S Zarich. Evaluating a new graphical ordinal logit method (GOLDminer) in the diagnosis of myocardial infarction utilizing clinical features and laboratory data. Yale J Biol Med 1999; 72(4):259-268. ICID: 825617

The quantitative measure of information, Shannon entropy treats data as a message transmission. We are interested in classifying data with near errorless discrimination. The method assigns upper limits of normal to tests computed from Rudolph’s maximum entropy definitions of group-based normal reference. Using the Bernoulli trial to determine maximum entropy reference, we determine from the entropy in the data a probability of a positive result that is the same for each test and conditionally independent of other results by setting the binary decision level for each test. The entropy of the discrete distribution is calculated from the probabilities of the distribution. The probability distribution of the binary patterns is not flat and the entropy decreases when there is information in the data. The decrease in entropy is the Kullback-Liebler distance.

The basic principle of separatory clustering is extracting features from endogenous data that amplify or maximize structural information into disjointed or separable classes. This differs from other methods because it finds in a database a theoretic – or more – number of variables with required VARIETY that map closest to an ideal, theoretic, or structural information standard. Scaling allows using variables with different numbers of message choices (number bases) in the same matrix, binary, ternary, etc (representing yes-no; small-modest, large, largest). The ideal number of class is defined by x^n. In viewing a variable value we think of it as low, normal, high, high high, etc. A system works with related parts in harmony. This frame of reference improves the applicability of S-clustering. By definition, a unit of information is log.r r = 1.

The method of creating a syndromic classification to control variety in the system also performs a semantic function by attributing a term to a Port Royal Class. If any of the attributes are removed, the meaning of the class is made meaningless. Any significant overlap between the groups would be improved by adding requisite variety. S-clustering is an objective and most desirable way to find the shortest route to diagnosis, and is an objective way of determining practice parameters.

Multiple Test Binary Decision Patterns where CK-MB = 18 u/l, LD-1 = 36 u/l, %LD1 = 32 u/l.

No. Pattern Freq P1 Self information Weighted information

0 000 26 0.1831 2.4493 0.4485
1 001 3 0.0211 5.5648 0.1176
2 010 4 0.0282 5.1497 0.1451
3 011 2 0.0282 6.1497 0.0866
4 100 6 0.0423 4.5648 0.1929
6 110 8 0.0563 4.1497 0.2338
7 111 93 0.6549 0.6106 0.3999

Entropy: sum of weighted information (average) 1.6243 bits

The effective information values are the least-error points. Non AMI patients exhibit patterns 0, 1, 2, 3, and 4: AMI patients are 6 and 7. There is 1 fp 4, and 1 fn 6. The error rate is 1.4%.

Summary:

A major problem in using quantitative data is lack of a justifiable definition of reference (normal). Our information model consists of a population group, a set of attributes derived from observations, and basic definitions using Shannon’s information measure entropy. In this model, the population set and its values for its variables are considered to be the only information available. The finding of a flat distribution with the Bernoulli test defines the reference population that has no information. The complementary syndromic group, treated in the same way, produces a distribution that is not flat and has a less than maximum information uncertainty.

The vector of probabilities – (1/2), (1/2), …(1/2), can be related to the path calculated from the Rypka-Fletcher equation, which

Ct = 1 – 2^-k/1 -2^-n

determines the theoretical maximum comprehension from the test of n attributes. We constructed a ROC curve from theoriginal IRIS data of R Fisher from four measurements of leaf and petal with a result obtained using information-based induction principles to determine discriminant points without the classification that had to be used for the discriminant analysis. The principle of maximum entropy, as formu;ated by Jaynes and Tribus proposes that for problems of statistical inference – which as defined, are problems of induction – the probabilities should be assigned so that the entropy function is maximized. Good proposed that maximum entropy be used to define the null hypothesis and Rudolph proposed that medical reference be defined as at maximum entropy.

Rudolph RA. A general purpose information processing automation: generating Port Royal Classes with probabilistic information. Intl Proc Soc Gen systems Res 1985;2:624-30.

Jaynes ET. Information theory and statistical mechanics. Phys Rev 1956;106:620-30.

Tribus M. Where do we stand after 30 years of maximum entropy? In: Levine RD, Tribus M, eds. The maximum entropy formalism. Cambridge, Ma: MIT Press, 1978.

Good IJ. Maximum entropy for hypothesis formulation, especially for multidimensional contingency tables. Ann Math Stat 1963;34:911-34.

The most important reason for using as many tests as is practicable is derived from the prominent role of redundancy in transmitting information (Noisy Channel Theorem). The proof of this theorem does not tell how to accomplish nearly errorless discrimination, but redundancy is essential.

In conclusion, we have been using the effective information (derived from Kullback-Liebler distance) provided by more than one test to determine normal reference and locate decision values. Syndromes and patterns that are extracted are empirically verifiable.

Entropy and Syntropy (photoatelier.org)
A Software Agent for Diagnosis of ACUTE MYOCARDIAL INFARCTION (pharmaceuticalintelligence.com)
K-Nearest-Neighbors and Handwritten Digit Classification (jeremykun.wordpress.com)
Data Mining: Classification VS Clustering (cluster analysis) (parasdoshi.com)
Myocardial Infarction Algorithm Strategy 77% Effective In One Hour (guardianlv.com)
Scale‑Free Diagnosis of AMI from Clinical Laboratory Values (pharmaceuticalintelligence.com)
The great healthcare chasm: Patients want to email, access EMRs, but physicians still can’t (medcitynews.com)
Guidelines Updated for Unstable Angina/Non-ST Elevation Myocardial Infarction (pharmaceuticalintelligence.com)

Read Full Post »

A Software Agent for Diagnosis of ACUTE MYOCARDIAL INFARCTION

Posted in Biomarkers & Medical Diagnostics, Computational Biology/Systems and Bioinformatics, HealthCare IT, Medical Devices R&D Investment, Statistical Methods for Research Evaluation, tagged acute myocardial infarct, artificial neural network, cardiac biomarkers, classification by information theory, combinatorial clusters, Euclidean distance, Kullback-Liebler Distance, multivariate statistical analysis, nonlinear discrimination, predictive analytics, training and validation on August 12, 2012| 7 Comments »

A Software Agent for Diagnosis of ACUTE MI

Authors: Isaac E. Mayzlin, Ph.D.1, David Mayzlin1,Larry H. Bernstein, M.D.2

1MayNet, Carlsbad, CA, 2Department of Pathology and Laboratory Medicine, BridgeportHospital, Bridgeport, CT.

Agent-based decision support systems are designed to provide medical staff with information needed for making critical decisions. We describe a Software Agent for evaluating multiple tests based on a large data base especially efficient when time for making the decision is critical for successful treatment of serious conditions, such as stroke or acute myocardial infarction (AMI).

Goldman and others (1) developed a screening algorithm based on characteristics of the chest pain, EKG changes, and key clinical findings to separate high-risk from low-risk patients at the time they present using clinical features without using a serum marker. The Goldman algorithm was not widely used because of a 7 percent misclassification error, mostly false positives. Nonetheless, A third of emergency room visits by patients presenting with symptoms of rule out AMI are not associated with chest pain. A related issue is the finding that a significant number of patients who are at high risk have to be identified using a cardiac marker. The use of cardiac isoenzymes has been to classify patients meeting the high risk criteria, many of whom are not subsequently found to have AMI.

Software Agent for Diagnosis based on the Knowledge incorporated in the Trained Artificial Neural Network and Data Clustering

This Software Agent is based on the combination of clustering by Euclidean distances in multi-dimensional space and non-linear discrimination fulfilled by the Artificial Neural Network (ANN) trained on clusters’ averages. Our studies indicate that at an optimum clustering distance the number of classes is minimized with efficient training on the ANN, retaining accuracy of classification by the ANN at 97%. The studies conducted involve training and testing on separate clinical data sets. We perform clustering using the geometrical (Euclidean) distance between two points in n-dimensional space, formed by n variables, including both input and output variables. Since this distance assumes compatibility of different variables, the values of all input variables are linearly transformed (scaled) to the range from 0 to 1.

The ANN technique for readers accustomed to classical statistics can be viewed as an extension of multivariate regression analyses with such new features as non-linearity and ability to process categorical data. Categorical (not continuous) variables represent two or more levels, groups, or classes of correspondent features, and in our case this concept is used to signify patient condition, for example existence or not of AMI.

Process description. We implemented the proposed algorithm for diagnosis of AMI. All the calculations were performed on the authors’ unique Software Agent Maynet. First, using the automatic random extraction procedure, the initial data set (139 patients) was partitioned into two sets — training and testing. This randomization also determined the size of these sets (96 and 43, respectively) since the program was instructed to assign approximately 70 % of data to the training set.

The main process consists of three successive steps:

(1) clustering performed on training data set,

(2) neural network’s training on clusters from previous step, and

(3) classifier’s accuracy evaluation on testing data.

The classifier in this research will be the ANN, created on step 2, with output in the range [0,1], that provides binary result (1 – AMI, 0 – not AMI), using decision point 0.5.

In this paper we used the data of two previous studies (2,3) with three patients, potential outliers, removed (n = 139). The data contains three input variables, CK-MB, LD-1, LD-1/total LD, and one output variable, diagnoses, coded as 1 (for AMI) or 0 (non-AMI).

Table 1. Effect of selection of maximum distance on the number of classes formed and on the accuracy of recognition by ANN

Clustering Distance Factor F(D = F * R)

Number ofClasses

Number of Nodes in The Hidden Layers

Number of Misrecognized Patterns inThe TestingSet of 43

Percent ofMisrecognized

10.90.8

0.7

241413

1, 02, 03, 0

1, 0

2, 0

3, 0

3, 2

121

2.34.62.3

2.3

4.6

2.3

Abbreviations: creatine kinase MB isoenzyme: CK-MB; lactate dehydrogenase isoenzyme-1: LD1; LD1/total LD ratio: %LD1; acute myocardial infarction: AMI; artificial neural network: ANN