Chapter 3. Evidence-Based Endocrinology and Clinical Epidemiology

• ARR Absolute risk reduction
• CI Confidence interval
• CV Coefficient of variation
• EBM Evidence-based medicine
• ER Event rate
• NiH Negativity in health
• NNT Number needed to treat
• PiD Positivity in disease
• QALY Quality-adjusted life year
• RCT Randomized controlled trial
• ROC Receiver-operating characteristic
• RRR Relative risk reduction
• USPSTF U.S. Preventive Services Task Force

The individual practitioner faces a multiplicity of potential diagnoses, limitations in diagnostic capacity, subclinical disease identified by tests rather than by clinical manifestations, and rapid changes in scientific knowledge. The paradigm of clinical decision-making based on the assumption that all that is needed to guide clinical practice is personal experience (however unsystematic), understanding of pathophysiology, and thorough training plus common sense is insufficient to address these challenges. Moreover, the integration of relevant research findings into clinical practice has been haphazard; the lag time between development of scientific knowledge and introduction into practice can be many years, and there is marked variation in practice. A systematic approach based on principles of clinical epidemiology can help address some of these issues. This quantitative approach has formed the primary, albeit not the only, basis of the evidence-based medicine movement. This movement posits that understanding certain rules of evidence is necessary to interpret the literature correctly and that physicians who practice based on the above understanding will provide superior care. This chapter will summarize some of the principles of clinical epidemiology and evidence-based endocrinology and some of their limitations.

Clinical epidemiology consists of the application of epidemiologic principles and methods to problems encountered in clinical medicine. Clinical epidemiology emphasizes a quantitative approach and is therefore concerned with counts of clinical events. Its applications are paramount (1) in diagnostic testing and how the results modify the probability of a particular disorder being present and (2) in treatment decisions in which the potential benefits and harms must be addressed. The techniques of clinical epidemiology have become increasingly important as practitioners confront the complexity of contemporary medical practice.

The appropriate choice and interpretation of diagnostic tests, whether biochemical assays, radiologic procedures, or clinical findings, have always been essential to the clinical practice of endocrinology. These tests, when introduced in the medical literature, are accompanied by varying degrees of validation. The clinician's assessment of the utility of tests, now even more important with the emphasis on cost-effectiveness, can be improved by knowledge of test EBM principles. We review some of these concepts as they apply to diagnosis and management of endocrine disorders, including the topics of test characteristics such as sensitivity and specificity, receiver-operating characteristic (ROC) curves, likelihood ratios, predictive values, and diagnostic accuracy.

The evaluation of endocrine function begins with a clinical question. The more vague the question, the more difficult it is to obtain a clear answer. Part of this step involves a clinical judgment about the likelihood of the disease prior to obtaining a test and its results. This pretest probability is combined with the performance characteristics of the test and its use (sensitivity and specificity, ROC curves, likelihood ratios, predictive values, and diagnostic accuracy) in order for proper interpretation. Variation is inherent in biological systems. Therefore, diagnostic tests must take into account not only variability of the tests themselves and how they are performed but also variability in the populations in whom the tests were developed, both with the disease and without the disease. Key aspects in the analysis of a test include reproducibility (precision) and accuracy. Reproducibility describes how close the test comes to producing the same results every time and depends on such factors as intraobserver and interobserver variability (as in the assessment of a physical finding or x-ray) and, in the case of biochemical tests, characteristics such as intra-assay and interassay coefficients of variation (CVs). Although studies utilizing radioimmunoassays and other assays routinely report intra-assay and interassay CVs, few papers publish multiple results performed on the same patient (intraindividual variation). There have also been relatively few studies on the reliability of measurements (ie, the degree of intraindividual variation). One study found that the minimum number of replicate measurements necessary to achieve satisfactory reliability of the mean of basal levels was 3 for plasma cortisol and 18 for salivary cortisol. Responses to dynamic tests required fewer replicates to achieve the same reliability (one to two samples).

Reproducibility depends on the conditions under which the test is performed. Clinicians must be aware of the distinction between efficacy and effectiveness when translating published results into practice. As applied to diagnostic testing, efficacy refers to the degree to which the test has been shown scientifically to accomplish the desired outcome. In contrast, effectiveness refers to the degree to which the test achieves this outcome in actual clinical practice. Most large studies have been performed in research venues and thus are efficacy studies, whereas the effectiveness of most tests in practice has not been extensively evaluated. In comparing one's own results with a published report or laboratory normal range, it is important to take into account those conditions (eg, test performed in a hospital vs a physician's office).

Accuracy describes how close the test comes to producing results that are a true measure of the phenomenon of interest; systematic bias of a highly reproducible test may produce the same incorrect result every time. Like reproducibility, accuracy depends on the conditions under which the test is performed; accuracy in the clinical practice setting may differ from that in the experimental setting where many extraneous influences are controlled.

When interpreting a test, the result is usually compared to a normal range. Frequently, the normal range is developed by using a reference population assumed (or preferably shown) to be disease free. For example, when designing a test to be used in the diagnosis of Cushing syndrome, the reference group should be made up of individuals who have clinical features suggestive of Cushing syndrome but who in fact do not have the disorder. However, reference groups may be made up of individuals who are readily accessible and not more appropriate comparisons. It is also important to note that in establishing a normal range based on a Gaussian or normal distribution, encompassing the mean ± two standard deviations, 5% of disease-free individuals have a result outside the limits of normal. (It is important to note that the definition of normal based on a Gaussian distribution of values is only one of a number of definitions of normal. Some others include the one most representative or most common in a class, the one most suited to survival, and the one that carries no penalty [ie, no risk].) Figure 3–1 illustrates a normal range and a range in a population with disease. A result outside normal limits is not equivalent to disease. Moreover, values within the normal range do not necessarily exclude disease. Values in the population of individuals with disease are determined separately, and the overlap with the normal range is assessed.

Sensitivity and Specificity

Ideally, a diagnostic test has no overlap between the results of individuals with the disease and those without the disease. The reality, however, is different. Test characteristics that describe this overlap are sensitivity and specificity, and they are typically illustrated in a 2 × 2 table. As shown in Figure 3–2, sensitivity and specificity are collectively known as operating characteristics. Sensitivity refers to the ability to identify correctly individuals with a disease. The sensitivity of a sign or symptom (or diagnostic test) is the proportion of patients with disease who has a positive test result, sign, or symptom. In contrast, specificity refers to the ability to identify correctly individuals without a disease. The specificity of a test is the proportion of healthy patients who have a negative test result or who lack that sign or symptom.

Thus, the sensitivity of a test equals the number of individuals with disease who have a positive test (true positive [TP] divided by the number of individuals with disease (true positives plus false negatives [FN]), whereas specificity equals the number of individuals without the disease who have a negative test (true negatives) divided by the number of individuals without disease (true negatives plus false positives [FP]). Sensitivity is sometimes termed PiD or positivity in disease, and specificity is sometimes termed NiH or negativity in health. In theory, sensitivity and specificity are characteristics of the test itself and not of the patients on whom the test is applied. However, this may not be correct in practice. The sensitivity of a test may be affected by the stage or severity of the disease. The specificity of a test may depend on the characteristics of the reference population. The nature of the groups used to establish the cut-points that differentiate normal from abnormal must be appropriate and should be specified in any report of a diagnostic test. The value chosen for a cutoff point also affects the sensitivity and specificity. To assist the clinician in assessing a report on a diagnostic test, a series of questions has been proposed (Table 3–1).

The overnight dexamethasone suppression test is commonly used as a screening test in the diagnosis of Cushing syndrome, and its use illustrates some of the issues in diagnostic testing. As shown in Figure 3–3, combining the results of a number of studies indicates a sensitivity of 98.1% and specificity of 98.9%. However, the individual studies varied, with sensitivities ranging from 83% to 100% and specificities from 77% to 100%. Moreover, the studies used different cortisol assays, doses of dexamethasone, and criteria for a positive test. Caution must be exercised in drawing conclusions from the combining of such data. In order to apply the sensitivity and specificity of a test derived from one study sample to a different population, the test cannot deviate from methods used (eg, dose of dexamethasone, type of cortisol assay, timing of dexamethasone administration, and cortisol assay) when the optimal cutoff was determined, and the sample studied must be similar to the new population to be tested. To meet this latter prerequisite, the sample studied must account for variability of diseased individuals. This requires that subjects with disease be defined using the best available gold standard (independent of the test in question) and include a broad enough cross-section of those with disease (eg, mild versus severe disease, different etiologies of disease, as well as age, sex, and race) to establish a reliable range of measurements. The characteristics of the reference sample of subjects without the disease are equally important. Although the 1-mg overnight dexamethasone suppression test is still believed to have excellent, albeit less than 100% sensitivity, it has serious problems with specificity, and false-positive results have been described with a variety of drugs as well as medical, surgical, and psychiatric conditions.

One additional method of reporting the performance of a test is diagnostic accuracy. This can also be derived from the 2 × 2 table. Diagnostic accuracy is defined as the ability of a test to identify correctly those with the disease and those without the disease:

From Crapo composite data on the 1-mg overnight dexamethasone suppression test, the diagnostic accuracy is calculated as 98.7%. In addition to the characteristics of the study subjects, the number of subjects included in the sample is also critical to assess the accuracy of the test. Each operating characteristic determined in a study should be accompanied by a confidence interval (CI)—a range of values calculated from sample size and standard error which expresses a degree of confidence (usually 90%, 95%, or 99%) that the unknown true sensitivity or specificity lies in that interval. CIs are a measure of the precision of an estimate. The range of a CI depends on two factors: (1) the number of observations; and (2) the spread in the data (commonly measured as a standard deviation). The fewer the number of observations, the larger the range of the CI, and the greater the standard deviation of the data, the larger the range of the CI.

In addition to the limitations on the operating characteristics based on the samples from which the data are derived, sensitivity and specificity are not independent of each other. They vary with the cutoff level chosen to represent positive and negative test results. In general, as sensitivity increases, specificity decreases and as specificity increases, sensitivity decreases. This phenomenon is depicted graphically in an ROC curve.

Receiver-Operating Characteristic Curves

An ROC curve graphically illustrates the trade-off between the false-negative and false-positive rates for different cutoff points of a diagnostic test. In an ROC curve, the true-positive rate (sensitivity) is plotted on the vertical axis and the false-positive rate (1−specificity) is plotted on the horizontal axis for different cutoff points for the test. The dotted diagonal line in Figure 3–4 corresponds to a test that is positive or negative just by chance (ie, the true-positive rate equals the false-positive rate). Such a test provides no useful information. Ideally, a test would provide results that could be plotted on one point in the top left corner—100% true-positive rate and 100% true-negative rate. The closer an ROC curve is to the upper left-hand corner of the graph, the more accurate it is, because the true-positive rate is 1 and the false-positive rate is 0. As the criterion for a positive test becomes more stringent, the point on the curve corresponding to sensitivity and specificity (point A) moves down and to the left (lower sensitivity, higher specificity); if less evidence is required for a positive test, the point on the curve corresponding to sensitivity and specificity (point B) moves up and to the right (higher sensitivity, lower specificity). Analysis of the area between the actual results and the straight line indicates how good the test is. The greater the area under the curve, the better the test.

Depending on the purpose of the test, the curves may be used to decide an optimal cutoff level for a single test. For example, with a screening test, high sensitivity is typically desired, and the trade-off is lower specificity. The cutoff point may also be chosen depending on health costs (morbidity and mortality associated with an error in diagnosis), financial costs, or need for maximal information (the operating position giving the greatest increase in posttest probability).

ROC curves may also be used to compare two or more tests by comparing the areas under the curves, which represent the inherent accuracy of each test. An example of the comparison of the performance of different tests for the diagnosis of pheochromocytoma is given in Figure 3–5. It is important to remember, however, that ROC curves are only as good as the operating characteristics from which they are generated.

Finally, determining cost-effective diagnostic strategies requires careful evaluation not only of a test in isolation, but also in the context of the other information available and the likelihood of disease. This is the essence of Bayesian models of decision-making. In this model, the physician updates his or her belief in a hypothesis with each new item of information, with different weights given to the new information depending on its operating characteristics. Consideration must be given to the question of the value added by a test or procedure. This can be assessed with ROC curves and statistical models.

Predictive Values, Likelihood Ratios, and Diagnostic Accuracy

Sensitivity and specificity are important test characteristics, yet the clinician wants to know how to interpret a test result. Predictive values help in this regard. As shown in Figure 3–6, the positive predictive value is the proportion of patients with a positive test who actually have the disease. Similarly, the negative predictive value is the proportion of those with a negative test who do not have the disease. Because each of these values are calculated using results from both individuals with and without the disease in question, the prevalence of the disease has a great impact on the values. For any given sensitivity and specificity, the lower the prevalence of disease (or the lower the pretest probability), the more false-positive results there are (Figure 3–6).

The likelihood ratio, which is derived from sensitivity and specificity, is an expression of the odds that a sign, symptom, or test result is expected in a patient with a given disease as opposed to one without the disease. Two forms of the likelihood ratio exist, the likelihood ratio for a positive finding and the likelihood ratio for a negative finding. Calculations are shown in Figure 3–2. Likelihood ratios offer some advantages over sensitivity and specificity. They are the most useful in calculating posttest probabilities given prevalence (a probability) and likelihood ratios. A convenient nomogram for this has been published (Figure 3–7).

In approaching a case, the clinician makes a series of inferences about the nature of a patient's condition and proceeds toward a working diagnosis along with some alternatives—generation of a differential diagnosis. Although one could create a listing of all possible causes of the patient's complaint (ie, a possibilistic differential diagnosis), experienced clinicians generate a differential diagnosis that is a combination of probabilistic (ie, considering first those disorders that are more likely), prognostic (ie, considering first those disorders that are more serious if missed), and pragmatic (ie, considering first those disorders that are most responsive to treatment). The clinician then refines the diagnostic hypotheses, sometimes using clues from the history and physical examination and often with the use of laboratory or radiologic tests. In so doing, the clinician tries to reduce the inherent uncertainty so that the most appropriate course of treatment can be prescribed.

The first step in this process is to understand the concept of probability. A probability is an expression of likelihood and thus represents an opinion of the relative frequency with which an event is likely to occur. In the case of diagnosis, probability is a numerical expression of the clinician's uncertainty about a diagnosis; expressing a clinical opinion in subjective terms such as likely and possible is fraught with imprecision and misunderstanding. Certainty that a disease is present is assigned a probability of one, certainty that a disease is not present is assigned a probability of zero, and a clinician's opinion of the disease being present or absent usually falls somewhere in between. Of course, probabilities are derived from different data sources that vary in their reliability and application to a given patient, such as the clinician's experience (remembered cases), research studies, and population-based epidemiologic studies. Therefore, some degree of uncertainty is inherent in a given probability, and the confidence with which one can rely on a given probability depends to a large extent on the data underlying it.

The diagnostic approach to minimizing uncertainty requires four steps. First, the clinician starts with an estimate of probability based on initial clinical impressions. This starting point is dubbed the pretest or a priori probability and is a number between zero and one that expresses the likelihood of disease. For example, a clinician who sees a large population of patients with diabetes mellitus may think that a 55-year-old patient with polyuria, polydipsia, weight loss, and fatigue has a probability of 0.70 of having diabetes mellitus (ie, if there were 100 such individuals, 70 would have diabetes).

Second, the clinician determines the threshold probability for treatment. The treatment threshold probability is defined as the probability of disease at which one is indifferent between giving treatment and withholding treatment. Establishing a treatment threshold probability takes into account the costs (not just in the monetary sense) and benefits of treating or not treating. Because probability is predicated on the lack of certainty about the presence or absence of disease, it is inevitable that some patients who are not diseased receive treatment and others who are diseased do not receive treatment. Third, if the pretest probability is greater than the threshold probability, the clinician chooses to treat; if it is less than the threshold probability, the clinician opts not to treat. If the clinician is not comfortable enough about the presence or absence of disease, he or she may choose to order further tests with the objective of getting closer to certainty. The fourth step involves taking information gained from the test and using it to update the pretest probability. The updated, or posttest, probability can also serve as a new pretest probability for the next step in hypothesis testing (Figure 3–8).

Treatment threshold probability depends on the effectiveness of treatment, including its adverse effects. In general, treatment threshold probability is generally low when treatment has a high benefit for diseased patients and/or a low risk of harm for nondiseased patients. Likewise, the treatment threshold probability is generally high when there is a high risk to nondiseased patients and/or a low benefit to diseased patients. For example, an endocrinologist evaluating an incidentally found pituitary microadenoma would need to be as certain as possible that the patient suffered from hormone hypersecretion before recommending surgical resection because of the high risk of harm to nondiseased patients relative to the benefit to diseased patients. The same principles apply to diagnostic testing. There is a test-threshold defined as the probability of disease at which one is indifferent between performing a test and not performing a test. Depending on the circumstances, one would treat without performing the test or not perform the test and ignore the issue entirely.

Tests can be combined in the hope that diagnostic accuracy is enhanced. Two tests can be performed in parallel (simultaneously) or in series (sequentially). When two tests are performed in parallel, a positive result in either test establishes the diagnosis; when two tests are performed in series, positive results in both tests are required to make the diagnosis. Thus, parallel testing increases sensitivity at the cost of specificity, whereas series testing increases specificity at the cost of sensitivity. For this approach to be better than one test alone, the second test has to provide information not provided by the first test. A common practice is to repeat the same test. This approach may be quite logical when the disorder is intermittently active (eg, intermittent Cushing syndrome) or when there is concern about the way the first test was performed. When tests are performed in series, the first test is usually the one with the higher sensitivity so that as many patients with the disease are discovered. False-positive results are then identified by a second test with higher specificity. Another approach to sequential testing uses as the first test the one with highest specificity so that fewer patients without the disease go on to further testing. This approach makes sense when the second test is associated with more morbidity. Adjusting cut-points for the first test can make it 100% specific (at the cost of some sensitivity). This approach has been used frequently in tests related to Cushing syndrome. One caveat applies to the method of choosing the cutoff point. In addition to considering whether the patients without the disease are an appropriate control group, attention should be paid to sample size. Caution must be exercised when applying test characteristics derived from small samples.

Critical to decision-making is the appropriate interpretation of the results of clinical trials and the impact of treatments on clinical events. The use of a 2 × 2 table can facilitate this process (Figure 3–9).

The event rate (ER) is the rate of occurrence of the outcome of interest. The relative risk reduction (RRR) is the proportional reduction in rates of bad outcomes between experimental and control participants in a trial, and it can be calculated as:

RRR is usually reported accompanied by a 95% CI. Absolute risk reduction (ARR) is the absolute arithmetic difference in rates of poor outcomes between experimental and control participants in a trial and is usually accompanied by a 95% CI. Another way to think about this is to ask how many patients would have to be treated in order to prevent one negative outcome or achieve one additional favorable outcome. This is termed number needed to treat (NNT), and it is calculated as 1/ARR and accompanied by a 95% CI. An example is provided (see Figure 3–9) from the results of a clinical trial. In the United Kingdom Prospective Diabetes Study (UKPDS-38), hypertensive patients with type 2 diabetes were randomized to tight versus less tight blood pressure control. The patients with tight blood pressure control had fewer diabetesrelated complications and fewer deaths due to diabetes compared to the patients with less tight blood pressure control. Over a period of 8 years of treatment, 15.9% of patients in the less tight control group had died of diabetes complications compared with 10.9% in the tight control group.

To apply these statistics in clinical decision-making to patients, it is necessary to calculate ARR, RRR, and NNT. The RRR of 32% indicates that approximately one-third of the expected deaths were prevented by tight control of blood pressure. However, although the RRR may be similar across different risk groups, the ARRs are not. ARRs become smaller when event rates are low, whereas RRR remains constant. Thus, RRR can be misleading. The lower the ER in the control group, the larger the difference between RRR and ARR. In other words, the benefits of treatment as reflected in ARR are usually greater in patients at higher risk of negative outcomes than in patients at lower risk of negative outcomes.

The RRR fails to discriminate absolute treatment effects that are clinically significant from those that are trivial. The NNT is 20, so that one would need to treat 20 patients for 8 years with tight blood pressure control to prevent one death from diabetes. NNT can be readily used to compare different treatments. In this case, RRR was 32% and ARR was 5.1%. If a different treatment with the same RRR of 32% has an ARR of 0.51%, the NNT would be 196, indicating that a much greater effort would be needed to achieve one additional favorable outcome. This might or might not be worth the effort, depending on a variety of factors. One method by which treatment thresholds can be determined is with the method of decision analysis.

Decision analysis is a mathematical tool designed to facilitate complex clinical decisions in which many variables must be considered simultaneously. This analytical procedure selects among available diagnostic or therapeutic options based on the probability and predetermined value (utility) of all possible outcomes of those options. Decision analysis provides a systematic framework for organizing all data relevant to the decision so that relevant uncertainties are less likely to be overlooked. Performing a decision analysis requires clear definition of the relationship between possible courses of action and their associated outcomes and assignment of numerical values to various courses of action. In so doing, decision analysis simplifies comparisons among different strategies.

There is substantial variation in duration and severity of disease between individuals. Choosing a treatment option in the setting of unpredictable effects is a difficult problem, and expected value decision-making is a useful tool. When individual outcomes are uncertain, expected value is the result that is expected on average. In preparing to perform a decision analysis, one first must define the problem with a clear statement of the strategies to be examined. Two or more strategies may be included. Clinical decisions have effects over different time frames. An important step in defining a decision problem is to decide the time horizon for outcomes to be evaluated. For example, if strategies for diabetes treatment are to be compared, a period of weeks to months may be appropriate for evaluating the risk of hypoglycemic episode. However, this relatively short time horizon would not be appropriate when the study outcome is diabetic retinopathy, where a time horizon of years would be a better choice. After choosing the problem and a single time horizon, carrying out a formal analysis typically involves six general steps:

1. Construction of a decision tree that maps out all the possibilities

2. Determination and assignment of probabilities

3. Assignment of utilities to each potential outcome

4. Determination of the expected utility

5. Choosing the course of action with the highest expected utility

6. Evaluation of the sensitivity of the chosen course of action to changes in probabilities and utilities

In a decision tree, the term decision alternative refers to one of the potential strategies to be analyzed (Figure 3–10). Each alternative should be listed. Figure 3–11 shows a sample decision tree that outlines strategies of medical versus surgical treatment for disease. The decision itself is represented by a box called a decision node. All of the possible outcomes for each decision alternative are listed. An event that has outcomes under the control of chance is denoted by a chance node. The symbol for a chance node is a circle. The series of events leading to the clinical outcomes is represented by a series of chance nodes and decision nodes. The decision tree is usually written from left to right, with the initial decision node on the far left and the final outcomes on the far right. A final outcome is represented by a terminal node. There may be any number of outcomes at a chance node. The listed outcomes should include all possible outcomes, and they must not overlap. In addition to this assumption of mutual exclusivity, structuring a tree in this fashion assumes that the probability of occurrence of one event does not influence the probability of occurrence of another event(s). The decision tree structure should be as similar as possible for all strategies, because differences may lead to a structural bias in the analysis.

The decision tree described above is relatively simple, and this approach cannot readily represent complex, dynamic clinical situations with recurring events. More elaborate models, such as Markov models, can be used to represent the passage through multiple health states. Patients may transition from one health state to another with some probability within a specified time period or model cycle. Like each terminal node in a static decision tree, each health state in the Markov model is associated with a specific clinical measure, utility, or cost. A key assumption of the Markov model is that the future is determined only by the individual's present health state; events prior to that health state or how long it took to arrive there do not affect the individual's future, a simplifying assumption that may not hold true for some health problems.

One strength of the decision analysis process is that it may be used for a variety of outcome measures. The outcome measure of interest determines the information needed for analysis. For example, one may use clinical measures such as survival following total thyroidectomy for differentiated thyroid cancer, preservation of vision after laser photocoagulation, or meeting a target level for glycosylated hemoglobin. Economic measures provide measures of cost and resource use. When cost measures are included, it is important to consider whose perspective is represented; analyses may reflect the viewpoint of society as a whole, payers, or patients.

A detailed analysis of costs may be used for cost-effectiveness analysis or cost–utility analysis. Utility measures reflect outcome preferences. For an individual, utilities represent quality of life preferences. They are quantitative values used to summarize multiple dimensions, which may be conflicting. For example, decision-makers frequently must choose among strategies that have differing effects on length of life and quality of life and involving trade-offs between the two. Utilities are traditionally scored on a scale from 0 to 1. The ideal situation, often perfect health, is scored as a 1. The worst situation, death, is scored as a 0, and this assumes that there are no utilities worse than death. Intermediate states are assigned values between 0 and 1. For example, living with diabetes mellitus may receive a score of 0.80—less than perfect, but preferable to death. Among the assumptions underlying this approach are: stability of estimates of utilities, that is, the estimates obtained prior to experiencing an event will not change after the event has been experienced; and the it-does-not-matter-how-you-get-there assumption (ie, death or other outcomes have the same utility regardless of the route a patient takes in getting there).

There are different ways to obtain utility values for different health states. One relatively simple method is the visual analog scale, in which a subject is asked to rate a given health state on a scale from 0 to 100. There also are more specific ways of determining utility. The standard gamble approach was developed by von Neumann and Morgenstern as a method for assessing utility. One advantage of this method is that it incorporates the participant's attitudes about risk taking, because the process involves consideration of a hypothetical gamble. For example, consider a 74-year-old diabetic man with known coronary artery disease and a chronically infected lower extremity ulcer. His physicians suggest a below-the-knee amputation (BKA), but they also discuss the option of a course of antibiotics instead of surgery. Assume that antibiotics have a 10% cure rate and that an amputation has a 95% cure rate. To assess utility from the patient's perspective, it is necessary to list all of the possible outcomes; rank the outcome states in the order of preference; assign a utility of 1 to the most preferred outcome and 0 to the least preferred outcome; and formulate situations where a patient is indifferent about choosing between a gamble (between outcomes of known utility) and a sure thing (involving an outcome with unknown utility). This is how the utility of each intermediate outcome is determined. For example, suppose the patient decides that a cure with antibiotics is an appealing outcome. This outcome is compared to those with known utilities—perfect health and death. When asked to choose between a cure with antibiotics and a gamble in which he has a 90% chance of achieving perfect health and a 10% chance of dying, he is unable to choose (indifferent). Thus, the utility for the outcome of cure with antibiotics is 0.9. However, a surgical cure with an amputation is a less appealing scenario. The utility for an amputation is only 0.75. By a similar process, the utility of a failure of antibiotics necessitating an amputation is 0.5. The utility scores can then be incorporated into a decision tree (see Figure 3–11). The expected utility for each decision alternative may be obtained by adding each (utility × probability) value. The utility for each outcome is on the right. The probabilities are below the outcome branches. The utility for antibiotic treatment = (0.1 × 0.9) + (0.9 × 0.5) = 0.54, whereas the utility for BKA = (0.95 × 0.75) + (0.05 × 0) = 0.71. In terms of the patient's quality of life, the expected value of amputation is greater than the expected value of antibiotic treatment. Different utilities for these health states would alter the analyses.

The time trade-off method is another way of determining utility. The utility may be described as a number between 0 and 1. For example, a subject may consider 10 years with pain from diabetic neuropathy equivalent to 5 years in perfect health. The utility of diabetic neuropathy would be 0.5. Alternatively, the utility may be expressed in quality-adjusted life years (QALYs). To determine the number of QALYs associated with an outcome, the time horizon for the outcome state is specified. Often, this is a patient's life expectancy in a particular outcome state. The number of years in full health that the subject sees as equivalent to the specified time with that outcome is the corresponding number of QALYs.

Determine the Probability of Each Chance Event

Once the decision tree structure is formed, the probability of each chance event may be determined. In general, this is best performed by taking a systematic review of published, peer-reviewed literature. However, this approach assumes that probabilities derived from a period in the past accurately reflect the probabilities in the future. Moreover, it assumes that the probabilities derived from other settings apply to the one in question. However, all these probabilities must reflect actual practice. For example, if decision analysis is being used to determine the best strategy for specific individual patients, the probabilities must be those of the site when the care is to be delivered. At times, not all of the information needed for the decision tree is available. This frequently occurs during analysis of a relatively new practice. If possible, primary data may be collected or secondary data may be analyzed. Expert opinion may be used in the absence of relevant data. At a given chance node, the sum of probabilities equals one.

Deciding on a Strategy: Averaging Out and Folding Back the Tree

The goal of decision analysis is to identify the strategy that leads to the most favorable expected outcome. To calculate the expected outcome, one starts at the outcome measures (typically the far right). Each group of branches, which start at a single chance node, is reduced to a single numerical value by multiplying each outcome measure by the probability associated with that outcome and adding all of the values. This is the process of folding back the decision tree. This process is repeated until there is a single numerical value for each strategy at the initial decision node. At this point, the expected outcome for each strategy has been calculated. The strategy with the more favorable outcome is the preferred strategy.

Discounting Future Events

A greater value is placed on current events than future events. It is better to pay $100 in 10 years rather than pay $100 now. Similarly, if one is to have a disease, it is preferable to have a disease in the future than to have it today. The value of a future event then depends on how far in the future it occurs. Discounting refers to calculating the present value of an outcome that occurs in the future. The discount rate is the annual rate at which costs are discounted, which is usually the rate of interest that money would bring if it were invested.

Sensitivity Analysis

Sensitivity analysis is an important part of the decision analysis process that tests the stability or robustness of a conclusion over a range of structural assumptions, value judgments, and estimates of probability. The initial analysis, or base case analysis, uses the best estimates for each part of the model. In sensitivity analysis, a plausible range of values is determined for each portion of the model. Variables that have the most influence on the model are determined. Different time horizons or perspectives may be considered. The objective is to see whether conclusions change when possibilities within a reasonable range are included.

Cost-Effectiveness Analysis Using Decision Analysis

Cost-effectiveness analysis is the use of decision analysis to compare strategies in terms of their cost per unit of output. Output is an outcome such as years of life, utility, or cases of disease prevented. Cost-effectiveness ratios are interpreted by comparing them to ratios for other strategies. An incremental cost-effectiveness ratio indicates how much additional money needs to be spent for a better, but more expensive strategy to generate one additional unit of outcome.

Of practical importance, there usually is a limit to the amount of money a policy-maker is willing to spend to gain one QALY; this is termed the willingness-to-pay threshold. Cost–utility analysis is a specific type of cost-effectiveness analysis that uses QALYs (or other measures of utility) as the effectiveness end-point. By convention, cost–utility analyses are often called cost-effectiveness analyses. However, not all cost-effectiveness studies use the cost–utility methodology. Because they use QALYs as an end-point, cost–utility analyses generate information that may be compared across disease states. It is important to note that although they are useful in providing a common metric among outcomes, a number of stringent assumptions must be made to compare cost-effectiveness analyses. Principles for assessing decision analysis have been developed by the evidence-based medicine (EBM) group and are shown in Table 3–2. Representative decision analyses related to endocrine disorders are shown in Table 3–3.

The methods of clinical epidemiology have been applied to risk, prognosis, the assessment of studies of different designs, and many other issues in clinical medicine. Readers are referred to standard texts.

The definition of evidence-based medicine (EBM) has evolved over time. A 2005 definition states that EBM requires the integration of the best research evidence with clinical expertise and patient's unique values and circumstances. The best research evidence refers to valid and clinically relevant research, often from the basic sciences of medicine, but especially from patient-centered clinical research. Clinical expertise refers to the ability to use physicians' clinical skills and past experience to rapidly identify each patient's unique health state and diagnosis, their individual risks and benefits of potential interventions, and their personal circumstances and expectations. Patient values consist of the unique preferences, concerns, and expectations each patient brings to a clinical encounter that must be integrated into clinical decisions if they are to serve the patient. Patient circumstances refer to the patient's individual clinical state and the clinical setting.

The five steps of EBM as defined in 1997 are discussed below. Entire textbooks are devoted to the principles of EBM. In this chapter, we will briefly review these five steps and focus on how EBM views evidence and review some of the limitations.

Step One: Translation of the Clinical Problem into Answerable Questions

One of the fundamental skills required for practicing EBM is the asking of well-built clinical questions. To benefit patients and clinicians, such questions need to be both directly relevant to patients' problems and phrased in ways that direct a literature search to relevant and precise answers. In practice, well-built clinical questions usually contain four elements: patient, intervention, comparison, and outcome. The patient refers to the individual to whom the evidence will be applied (eg, a hypertensive patient with diabetes). (The evidence-based approach can also be applied to a group to inform policy.) The intervention is the diagnostic test or therapy being considered for application to the particular patient (or group) (eg, tight control of blood pressure). The comparison is that intervention to which a new intervention being considered is to be compared (eg, less tight control of blood pressure). The outcome is the end point of interest to the physician or patient (eg, stroke, myocardial infarction, or death from diabetes). Background questions relate to general information or basic facts about a disease, and this information can usually be found in reliable textbooks, whether published in book form or on-line. However, more involved clinical questions, especially those that require the most up-to-date information about diagnosis, prognosis, or treatment, have been termed foreground questions and require a different approach.

Step Two: Finding the Best Evidence

The finding of best research evidence from the medical literature is a critical feature of the EBM approach and is an iterative process that involves medical informatics. Given the enormous breadth and depth of the medical literature, with more than two million articles published each year in about 20,000 biomedical journals (with more being established each year), efficient strategies must be used to take advantage of the best original journal articles, reviews and synopses of primary studies, and practice guidelines, along with textbooks, both traditional and innovative. Among the useful sources of information are PubMed, which allows sophisticated search strategies of the MEDLINE database. A variety of tools have been developed to facilitate this process. However, these are beyond the scope of this chapter, and readers are referred to references on the subject. Certain journals are specifically devoted to critically appraised articles (eg, ACP Journal Club, BMJ Evidence-Based Medicine). It is important to remember, however, that the presence of the label evidence based does not ensure that the information meets the highest standards. Systematic reviews are another useful source for evidence. In addition to the professional journals, such reviews are available from both governmental agencies (eg, Agency for Healthcare Research and Quality) and nongovernmental agencies (eg, the Cochrane Collaboration). Standards have been developed for reporting systematic reviews: preferred reporting items for systematic reviews and meta-analyses (PRISMA). Among the issues addressed specifically is that of publication bias and its potential effects on the results. Publication bias reflects, among other things, the tendency to not publish studies with negative findings.

Step Three: Appraising the Evidence for Its Validity and Usefulness

Critical appraisal is the process of systematically examining research evidence to assess its validity, results, and relevance before using it to inform a decision. There are three basic questions to be addressed in evaluating any kind of research: (1) Is it valid? (2) Is it important? and (3) Is it applicable to my patient? This involves assessment of the study's methods in order to assess the internal validity of the conclusions. Once criteria for internal validity are satisfied, then the importance of the finding can be assessed. It is critical that statistical significance not be equated with clinical significance. Assuming the finding is important, it is necessary to consider the study's relevance to the patient (ie, the external validity of the study). A series of guides has been developed to help the clinician to critically appraise different types of studies. An example is shown in Table 3–4. Underlying this appraisal process is the EBM model of the hierarchy of evidence.

Hierarchies of Evidence

Any empirical observation about the apparent relation between events constitutes evidence. This includes the thoughtful observations of an experienced clinician, the observations derived from physiologic experiments, and the results of large, randomized clinical trials. Methodological safeguards to protect against systematic error (bias) are inherent in experimental design. In this way, experiments differ from clinical observations. Different study designs lend themselves to different safeguards, and these safeguards vary in their efficacy both in theory and in practice.

The EBM approach uses a hierarchy of study designs based on their inherent safeguards. A hierarchy is shown in Table 3–5. In this hierarchy, randomized controlled trials (RCTs) are given the highest weight, followed by quasi-experimental studies (ie, no-RCTs), followed by other study designs, and finally opinions of authorities and unsystematic clinical observations. Evidence from the top of the hierarchy should have greater impact in clinical decision-making than observations from the lower levels. There is disagreement about what the hierarchy should be. Some have an N of 1 randomized trial (the equivalent of an RCT with one subject) at the top of the hierarchy, whereas others place meta-analyses of randomized trials at the top. However, the term evidence is not synonymous with evidence from RCTs (or whatever study design occupies at the preferred level of the hierarchy). Rather, it is preferred to use that sort of evidence when it is available. Evidence from other sources is still evidence and may be more than sufficient. For example, the introduction of thyroid replacement therapy for hypothyroidism or insulin therapy for type 1 diabetes has never been subjected to RCTs.

While recognizing the central role of evaluating the quality of the evidence in decision-making, the U.S. Preventive Services Task Force (USPSTF) has developed a different approach to address some of the shortcomings of the EBM approach. The USPSTF adopted three major changes to the process. First, a rating of internal validity was added to the study criterion for judging individual studies. A well-designed cohort or case-control study could be more compelling than an inadequately powered or poorly conducted RCT. Second, evidence was explicitly assessed at three different levels—the level of the individual study, both in terms of internal and external validity; the body of evidence supporting a given linkage in the analytic framework; and, because of the focus of the USPSTF, the level of evidence for the entire preventive service. Third, the magnitude of effect was separated from the assessment of quality. Both benefit and harm are considered in assessing the magnitude of the effect(s). Overall, the USPSTF grades the quality of the overall evidence for a service on a three-point scale (good, fair, poor) and grades its recommendations according to one of five classifications (A, B, C, D, I) reflecting the strength of evidence and magnitude of net benefit (benefits minus harms) (Table 3–6). This approach of assessing both the quality of evidence and strength of recommendations has been expanded to include practice guidelines: grading of recommendations, assessment, development, and evaluation (GRADE) system. Although a discussion of practice guidelines is beyond the scope of this chapter, it is important to recognize that they vary widely in quality, strength of evidence, and bias. Before a practitioner bases a decision on a practice guideline, the basis of that guideline must be clear.

Other Evidence-Related Issues

Notwithstanding these improvements, there are three other major issues concerning the evidence. First, it is important to recognize that all of this evidence is essentially quantitative in nature. Thus, the EBM definition of evidence is relatively narrow and excludes information important to clinicians; many qualitative factors are involved in clinical decision-making concerning individual patients. Second, there are problems in the evidence in terms of the quality and scope of the data (eg, the soft data that clinicians usually use); the scope of the topics (average patients, gray zones of practice, lack of RCTs); sources of authority, which however explicit, require judgment; and potential abuses (eg, key findings could be ignored). Third, there is the issue of applying results in populations (ie, average results) to individuals.

Steps Four and Five: Applying the Results in Practice and Evaluating Performance

Knowledge of the best available evidence of treatment is not enough to recommend a particular course of action. Underlying this issue is heterogeneity of treatment effects. Treatment effect heterogeneity is the term given to the phenomenon in which the same treatment produces different results in different patients. For the average effect observed in a clinical trial to occur with certainty in one patient, all patients in the trial must have had that average response. Yet average response may also reflect a large response in some and small or no response or even harm in others. Part of clinical decision-making involves addressing this phenomenon to individualize therapy in the most effective way. To do so, four major factors about the patient must be taken into account: (1) baseline probability of incurring a disease-related adverse event (risk without treatment or susceptibility/prognosis), (2) responsiveness to the treatment, (3) vulnerability to the adverse side-effects of the treatment, and (4) utilities for different outcomes. When lacking good data on a patient's individual level of risk, responsiveness, and vulnerability, the average treatment effect as reported in good clinical trials provides the most reasonable guide for decision-making. However, it is important that physicians evaluate the results in their own patients so that the course can be modified when necessary.

Practicing and perfecting the art of medicine demands recognition that uncertainty permeates all clinical decisions. How clinicians make decisions, whether diagnostic or therapeutic, by combining the art and the science is a complicated matter replete with unknowns. There is a variety of modes of clinical problem-solving ranging from automatic decisions in which a clinician makes a decision without consciously considering alternatives or even specific features of the problem to rational decision-making using simplified strategies to take into account probabilities and values of outcomes to formal decision analysis. The potential deficiencies of automatic decisions (nondecisions one might say) are obvious. Formal decision analyses, even if accepted as gold standards (which they are not), are beyond practical use for the vast majority of clinicians, so that we are left with simplified processes to combine likelihoods and values. However, even in striving to reduce uncertainty and coming up with the best course of action for the patient, it is critical to recognize that uncertainty will always be with us. In using a quantitative approach, we can only become more certain about the probability of the outcome. We cannot guarantee the desired outcome.

General

Diagnostic Testing and Treatment Decisions (Clinical Decision Making and Decision Analysis)

Evidence-Based Medicine: General

Translation of the Clinical Problem into Answerable Questions

Finding the Best Evidence

Critical Appraisal and Applying the Results

Practice Guidelines

Internet Sites Related to Evidence-Based Medicine