RATIO-SCALE ELICITATION OF DEGREES OF BELIEF Shimon Schocken Department of Information Systems NW Stern School of Business Room 9-80, 44 West 4th Street New York, NY 1001 2 (212) 998-0841 (212) 995-4228 (fax) sschocke@stern.nyu.edu September 1988 Working Paper Series STERN IS-88-95 Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Abstract Most research on rule-based inference under uncertainty has focused on the normative validity and efficiency of various belief-update algorithms. In this paper we shift the attention to the inputs of these algorithms, namely, to the degrees of beliefs elicited from domain experts. Classical methods for eliciting continuous probability functions are of little use in a rule-based model, where propositions of interest are taken to be causally related and, typically, discrete, random variables. We take the position that the numerical encoding of degrees of belief in such propositions is somewhat analogous to the measurement of physical stimuli like brightness, weight, and distance. With that in mind, we base our elicitation techniques on statements regarding the relative likelihoods of various clues and hypotheses. We propose a formal procedure designed to (a) elicit such inputs in a credible manner, and, (b) transform them into the conditional probabilities and likelihood-ratios required by Bayesian inference systems. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 1. Introduction Most rule-based expert systems fall in the category of deductive, inexact, classification models: given a set of observable clues, MI G M, an inference engine algorithm attempts to discern a set of hypotheses, H'G H, which provides the best explanation to MI. M and H are sets of propositions related to each other through inexact inference rules. For example, consider the following reasoning chain: the act of smoking (a disposition) increases the likelihood of a heart disease (an hypothesis), which, in turn, is sometimes manifested through a chest ache radiating to the left arm (a manifestation). This line of reasoning is plausible, but not necessarily categorical; many smokers will not develop any heart problems; likewise, chest ache is not a unique manifestation of heart disease. Hence, although causal information is indeed useful, any inference drawn from it must be qualified by the impreciseness of the underlying rules. During the past decade, a number of models were put forward to represent and carry out rule-based inference under uncertainty. are rule-based belief-update algorithms (Duda and Shortliffe, 1977), influence diagrams (Howard and Matheson, 1981), and belief networks (Pearl, 1986). These models are closely related to each other at the elicitation level, requiring human experts to specify a coherent set of degrees of belief reflecting the uncertainty associated with inference rules. This elicitation task, which is normally delegated to a knowledge Center for Digital Economy Research Stem School of Business Working Paper 19-88-95 2 engineer, is the focus of this paper. Throughout the paper, we use the belief network paradigm as our working environment; at the same time, our results are equally applicable to any inference model involving inexact rules. A belief network is an acyclic, directed graph, consisting of propositional nodes and causal arcs. The directed arc (x,y) emanating from node x to node y represents our belief that x causes y directly. The strength of this causal relationship is modeled through the conditional probability P(y1x). If a node y has multiple causes, {xl, ...,xn), the degree of belief associated with this complex relationship is the conditional probability tensor ~(ylx~, xn). A belief network can be encoded as a set of inexact inference rules. In the rule-based terminology, an arc (x,y) and its label ~(y/x) to the rule IF x THEN y WITH DEGREE OF BELIEF ~(ylx). These rules are elicited from human experts. Automatic inference is carried out by following reasoning chains from the observable evidence, M1, back to its possible explanations, HI. This backward reasoning process is accompanied by a belief-update algorithm designed to order the prospective hypotheses h'E HI in terms of the posterior beliefs P(h1 ]MI). Unfortunately, this belief-update procedure is exponential in the size of the network, and, in general, is NP- hard (Cooper, 1987). If, however, the topology of the network Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 meets certain criteria, posterior beliefs may be computed efficiently (Pearl, 1986) . A belief network is constructed through an elaborate knowledge elicitation procedure involving domain experts and knowledge engineers. Now, humans are normally good at suggesting causal relationships between evidence and hypotheses. At the same time, humans have serious problems in estimating the uncertainty associated with such conjectures. For example, a physician can swiftly suggest that the cause of swollen ankles (m) might be a certain heart disease (h). However, this same expert might be at loss when asked to estimate the subjective probability associated with this rule, ~(mlh). When pressed to do so, the expert will probably produce a number, but the validity of this estimate is clearly questionable. We propose a global approach to elicitation which minimizes the guesswork and yields a credible set of degrees of belief. 2. The Problem Unlike early rule-based architectures, the belief network model has a sound interpretation on deductive as well as on probabilistic grounds. From a deductive standpoint, the network's nodes are viewed as propositions, and the network's topology reflects causal relationships among these propositions. From a probabilistic standpoint, these nodes are viewed as random variables associated with an unknown joint probability Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 distribution function (PDF). The network's topology is analogous to a set of independence assumptions imposed on the PDF. Generally speaking, if two nodes are not connected, the random variables that they represent are assumed independent. These assumptions simplify the PDF considerably; hence, they also place an explicit constraint on the family of PDF's that might be modeled as belief networks. Consider, for example, the simple network depicted in Figure 1, taken from the domain of diagnosing heart diseases. This network has the following interpretation: the disease h may be caused by either cl, c2, or both. h, in turn, manifests itself through subsets of the symptoms {ml,m2,m3). These relationships are taken to be inexact. For example, it is possible that both cl and c2 obtain but h does not obtain. Likewise, it is possible (but not likely) that h obtains and none of the m's obtain. Fiqure 1 Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Hypothesis: Predispositions: h: heart disease cl: high Cholesterol count c2: smoking Symptoms : ml: chest ache radiating to the left arm m2: swollen ankles m3: shortness of breathing How can we elicit and represent the PDF P(cl,c2,h,ml,m2,m3)? one naive approach is to set up a global contingency table which specifies the joint frequencies of all possible combinations of propositions. Clearly, this is not a practical solution. Alternatively, if the propositions in question are arranged in a network like Figure 1, one can interpret the topology of the network as a graph-theoretic definition of the PDF: This derivation is based on a standard theorem in probability, the "chain rule," along with a set of marginal and conditional independence assumptions imposed on the PDF by the network's topology (Pearl, 1986) . Pearl has shown that a PDF like (1) is suspectable to an efficient belief-update algorithm which is consistent with probability theory. This algorithm computes posterior beliefs in hypotheses given any subset of observable clues (terminal nodes in the network), in time linear to the network's size. The algorithm involves only local computations, Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 and, in principle, can proceed in parallel. We now turn to the elicitation problem: in order to fully specify the belief network associated with a PDF like (I), the knowledge engineer must elicit three types of probabilities: (a) marginal probabilities associated with nodes with no parents, e.g. P(c1) and P(c2). (b) conditional probability tensors associated with multi- cause relationships, e.g. P(hlcl,c2)- (c) elementary conditional probabilities, e.g. P(mllh), ~(m2lh), ~(m3lh). The elicitation of conditional probability tensors is largely impractical, as the number of questions that one is required to ask grows exponentially with the number of propositions following the conditioning bar. There exist heuristic techniques, though, designed to approximate ~(hlc~,...,c~) the set of elementary probabilities ~(hlc~), P(hlcn) (Kim and Pearl, 1987). Hence, we see that the elicitation problem is primarily one of assessing elementary conditional probabilities. Therefore, we'll restrict our attention for now to the bottom tier of Figure 1, focusing on the relationships among h and its 3 manifestations. Our analysis can be easily extended to any number n>3 of propositions, so we'll sometimes use n=3 for brevity. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 3. Two Propositions on Elicitation Our approach to elicitation is motivated by two key propositions regarding the direction and ratio-scale properties of implicit degrees of belief. These propositions are briefly discussed below. Proposition 1: some questions are easier to answer if you turn them around. Consider the rule . This is sometimes referred to as llabductivell nbackwardll reasoning. That is, although the causal relationship between the hypothesis h and its manifestation m is h->m, we are typically faced with the problem of assessing the likelihood of the unknown h in light of the observable fact m. The interpretation of the belief function d(.l.) depends on our choice of a belief language. Most of these languages, though, are wunidirectional." That is, they consist of either diasnostic or causal inference, but not of both. For example, the certainty factors calculus and the Dempster-Shafer model require the expert to specify diagnostic degrees of belief in terms of C~(hlm) and ~el(h/m), Conversely, the Bayesian belief network specifies the causal relationship "h causes mm directly, requiring the expert to estimate the causal degree of belief ~(mlh), being a probability. Indeed, there exist a growing body of literature suggesting that Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 8 humans find it easier to "think forward in reverse," preferring causal on diagnostic explanations of evidential reasoning (Einhorn and Hogarth, 1987, Shachter and Heckerman, 1987). These findings clearly render cognitive justification to the Bayesian belief network formalism. At the same time, the insistence that in any given situation human reasoning is confined to proceed in only one direction seems to be overly restrictive. For example, consider the rule . Denote this rule by S -> HD. Which subjective probability is more credibly available from a human expert, P (HD ( S) or P (S I HD) ? The answer seems to depend largely on the experience of the expert, and, in particular, on his or her ability to retrieve examples from the S and HD populations. If the expert knows relatively far more smokers than she knows people with heart diseases, it is probably safer to use the smokers population as a reference group (Figure 2-a) and go on to assess P(HD[S). If, alternatively, we force this expert to specify P(S/HD), she will have to resort, in her mind, to a small sample (Figure 2-b), yielding a highly unreliable estimate of P(S/HD). The situation changes if the expert happens to be a heart disease specialist. his latter expert will probably find it easier to assess P(S~HD), due to the large sample of people with heart disease that she can retrieve from her clinical work experience. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Fiaure 2-a Fiaure 2-b We see that the factors that influence implicit beliefs are both availability and representativeness (Tversky and Kahneman, 1974). The availability heuristic leads the expert to focus on the population which is more salient or vivid in his mind. Strictly speaking, this heuristic is beneficial only if it coincides with a larger population. If the selected background population is small, most people will still be willing to use it as a representative image of the overall population. This insensitivity to sample size, or the Itlaw of small numbers,If is a manifestation of the well-known representativeness bias. To debias this flaw, the expert should be encouraged to retrieve as many examples as possible from both populations. The larger sample should then be selected as the conditioning assumption. Hence, we wish to support experts who are willing to express their reasoning using diagnostic inference instead of or in tandem with causal inference. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Proposition 2: relative degrees of belief are more credibly available then absolute degrees of belief. We take the position that the numerical encoding of degrees of belief is somewhat analogous to the measurement of such physical stimuli as brightness, weight, and distance. Can you specify the aerial distances between New York, Tokyo, Cairo, and Seattle with good confidence? probably not. It seems obvious that you would rather prefer a statement like: Tokyo is about twice as far away from New York as Cairo is. In general, people find it easier to express differences between physical quantities (as well as preferences) using relative, pair-wise judgement, rather than a cardinal scale of measurement (Stevens, 1957, Stevens and Galanter, 1964, Krantz, 1973). Going back to the previous question, let's take New York as our point of departure, and consider the following pair-wise distance comparisons: Cairo Seattle New York Tokyo The entries in this table read as follows: from New York, Tokyo is 2 times as far as Cairo, and 3 times as far as Seattle. Also, Cairo is 2 times as far as Seattle. Note that these judgement are not only inaccurate but also inconsistent, as is normally the case with human inputs (the inconsistency may be resolved, for Tokyo Cairo Seattle 1 2 3 Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 11 example, by setting the third entry in the second row to 1.5). How can we synthesize this set of n(n-1)/2 pair-wise comparisons into a vector of n weights reflecting the true distances implicit in the human's answers? Several such methods have been proposed, e.g. least squares (Cogger and Yu, 1983), and logarithmic least squares (De Graan, 1980). However, perhaps the only method capable of synthesizing inconsistent inputs in a credible manner is the Eigenvector method proposed by T.L. Saaty (1980). This method is illustrated in the next section. 3. One-Way Elicitation TO restate the elicitation problem, consider an hypothesis h and a series of related manifestations {ml, ..*,mn). Our goal is to estimate the probabilities vector ~=<~(m~lh), using pair-wise comparisons, elicited from a human expert. That is, rather than asking the expert to specify the absolute probabilities p(mi 1 h) and p(mj 1 h) , we ask her to estimate the extent to which mi is more likely than mj in light of h. These local judgments, which are likely to be inconsistent, will be further synthesized into a ratio-scale of probabilities. To illustrate, consider the following set of questions: Assume that a person X has a heart disease (h). Now consider the following two observations: X suffers from a chest ache radiating to the left arm (ml) X suffers from swollen ankles (m2). In your opinion, which observation (ml,m2) is more likely in light of h? (assume the expert answered ml) Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 To what extent is you think that ml number 1. If you ml more likely than m2? For example, if and m2 are equally likely, enter the think that ml is twice as likely as m2, enter the number 2. Feel free to enter any number greater than 1 that you think describes the extent to which ml is more likely than m2: With n manifestations, we have to ask the expert n(n-1)/2 such questions. The expert's answers are recorded in an nxn likelihood matrix, A, in which aij=P(milh)/~(m~lh). This approach is similar to Gupta and Wilson's (1987), who had experts express their opinions regarding the performance of competing forecast models. These inputs were recorded in a matrix A in which the element aij was the perceived odds that forecast model i will outperform forecast model j. The cognitive complexity of the elicitation may be somewhat mitigated if we ask the expert to first rank-order the manifestations in terms of decreasing perceived likelihoods, yielding (without loss of generality) an ordered set with p(mi 1 h) >p(mj 1 h) if i0 and aij=l/aji. We are thus in the familiar domain of Saaty's Center for Digital Economy Research Stem School of Business Working Paper IS-88-95 13 analytic hierarchy process, and we can proceed to compute the principal Eigenvector of A, denoted W. The crux of this approach is the notion that W is "ratio equivalentw to the unknown PI with wi/wj=~(miih)/~(mj1h) for all pairs ~{l, n)x{l,..., n). For example, suppose n=3 and the true probabilities vector ~=<~(mllh),P(m2/h),P(m31h)> as follows: Had we had access to an expert whose judgement are perfectly calibrated with reality, we would end up with the following matrix: Each entry aij in the matrix represents the degree to which the manifestation mi is more likely than the manifestation mj in light of h, in the expert's mind. For example, Since A in the above ideal example is consistent, its principal Eigenvector is given by any of its columns. Thus, focusing for Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 14 example on the last column, we obtain the following weights vector: Due to A's consistency, the true probabilities vector P (2) and the weights vector W (3) are ratio-identical. In reality, though, the vector P is unknown, and the matrix A is inconsistent. That is, the set of answers to the n(n-1)/2 questions presented to the expert will yield intransitive responses, with P (mi 1 h) P(mj lh) ------- * ------- P (mi 1 h) ------- P (mj I h) P (mk / h) P(mkfh) Hence, in the general case, we have to deal with an inconsistent matrix. To illustrate, refer to the A matrix, and assume that the expert's above-diagonal judgement alar a13, and a23 were +25%, -25% and +50% off mark (with respect to A), respectively, yielding the following inconsistent matrix: The normalized principal Eigenvector of this matrix is: Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Where do we proceed from here? the key assumption underlying our approach is that in spite of A's inconsistency, W' is still a good "ratio-estimatew of the unknown probabilities vector P. Thus, had we had a-priori knowledge, say, that ~(m~lh)=0.2, could have used it as an anchor to compute the following estimate P' of P: The difference between this result and the true probabilities vector P=<0.8,0.2,0.1> stems from the inconsistency of At and the imperfect knowledge that it represents. However, considering the expert's biasdeness and inconsistency, this result is surprisingly good, illustrating the robustness of the normalized Eigenvector to data perturbations. Of course, the ability to construct PI hinges on our a-priori knowledge of any one of its underlying probability elements. In some situations, we may anticipate this requirement ex-ante and augment the initial set of manifestation with an additional clue, m*, whose conditional probability ~(m*/h) either known or can be credibly estimated. For example, in the Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 above heart disease scenario, m* might be the fact that the underlying patient is a male, the assumption being that the probability ~(malefheart is well-known. There exist situations, however, in which we have no prior knowledge of any one of the underlying degrees of belief. In these cases, we can use an extension of the above technique in order to estimate the likelihood-ratio vector <~(m~jh)/~(m~lh),...,P(m~lh)/~(m~l~)>, "not h." Methods to compute such vectors for dichotomous and multi-valued propositions are described in the next section. We conclude the present section with a brief comment regarding the scale of measure used throughout the paper. Our approach to elicitation is based on the assumption that humans are capable of describing relative likelihoods using numbers. This controversial assumption was challenged by many, not the least of them is H.R. Haldeman, Chief of Staff of President Nixon. Describing Kissingerls persistent concern about a Russian attack on China, Haldeman recalls how used to tease him about his use of percentages. He would say there was a 60% chance of a Soviet strike on China, for example, and I would say: why 60, Henry? Couldnlt it be 65% or 58%? (Kotz and Stroup, 1983). Clearly, Haldeman's point is well taken. Although there exist evidence that people find it easier to assess odds rather than probabilities, the credibility of any numeric measure of Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 intuitive judgement is an open question. It was this motivation which led to Saaty's (1980) 1 to 9 scale, which is often accompanied with a verbal interpretation. This scale can be slightly modified to suit our elicitation context, as follows: 1 - propositions p and q are equally likely 3 - p is weakly more likely than q 5 - p is strongly more likely than q 7 - p is very strongly more likely than q 9 - p is absolutely more likely than q The numbers 2,4,6, and 8 are used to express intermediate judgement between adjacent scale values. This scale has been justified by Saaty on analytical, psychological, and experimental grounds. In a similar vein, Lichtenstein and Newman (1967) have shown empirically that verbal descriptions of uncertainty may be mapped on ranges of probabilities. These findings are especially relevant to the Eigenvector method, which is insensitive to the type of scale being used. As Harker and Vargas (1987) argue, "One scale may be appropriate for one application and may not be appropriate for another. In this situation, a different scale could and should be chosen for each application." For example, in situations where little is known about a particular set of hypotheses, a 1 to 3 scale might be used. Clearly, the freedom to modify the scale of measure or develop a totally new one adds significant flexibility to the elicitation task. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 4. Two-way Elicitation Most realistic inference problems involve multi-valued propositions. For example, the dichotomous "heart diseasew hypothesis can be made finer by considering the following possibilities: hl: X is likely to suffer a fatal stroke h2: X is likely to suffer a mild stroke h3: X has no heart disease From a logical perspective, to say that h assumes one of the values in H={hl,hZ,h3} is equivalent to assigning truth values to the three dichotomous propositions hl, h2, and h3. We will assume henceforth that H enumerates all the possible values that h can attain and that these values are mutually exclusive. Using the techniques described in the previous section, we can condition our set of causal elicitation questions on each value of h, yielding three matrices and three weight vectors Whl, Wh2, and Wh3, where Whi is conditioned by the background hypothesis hi. Now, these three matrices and respective vectors are disjoint: one cannot use them to calculate likelihood-ratios across competing hypotheses. That is, although we can calculate the ratios, say, P(mi 1 hl)/p(mj 1 hl) and p(mi 1 h2)/p(mj / h2) within the vectors Whl and Wh2 there is no sufficient information to compute the more useful likelihood-ratios ~(rn~lh~)/~(m~lh~) ~(mj lh2). These ratios were termed by Alan Turing the 'Iweights of evidencew carried by mi and mj, respectively, to the Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 19 statement Ifhl is preferred on h2" (Good, 1950). These weights of evidence play an important role in many Bayesian belief-update algorithms. Recall that the ultimate objective of any belief-update algorithm is to compute the posterior beliefs in the competing hypotheses H in light of {m lr.*.rmn). If these manifestations are ratio-independent with respect to H (Grosof, 1986), the posterior odds favoring hi on hj is given in terms of Bayes rule, as follows: This formula has the following tfmechanicalw in the absence of any relevant evidence, the posterior odds are set to the prior odds, P(hi)/P(hj). When we know that a certain manifestation is present, say mk, we multiply these odds by the likelihood-ratio p(mklhi)/p(mk/hj). As more manifestations become available, the posterior odds are updated in a similar fashion. Hence, what we are after are likelihood-ratio vectors of the form Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 20 These vectors must be specified for all pairs of competing hypotheses eH~H. How do we go about computing these vectors? we propose a solution, called two-way elicitation. The first stage of this procedure is identical to the one-way elicitation described in the previous section. Having completed this line of causal questions, we ask the expert to go through a second stage of diaanostic questions, The expert's answers are then synthesized into a set of n(n-1)/2 likelihood-ratio vectors like (5). For now, we assume that the second stage involves roughly the same number of questions as the first stage. As we'll see shortly, the second stage is far less demanding. A typical diaanostic question has the following form: Assume that a person X suffers from swollen ankles (m2). Now consider the following two possibilities: X will suffer a mild heart attack (h2), X has no heart disease (h3). In your opinion, which possibility (h2,h3) is more likely in light of m2 (assume the expert answered h2) To what extent is h2 more likely than h3? For example, if you think that h2 and h3 are equally likely, enter the number 1. If you think that h2 is twice as likely as h3, enter the number 2. Feel free to enter any positive real number that you think describes the extent to which h2 is more likely than h3: With the manifestation m2 fixed, three such pair-wise comparisons are required to construct a likelihood matrix and a diagnostic weight vector, denoted Wm2. Wm2 is taken to be ratio-equivalent Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 21 to the probabilities vector . Two- way elicitation thus consists of two independent elicitation stages: in the first stage (described in the previous section), the three causal weight vectors WhlrWh2,Wh3 are computed. The second stage yields the three diagnostic vectors WmllWm2,Wm3. The restriction of n=3 can be easily lifted, and the number of hypotheses and manifestations need not be equal. We now show how these disjoint sets of vectors can be synthesized into the n(n-1)/2 likelihood-ratio vectors (5). The data gathered in the two-way elicitation procedure can be represented in two related, directed graphs (Figure 3). The nodes Mij and Hij represent the unknown probabilities p(rnilhj) and P (hi / mj ) , respectively. The directed arcs (Mi j ,Mkl) and (HijI are labeled Mijk- and Hijkl , respectively. Each triangle Hk in the causal graph represents the causal weights vector WhkI k=1,2,3, with M~~~~=P . As was argued earlier, these ratios are not particularly useful in their present form, and we are more interested in the likelihood-ratios ~~~~~'~(m~lh~)/~(m~lh~) to which the manifestation mi serves to discriminate between the hypotheses hj and hk. In Figure 3, Mijik is the label of the "externalH arc connecting Mij and Mik across the two triangles Hj and Hk. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 PI33 Causal graph Diagnostic graph Fiaure 3 Let's focus on the following two sets of labels: Mij ik=P (mi I hj ) /P (mi I hk) (external, causal arcs) (6) Hj iki'P(hj 1 mi) /P (hkl mi) (internal, diagnostic arcs) (7) Also, denote the prior odds favoring hypothesis hk on hj by: okj = P(hk)/P(hj) (8) Given this notation, (6), (7), and (8) are related through Bayes rule: Hence, we see that the two graphs mirror each other. That is, every cro~s-trianqle arc Mijik at the causal graph is proportionally related to a within-trianale arc Hjiki at the Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 23 diagnostic graph. For example, the bold circuit (triangle M2) of the diagnostic graph is the "dual imagetf the external circuit connecting a11 M2k nodes, k=1,2,3 across all triangles in the diagnostic graph. Hence, had we had access to a prior weights vector W0= which is ratio-equivalent to the prior probabilities vector , we could have used (9) to compute the desired likelihood-ratio vectors <~(m~lh~)/~(m~lhj) for any pair of competing hypotheses HxH. It turns out that one half of the two-way elicitation procedure can be cut down considerably, due to some favorable graph properties. To illustrate, consider Figure 3. Once we nconstruct" any one triangle in the diagnostic graph, we can compute its dual external circuit in the causal graph, through (9). And, due to the topology of the causal graph, any such single circuit makes the entire causal graph connected, meaning that there is a path between any two given nodes. Let us define the "intensitytt a directed path as the product of all the labels along the path. We immediately get, from the label's definition, that this is a telescopic product. For example, the intensity of the path <(M31,Mll), (Mll,M21) , (M2ltM22) (M22,M32)> is: P(m3 1 hl) p p (6) p-2, P (m3 1 hl) ,,-,,,,, * -------- * -------- * -------- = -------- P~) pW1) P (F2 ) P(m3 / h2) P(m3 1 h2) Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Similarly, the intensity of a loop is 1. This implies that the intensities of all the paths connecting a given pair of nodes are the same. This is true because the paths are directed, and the only difference between two paths with the same end-points is that one might consist of one or more loops. However, the intensity of these loops is 1, so they have no effect on the overall path intensity. Let us illustrate the relevance of this analysis to the elicitation problem. Suppose that we have gone through the first stage of the elicitation procedure, yielding the three triangles HI, HZ, and 83. We now turn to the diagnostic, second stage of the elicitation. After reviewing the various manifestations, we find out that M2, say, is the most reliable background population (recall proposition 2). Therefore, we proceed to compute the diagnostic weight vector WmZI which is represented by the bold triangle M2 in Figure 3. At that stage, if we have access to the prior weights vector, Wo, we can proceed to construct the circuit connecting all M2k nodes, k=1,2,3, in the causal graph. By llconstruct'v mean that we can now compute the circuit's labels {M2122,M2223,M2321), which, in turn, allow us to compute any likelihood-ratio pertaining to m2. Moreover, we can now proceed to compute other desired likelihood-ratio Mikil=P (mi I hk) /P (mi 1 hl) , by simply taking the intensity of any path connecting the nodes Mik and Mil. Since the causal graph is Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 25 connected, at least one such path must exist. And due to the intensity uniqueness, this solution is also unique. Estimating prior probability vectors: We now turn to discuss the elicitation of the prior weights vector Wo= which is assumed to be ratio-identical to the prior probability vector . In some situations, e.g. when the {hi} represent well-known diseases, this vector can be estimated from textbook information, field records, or other relevant background knowledge. In other situations, though, we have to elicit Wo from a domain expert. In the latter case, we can simply use the one-way elicitation procedure described in Section 3. That is, the expert will be asked to compare the relative likelihoods of all pairs E HxH, forming a likelihood matrix. The desired vector, Wo, will then be taken to be the Eigenvector of this matrix. 5. conclusion Our elicitation approach relies heavily on the Eigenvector method, the cornerstone of Saatyts Analytic Hierarchy Process (AHP) . As Harker and Vargas (1987) put it, the AHP framework is designed to cope with intuitive, rational, and irrational judgement, with and without certainty. It is thus natural, in our opinion, to apply it to the problem of eliciting degrees of belief, where rational knowledge is often combined with intuitive guts feeling and, on occasion, with inconsistent judgement. It Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 was argued by Fischhoff et a1 (1980) that it is inappropriate to think of a person's opinion about a set of events as existing within that person in a precise, fixed fashion, just waiting to be measured. And yet, asking experts to provide numeric degrees of belief and adding this information verbatim to a knowledge- base is a common practice among many knowledge engineers. We think that the elicitation problem deserves a more serious treatment. This paper is a step in this direction. 6, References Cogger, K.O. and Yu, P.L., "Eigen Weight Vectors and Least Distance Approximation for Revealed Preference in Pair-Wise Weight Ratios," School of Business, University of Kansas, 1983. Cooper, G.F., "Probabilistic Inference Using Belief Networks is NP-Hard," Tech. Report KSL-87-27, Medical Computer Science Group, Knowledge Systems Laboratory, Stanford University, Stanford, CA, May, 1987. de Graan, J.G., ~lExtensions the Multiple Criteria Analysis Method of T.L. Saaty," Paper presented at EURO IV, Cambridge, July 22-25, 1980. Duda, R, O., and Shortliffe, E. H., "Expert Systems ResearchfV1 Science, 220, 1981, pp. 261-268. Einhorn, H. J. and Hogarth, R. M:, "Decision Making: going Forward in Reverse," Haward Buslness Review, January-February 1987, pp. 66-70. Fischhoff, B., Slovic, P., and Lichtenstein, S., ''Knowing What You Want: Measuring Labile Values," in T.S. Wallsten (ed.), Cognitive Processes in Choice and Decision Behavior, Erlbaum Associates, Hillsdale, New Jersey, 1980. Good, I. J., "Probability and the Weighing of Evidence," New York: Hafner, 1950. Grosof, B. N., "Evidential Information as Transformed ProbabilityIw in Lernmer, J. F., and Kanal, L. (eds.) , Uncertainty in Artificial Intelligence. North Holland, 1986. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95 Gupta, S., and Wilton, P.! 'Combinations of Forecasts: an Extension," Management Sclence, Vol. 33, No. 3, March 1987, 356- 372. Harker, P.T. and Vargas, L.G., "The Theory of Ratio-Scale Estimation: Saatyrs Analytic Hierarchy Process," Management Science, Vol. 33, No. 11, November 1987, pp. 1383-1403 Howard, R.A. and Matheson, J.E., I1Influence Diagrams," in nReadings on the Principles and Applications of Decision AnalysisN (R.A. Howard and J.E. Matheson, Eds.) Strategic Decision Group, Menlo Park, CA pp. 719-762, 1981, Kim, J.H., and Pearl, J., "CONVINCE: A Conversational Inference consolidation Engine," IEEE Trans. on Systems, Man, and Cybernetics, Vol. SMC-17, No. 2, March/April 1987, 120-131. Kotz, S. and Stroup, D.F., ItEducated Guessing," Marcel Dekker, 1983, p, 18. Krantz, D.H., ''A Theory of Magnitude Estimation and Cross- Modality ~atching,~! Math. Psychology, 9, 2 (1972), 168-199. Lichtenstein, S. and Newman, J.R., wEmpirical Scaling of Common Verbal Phrases Associated with Numerical Probabilitie~,~~ Psychonomic Science, 9, 1967, 563-564. Pearl, J., wFusion, Propagation, and Structuring in Belief Networks," Artificial Intelligence, September, 1986. Saaty, T.L., "The Analytic Hierarchy ProcessIfl New York, 1980. Shachter, R.D.! and Heckerman, D.E., "Thinking Backward for Knowledge Acq~lsition,~~ Journal, Fall, 1987, pp. 55-61. Stevens, S. S., "On the Psychological LawsIf1 Review, 64, 1957, pp. 153-181. Stevens, S.S. and Galanter, E., "Ratio-Scale and Category-Scale for a Dozen Perceptual Continua," J. Experimental Psychology, 54 (1964), 377-411. Tversky, A. and Kahneman, D., IsJudgment Under Uncertainty: Heuristics and Biases," Science, 185, September, 1974, pp. 1124-1131. Center for Digital Economy Research Stem School of Business IVorking Paper IS-88-95