Final Report Summary - BAYINNO (Bayesian Truth Serum and its applications to conjoint analysis: a reliable way to assess user preferences for new products, services and policies)
Many areas of economics use subjective data, gathered from respondents with traditional surveys as well as through new web-based information exchanges and markets. For example, prediction markets, which have been attracting increasing attention recently, rely on data from participants who provide probabilistic estimates of market-related and other events. Economic forecasting is another area that relies on subjective data, where “hard” data such as unemployment figures, actual consumer spending, exports etc., are often supplemented by qualitative judgments and surveys. Economic indexes such as Consumer Confidence Index and Happiness Index are used by economists to evaluate policy effects and plan future actions, while Labor Survey and Community Innovation Survey are just two examples of government surveys that are used to develop indicators and evaluate policies.
Subjective data is important for the private sector as much as for the public sector. Reliable and simple methods of eliciting data are especially important for small and medium enterprises that have very limited resources for market research. In use of subjective data the crucial question is its overall quality. More precisely, the key issue is how much confidence one can have in truthfulness of our respondents’ answers and consequently in the findings derived from them. Subjective data is known to pose reliability problems, although this may not necessarily be a result of intention to deceive. One solution to this problem is presented by Bayesian Truth Serum (BTS), developed by Drazen Prelec (Science, 2004). This is a game-theoretic scoring system that provides incentives for honest reporting of private judgments. The method assigns a score to each of the respondents according to their answers, their predictions of how other people will respond, and on the actual answers by the other respondents. An important feature of the BTS method is that respondents are rewarded not only for their knowledge of the topic, but also for how well they know their peers, i.e. for their meta-knowledge.
Innovation development relies crucially on input elicited from potential innovation adopters. Regardless whether in private or public sphere, innovation is inherently an expensive and risky activity. To minimize the risk and improve chances for success, innovation developers aim to get information from the stakeholders into the development process starting at the beginning and continuing throughout the process. By improving the quality and reliability of subjective data, the BTS method opens up completely new possibilities for subjective judgment to be incorporated into economic research, most notably innovation research.
The aim of this project was to show how BTS could be incorporated in existing methodologies for innovation development to create improvements and deliver new value. Conjoint analysis was chosen as a test bed for this approach, being a very popular methodology within innovation development. Conjoint analysis is actually a set of methodologies with a predominantly statistical content that originated from mathematical psychology. It is an experimental approach for measuring consumer’s preferences about a product, service, project or policy, and has been widely used in social and applied sciences. Although consumer measurement was recognized as a serious problem of this method due to wear-out, self-perception biases, and other phenomena, no solution was offered yet.
Since choice based conjoint analysis involves a series of repeated choices, this project started with the study of how BTS can be incorporated in modeling of one choice event where people choose between two alternatives. An example for such an event is purchase intention survey, which asks subjects whether they want to buy a product or not. Purchase intention survey is often used for early sales forecasting and elimination of potentially failing concepts, although it is known that there is a systematic discrepancy between stated purchase intentions and purchase incidence. This is usually corrected by use of sophisticated statistical models, which require large quantities of respondent and product related data. A novel explanation for this discrepancy is recognizing that those respondents who are not sufficiently knowledgeable about the product report unreliable purchase intentions. Consequently, in order to improve forecasting precision, only the data from the knowledgeable respondents should be used for forecasting. So instead of collecting additional product and respondent specific data, the new approach championed in the BayInno project is to identify “high quality” data and discard the rest. The problem lies in the inability to recognize the knowledgeable respondents ex ante, as their understanding is usually tacit. A theoretical model is developed and an algorithm is produced that shows how to identify the tacit knowledge holders from their meta-predictions only. At the same time the intervention in data collection is minimal, as only one question needs to be added to the questionnaire. The algorithm was tested in three online experiments.
Having resolved the incorporation of BTS in one choice set, the ground was set for consideration of BTS and repeated choices. The resulting new hybrid methodology is called predicted preference conjoint analysis. A theoretical argument mathematically demonstrates that replacing the own choice (the alternative that the respondent would choose for him/herself) with the predicted choice (the alternative the respondent thinks will be chosen by majority of others) yields a more accurate estimate of personal utilities and more precise forecast of preference shares for the alternatives. In addition, it is shown that these results remain valid even when the information from respondents’ social circles is biased, provided that the net bias across individuals is zero. These theoretical results are confirmed in three online experiments.
The advantage of both methods (purchase intentions with BTS and predicted preference conjoint analysis) is that engaging respondents’ meta-knowledge employs the information contained in their social circle. Although this information can be tacit, the new approach developed in BayInno project shows that it can be effectively used. In addition, the use of meta-predictions enlarges the sample as it implicitly includes all the members of respondents' social circles. Yet another practical advantage is that using social circles by way of meta-predictions relaxes requirements on sample representativeness. For example, although conjoint analysis is traditionally performed on carefully chosen representative samples (which raises the cost of the study), with new preference predicted conjoint it is possible to use online non-representative inexpensive samples as long as they are large enough so that the various biases cancel out. This makes conjoint analysis more accessible to small businesses, individual innovators, public sector entities, and anyone else who wants to develop an innovation and forecast its adoption, but has limited budget.
Subjective data is important for the private sector as much as for the public sector. Reliable and simple methods of eliciting data are especially important for small and medium enterprises that have very limited resources for market research. In use of subjective data the crucial question is its overall quality. More precisely, the key issue is how much confidence one can have in truthfulness of our respondents’ answers and consequently in the findings derived from them. Subjective data is known to pose reliability problems, although this may not necessarily be a result of intention to deceive. One solution to this problem is presented by Bayesian Truth Serum (BTS), developed by Drazen Prelec (Science, 2004). This is a game-theoretic scoring system that provides incentives for honest reporting of private judgments. The method assigns a score to each of the respondents according to their answers, their predictions of how other people will respond, and on the actual answers by the other respondents. An important feature of the BTS method is that respondents are rewarded not only for their knowledge of the topic, but also for how well they know their peers, i.e. for their meta-knowledge.
Innovation development relies crucially on input elicited from potential innovation adopters. Regardless whether in private or public sphere, innovation is inherently an expensive and risky activity. To minimize the risk and improve chances for success, innovation developers aim to get information from the stakeholders into the development process starting at the beginning and continuing throughout the process. By improving the quality and reliability of subjective data, the BTS method opens up completely new possibilities for subjective judgment to be incorporated into economic research, most notably innovation research.
The aim of this project was to show how BTS could be incorporated in existing methodologies for innovation development to create improvements and deliver new value. Conjoint analysis was chosen as a test bed for this approach, being a very popular methodology within innovation development. Conjoint analysis is actually a set of methodologies with a predominantly statistical content that originated from mathematical psychology. It is an experimental approach for measuring consumer’s preferences about a product, service, project or policy, and has been widely used in social and applied sciences. Although consumer measurement was recognized as a serious problem of this method due to wear-out, self-perception biases, and other phenomena, no solution was offered yet.
Since choice based conjoint analysis involves a series of repeated choices, this project started with the study of how BTS can be incorporated in modeling of one choice event where people choose between two alternatives. An example for such an event is purchase intention survey, which asks subjects whether they want to buy a product or not. Purchase intention survey is often used for early sales forecasting and elimination of potentially failing concepts, although it is known that there is a systematic discrepancy between stated purchase intentions and purchase incidence. This is usually corrected by use of sophisticated statistical models, which require large quantities of respondent and product related data. A novel explanation for this discrepancy is recognizing that those respondents who are not sufficiently knowledgeable about the product report unreliable purchase intentions. Consequently, in order to improve forecasting precision, only the data from the knowledgeable respondents should be used for forecasting. So instead of collecting additional product and respondent specific data, the new approach championed in the BayInno project is to identify “high quality” data and discard the rest. The problem lies in the inability to recognize the knowledgeable respondents ex ante, as their understanding is usually tacit. A theoretical model is developed and an algorithm is produced that shows how to identify the tacit knowledge holders from their meta-predictions only. At the same time the intervention in data collection is minimal, as only one question needs to be added to the questionnaire. The algorithm was tested in three online experiments.
Having resolved the incorporation of BTS in one choice set, the ground was set for consideration of BTS and repeated choices. The resulting new hybrid methodology is called predicted preference conjoint analysis. A theoretical argument mathematically demonstrates that replacing the own choice (the alternative that the respondent would choose for him/herself) with the predicted choice (the alternative the respondent thinks will be chosen by majority of others) yields a more accurate estimate of personal utilities and more precise forecast of preference shares for the alternatives. In addition, it is shown that these results remain valid even when the information from respondents’ social circles is biased, provided that the net bias across individuals is zero. These theoretical results are confirmed in three online experiments.
The advantage of both methods (purchase intentions with BTS and predicted preference conjoint analysis) is that engaging respondents’ meta-knowledge employs the information contained in their social circle. Although this information can be tacit, the new approach developed in BayInno project shows that it can be effectively used. In addition, the use of meta-predictions enlarges the sample as it implicitly includes all the members of respondents' social circles. Yet another practical advantage is that using social circles by way of meta-predictions relaxes requirements on sample representativeness. For example, although conjoint analysis is traditionally performed on carefully chosen representative samples (which raises the cost of the study), with new preference predicted conjoint it is possible to use online non-representative inexpensive samples as long as they are large enough so that the various biases cancel out. This makes conjoint analysis more accessible to small businesses, individual innovators, public sector entities, and anyone else who wants to develop an innovation and forecast its adoption, but has limited budget.