Community Research and Development Information Service - CORDIS

  • European Commission
  • Projects and Results
  • Periodic Report Summary 1 - BAYINNO (Bayesian Truth Serum and its applications to conjoint analysis: a reliable way to assess user preferences for new products, services and policies)

Periodic Report Summary 1 - BAYINNO (Bayesian Truth Serum and its applications to conjoint analysis: a reliable way to assess user preferences for new products, services and policies)

Many areas of economics use subjective data, gathered from respondents with traditional surveys as well as through new web-based information exchanges and markets. For example, innovation development relies crucially on input elicited from potential innovation adopters. Subjective data is known to present problems regarding its truthfulness, although this may not necessarily be a result of intention to deceive. Resolving the problem of truthfulness would improve reliability of current uses of subjective data and also open up new uses.
One solution to this problem is presented by Bayesian Truth Serum (BTS). BTS is a game-theoretic scoring system that provides incentives for honest reporting of private judgments, such as forecasts. It was developed by Drazen Prelec and published in Science in 2004.
The method works in the following way: for a particular question, respondents are asked (1) to choose among a finite number of possible alternative answers to the question, and (2) to give their estimate what percentage of the respondents endorsed any of the alternatives. The method assigns a score to each of the respondents according to their answers, their predictions of how other people will respond, and on the actual answers by the other respondents. An important feature of the BTS method is that respondents are rewarded not only for their knowledge of the topic, but also for how well they know their peers, i.e. for their meta-knowledge (their knowledge about what other think).
By improving the quality and reliability of subjective data, the BTS method opens completely new possibilities for subjective judgment to be incorporated into economic research, most notably innovation research. One of the most popular methods in innovation design is conjoint analysis, a set of methodologies with a predominantly statistical content that originated from mathematical psychology. It is an experimental approach for measuring consumer’s preferences about a product, service, project or policy, and has been widely used in social and applied sciences. A central feature of conjoint approach is that the utility derived from the object can be decomposed into part-worths (or utility scores) that relate to different attributes of that object. By combining the utilities for different attributes/features, the individual’s overall relative utility is provided for each of the hypothetical bundles. This allows researchers to identify optimal bundle of attributes and levels that represents the optimal product, service or new public policy. Results from conjoint analysis can be further used for forecasting purposes.
Conjoint analysis has been evolving since its beginnings searching for ways to improve predictive accuracy. Two main research streams are concerned with: (1) the relative merits of the various data-collection methods including full-profile ratings, paired comparison data, self-explicated data, and stated preference; (2) better estimation approaches which led to Adaptive Conjoint Methods, Hierarchical Bayes methods and Hybrid Conjoint approaches.
In this project it is proposed that another novel way to improve conjoint analysis is by combining it with the BTS algorithm, which addresses both the data and forecasting aspect. BTS has the ability to improve both reliability and accuracy of collected data through incentivizing truthfulness, and to improve forecasting accuracy without resorting to complicated estimation techniques.
To show that BTS can improve forecasting accuracy, classic conjoint model is compared with the one where utility structure is derived from meta-predictions (each respondent gives meta-predictions for alternatives in every choice set, and the alternative with the highest meta-prediction is defined as meta-predicted choice). On the data collected from a series of experiments, it was demonstrated that meta-predicted choice indeed produces smaller forecasting errors. Recognizing heterogeneity within the respondents, it was also showed that those subjects who are better predictors (and have higher BTS scores) are more accurate and have better internal precision. This shows the usefulness of meta-expertise as a more accurate source of information on utility.
Since conjoint analysis involves a series of repeated choices, the first step was to gain in depth understanding of how BTS can be incorporated in modelling of one choice event where people choose between two alternatives. An example for such an event is purchase intention survey, which asks subjects whether they want to buy a product or not. It was shown that incorporation of BTS opens completely new ways to approach purchase intentions forecasting.
Purchase intention survey is often used for early sales forecasting and elimination of potentially failing concepts, although it is known that there is a systematic discrepancy between stated purchase intentions and purchase incidence. This is usually corrected by use of sophisticated statistical models, which require large quantities of respondent and product related data. This project proposes a novel explanation for this discrepancy by suggesting that those respondents who are not sufficiently knowledgeable about the product report unreliable purchase intentions. So in order to improve forecasting precision, the knowledgeable respondents should be identified and only their data should be used for forecasting. To show that, a theoretical model is developed and an easy to follow algorithm is produced; the algorithm shows how to identify the tacit knowledge holders from their meta-predictions only. At the same time the intervention in data collection is minimal, as only one question needs to be added to the questionnaire. This is much simpler than the state of the art that involves laborious and lengthy procedures that are described in the literature.
This work led to the exploration of the question of expertise in general. There is a pro and contra evidence in the literature for the accuracy of experts in prediction of market phenomena, but the explanation of when and why it happens has been lacking. The theoretical model proposed in this project gave insight into why product experts are not always experts on predicting the realized state of the world, and showed that the existing pro and contra evidence can be explained by one model. The only exception when model shows that product experts will always be experts on the state of the world is in case of attractive radical innovations, which coincides with the findings in the literature.

Reported by



Life Sciences
Follow us on: RSS Facebook Twitter YouTube Managed by the EU Publications Office Top