Contemporary methods for identifying and leveraging expertise in collective decision-making

Abstract

From company board committees to grand juries to political parties, decision making often involves a group of individuals with differing views on what the best decision may be. Individuals have different views, in part, because some individuals have more knowledge than others. In theory, decision makers can make better decisions by leveraging the expertise of individuals in the crowd by identifying those who have more knowledge than others. Unfortunately, it is often difficult to identify and leverage expertise in practice. Decision makers often have no records of individuals' past performance from which they can estimate expertise, and subjective measures such as confidence and self-ratings of expertise are often considered as unreliable. In this thesis, I demonstrate how individuals' meta-predictions -- predictions about what other individuals will predict -- can be used to identify and leverage expertise in the crowd on 'single-question' problems, where records on individuals' past performance are unavailable. I first examine how a recent algorithm in the literature can be used to distinguish between subsets of high-performing and low-performing individuals in the crowd on binary decision problems. I show that this algorithm is in fact weighting individuals by the absolute difference between an individual's vote and their meta-prediction about the votes of others, and thus this weighting metric can be used to identify and leverage expertise in the crowd. I develop an improved weighting approach that uses individuals' probabilistic forecasts and their meta-predictions about the average probability forecast of others, and show that this outperforms the top alternative probabilistic aggregation approaches in the literature on a large range of decision problems. Furthermore, I demonstrate that this improved weighting approach provides a superior measure of expertise than existing single-question approaches to identifying expertise in the literature. As an additional test, I compare this single-question approach with cross-domain weighting -- weighting by individuals' performance on problems on unrelated domains -- and show that cross-domain weighting is favoured over single-question approaches in cases where it is possible to obtain individuals' past performance on problems -- even if those questions are from unrelated domains. In general, our results demonstrate the potential for using individuals' meta-predictions and performance on problems from unrelated domains to identify expertise in cases where other approaches might be ineffective or unavailable.