Journal article

Mathematically aggregating experts' predictions of possible futures

AM Hanea, DP Wilkinson, M McBride, A Lyon, D van Ravenzwaaij, F Singleton Thorn, C Gray, DR Mandel, A Willcox, E Gould, ET Smith, F Mody, M Bush, F Fidler, H Fraser, BC Wintle



Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the "best" final prediction. When experts' performance can be scored on similar questions ahead of time, these scores can..

View full abstract


Awarded by Defence Advanced Research Projects Activity (DARPA), SCORE program

Funding Acknowledgements

This research is based upon work funded by the Defence Advanced Research Projects Activity (DARPA), under Contract [HR001118S0047] as part of the SCORE program. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Cognimotive Consulting Inc. (DRM's affiliation) and DelphiCloud (AL's affiliation) did not provide any financial support and did not have any role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of the author affiliated with Cognimotive Consulting Inc. and DelphiCloud are articulated in the `author contributions' section.