ORCID Profile
0000-0003-3870-5949
Current Organisation
University of Melbourne
Does something not look right? The information on this page has been harvested from data sources that may not be up to date. We continue to work with information providers to improve coverage and quality. To report an issue, use the Feedback Form.
Publisher: Center for Open Science
Date: 23-02-2021
Abstract: Experts are often asked to represent their uncertainty as a subjective probability. Structured protocols offer a transparent and systematic way to elicit and combine probability judgements from multiple experts. As part of this process, experts are asked to in idually estimate a probability (e.g., of a future event) which needs to be combined, or aggregated, into a final group prediction. The experts' judgements can be aggregated behaviourally (by striving for consensus), or mathematically (by using a mathematical rule to combine in idual estimates). Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. However, the choice of a rule is not straightforward, and the quality of the aggregated group judgement depends on it. The quality of an aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the ``best” final prediction.In the ideal case, in idual experts' performance (as probability assessors) would be scored on similar questions ahead of time, these scores translated into performance-based weights, and a performance-based weighted aggregation could then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. Next, we investigate the relative performance of these aggregation methods using three datasets. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst.
Publisher: Informa UK Limited
Date: 02-10-2018
Publisher: Frontiers Media SA
Date: 22-11-2018
Publisher: Wiley
Date: 02-07-2021
DOI: 10.1111/GCB.15750
Abstract: Conservation managers are under increasing pressure to make decisions about the allocation of finite resources to protect bio ersity under a changing climate. However, the impacts of climate and global change drivers on species are outpacing our capacity to collect the empirical data necessary to inform these decisions. This is particularly the case in the Australian Alps which have already undergone recent changes in climate and experienced more frequent large‐scale bushfires. In lieu of empirical data, we use a structured expert elicitation method (the IDEA protocol) to estimate the change in abundance and distribution of nine vegetation groups and 89 Australian alpine and subalpine species by the year 2050. Experts predicted that most alpine vegetation communities would decline in extent by 2050 only woodlands and heathlands are predicted to increase in extent. Predicted species‐level responses for alpine plants and animals were highly variable and uncertain. In general, alpine plants spanned the range of possible responses, with some expected to increase, decrease or not change in cover. By contrast, almost all animal species are predicted to decline or not change in abundance or elevation range more species with water‐centric life‐cycles are expected to decline in abundance than other species. While long‐term ecological data will always be the gold standard for informing the future of bio ersity, the method and outcomes outlined here provide a pragmatic and coherent basis upon which to start informing conservation policy and management in the face of rapid change and a paucity of data.
Publisher: Public Library of Science (PLoS)
Date: 02-09-2021
DOI: 10.1371/JOURNAL.PONE.0256919
Abstract: Structured protocols offer a transparent and systematic way to elicit and combine/aggregate, probabilistic predictions from multiple experts. These judgements can be aggregated behaviourally or mathematically to derive a final group prediction. Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. The quality of this aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the “best” final prediction. When experts’ performance can be scored on similar questions ahead of time, these scores can be translated into performance-based weights, and a performance-based weighted aggregation can then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. We differentially weight our experts’ estimates by measures of reasoning, engagement, openness to changing their mind, informativeness, prior knowledge, and extremity, asymmetry or granularity of estimates. Next, we investigate the relative performance of these aggregation methods using three datasets. The main goal of this research is to explore how measures of knowledge and behaviour of in iduals can be leveraged to produce a better performing combined group judgment. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst. Moreover, the majority of methods outperform the usual benchmarks provided by the simple average or the median of estimates.
Publisher: MDPI AG
Date: 03-02-2020
Abstract: In levee system reliability, the length effect is the term given to the phenomenon that the longer the levee, the higher the probability that it will have a weak spot and fail. Quantitatively, it is the ratio of the segment failure probability to the cross-sectional failure probability. The literature is lacking in methods to calculate the length effect in levees, and often over-simplified methods are used. An efficient (but approximate) method, which we refer to as the modified outcrossing (MO) method, was developed for the system reliability model used in Dutch national flood risk analysis and for the provision of levee assessment tools, but it is poorly documented and its accuracy has not been tested. In this paper, we propose a method to calculate the length effect in levees by s ling the joint spatial distribution of the resistance variables using a copula approach, and represented by a Bayesian Network (BN). We use the BN to verify the MO method, which is also described in detail in this paper. We describe how both methods can be used to update failure probabilities of (long) levees using survival observations (i.e., high water levels and no levee failure), which is important because we have such observations in abundance. We compared the methods via a numerical ex le, and found that the agreement between the segment failure probability estimates was nearly perfect in the prior case, and very good in the posterior case, for segments ranging from 500 m to 6000 m in length. These results provide a strong verification of both methods, either of which provide an attractive alternative to the more simplified approaches often encountered in the literature and in practice.
Publisher: The Royal Society
Date: 06-2023
DOI: 10.1098/RSOS.221553
Abstract: This paper explores judgements about the replicability of social and behavioural sciences research and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited from groups using a structured approach called the IDEA protocol (‘investigate’, ‘discuss’, ‘estimate’ and ‘aggregate’). Five groups of five people with relevant domain expertise evaluated 25 research claims that were subject to at least one replication study. Participants assessed the probability that each of the 25 research claims would replicate (i.e. that a replication study would find a statistically significant result in the same direction as the original study) and described the reasoning behind those judgements. We quantitatively analysed possible correlates of predictive accuracy, including self-rated expertise and updating of judgements after feedback and discussion. We qualitatively analysed the reasoning data to explore the cues, heuristics and patterns of reasoning used by participants. Participants achieved 84% classification accuracy in predicting replicability. Those who engaged in a greater breadth of reasoning provided more accurate replicability judgements. Some reasons were more commonly invoked by more accurate participants, such as ‘effect size’ and ‘reputation’ (e.g. of the field of research). There was also some evidence of a relationship between statistical literacy and accuracy.
Publisher: Wiley
Date: 13-06-2023
Abstract: Here, we demonstrate how IDEAcology aids in preparing for and implementing a structured expert elicitation using the IDEA protocol, an iterative quantitative expert elicitation framework. Expert judgement is used to inform decision‐making on environmental assessment and management when imminent decisions are required, and quantitative data are absent or uninformative. Structured elicitation protocols can help improve the final judgements derived from experts, but they can also be administratively heavy and time‐consuming, requiring manual collation of experts' estimates and rationales, construction and dissemination of summary plots for discussion and collating final estimates post‐discussion. These challenges highlight the need for a centralised portal that enables synchronous access by all contributors, real‐time structured facilitation of discussion, whether in person or online, and streamlined data management. To meet this need, we developed the IDEAcology interface ( www.ideacology.com ) to support data collation, summary, interactions and ultimately the deployment of structured expert elicitation using the IDEA protocol. The IDEAcology interface is designed to be a central portal for scientists and practitioners to easily implement structured expert elicitation projects, while also facilitating data management by providing a reliable and efficient way for elicitation managers to design and run an elicitation, and for experts to input, visualise and cross‐examine estimates. The key advantages that IDEAcology provides include an easy‐to‐use interface with synchronous access to a single platform, reducing logistic difficulties, facilitating transparent discussion, improving the accuracy of estimates, enabling fast and efficient reporting by providing analysis‐ready data outputs and lastly, flexibility in the types of elicitation questions that can be accommodated in the interface.
Publisher: Center for Open Science
Date: 04-05-2021
Abstract: This paper explores judgements about the replicability of social and behavioural sciences research, and what drives those judgements. Using a mixed methods approach, it draws on qualitative and quantitative data elicited using a structured iterative approach for eliciting judgements from groups, called the IDEA protocol (‘Investigate’, ‘Discuss’, ‘Estimate’ and ‘Aggregate’). Five groups of five people separately assessed the replicability of 25 ‘known-outcome’ claims. That is, social and behavioural science claims that have already been subject to at least one replication study. Specifically, participants assessed the probability that each of the 25 research claims will replicate (i.e. a replication study would find a statistically significant result in the same direction as the original study). In addition to their quantitative judgements, participants also outlined the reasoning behind their judgements. To start, we quantitatively analysed some possible correlates of predictive accuracy, such as self-rated understanding and expertise in assessing each claim, and updating of judgements after feedback and discussion. Then we qualitatively analysed the reasoning data (i.e., the comments and justifications people provided for their judgements) to explore the cues and heuristics used, and features of group discussion that accompanied more and less accurate judgements.
Publisher: Center for Open Science
Date: 25-02-2021
Abstract: Expert elicitation is deployed when data are absent or uninformative and critical decisions must be made. In designing an expert elicitation, most practitioners seek to achieve best practice while balancing practical constraints. The choices made influence the required time and effort investment, the quality of the elicited data, experts’ engagement, the defensibility of results, and the acceptability of resulting decisions. This piece outlines some of the common choices practitioners encounter when designing and conducting an elicitation. We discuss the evidence supporting these decisions and identify research gaps. This will hopefully allow practitioners to better navigate the literature, and will inspire the expert judgement research community to conduct well powered, replicable experiments that properly address the research gaps identified.
Publisher: Informa UK Limited
Date: 09-08-2018
Publisher: Elsevier BV
Date: 10-2019
Publisher: Public Library of Science (PLoS)
Date: 22-06-2018
Publisher: Center for Open Science
Date: 22-02-2021
Abstract: Replication is a hallmark of scientific research. As replications of in idual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based on the IDEA protocol. The repliCATS process is delivered through an underpinning online platform and applied to the evaluation of research claims in social and behavioural sciences. This process can be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period. Pilot data suggests that the accuracy of the repliCATS process meets or exceeds that of other techniques used to predict replicability. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to assist with problems like understanding the limits of generalizability of scientific claims. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.
Publisher: Wiley
Date: 23-03-2020
DOI: 10.1002/EAP.2075
Abstract: Performance weighted aggregation of expert judgments, using calibration questions, has been advocated to improve pooled quantitative judgments for ecological questions. However, there is little discussion or practical advice in the ecological literature regarding the application, advantages or challenges of performance weighting. In this paper we (1) illustrate how the IDEA protocol with four-step question format can be extended to include performance weighted aggregation from the Classical Model, and (2) explore the extent to which this extension improves pooled judgments for a range of performance measures. Our case study demonstrates that performance weights can improve judgments derived from the IDEA protocol with four-step question format. However, there is no a-priori guarantee of improvement. We conclude that the merits of the method lie in demonstrating that the final aggregation of judgments provides the best representation of uncertainty (i.e., validation), whether that be via equally weighted or performance weighted aggregation. Whether the time and effort entailed in performance weights can be justified is a matter for decision-makers. Our case study outlines the rationale, challenges, and benefits of performance weighted aggregations. It will help to inform decisions about the deployment of performance weighting and avoid common pitfalls in its application.
Publisher: Springer Science and Business Media LLC
Date: 13-06-2020
Publisher: Wiley
Date: 12-12-2020
DOI: 10.1002/QRE.2596
Publisher: Public Library of Science (PLoS)
Date: 26-01-2023
DOI: 10.1371/JOURNAL.PONE.0274429
Abstract: As replications of in idual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.
Publisher: Center for Open Science
Date: 15-02-2023
Abstract: Replication is an important “credibility control” mechanism for clarifying the reliability of published findings. However, replication is costly, and it is infeasible to replicate everything. Accurate, fast, lower cost alternatives such as eliciting predictions from experts or novices could accelerate credibility assessment and improve allocation of replication resources for important and uncertain findings. We elicited judgments from experts and novices on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using a new interactive structured elicitation protocol and we conducted 35 new replications. Participants’ average estimates were similar to the observed replication rate of 60%. After interacting with their peers, novices updated both their estimates and confidence in their judgements significantly more than experts and their accuracy improved more between elicitation rounds. Experts’ average accuracy was 0.54 (95% CI: [0.454, 0.628]) after interaction and they correctly classified 55% of claims novices’ average accuracy was 0.55 (95% CI: [0.455, 0.628]), correctly classifying 61% of claims. The difference in accuracy between experts and novices was not significant and their judgments on the full set of claims were strongly correlated (r=.48). These results are consistent with prior investigations eliciting predictions about the replicability of published findings in established areas of research and suggest that expertise may not be required for credibility assessment of some research findings.
Publisher: Informa UK Limited
Date: 26-02-2018
Location: Romania
No related grants have been discovered for Anca Hanea.