ORCID Profile
0000-0002-5030-4091
Current Organisation
University of Groningen
Does something not look right? The information on this page has been harvested from data sources that may not be up to date. We continue to work with information providers to improve coverage and quality. To report an issue, use the Feedback Form.
Publisher: Elsevier BV
Date: 02-2017
Publisher: Elsevier BV
Date: 03-2017
Publisher: Elsevier BV
Date: 06-2012
Publisher: Springer Science and Business Media LLC
Date: 26-11-2009
Publisher: Cambridge University Press (CUP)
Date: 14-05-2013
DOI: 10.1017/S0140525X12003068
Abstract: We focus on two issues: (1) an unusual, counterintuitive prediction that quantum probability (QP) theory appears to make regarding multiple sequential judgments, and (2) the extent to which QP is an appropriate and comprehensive benchmark for assessing judgment. These issues highlight how QP theory can fall prey to the same problems of arbitrariness that Pothos & Busemeyer (P& B) discuss as plaguing other models.
Publisher: Center for Open Science
Date: 19-01-2018
Abstract: The debate between Bayesians and frequentist statisticians has been going on for decades. Whilst there are fundamental theoretical and philosophical differences between both schools of thought, we argue that in two most common situations the practical differences are negligible when off-the-shelve Bayesian analysis (i.e., using ‘objective’ priors) is used. We emphasize this reasoning by focusing on interval estimates: confidence intervals and credible intervals. We show that this is the case for the most common empirical situations in the social sciences, the estimation of a proportion of a binomial distribution and the estimation of the mean of a unimodal distribution. Numerical differences between both approaches are small, sometimes even smaller than those between two competing frequentist or two competing Bayesian approaches. We outline the ramifications of this for scientific practice.
Publisher: American Psychological Association (APA)
Date: 06-2022
DOI: 10.1037/MET0000415
Abstract: Tendeiro and Kiers (2019) provide a detailed and scholarly critique of Null Hypothesis Bayesian Testing (NHBT) and its central component-the Bayes factor-that allows researchers to update knowledge and quantify statistical evidence. Tendeiro and Kiers conclude that NHBT constitutes an improvement over frequentist
Publisher: Center for Open Science
Date: 13-03-2023
Abstract: BackgroundPublishing study results in scientific journals has been the standard way of disseminating science. However, getting results published may depend on their statistical significance. The consequence of this is that the representation of scientific knowledge might be biased. This type of bias has been called publication bias. The main objective of the present study is to get more insight into publication bias by examining it at the author, reviewer, and editor level. Additionally, we make a direct comparison between publication bias induced by authors, by reviewers, and by editors. We approached our participants by e-mail, asking them to fill out an online survey. ResultsOur findings suggest that statistically significant findings have a higher likelihood to be published than statistically non-significant findings, because (1) authors (n = 65) are more likely to write up and submit articles with significant results compared to articles with non-significant results (median effect size 1.10, BF10 = 1.09*10^7) (2) reviewers (n = 60) give more favourable reviews to articles with significant results compared to articles with non-significant results (median effect size 0.58, BF10 = 4.73*10^2) and (3) editors (n = 171) are more likely to accept for publication articles with significant results compared to articles with non-significant results (median effect size, 0.94, BF10 = 7.63*10^7). Evidence on differences in the relative contributions to publication bias by authors, reviewers, and editors is ambiguous (editors vs reviewers: BF10 = 0.31, reviewers vs authors: BF10 = 3.11, and editors vs authors: BF10 = 0.42). DiscussionOne of the main limitations was that rather than investigating publication bias directly, we studied potential for publication bias. Another limitation was the low response rate to the survey.
Publisher: Springer Science and Business Media LLC
Date: 26-07-2023
Publisher: Center for Open Science
Date: 30-10-2018
Abstract: The crisis of confidence has played a primary role in undermining the trust researchers place in the findings of their peers, and our beliefs about the credibility of research results. Thus, the importance of increasing trust in credible reported research is paramount. Incentives such as preregistration are aimed at establishing a more trustworthy scientific literature, in that they help prevent various questionable research practices. As it stands, however, no empirical evidence exists demonstrating that preregistration does increase trust. Indeed, the objective merits of preregistration greatly lose in value if a researcher's subjective assessment of the value of preregistration does not align. Additionally, the picture may be complicated by a researcher's familiarity with the author of the study, regardless of the preregistration status of the research. The following proposal describes how we aim to test the extent to which preregistration increases the trust of participants in the reported outcomes. We also aim to assess how familiarity with another researcher might influence trust. We expect that preregistration increases researchers' trust in findings, relative to no preregistration, and that registered reporting increases trust more than preregistration alone. We also expect that familiarity enhances trust judgments to some extent, however we do not have specific expectations regarding the nature of this effect. We therefore include familiarity as an exploratory effect in our analyses. The OSF page for this registered report proposal and its pilot can be found here: 0.17605/OSF.IO/B3K75
Publisher: Center for Open Science
Date: 06-06-2023
Abstract: A large amount of scientific literature in social and behavioural sciences bases their conclusions on one or more hypothesis tests. As such, it is important to obtain more knowledge about how researchers in social and behavioural sciences interpret quantities that result from such hypothesis tests, such as p-values and Bayes factors. In the present study, we will explore the relationship between obtained statistical evidence and the degree of belief or confidence that there is a positive effect in the population of interest. In particular, we are interested in the existence of a so-called cliff effect: A qualitative drop in the degree of belief that there is a positive effect around certain threshold values of statistical evidence. We will compare this relationship for p-values to the relationship for corresponding degrees of evidence quantified through Bayes factors, and we will examine whether this relationship is affected by two different modes of presentation.
Publisher: Center for Open Science
Date: 21-04-2021
Abstract: We present consensus-based guidance for conducting and documenting multi-analyst studies. We discuss why broader adoption of the multi-analyst approach will strengthen the robustness of results and conclusions in empirical sciences.
Publisher: SAGE Publications
Date: 04-11-2017
Abstract: Parenthood is central to the personal and social identity of many people. For in iduals with psychotic disorders, parenthood is often associated with formidable challenges. We aimed to identify predictors of adequate parenting among parents with psychotic disorders. Data pertaining to 234 parents with psychotic disorders living with dependent children were extracted from a population-based prevalence study, the 2010 second Australian national survey of psychosis, and analysed using confirmatory factor analysis. Parenting outcome was defined as quality of care of children, based on participant report and interviewer enquiry/exploration, and included level of participation, interest and competence in childcare during the last 12 months. Five hypothesis-driven latent variables were constructed and labelled psychosocial support, illness severity, substance abuse/dependence, adaptive functioning and parenting role. Importantly, 75% of participants were not identified to have any dysfunction in the quality of care provided to their child(ren). Severity of illness and adaptive functioning were reliably associated with quality of childcare. Psychosocial support, substance abuse/dependence and parenting role had an indirect relationship to the outcome variable via their association with either severity of illness and/or adaptive functioning. The majority of parents in the current s le provided adequate parenting. However, greater symptom severity and poorer adaptive functioning ultimately leave parents with significant difficulties and in need of assistance to manage their parenting obligations. As symptoms and functioning can change episodically for people with psychotic illness, provision of targeted and flexible support that can deliver temporary assistance during times of need is necessary. This would maximise the quality of care provided to vulnerable children, with potential long-term benefits.
Publisher: Center for Open Science
Date: 06-12-2018
Abstract: When data analysts operate within different statistical frameworks (e.g., frequentist versus Bayesian, emphasis on estimation versus emphasis on testing), how does this impact the qualitative conclusions that are drawn for real data? To study this question empirically we selected from the literature two simple scenarios --involving a comparison of two proportions and a Pearson correlation-- and asked four teams of statisticians to provide a concise analysis and a qualitative interpretation of the outcome. The results showed considerable overall agreement nevertheless, this agreement did not appear to diminish the intensity of the subsequent debate over which statistical framework is more appropriate to address the questions at hand.
Publisher: Elsevier BV
Date: 02-2011
Publisher: SAGE Publications
Date: 2021
Abstract: When social scientists wish to learn about an empirical phenomenon, they perform an experiment. When they wish to learn about a complex numerical phenomenon, they can perform a simulation study. The goal of this Tutorial is twofold. First, it introduces how to set up a simulation study using the relatively simple ex le of simulating from the prior. Second, it demonstrates how simulation can be used to learn about the Jeffreys-Zellner-Siow (JZS) Bayes factor, a currently popular implementation of the Bayes factor employed in the BayesFactor R package and freeware program JASP. Many technical expositions on Bayes factors exist, but these may be somewhat inaccessible to researchers who are not specialized in statistics. In a step-by-step approach, this Tutorial shows how a simple simulation script can be used to approximate the calculation of the Bayes factor. We explain how a researcher can write such a s ler to approximate Bayes factors in a few lines of code, what the logic is behind the Savage-Dickey method used to visualize Bayes factors, and what the practical differences are for different choices of the prior distribution used to calculate Bayes factors.
Publisher: University of California Press
Date: 2021
Abstract: A majority of statistically educated scientists draw incorrect conclusions based on the most commonly used statistical technique: null hypothesis significance testing (NHST). Frequentist techniques are often claimed to be incorrectly interpreted as Bayesian outcomes, which suggests that a Bayesian framework may fit better to inferences researchers frequently want to make (Briggs, 2012). The current study set out to test this proposition. Firstly, we investigated whether there is a discrepancy between what researchers think they can conclude and what they want to be able to conclude from NHST. Secondly, we investigated to what extent researchers want to incorporate prior study results and their personal beliefs in their statistical inference. Results show the expected discrepancy between what researchers think they can conclude from NHST and what they want to be able to conclude. Furthermore, researchers were interested in incorporating prior study results, but not their personal beliefs, into their statistical inference.
Publisher: Proceedings of the National Academy of Sciences
Date: 12-03-2018
Abstract: We describe and demonstrate an empirical strategy useful for discovering and replicating empirical effects in psychological science. The method involves the design of a metastudy, in which many independent experimental variables—that may be moderators of an empirical effect—are indiscriminately randomized. Radical randomization yields rich datasets that can be used to test the robustness of an empirical claim to some of the vagaries and idiosyncrasies of experimental protocols and enhances the generalizability of these claims. The strategy is made feasible by advances in hierarchical Bayesian modeling that allow for the pooling of information across unlike experiments and designs and is proposed here as a gold standard for replication research and exploratory research. The practical feasibility of the strategy is demonstrated with a replication of a study on subliminal priming.
Publisher: Center for Open Science
Date: 02-03-2021
Abstract: We argue that statistical practice in the social and behavioral sciences benefits from transparency, a fair acknowledgement of uncertainty, and openness to alternative interpretations. To promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data (2) quantifying inferential uncertainty (3) assessing data preprocessing choices (4) reporting multiple models (5) involving multiple analysts (6) interpreting results modestly and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness, and organized skepticism. We believe that these ethical considerations --and their statistical consequences-- establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.
Publisher: Center for Open Science
Date: 03-09-2019
Abstract: Tendeiro and Kiers (2019) provide a detailed and scholarly critique of Null Hypothesis Bayesian Testing (NHBT) and its central component –the Bayes factor– that allows researchers to update knowledge and quantify statistical evidence. Tendeiro and Kiers conclude that NHBT constitutes an improvement over frequentist p-values, but primarily elaborate on a list of eleven ‘issues’ of NHBT. In this commentary, we provide context to each issue and conclude that many issues may in fact be conceived as pronounced advantages of NHBT.
Publisher: Springer Science and Business Media LLC
Date: 28-06-2016
Publisher: American Psychological Association (APA)
Date: 06-2021
DOI: 10.1037/CPS0000013
Publisher: Springer Science and Business Media LLC
Date: 08-11-2012
DOI: 10.3758/S13421-012-0268-3
Abstract: Probability matching in sequential decision making is a striking violation of rational choice that has been observed in hundreds of experiments. Recent studies have demonstrated that matching persists even in described tasks in which all the information required for identifying a superior alternative strategy-maximizing-is present before the first choice is made. These studies have also indicated that maximizing increases when (1) the asymmetry in the availability of matching and maximizing strategies is reduced and (2) normatively irrelevant outcome feedback is provided. In the two experiments reported here, we examined the joint influences of these factors, revealing that strategy availability and outcome feedback operate on different time courses. Both behavioral and modeling results showed that while availability of the maximizing strategy increases the choice of maximizing early during the task, feedback appears to act more slowly to erode misconceptions about the task and to reinforce optimal responding. The results illuminate the interplay between "top-down" identification of choice strategies and "bottom-up" discovery of those strategies via feedback.
Publisher: Springer Science and Business Media LLC
Date: 08-07-2014
Publisher: eLife Sciences Publications, Ltd
Date: 09-11-2021
DOI: 10.7554/ELIFE.72185
Abstract: Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
Publisher: Cold Spring Harbor Laboratory
Date: 05-09-2023
Publisher: No publisher found
Date: 2013
Publisher: Elsevier BV
Date: 12-2018
Publisher: Center for Open Science
Date: 18-04-2023
Abstract: In recent years, the advancement of open science has led to data sharing becoming more common practice. Data availability has clear merits for science as it opens up possibilities for re-use of datasets by others, leading to less redundancy, more efficiency, and more transparency. The ideal is for scientific data to be as open as possible and FAIR: Findable, Accessible, Interoperable, and Reusable. Parallel to this development, recent times have seen more stringent guidelines with respect to data privacy, culminating in the General Data Protection Regulation law, or GDPR. The GDPR is a stringent European law on data privacy that makes it more difficult for researchers to navigate the balance between protecting participants’ privacy and making one's dataset as open as possible. In this paper, we provide two worked ex les of how to pseudonymise real datasets and share them in a FAIR fashion.
Publisher: Center for Open Science
Date: 14-12-2018
Abstract: Despite its many advocates, Bayesian inference is currently employed by only a minority of social and behavioural scientists. One possible barrier is a lack of consensus on how best to conduct and report such analyses. Employing Bayesian methods involves making choices about prior distributions, likelihood functions and robustness checks, as well as about how to present, visualize and interpret the results (for a glossary of the main Bayesian statistical concepts, see Box 1). Some researchers may find this wide range of choices too daunting to use Bayesian inference in their own study. This paper highlights the areas of agreement and the arguments behind disagreements, established on the back of a self-questionnaire provided and explained in detail on OSF (osf.io/6eqx5/).
Publisher: Center for Open Science
Date: 30-07-2020
Abstract: Background: Following testing in clinical trials, the use of remdesivir for treatment of COVID-19 has been authorized for use in parts of the world, including the USA and Europe. Early authorizations were largely based on results from two clinical trials. A third study published by Wang et al. was underpowered and deemed inconclusive. Although regulators have shown an interest in interpreting the Wang et al. study, under a frequentist framework it is difficult to determine if the non-significant finding was caused by a lack of power or by the absence of an effect. Bayesian hypothesis testing does allow for quantification of evidence in favor of the absence of an effect.Findings: Results of our Bayesian reanalysis of the three trials show ambiguous evidence for the primary outcome of clinical improvement and moderate evidence against the secondary outcome of decreased mortality rate. Additional analyses of three studies published after initial marketing approval support these findings. Conclusions: We recommend that regulatory bodies take all available evidence into account for endorsement decisions. A Bayesian approach can be beneficial, in particular in case of statistically non-significant results. This is especially pressing when limited clinical efficacy data is available.
Publisher: Springer Science and Business Media LLC
Date: 22-09-2021
DOI: 10.1007/S42113-021-00117-Y
Abstract: Nested data structures, in which conditions include multiple trials and are fully crossed with participants, are often analyzed using repeated-measures analysis of variance or mixed-effects models. Typically, researchers are interested in determining whether there is an effect of the experimental manipulation. These kinds of analyses have different appropriate specifications for the null and alternative models, and a discussion on which is to be preferred and when is sorely lacking. van Doorn et al. (2021) performed three types of Bayes factor model comparisons on a simulated data set in order to examine which model comparison is most suitable for quantifying evidence for or against the presence of an effect of the experimental manipulation. Here, we extend their results by simulating multiple data sets for various scenarios and by using different prior specifications. We demonstrate how three different Bayes factor model comparison types behave under changes in different parameters, and we make concrete recommendations on which model comparison is most appropriate for different scenarios.
Publisher: American Psychological Association (APA)
Date: 04-2020
DOI: 10.1037/DEC0000103
Publisher: Center for Open Science
Date: 29-09-2021
Abstract: Adopting some practical elements of open science – a movement whose goal is to make scientific research available for everyone – presents unique chal- lenges for qualitative researchers, particularly when it comes to data sharing. In this article, we discuss the issue of open qualitative data, arguing that while concerns about ethics and loss of data quality are legitimate, they do not pose so great a problem as to preclude qualitative researchers from ef- fectively practicing open science. We describe the cost-benefit balance that each qualitative researcher takes into account as they choose whether or not to share their data, and highlight the fact that qualitative research practice lends itself to transparency and integrity by its reliance on reflexivity, and other practices such as member checking and using multiple coders. We con- clude with a reminder to readers that fruitful open science does not require for one to engage in all possible practices for a given study only those which are appropriate and feasible.
Publisher: Wiley
Date: 23-01-2015
DOI: 10.1002/9781118625392.WBECP453
Abstract: A Bayesian analysis concerns the quantification and updating of knowledge according to the laws of probability theory. A Bayesian analysis allows researchers to estimate parameters (e.g., how effective is a new treatment?) or to conduct hypothesis testing and model selection (e.g., is the new treatment more effective than the standard treatment?). The Bayesian paradigm offers concrete advantages for the practical research worker one of these is the ability to attach probabilities to hypotheses and parameters, and another is the ability to collect data until the evidence conclusively supports the null hypothesis or the alternative hypothesis.
Publisher: Center for Open Science
Date: 20-01-2018
Abstract: Copyright Notice. The document distributed here has been provided as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.
Publisher: Center for Open Science
Date: 31-10-2022
Abstract: Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic, and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications, and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of in iduals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target.
Publisher: The Royal Society
Date: 02-2023
DOI: 10.1098/RSOS.210586
Abstract: Increased execution of replication studies contributes to the effort to restore credibility of empirical research. However, a second generation of problems arises: the number of potential replication targets is at a serious mismatch with available resources. Given limited resources, replication target selection should be well-justified, systematic and transparently communicated. At present the discussion on what to consider when selecting a replication target is limited to theoretical discussion, self-reported justifications and a few formalized suggestions. In this Registered Report, we proposed a study involving the scientific community to create a list of considerations for consultation when selecting a replication target in psychology. We employed a modified Delphi approach. First, we constructed a preliminary list of considerations. Second, we surveyed psychologists who previously selected a replication target with regards to their considerations. Third, we incorporated the results into the preliminary list of considerations and sent the updated list to a group of in iduals knowledgeable about concerns regarding replication target selection. Over the course of several rounds, we established consensus regarding what to consider when selecting a replication target. The resulting checklist can be used for transparently communicating the rationale for selecting studies for replication.
Publisher: Center for Open Science
Date: 11-07-2017
Abstract: We describe and demonstrate an empirical strategy useful for discovering and replicating empirical effects in psychological science. The method involves the design of a meta-study, in which many independent experimental variables—that may be moderators of an empirical effect—are indiscriminately randomized. Radical randomization yields rich data sets that can be used to test the robustness of an empirical claim to some of the vagaries and idiosyncrasies of experimental protocols and enhances the generalizability of these claims. The strategy is made feasible by advances in hierarchical Bayesian modeling which allow for the pooling of information across unlike experiments and designs, and is proposed here as a gold standard for replication research and exploratory research. The practical feasibility of the strategy is demonstrated with a replication of a study on subliminal priming. All materials and data are freely available online via osf.io/u2vwa/.
Publisher: The Royal Society
Date: 04-2020
DOI: 10.1098/RSOS.181351
Abstract: The crisis of confidence has undermined the trust that researchers place in the findings of their peers. In order to increase trust in research, initiatives such as preregistration have been suggested, which aim to prevent various questionable research practices. As it stands, however, no empirical evidence exists that preregistration does increase perceptions of trust. The picture may be complicated by a researcher's familiarity with the author of the study, regardless of the preregistration status of the research. This registered report presents an empirical assessment of the extent to which preregistration increases the trust of 209 active academics in the reported outcomes, and how familiarity with another researcher influences that trust. Contrary to our expectations, we report ambiguous Bayes factors and conclude that we do not have strong evidence towards answering our research questions. Our findings are presented along with evidence that our manipulations were ineffective for many participants, leading to the exclusion of 68% of complete datasets, and an underpowered design as a consequence. We discuss other limitations and confounds which may explain why the findings of the study deviate from a previously conducted pilot study. We reflect on the benefits of using the registered report submission format in light of our results. The OSF page for this registered report and its pilot can be found here: 0.17605/OSF.IO/B3K75 .
Publisher: Center for Open Science
Date: 17-08-2021
Abstract: Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalisation of phenomena. Operationalisation is the process of translating theoretical constructs into measurable lab-quantities. Thus, the validity of operationalisation is central for the quality of empirical studies. But do differences in the validity of operationalisation affect the way scientists evaluate scientific literature? To investigate this, we adapted three published studies and manipulated the strength of operationalisation. We preregistered our planned study as a registered report. For each study, we designed two tasks in which the validity of the used operationalisation is manipulated and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalisation is more valid but (2) the different validity is only to some extent reflected in a different judgment of the quality of the study. These results combined give partial evidence for the notion that researchers’ evaluations of research results are not affected by operationalisation validity.
Publisher: Center for Open Science
Date: 19-12-2018
Abstract: Background: Until recently a typical rule that has often been used for the endorsement of new medications by the Food and Drug Administration has been the existence of at least two statistically significant clinical trials favoring the new medication. This rule has consequences for the true positive (endorsement of an effective treatment) and false positive rates (endorsement of an ineffective treatment). Methods: In this paper, we compare true positive and false positive rates for different evaluation criteria through simulations that rely on (1) conventional p-values (2) confidence intervals based on meta-analyses assuming fixed or random effects and (3) Bayes factors. We varied threshold levels for statistical evidence, thresholds for what constitutes a clinically meaningful treatment effect, and number of trials conducted. Results: Our results show that Bayes factors, meta-analytic confidence intervals, and p-values often have similar performance. Bayes factors may perform better when the number of trials conducted is high and when trials have small s le sizes and clinically meaningful effects are not small, particularly in fields where the number of non-zero effects is relatively large. Conclusions: Thinking about realistic effect sizes in conjunction with desirable levels of statistical evidence, as well as quantifying statistical evidence with Bayes factors may help improve decision-making in some circumstances.
Publisher: Center for Open Science
Date: 05-08-2020
Abstract: To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.
Publisher: Cambridge University Press (CUP)
Date: 24-01-2014
DOI: 10.1017/S0140525X13000769
Abstract: Newell & Shanks (N& S) show that there is no convincing evidence that processes assumed to be unconscious and superior are indeed unconscious. We take their argument one step further by showing that there is also no convincing evidence that these processes are superior. We review alternative paradigms that may provide more convincing tests of the superiority of (presumed) unconscious processes.
Publisher: Center for Open Science
Date: 08-07-2022
Abstract: Theoretical arguments and empirical investigations indicate that a high proportion of published findings are false or do not replicate. The current position paper provides a broad perspective on this scientific error, focusing both on reform history and on opportunities for future reform. Talking points are organised along four main themes: methodological reform, statistical reform, publishing reform, and institutional reform. For each of these four themes, we discuss the current state of affairs, existing knowledge gaps, and opportunities for future reform. The resulting agenda should provide a useful resource for ushering in an era that is marked by a lower prevalence of scientific error.
Publisher: American Psychological Association (APA)
Date: 12-2016
DOI: 10.1037/XGE0000067
Abstract: Priming is arguably one of the key phenomena in contemporary social psychology. Recent retractions and failed replication attempts have led to a ision in the field between
Publisher: Springer Science and Business Media LLC
Date: 11-03-2016
Publisher: University of California Press
Date: 2019
DOI: 10.1525/COLLABRA.218
Abstract: The crisis of confidence in psychology has prompted vigorous and persistent debate in the scientific community concerning the veracity of the findings of psychological experiments. This discussion has led to changes in psychology’s approach to research, and several new initiatives have been developed, many with the aim of improving our findings. One key advancement is the marked increase in the number of replication studies conducted. We argue that while it is important to conduct replications as part of regular research protocol, it is neither efficient nor useful to replicate results at random. We recommend adopting a methodical approach toward the selection of replication targets to maximize the impact of the outcomes of those replications, and minimize waste of scarce resources. In the current study, we demonstrate how a Bayesian re–analysis of existing research findings followed by a simple qualitative assessment process can drive the selection of the best candidate article for replication.
Publisher: Public Library of Science (PLoS)
Date: 25-04-2018
Publisher: Public Library of Science (PLoS)
Date: 08-03-2017
Publisher: eLife Sciences Publications, Ltd
Date: 05-10-2021
Publisher: Cold Spring Harbor Laboratory
Date: 30-06-2023
DOI: 10.1101/2023.06.30.23292074
Abstract: To quantify the strength of statistical evidence of randomised controlled trials (RCTs) for novel cancer drugs approved by the Food and Drug Administration (FDA) in the last two decades. We used data on overall survival (OS), progression-free survival (PFS), and tumour response (TR) for novel cancer drugs approved for the first time by the FDA between January 2000 and December 2020. We assessed strength of statistical evidence by calculating Bayes Factors ( BF s) for all available endpoints, and we pooled evidence using Bayesian fixed-effect meta-analysis for indications approved based on two RCTs. Strength of statistical evidence was compared between endpoints, approval pathways, lines of treatment, and types of cancer. We analysed the available data from 82 RCTs corresponding to 68 indications supported by a single RCT and seven indications supported by two RCTs. Median strength of statistical evidence was ambiguous for OS ( BF = 1.9 IQR 0.5-14.5), and strong for PFS ( BF = 24,767.8 IQR 109.0-7.3*10 6 ) and TR ( BF = 113.9 IQR 3.0-547,100). Overall, 44 indications (58.7%) were approved without clear statistical evidence for OS improvements and seven indications (9.3%) were approved without statistical evidence for improvements on any endpoint. Strength of statistical evidence was lower for accelerated approval compared to non-accelerated approval across all three endpoints. No meaningful differences were observed for line of treatment and cancer type. This analysis is limited to statistical evidence. We did not consider non-statistical factors (e.g., risk of bias, quality of the evidence). BF s offer novel insights into the strength of statistical evidence underlying cancer drug approvals. Most novel cancer drugs lack strong statistical evidence that they improve OS, and a few lack statistical evidence for efficacy altogether. These cases require a transparent and clear explanation. When evidence is ambiguous, additional post-marketing trials could eliminate uncertainty. Novel cancer drugs are approved based on limited evidence. Between 2000 and 2020, novel cancer drugs were typically approved based on a single pivotal trial, with only half of these being randomised controlled trials and many relying on surrogate endpoints to demonstrate efficacy. Using a newly developed Bayesian survival analysis strategy, we provide novel insights into the strength of statistical evidence given the limited data available at approval. Most novel cancer drugs lack strong statistical evidence that they improve overall survival, and a minority lack evidence for efficacy altogether.
Publisher: Springer Science and Business Media LLC
Date: 11-11-2021
DOI: 10.1038/S41562-021-01211-8
Abstract: We argue that statistical practice in the social and behavioural sciences benefits from transparency, a fair acknowledgement of uncertainty and openness to alternative interpretations. Here, to promote such a practice, we recommend seven concrete statistical procedures: (1) visualizing data (2) quantifying inferential uncertainty (3) assessing data preprocessing choices (4) reporting multiple models (5) involving multiple analysts (6) interpreting results modestly and (7) sharing data and code. We discuss their benefits and limitations, and provide guidelines for adoption. Each of the seven procedures finds inspiration in Merton's ethos of science as reflected in the norms of communalism, universalism, disinterestedness and organized scepticism. We believe that these ethical considerations-as well as their statistical consequences-establish common ground among data analysts, despite continuing disagreements about the foundations of statistical inference.
Publisher: Springer Science and Business Media LLC
Date: 27-11-2019
DOI: 10.1186/S12874-019-0865-Y
Abstract: Until recently a typical rule that has often been used for the endorsement of new medications by the Food and Drug Administration has been the existence of at least two statistically significant clinical trials favoring the new medication. This rule has consequences for the true positive (endorsement of an effective treatment) and false positive rates (endorsement of an ineffective treatment). In this paper, we compare true positive and false positive rates for different evaluation criteria through simulations that rely on (1) conventional p -values (2) confidence intervals based on meta-analyses assuming fixed or random effects and (3) Bayes factors. We varied threshold levels for statistical evidence, thresholds for what constitutes a clinically meaningful treatment effect, and number of trials conducted. Our results show that Bayes factors, meta-analytic confidence intervals, and p -values often have similar performance. Bayes factors may perform better when the number of trials conducted is high and when trials have small s le sizes and clinically meaningful effects are not small, particularly in fields where the number of non-zero effects is relatively large. Thinking about realistic effect sizes in conjunction with desirable levels of statistical evidence, as well as quantifying statistical evidence with Bayes factors may help improve decision-making in some circumstances.
Publisher: Springer Science and Business Media LLC
Date: 15-02-2018
Publisher: Cambridge University Press (CUP)
Date: 08-10-2021
DOI: 10.1017/S0033291721003950
Abstract: Approval and prescription of psychotropic drugs should be informed by the strength of evidence for efficacy. Using a Bayesian framework, we examined (1) whether psychotropic drugs are supported by substantial evidence (at the time of approval by the Food and Drug Administration), and (2) whether there are systematic differences across drug groups. Data from short-term, placebo-controlled phase II/III clinical trials for 15 antipsychotics, 16 antidepressants for depression, nine antidepressants for anxiety, and 20 drugs for attention deficit hyperactivity disorder (ADHD) were extracted from FDA reviews. Bayesian model-averaged meta-analysis was performed and strength of evidence was quantified (i.e. BF BMA ). Strength of evidence and trialling varied between drugs. Median evidential strength was extreme for ADHD medication ( BF BMA = 1820.4), moderate for antipsychotics ( BF BMA = 365.4), and considerably lower and more frequently classified as weak or moderate for antidepressants for depression ( BF BMA = 94.2) and anxiety ( BF BMA = 49.8). Varying median effect sizes ( ES schizophrenia = 0.45, ES depression = 0.30, ES anxiety = 0.37, ES ADHD = 0.72), s le sizes ( N schizophrenia = 324, N depression = 218, N anxiety = 254, N ADHD = 189.5), and numbers of trials ( k schizophrenia = 3, k depression = 5.5, k anxiety = 3, k ADHD = 2) might account for differences. Although most drugs were supported by strong evidence at the time of approval, some only had moderate or ambiguous evidence. These results show the need for more systematic quantification and classification of statistical evidence for psychotropic drugs. Evidential strength should be communicated transparently and clearly towards clinical decision makers.
Publisher: Informa UK Limited
Date: 20-03-2019
Publisher: American Psychological Association (APA)
Date: 10-2014
DOI: 10.1037/A0036923
Publisher: Frontiers Media SA
Date: 2000
Publisher: Elsevier BV
Date: 08-2018
DOI: 10.1016/J.JAD.2018.04.040
Abstract: Studies have shown similar efficacy of different antidepressants in the treatment of depression. Data of phase-2 and -3 clinical-trials for 16 antidepressants (levomilnacipran, desvenlafaxine, duloxetine, venlafaxine, paroxetine, escitalopram, vortioxetine, mirtazapine, venlafaxine XR, sertraline, fluoxetine, citalopram, paroxetine CR, nefazodone, bupropion, vilazodone), approved by the FDA for the treatment of depression between 1987 and 2016, were extracted from the FDA reviews that were used to evaluate efficacy prior to marketing approval, which are less liable to reporting biases. Meta-analytic Bayes factors, which quantify the strength of evidence for efficacy, were calculated. In addition, posterior pooled effect-sizes were calculated and compared with classical estimations. The resulted Bayes factors showed that the evidence load for efficacy varied strongly across antidepressants. However, all tested drugs except for bupropion and vilazodone showed strong evidence for their efficacy. The posterior effect-size distributions showed variation across antidepressants, with the highest pooled estimated effect size for venlafaxine followed by paroxetine, and the lowest for bupropion and vilazodone. Not all published trials were included in the study. The results illustrate the importance of considering both the effect size and the evidence-load when judging the efficacy of a treatment. In doing so, the currently employed Bayesian approach provided clear insights on top of those gained with traditional approaches.
Publisher: Elsevier BV
Date: 06-2011
DOI: 10.1016/J.COGNITION.2011.02.002
Abstract: Research in the field of mental chronometry and in idual differences has revealed several robust regularities (Jensen, 2006). These include right-skewed response time (RT) distributions, the worst performance rule, correlations with general intelligence (g) that are more pronounced for RT standard deviations (RTSD) than they are for RT means (RTm), an almost perfect linear relation between in idual differences in RTSD and RTm, linear Brinley plots, and stronger correlations between g and inspection time (IT) than between g and RTm. Here we show how all these regularities are manifestations of a single underlying relationship, when viewed through the lens of Ratcliff's diffusion model (Ratcliff, 1978 Ratcliff, Schmiedek, & McKoon, 2008). The single underlying relationship is between in idual differences in general intelligence and in idual differences in "drift rate", which is just the speed of information processing in Ratcliff's model. We also test and confirm a strong prediction of the diffusion model, namely that the worst performance rule generalizes to phenomena outside of the field of intelligence. Our approach provides an integrative perspective on intelligence findings.
Publisher: Cold Spring Harbor Laboratory
Date: 07-11-2022
DOI: 10.1101/2022.11.02.22281762
Abstract: The use of Cox proportional hazards regression to analyze time-to-event data is ubiquitous in biomedical research. Typically, the frequentist framework is used to draw conclusions about whether hazards are different between patients in an experimental and a control condition. We offer a procedure to calculate Bayes factors for simple Cox models, both for the scenario where the full data is available and for the scenario where only summary statistics are available. The procedure is implemented in our “baymedr” R package. The usage of Bayes factors remedies some shortcomings of frequentist inference and has the potential to save scarce resources.
Publisher: Wiley
Date: 19-03-2014
DOI: 10.1111/COGS.12119
Abstract: In most decision-making situations, there is a plethora of information potentially available to people. Deciding what information to gather and what to ignore is no small feat. How do decision makers determine in what sequence to collect information and when to stop? In two experiments, we administered a version of the German cities task developed by Gigerenzer and Goldstein (1996), in which participants had to decide which of two cities had the larger population. Decision makers were not provided with the names of the cities, but they were able to collect different kinds of cues for both response alternatives (e.g., "Does this city have a university?") before making a decision. Our experiments differed in whether participants were free to determine the number of cues they examined. We demonstrate that a novel model, using hierarchical latent mixtures and Bayesian inference (Lee & Newell, ) provides a more complete description of the data from both experiments than simple conventional strategies, such as the take-the-best or the Weighted Additive heuristics.
Publisher: The Royal Society
Date: 05-2021
DOI: 10.1098/RSOS.201697
Abstract: To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.
Publisher: Elsevier BV
Date: 05-2015
DOI: 10.1016/J.COGPSYCH.2015.03.002
Abstract: In a world of limited resources, scarcity and rivalry are central challenges for decision makers-animals foraging for food, corporations seeking maximal profits, and athletes training to win, all strive against others competing for the same goals. In this article, we establish the role of competitive pressures for the facilitation of optimal decision making in simple sequential binary choice tasks. In two experiments, competition was introduced with a computerized opponent whose choice behavior reinforced one of two strategies: If the opponent probabilistically imitated participant choices, probability matching was optimal if the opponent was indifferent, probability maximizing was optimal. We observed accurate asymptotic strategy use in both conditions irrespective of the provision of outcome probabilities, suggesting that participants were sensitive to the differences in opponent behavior. An analysis of reinforcement learning models established that computational conceptualizations of opponent behavior are critical to account for the observed ergence in strategy adoption. Our results provide a novel appraisal of probability matching and show how this in idually 'irrational' choice phenomenon can be socially adaptive under competition.
Publisher: Center for Open Science
Date: 17-10-2022
Abstract: van Doorn et al. (2021) outlined various questions that arise when conducting Bayesian model comparison for mixed effects models. Seven response articles offered their own perspective on the preferred setup for mixed model comparison, on the most appropriate specification of prior distributions, and on the desirability of default recommendations. This article presents a round-table discussion that aims to clarify outstanding issues, explore common ground, and outline practical considerations for any researcher wishing to conduct a Bayesian mixed effects model comparison.
Publisher: Center for Open Science
Date: 21-10-2019
Abstract: The practice of sequentially testing a null hypothesis as data are collected until the null hypothesis is rejected is known as optional stopping. It is well-known that optional stopping is problematic in the context of p-value based null hypothesis significance testing: The false positive rates quickly overcome the single test’s significance level. However, the state of affairs under null hypothesis Bayesian testing, where p-values are replaced by Bayes factors, has perhaps surprisingly been much less consensual. Rouder (2014) used simulations to defend the use of optional stopping under null hypothesis Bayesian testing. The idea behind these simulations is closely related to the idea of s ling from prior predictive distributions. Deng et al. (2016) and Hendriksen et al. (2020) have provided mathematical evidence to the effect that optional stopping under null hypothesis Bayesian testing does hold under some conditions. These papers are, however, exceedingly technical for most researchers in the applied social sciences. In this paper we provide some mathematical derivations concerning Rouder’s approximate simulation results for the two Bayesian hypothesis tests that he considered. The key idea is to consider the probability distribution of the Bayes factor, which is regarded as being a random variable across repeated s ling. This paper therefore offers an intuitive perspective to the literature and we believe it is a valid contribution towards understanding the practice of optional stopping in the context of Bayesian hypothesis testing.
Publisher: Center for Open Science
Date: 24-03-2020
Abstract: A majority of statistically educated scientists draw incorrect conclusions based on, arguably, the most commonly used statistical technique: null hypothesis significance testing (NHST). Frequentist techniques are often claimed to be incorrectly interpreted as Bayesian outcomes, which suggests that a Bayesian framework may fit better to inferences researchers frequently want to make (Briggs, 2012). The current study set out to test this proposition. Firstly, we investigated whether there is a discrepancy between what researchers think they can conclude and what they want to be able to conclude from NHST. Secondly, we investigated to what extent researchers want to incorporate prior study results and subjective beliefs in their statistical inference. Results show the expected discrepancy between what researchers think they can conclude from NHST and what they want to be able to conclude. Furthermore, researchers were interested in incorporating prior study results, but not subjective beliefs, into their statistical inference.
Publisher: Springer Science and Business Media LLC
Date: 13-08-2011
Publisher: Center for Open Science
Date: 20-10-2020
Abstract: The last 25 years have shown a steady increase in attention for the Bayes factor as a tool for hypothesis evaluation and model selection. The present review highlights the potential of the Bayes factor in psychological research. We discuss six types of applications: Bayesian evaluation of point null, interval, and informative hypotheses, Bayesian evidence synthesis, Bayesian variable selection and model averaging, and Bayesian evaluation of cognitive models. We elaborate what each application entails, give illustrative ex les, and provide an overview of key references and software with links to other applications. The paper is concluded with a discussion of the opportunities and pitfalls of Bayes factor applications and a sketch of corresponding future research lines.
Publisher: Center for Open Science
Date: 30-03-2021
Abstract: Planning s le size often requires researchers to identify a statistical technique and to make several choices during their calculations. Currently, there is a lack of clear guidelines for researchers to find and use the applicable procedure. In the present tutorial, we introduce a web app and R package that offer nine different procedures to determine and justify the s le size for independent two-group study designs. The application highlights the most important decision points for each procedure and suggests ex le justifications for them. The resulting s le size report can serve as a template for preregistrations and manuscripts.
Publisher: Center for Open Science
Date: 21-12-2022
Abstract: The push for open science has produced many initiatives that arguably improve the quality of science, such as preregistration of research methods, more general availability of scientific articles in the form of preprints and open access articles, and the availability of (meta)data and code. Open science can have direct benefits for the career prospects of the in idual through higher quality of the work and increased chance of publication (Allen & Mehler, 2019). However, being an outspoken advocate of open science might also have indirect positive effects on in idual scientific careers, in the form of status in a scientific community, decisions for tenure, and eligibility for certain kinds of funding. In light of these benefits, it may be profitable for in idual labs to appear to engage in open science practices, without actually putting in the associated effort or doing only the bare minimum. In this article, we explore two types of academic behavior through a dynamic computational model (cf. Smaldino & McElreath, 2016) of an academic community that rewards open science: 1) actively practicing open science and/or 2) advocating open science. Crossing these two types of behavior leads to four different kinds of labs and we examine which of them thrive in this academic community. Implications of the model results are discussed.
Publisher: Center for Open Science
Date: 14-03-2018
Abstract: For many years the Diffusion Decision Model (DDM) has successfully accounted for behavioral data from a wide range of domains. Important contributors to the DDM’s success are the across-trial variability parameters, which allow the model to account for the various shapes of response time distributions encountered in practice. However, several researchers have pointed out that estimating the variability parameters can be a challenging task. Moreover, the numerous fitting methods for the DDM each come with their own associated problems and solutions. This often leaves users in a difficult position. In this collaborative project we invited researchers from the DDM community to apply their various fitting methods to simulated data and provide advice and expert guidance on estimating the DDM’s between-trial variability parameters using these methods. Our study establishes a comprehensive reference resource and describes methods that can help to overcome the challenges associated with estimating the DDM’s across-trial variability parameters.
Publisher: The Royal Society
Date: 07-2023
DOI: 10.1098/RSOS.230448
Abstract: Theoretical arguments and empirical investigations indicate that a high proportion of published findings do not replicate and are likely false. The current position paper provides a broad perspective on scientific error, which may lead to replication failures. This broad perspective focuses on reform history and on opportunities for future reform. We organize our perspective along four main themes: institutional reform, methodological reform, statistical reform and publishing reform. For each theme, we illustrate potential errors by narrating the story of a fictional researcher during the research cycle. We discuss future opportunities for reform. The resulting agenda provides a resource to usher in an era that is marked by a research culture that is less error-prone and a scientific publication landscape with fewer spurious findings.
Publisher: Elsevier BV
Date: 12-2009
Publisher: Center for Open Science
Date: 22-02-2021
Abstract: Replication is a hallmark of scientific research. As replications of in idual studies are resource intensive, techniques for predicting the replicability are required. We introduce a new technique to evaluating replicability, the repliCATS (Collaborative Assessments for Trustworthy Science) process, a structured expert elicitation approach based on the IDEA protocol. The repliCATS process is delivered through an underpinning online platform and applied to the evaluation of research claims in social and behavioural sciences. This process can be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period. Pilot data suggests that the accuracy of the repliCATS process meets or exceeds that of other techniques used to predict replicability. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to assist with problems like understanding the limits of generalizability of scientific claims. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.
Publisher: Hogrefe Publishing Group
Date: 02-2011
DOI: 10.1027/1618-3169/A000093
Abstract: Research using the Implicit Association Test (IAT) has shown that names labeled as Caucasian elicit more positive associations than names labeled as non-Caucasian. One interpretation of this result is that the IAT measures latent racial prejudice. An alternative explanation is that the result is due to differences in in-group/out-group membership. In this study, we conducted three different IATs: one with same-race Dutch names versus racially charged Moroccan names one with same-race Dutch names versus racially neutral Finnish names and one with Moroccan names versus Finnish names. Results showed equivalent effects for the Dutch-Moroccan and Dutch-Finnish IATs, but no effect for the Finnish-Moroccan IAT. This suggests that the name-race IAT-effect is not due to racial prejudice. A diffusion model decomposition indicated that the IAT-effects were caused by changes in speed of information accumulation, response conservativeness, and non-decision time.
Publisher: Springer Science and Business Media LLC
Date: 27-01-2020
Publisher: Center for Open Science
Date: 20-04-2022
Abstract: Linde et al. (2021) compared the "two one-sided tests", the "highest density interval -- region of practical equivalence", and the "interval Bayes factor" approaches to establishing equivalence in terms of power and Type I error rate using typical decision thresholds. They found that the interval Bayes factor approach exhibited a higher power but also a higher Type I error rate than the other approaches. In response, C bell and Gustafson (2022) showed that the performances of the three approaches can approximate one another when they are calibrated to have the same Type I error rate. In this article, we argue that these results have little bearing on how these approaches are used in practice a concrete ex le is used to highlight this important point.
Publisher: Springer Science and Business Media LLC
Date: 09-2009
Publisher: Public Library of Science (PLoS)
Date: 03-10-2023
Publisher: Center for Open Science
Date: 19-06-2023
Abstract: The importance of replication in the social and behavioural sciences has been emphasized for decades. Various frequentist and Bayesian approaches have been proposed to qualify a replication study as successful or unsuccessful. One of them is meta-analysis. The focus of the present study is on the way meta-analysis functions as a replication success metric. To investigate this, original and replication studies that are part of two large-scale replication projects were used. For each original study, the probability of replication success was calculated using meta-analysis under different assumptions of the underlying population effect when replication results were unknown. The accuracy of the predicted overall replication success was evaluated once replication results became available using adjusted Brier scores. Our results showed that meta-analysis performed poorly when used as a replication success metric. In many cases, quantifying replication success using meta-analysis resulted in the conclusion where the replication was deemed a success regardless of the results of the replication study.
Publisher: Center for Open Science
Date: 25-09-2023
Publisher: Center for Open Science
Date: 06-03-2020
Abstract: Low replication rates and ill-informed selection of replication targets can harm clinical practice. We take a pragmatic approach to suggest which studies to replicate in clinical psychology. We propose a two-step process entailing a quantitative assessment of evidence strength using Bayes factors, and a qualitative assessment concerning theory and methodology. We provide proof of concept on a series of published clinical studies. We included 75 studies with 94 in idual effects. Step 1 yielded 42 effects (45%) with Bayes Factors suggesting ambiguous evidence or absence of an effect. These 42 effects were qualitatively assessed by two raters. We illustrate their decision process, and discuss advantages and disadvantages of the proposed steps.
Publisher: Cold Spring Harbor Laboratory
Date: 07-03-2021
DOI: 10.1101/2021.03.04.21252887
Abstract: Approval and prescription of drugs should be informed by the strength of evidence for efficacy. While there is no formal policy towards different standards for drug approval, the typical strength of evidence might differ for different psychotropic drug groups. Using a Bayesian framework, we examine (1) whether psychotropic drugs are supported by substantial evidence (at the time of Food and Drug Administration [FDA] approval), and (2) whether there are systematic differences across drug groups. Data from short-term, placebo-controlled phase II/III clinical trials for 15 antipsychotics, 16 antidepressants for depression, nine antidepressants for anxiety, and 20 drugs for ADHD were extracted from FDA reviews evaluating efficacy prior to marketing approval. Bayesian model-averaged meta-analysis was performed and strength of evidence was quantified with the Bayes factor ( BF BMA ). We observed substantial variation in strength of evidence and trialling between approved psychotropic drugs: Median evidential strength was extremely strong for ADHD medication ( BF BMA = 1820.4), but considerably lower and more frequently classified as weak or moderate for antidepressants for both depression ( BF BMA = 94.2) and anxiety ( BF BMA = 49.8). Differences might be accounted for by varying median effect sizes (schizophrenia: ES BMA = 0.45, depression: ES BMA = 0.30, anxiety: ES BMA = 0.37, ADHD: ES BMA = 0.72), s le sizes (schizophrenia: N = 324, depression: N = 218, anxiety: N = 254, ADHD: N = 189.5), and numbers of trials (schizophrenia: Nr = 3, depression: Nr = 5.5, anxiety: Nr = 3, ADHD: Nr = 2). The analysis only included pre-marketing studies. Evidential strength varied across drug groups: Although most psychotropic drugs were supported by strong evidence at the time of approval, some drugs only had moderate or even ambiguous evidence. These results show the need for more systematic quantification and classification of statistical evidence for psychotropic drugs, and for transparent and clear communication of evidential strength toward clinical decision makers. osf.io/5jn2d
Publisher: Center for Open Science
Date: 04-04-2018
Abstract: Efficient medical progress requires that we know when a treatment effect is absent. We considered all 207 Original Articles published in the 2015 volume of the New England Journal of Medicine and found that 45 (21.7%) reported a null result for at least one of the primary outcome measures. Unfortunately, standard statistical analyses are unable to quantify the degree to which these null results actually support the null hypothesis. Such quantification is possible, however, by conducting a Bayesian hypothesis test. Here we reanalyzed a subset of 43 null results from 36 articles using a default Bayesian test for contingency tables. This Bayesian reanalysis revealed that, on average, the reported null results provided strong evidence for the absence of an effect. However, the degree of this evidence is variable and cannot be reliably predicted from the p-value. For null results, s le size is a better (albeit imperfect) predictor for the strength of evidence in favor of the null hypothesis. Together, our findings suggest that (a) the reported null results generally correspond to strong evidence in favor of the null hypothesis (b) a Bayesian hypothesis test can provide additional information to assist the interpretation of null results.
Publisher: Center for Open Science
Date: 18-12-2018
Abstract: The crisis of confidence in psychology has prompted vigorous and persistent debate in the scientific community concerning the veracity of the findings of psychological experiments. This discussion has led to changes in psychology's approach to research, and several new initiatives have been developed, many with the aim of improving our findings. One key advancement is the marked increase in the number of replication studies conducted. We argue that while it is important to conduct replications as part of regular research protocol, it is neither efficient nor useful to replicate results at random. We recommend adopting a methodical approach toward the selection of replication targets to maximize the impact of the outcomes of those replications, and minimize waste of scarce resources. In the current study, we demonstrate how a Bayesian re--analysis of existing research findings followed by a simple qualitative assessment process can drive the selection of the best candidate article for replication. Submitted to Collabra: Psychology 18 Dec 2018
Publisher: American Psychological Association (APA)
Date: 11-2017
DOI: 10.1037/XLM0000407
Abstract: Learning to choose adaptively when faced with uncertain and variable outcomes is a central challenge for decision makers. This study examines repeated choice in dynamic probability learning tasks in which outcome probabilities changed either as a function of the choices participants made or independently of those choices. This presence/absence of sequential choice-outcome dependencies was implemented by manipulating a single task aspect between conditions: the retention/withdrawal of reward across in idual choice trials. The study addresses how people adapt to these learning environments and to what extent they engage in 2 choice strategies often contrasted as paradigmatic ex les of striking violation of versus nominal adherence to rational choice: ersification and persistent probability maximizing, respectively. Results show that decisions approached adaptive choice ersification and persistence when sufficient feedback was provided on the dynamic rules of the probabilistic environments. The findings of ergent behavior in the 2 environments indicate that ersified choices represented a response to the reward retention manipulation rather than to the mere variability of outcome probabilities. Choice in both environments was well accounted for by the generalized matching law, and computational modeling-based strategy analyses indicated that adaptive choice arose mainly from reliance on reinforcement learning strategies. (PsycINFO Database Record
Publisher: Hogrefe Publishing Group
Date: 05-2018
DOI: 10.1027/1618-3169/A000402
Abstract: Abstract. As a research field expands, scientists have to update their knowledge and integrate the outcomes of a sequence of studies. However, such integrative judgments are generally known to fall victim to a primacy bias where people anchor their judgments on the initial information. In this preregistered study we tested the hypothesis that people anchor on the outcome of a small initial study, reducing the impact of a larger subsequent study that contradicts the initial result. Contrary to our expectation, undergraduates and academics displayed a recency bias, anchoring their judgment on the research outcome presented last. This recency bias is due to the fact that unsuccessful replications decreased trust in an effect more than did unsuccessful initial experiments. We recommend the time-reversal heuristic to account for temporal order effects during integration of research results.
Publisher: Center for Open Science
Date: 23-02-2021
Abstract: Experts are often asked to represent their uncertainty as a subjective probability. Structured protocols offer a transparent and systematic way to elicit and combine probability judgements from multiple experts. As part of this process, experts are asked to in idually estimate a probability (e.g., of a future event) which needs to be combined, or aggregated, into a final group prediction. The experts' judgements can be aggregated behaviourally (by striving for consensus), or mathematically (by using a mathematical rule to combine in idual estimates). Mathematical rules (e.g., weighted linear combinations of judgments) provide an objective approach to aggregation. However, the choice of a rule is not straightforward, and the quality of the aggregated group judgement depends on it. The quality of an aggregation can be defined in terms of accuracy, calibration and informativeness. These measures can be used to compare different aggregation approaches and help decide on which aggregation produces the ``best” final prediction.In the ideal case, in idual experts' performance (as probability assessors) would be scored on similar questions ahead of time, these scores translated into performance-based weights, and a performance-based weighted aggregation could then be used. When this is not possible though, several other aggregation methods, informed by measurable proxies for good performance, can be formulated and compared. Here, we develop a suite of aggregation methods, informed by previous experience and the available literature. Next, we investigate the relative performance of these aggregation methods using three datasets. Although the accuracy, calibration, and informativeness of the majority of methods are very similar, a couple of the aggregation methods consistently distinguish themselves as among the best or worst.
Publisher: University of California Press
Date: 2018
DOI: 10.1525/COLLABRA.149
Abstract: The debate between Bayesians and frequentist statisticians has been going on for decades. Whilst there are fundamental theoretical and philosophical differences between both schools of thought, we argue that in two most common situations the practical differences are negligible when off-the-shelf Bayesian analysis (i.e., using ‘objective’ priors) is used. We emphasize this reasoning by focusing on interval estimates: confidence intervals and credible intervals. We show that this is the case for the most common empirical situations in the social sciences, the estimation of a proportion of a binomial distribution and the estimation of the mean of a unimodal distribution. Numerical differences between both approaches are small, sometimes even smaller than those between two competing frequentist or two competing Bayesian approaches. We outline the ramifications of this for scientific practice.
Publisher: American Psychological Association (APA)
Date: 2012
DOI: 10.1037/A0026275
Abstract: In their influential Psychological Review article, Bogacz, Brown, Moehlis, Holmes, and Cohen (2006) discussed optimal decision making as accomplished by the drift diffusion model (DDM). The authors showed that neural inhibition models, such as the leaky competing accumulator model (LCA) and the feedforward inhibition model (FFI), can mimic the DDM and accomplish optimal decision making. Here we show that these conclusions depend on how the models handle negative activation values and (for the LCA) across-trial variability in response conservativeness. Negative neural activations are undesirable for both neurophysiological and mathematical reasons. However, when negative activations are truncated to 0, the equivalence to the DDM is lost. Simulations show that this concern has practical ramifications: the DDM generally outperforms truncated versions of the LCA and the FFI, and the parameter estimates from the neural models can no longer be mapped onto those of the DDM in a simple fashion. We show that for both models, truncation may be avoided by assuming a baseline activity for each accumulator. This solution allows the LCA to approximate the DDM and the FFI to be identical to the DDM.
Publisher: Center for Open Science
Date: 06-12-2019
Abstract: In this study we examined the consequences of ignoring violations of assumptions un-derlying the use of sum scores in assessing attention problems (AP), and if psychomet-rically more refined models improve predictions of relevant outcomes in adulthood. Tracking Adolescents’ In idual Lives data were used. AP symptom properties were examined using the AP scale of the Child Behavior Checklist at age 11. Consequences of model violations were evaluated in relation to psychopathology, educational attain-ment, financial status, and ability to form relationships in adulthood. Results showed that symptoms differed with respect to information and difficulty. Moreover, evidence of multidimensionality was found, with two groups of items measuring sluggish cogni-tive tempo and ADHD symptoms. Item response theory analyses indicated that a bifac-tor model fitted these data better than other competing models. In terms of accuracy of predicting functional outcomes, sum scores were robust against violations of assump-tions in some situations. Nevertheless, AP scores derived from the bifactor model showed some superiority over sum scores. These findings show that more accurate predictions of later-life difficulties can be made if one uses a more suitable psychomet-ric model to assess AP severity in children. This has important implications for research and clinical practice.
Publisher: Springer Science and Business Media LLC
Date: 15-09-2015
Publisher: The Royal Society
Date: 09-2021
DOI: 10.1098/RSOS.191354
Abstract: Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalization of phenomena. Operationalization is the process of translating theoretical constructs into measurable laboratory quantities. Thus, the validity of operationalization is central for the quality of empirical studies. But do differences in the validity of operationalization affect the way scientists evaluate scientific literature? To investigate this, we manipulated the strength of operationalization of three published studies and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalization is more valid, but (2) the different validity is only to some extent reflected in a judgement of the study's quality. These results combined give partial corroboration to the notion that researchers’ evaluations of research results are not affected by operationalization validity.
Publisher: Center for Open Science
Date: 10-11-2020
Abstract: Some important research questions require the ability to find evidence for two conditions being practically equivalent. This is impossible to accomplish within the traditional frequentist null hypothesis significance testing framework hence, other methodologies must be utilized. We explain and illustrate three approaches for finding evidence for equivalence: The frequentist two one-sided tests procedure, the Bayesian highest density interval region of practical equivalence procedure, and the Bayes factor interval null procedure. We compare the classification performances of these three approaches for various plausible scenarios. The results indicate that the Bayes factor interval null approach compares favorably to the other two approaches in terms of statistical power. Critically, compared to the Bayes factor interval null procedure, the two one-sided tests and the highest density interval region of practical equivalence procedures have limited discrimination capabilities when the s le size is relatively small: specifically, in order to be practically useful, these two methods generally require over 250 cases within each condition when rather large equivalence margins of approximately 0.2 or 0.3 are used for smaller equivalence margins even more cases are required. Because of these results, we recommend that researchers rely more on the Bayes factor interval null approach for quantifying evidence for equivalence, especially for studies that are constrained on s le size.
Publisher: BMJ
Date: 07-03-2018
DOI: 10.1136/ARCHDISCHILD-2017-314081
Abstract: Calgranulin-C (S100A12) is a new faecal marker of inflammation that is potentially more specific for inflammatory bowel disease (IBD) than calprotectin, since it is only released by activated granulocytes. We compared calgranulin-C and calprotectin to see which of the two tests best predicted IBD in children with chronic abdominal pain and diarrhoea. Delayed-type cross-sectional diagnostic study. Previously undiagnosed patients aged 6–17 years, who were seen in paediatric clinics in the Netherlands and Belgium, sent in a stool s le for analysis. Patients with a high likelihood of IBD underwent upper and lower endoscopy (ie, preferred reference test), while those with a low likelihood were followed for 6 months for latent IBD to become visible (ie, alternative reference test). We used Bayesian modelling to correct for differential verification bias. Primary outcome was the specificity for IBD using predefined test thresholds (calgranulin-C: 0.75 µg/g, calprotectin: 50 µg/g). Secondary outcome was the test accuracy with thresholds based on receiver operating characteristics (ROC) analysis. IBD was diagnosed in 93 of 337 patients. Calgranulin-C had significantly better specificity than calprotectin when predefined thresholds were used (97% (95% credible interval (CI) 94% to 99%) vs 71% (95% CI 63% to 79%), respectively). When ROC-based thresholds were used (calgranulin-C: 0.75 µg/g, calprotectin: 400 µg/g), both tests performed equally well (specificity: 97% (95% CI 94% to 99%) vs 98% (95% CI 95% to 100%)). Both calgranulin-C and calprotectin have excellent test characteristics to predict IBD and justify endoscopy. NCT02197780 .
Publisher: Wiley
Date: 07-2019
DOI: 10.1002/MPR.1795
Publisher: SAGE Publications
Date: 2022
DOI: 10.1177/25152459211054059
Abstract: Planning s le size often requires researchers to identify a statistical technique and to make several choices during their calculations. Currently, there is a lack of clear guidelines for researchers to find and use the applicable procedure. In the present tutorial, we introduce a web app and R package that offer nine different procedures to determine and justify the s le size for independent two-group study designs. The application highlights the most important decision points for each procedure and suggests ex le justifications for them. The resulting s le-size report can serve as a template for preregistrations and manuscripts.
Publisher: American Psychological Association (APA)
Date: 03-2020
DOI: 10.1037/REV0000166
Abstract: Independent racing evidence-accumulator models have proven fruitful in advancing understanding of rapid decisions, mainly in the case of binary choice, where they can be relatively easily estimated and are known to account for a range of benchmark phenomena. Typically, such models assume a one-to-one mapping between accumulators and responses. We explore an alternative independent-race framework where more than one accumulator can be associated with each response, and where a response is triggered when a sufficient number of accumulators associated with that response reach their thresholds. Each accumulator is primarily driven by the difference in evidence supporting one versus another response (i.e., that response's "advantage"), with secondary inputs corresponding to the total evidence for both responses and a constant term. We use Brown and Heathcote's (2008) linear ballistic accumulator (LBA) to instantiate the framework in a mathematically tractable measurement model (i.e., a model whose parameters can be successfully recovered from data). We show this "advantage LBA" model provides a detailed quantitative account of a variety of benchmark binary and multiple choice phenomena that traditional independent accumulator models struggle with in binary choice the effects of additive versus multiplicative changes to input values, and in multiple choice the effects of manipulations of the strength of lure (i.e., nontarget) stimuli and Hick's law. We conclude that the advantage LBA provides a tractable new avenue for understanding the dynamics of decisions among multiple choices. (PsycINFO Database Record (c) 2020 APA, all rights reserved).
Publisher: Springer Science and Business Media LLC
Date: 29-03-2019
Publisher: Elsevier BV
Date: 03-2012
Publisher: American Psychological Association (APA)
Date: 06-2023
DOI: 10.1037/MET0000402
Publisher: Springer Science and Business Media LLC
Date: 06-03-2017
DOI: 10.3758/S13423-017-1259-Y
Abstract: The lexical-decision task is among the most commonly used paradigms in psycholinguistics. In both the signal-detection theory and Diffusion Decision Model (DDM Ratcliff, Gomez, & McKoon, Psychological Review, 111, 159-182, 2004) frameworks, lexical-decisions are based on a continuous source of word-likeness evidence for both words and non-words. The Retrieving Effectively from Memory model of Lexical-Decision (REM-LD Wagenmakers et al., Cognitive Psychology, 48(3), 332-367, 2004) provides a comprehensive explanation of lexical-decision data and makes the prediction that word-likeness evidence is more variable for words than non-words and that higher frequency words are more variable than lower frequency words. To test these predictions, we analyzed five lexical-decision data sets with the DDM. For all data sets, drift-rate variability changed across word frequency and non-word conditions. For the most part, REM-LD's predictions about the ordering of evidence variability across stimuli in the lexical-decision task were confirmed.
Start Date: 2018
End Date: 2023
Funder: Nederlandse Organisatie voor Wetenschappelijk Onderzoek
View Funded ActivityStart Date: 2013
End Date: 2015
Funder: Australian Research Council
View Funded ActivityStart Date: 2014
End Date: 2018
Funder: Australian Research Council
View Funded Activity