ORCID Profile
0000-0002-3205-1983
Current Organisation
Deakin University
Does something not look right? The information on this page has been harvested from data sources that may not be up to date. We continue to work with information providers to improve coverage and quality. To report an issue, use the Feedback Form.
Publisher: Center for Open Science
Date: 11-05-2023
Abstract: We demonstrate that all conventional meta-analyses of correlation coefficients are biased, explain why, and offer solutions. Because the standard error of the correlation coefficient depends on the size of the coefficient, inverse-variance weighted averages will be biased even under ideal meta-analytical conditions (i.e., absence of publication bias, p-hacking, or other biases). Transformation to Fisher’s z often greatly reduces these biases but still does not mitigate them entirely. Although all are small-s le biases (n & 200), they will often have practical consequences in psychology where the typical s le size of correlational studies is 86. We offer several solutions: a newly developed estimator, UWLS+3 and two small-s le adjustments. UWLS+3 is the unrestricted weighted least squares weighted average (UWLS) that adjusts the degrees of freedom used to calculate correlations and thereby renders any remaining bias scientifically trivial. We also offer a simple small-s le correction, (n-2)/(n-1), for random-effects that works nearly as well as these other adjustments in most applications and a small-s le adjustment, (4n-2)/(4n-1), to Fisher’s z-transformation that is better still.
Publisher: Wiley
Date: 03-08-2021
DOI: 10.1002/JRSM.1512
Abstract: We introduce and evaluate three tests for publication selection bias based on excess statistical significance (ESS). The proposed tests incorporate heterogeneity explicitly in the formulas for expected and ESS. We calculate the expected proportion of statistically significant findings in the absence of selective reporting or publication bias based on each study's SE and meta‐analysis estimates of the mean and variance of the true‐effect distribution. A simple proportion of statistical significance test (PSST) compares the expected to the observed proportion of statistically significant findings. Alternatively, we propose a direct test of excess statistical significance (TESS). We also combine these two tests of excess statistical significance (TESSPSST). Simulations show that these ESS tests often outperform the conventional Egger test for publication selection bias and the three‐parameter selection model (3PSM).
Publisher: Wiley
Date: 07-2019
DOI: 10.1111/BJIR.12483
Publisher: Center for Open Science
Date: 06-2022
Abstract: Adjusting for publication bias is essential when drawing meta-analytic inferences. However,most methods that adjust for publication bias are sensitive to the particular researchconditions, such as the degree of heterogeneity in effect sizes across studies. Sladekovaet al. (2022) tried to circumvent this complication by selecting the methods that are mostappropriate for a given set of conditions, and concluded that publication bias on averagecauses only minimal over-estimation of effect sizes in psychology. However, this approachsuffers from a “catch-22” problem — to know the underlying research conditions, one needsto have adjusted for publication bias correctly, but to correctly adjust for publication bias,one needs to know the underlying research conditions. To alleviate this problem weconduct an alternative analysis, Robust Bayesian meta-analysis (RoBMA), which is notbased on model-selection but on model-averaging. In RoBMA, models that predict theobserved results better are given correspondingly higher weights. A RoBMA reanalysis ofSladekova et al.’s data reveals that more than 60% of meta-analyses in psychology notablyoverestimate the evidence for the presence of the meta-analytic effect and more than 50%overestimate its magnitude. Our results highlight the need for robust bias correction whenconducting meta-analyses and for the adoption of publishing formats such as RegisteredReports that are less prone to publication bias.
Publisher: SAGE Publications
Date: 10-2022
DOI: 10.1177/25152459221120427
Abstract: New meta-regression methods are introduced that identify whether the magnitude of heterogeneity across study findings is correlated with their standard errors. Evidence from dozens of meta-analyses finds robust evidence of this correlation and that small-s le studies typically have higher heterogeneity. This correlated heterogeneity violates the random-effects (RE) model of additive and independent heterogeneity. When small studies not only have inadequate statistical power but also high heterogeneity, their scientific contribution is even more dubious. When the heterogeneity variance is correlated with the s ling-error variance to the degree we find, simulations show that RE is dominated by an alternative weighted average, the unrestricted weighted least squares (UWLS). Meta-research evidence combined with simulations establish that UWLS should replace RE as the conventional meta-analysis summary of psychological research.
Publisher: Wiley
Date: 09-03-2020
DOI: 10.1111/JOES.12363
Publisher: Center for Open Science
Date: 10-05-2023
Abstract: In their book `Nudge: Improving Decisions About Health, Wealth and Happiness', Thaler and Sunstein argue that choice architectures are promising public policy interventions. This research programme motivated the creation of so-called `nudge units' which aim to apply insights from behavioural science to improve public policy. We take a close look at a meta-analysis of the evidence gathered by two of the largest and most influential nudge units using statistical techniques to detect reporting biases. We find evidence suggestive of selective reporting on the full dataset, although this pattern is not robust in subgroup analyses. We therefore additionally evaluate the public pre-analysis plans from the Office of Evaluation Sciences. We find that the analysis plans and reporting usually lack sufficient detail to evaluate (unintentional) reporting biases. Nevertheless, we identify several instances of excellent practice. We additionally highlight several improvements that would enhance the effectiveness of the pre-analysis plans and reports as a means to combat reporting biases. We believe our findings and suggestions can further improve the evidence base on which policy decisions are made.
Publisher: Center for Open Science
Date: 17-06-2021
Abstract: Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed however, recent simulation studies have shown the methods’ performance to depend on the true data generating process – no method consistently outperforms the others across a wide range of conditions. To avoid the condition-dependent, all-or-none choice between competing methods we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models of the relationship between effect sizes and their standard errors. The resulting estimator weights the models with the support they receive from the existing research record. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of competing publication bias adjustment methods.
Publisher: Wiley
Date: 25-04-2023
DOI: 10.1111/BJIR.12746
Abstract: We survey 20,439 estimates from 64 distinct research areas to assess power, bias and statistical significance in industrial relations research. The average estimate published in industrial relations research lacks adequate power average power is 33 per cent, and median power is only 14 per cent, much lower than the conventional 80 per cent standard. Low power means that industrial relations researchers will find it more difficult to detect important associations pertaining to workplace relations. Low power also leads to exaggerated research findings. We find substantial publication bias in industrial relations research, though nearly half of the research areas have little or no bias.
Publisher: Wiley
Date: 23-03-2018
DOI: 10.1111/JOES.12211
Publisher: Wiley
Date: 29-10-2021
DOI: 10.1002/JRSM.1529
Abstract: Recent, high‐profile, large‐scale, preregistered failures to replicate uncover that many highly‐regarded experiments are “false positives” that is, statistically significant results of underlying null effects. Large surveys of research reveal that statistical power is often low and inadequate. When the research record includes selective reporting, publication bias and/or questionable research practices, conventional meta‐analyses are also likely to be falsely positive. At the core of research credibility lies the relation of statistical power to the rate of false positives. This study finds that high ( %–60%) median retrospective power (MRP) is associated with credible meta‐analysis and large‐scale, preregistered, multi‐lab “successful” replications that is, with replications that corroborate the effect in question. When median retrospective power is low ( %), positive meta‐analysis findings should be interpreted with great caution or discounted altogether.
Publisher: Center for Open Science
Date: 12-10-2023
Publisher: American Psychological Association (APA)
Date: 12-05-2022
DOI: 10.1037/MET0000502
Abstract: We introduce a new meta-analysis estimator, the weighted and iterated least squares (WILS), that greatly reduces publication selection bias (PSB) when selective reporting for statistical significance (SSS) is present. WILS is the simple weighted average that has smaller bias and rates of false positives than conventional meta-analysis estimators, the unrestricted weighted least squares (UWLS), and the weighted average of the adequately powered (WAAP) when there is SSS. As a simple weighted average, it is not vulnerable to violations in publication bias corrections models' assumptions too often seen in application. WILS is based on the novel idea of allowing excess statistical significance (ESS), which is a necessary condition of SSS, to identify when and how to reduce PSB. We show in comparisons with large-scale preregistered replications and in evidence-based simulations that the remaining bias is small. The routine application of WILS in the place of random effects would do much to reduce conventional meta-analysis's notable biases and high rates of false positives. (PsycInfo Database Record (c) 2022 APA, all rights reserved).
Publisher: Center for Open Science
Date: 11-03-2022
Abstract: Using a s le of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been published as being non-significant. We estimate the extent of inflated significance to range between 56.2% and 71.3% of the significant p-values. Subs le analysis suggests that the extent of inflated significance is reduced in research fields that use experimental designs, analyze microeconomics research questions and have at least some adequately powered studies.
Publisher: Center for Open Science
Date: 08-02-2022
Abstract: Cash transfers are among the most popular poverty interventions. Indeed the charity evaluator GiveWell even lists GiveDirectly - a charity that directly sends your donations as cash to people in extreme poverty - as one of their top-rated charities [harities/give-directly]. McGuire, Kaiser, and Bach-Mortensen1 conducted a timely and comprehensive meta-analysis on the impact of cash transfers on subjective well-being and mental health, featuring 45 studies with a combined total of 116,999 in iduals. McGuire and colleagues1 conclude “CTs [cash transfers] have a small but statistically significant positive effect on both SWB [subjective well-being] (Cohen’s d = 0.13, 95% confidence interval (CI) 0.09, 0.18) and MH [mental health] (d = 0.07, 95% CI 0.05, 0.09) among recipients.” We show that once publication bias is properly accounted for, this effect is - depending on the outcome measure - either greatly reduced or completely diminished.
Publisher: Center for Open Science
Date: 14-08-2023
Abstract: This paper provides concise, nontechnical, step-by-step guidelines on how to conduct a modern meta-analysis, especially in social sciences. We treat publication bias, p-hacking, and heterogeneity as phenomena meta-analysts must always confront. To this end, we provide concrete methodological recommendations. Meta-analysis methods have advanced notably over the last few years. Yet many meta-analyses still rely on outdated approaches, some ignoring publication bias and systematic heterogeneity. While limitations persist, recently developed techniques allow robust inference even in the face of formidable problems in the underlying empirical literature. The purpose of this paper is to summarize the state of the art in a way accessible to aspiring meta-analysts in any field. We also discuss how meta-analysts can use advances in artificial intelligence to work more efficiently.
Publisher: Wiley
Date: 07-08-2022
DOI: 10.1002/JRSM.1594
Abstract: Publication bias is a ubiquitous threat to the validity of meta‐analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition‐dependent, all‐or‐none choice between competing methods and conflicting results, we extend robust Bayesian meta‐analysis and model‐average across two prominent approaches of adjusting for publication bias: (1) selection models of p ‐values and (2) models adjusting for small‐study effects. The resulting model ensemble weights the estimates and the evidence for the absence resence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi‐lab replications demonstrate the benefits of Bayesian model‐averaging of complementary publication bias adjustment methods.
Publisher: Center for Open Science
Date: 19-05-2023
Abstract: We demonstrate that all meta-analyses of partial correlations are biased, and yet hundreds of meta-analyses of partial correlation coefficients (PCC) are conducted each year widely across economics, business, education, psychology, and medical research. To address these biases, we offer a new weighted average, UWLS+3. UWLS+3 is the unrestricted weighted least squares weighted average that makes an adjustment to the degrees of freedom that are used to calculate partial correlations and, by doing so, renders trivial any remaining meta-analysis bias. Our simulations also reveal that these meta-analysis biases are small-s le biases (n & 200), and a simple correction factor of (n-2)/(n-1) greatly reduces these small-s le biases. In many applications where primary studies typically have hundreds or more observations, partial correlations can be meta-analyzed in standard ways with only negligible bias. However, in other fields in the social and the medical sciences that are dominated by small s les, these meta-analysis biases are easily avoidable by our proposed methods.
No related grants have been discovered for T. D. Stanley.