Blending Bayesian and classical tools to define optimal sample-size-dependent significance levels

Mark Andrew Gannon, Carlos Alberto de Braganca Pereira, Adriano Polpo

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

This article argues that researchers do not need to completely abandon the p-value, the best-known significance index, but should instead stop using significance levels that do not depend on sample sizes. A testing procedure is developed using a mixture of frequentist and Bayesian tools, with a significance level that is a function of sample size, obtained from a generalized form of the Neyman-Pearson Lemma that minimizes a linear combination of alpha, the probability of rejecting a true null hypothesis, and ,beta, the probability of failing to reject a false null, instead of fixing alpha and minimizing beta. The resulting hypothesis tests do not violate the Likelihood Principle and do not require any constraints on the dimensionalities of the sample space and parameter space. The procedure includes an ordering of the entire sample space and uses predictive probability (density) functions, allowing for testing of both simple and compound hypotheses. Accessible examples are presented to highlight specific characteristics of the new tests.

Original languageEnglish
Pages (from-to)213-222
Number of pages10
JournalAmerican Statistician
Volume73
DOIs
Publication statusPublished - 2019
Externally publishedYes

Cite this

Gannon, Mark Andrew ; de Braganca Pereira, Carlos Alberto ; Polpo, Adriano. / Blending Bayesian and classical tools to define optimal sample-size-dependent significance levels. In: American Statistician. 2019 ; Vol. 73. pp. 213-222.
@article{4a923a6c1c6149b991f9f4bdefc32346,
title = "Blending Bayesian and classical tools to define optimal sample-size-dependent significance levels",
abstract = "This article argues that researchers do not need to completely abandon the p-value, the best-known significance index, but should instead stop using significance levels that do not depend on sample sizes. A testing procedure is developed using a mixture of frequentist and Bayesian tools, with a significance level that is a function of sample size, obtained from a generalized form of the Neyman-Pearson Lemma that minimizes a linear combination of alpha, the probability of rejecting a true null hypothesis, and ,beta, the probability of failing to reject a false null, instead of fixing alpha and minimizing beta. The resulting hypothesis tests do not violate the Likelihood Principle and do not require any constraints on the dimensionalities of the sample space and parameter space. The procedure includes an ordering of the entire sample space and uses predictive probability (density) functions, allowing for testing of both simple and compound hypotheses. Accessible examples are presented to highlight specific characteristics of the new tests.",
keywords = "Hardy-Weinberg equilibrium, Neyman-Pearson lemma, Predictive distribution, Significance test, TESTS",
author = "Gannon, {Mark Andrew} and {de Braganca Pereira}, {Carlos Alberto} and Adriano Polpo",
year = "2019",
doi = "10.1080/00031305.2018.1518268",
language = "English",
volume = "73",
pages = "213--222",
journal = "American Statistician",
issn = "0003-1305",
publisher = "American Statistical Association",

}

Blending Bayesian and classical tools to define optimal sample-size-dependent significance levels. / Gannon, Mark Andrew; de Braganca Pereira, Carlos Alberto; Polpo, Adriano.

In: American Statistician, Vol. 73, 2019, p. 213-222.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Blending Bayesian and classical tools to define optimal sample-size-dependent significance levels

AU - Gannon, Mark Andrew

AU - de Braganca Pereira, Carlos Alberto

AU - Polpo, Adriano

PY - 2019

Y1 - 2019

N2 - This article argues that researchers do not need to completely abandon the p-value, the best-known significance index, but should instead stop using significance levels that do not depend on sample sizes. A testing procedure is developed using a mixture of frequentist and Bayesian tools, with a significance level that is a function of sample size, obtained from a generalized form of the Neyman-Pearson Lemma that minimizes a linear combination of alpha, the probability of rejecting a true null hypothesis, and ,beta, the probability of failing to reject a false null, instead of fixing alpha and minimizing beta. The resulting hypothesis tests do not violate the Likelihood Principle and do not require any constraints on the dimensionalities of the sample space and parameter space. The procedure includes an ordering of the entire sample space and uses predictive probability (density) functions, allowing for testing of both simple and compound hypotheses. Accessible examples are presented to highlight specific characteristics of the new tests.

AB - This article argues that researchers do not need to completely abandon the p-value, the best-known significance index, but should instead stop using significance levels that do not depend on sample sizes. A testing procedure is developed using a mixture of frequentist and Bayesian tools, with a significance level that is a function of sample size, obtained from a generalized form of the Neyman-Pearson Lemma that minimizes a linear combination of alpha, the probability of rejecting a true null hypothesis, and ,beta, the probability of failing to reject a false null, instead of fixing alpha and minimizing beta. The resulting hypothesis tests do not violate the Likelihood Principle and do not require any constraints on the dimensionalities of the sample space and parameter space. The procedure includes an ordering of the entire sample space and uses predictive probability (density) functions, allowing for testing of both simple and compound hypotheses. Accessible examples are presented to highlight specific characteristics of the new tests.

KW - Hardy-Weinberg equilibrium

KW - Neyman-Pearson lemma

KW - Predictive distribution

KW - Significance test

KW - TESTS

U2 - 10.1080/00031305.2018.1518268

DO - 10.1080/00031305.2018.1518268

M3 - Article

VL - 73

SP - 213

EP - 222

JO - American Statistician

JF - American Statistician

SN - 0003-1305

ER -