Benchmarks for models of short-term and working memory

Klaus Oberaur, Stephan Lewandowsky, Edward Awh, Gordon Brown, Andrew Conway, Nelson Cowan, Christopher Donkin, Simon Farrell, Graham Hitch, Mark John Hurlstone, Wei Ji Ma, Candice Morey, Derek Nee, Judith Schweppe, Evie Vergauwe, Geoff Ward

Research output: Contribution to journalArticle

  • 4 Citations

Abstract

Any mature field of research in psychology – such as short-term/working memory – is characterized by a wealth of empirical findings. It is currently unrealistic to expect a theory to explain them all; theorists must satisfice with explaining a subset of findings. The aim of the present article is to make the choice of that subset less arbitrary and idiosyncratic than is current practice. We propose criteria for identifying benchmark findings that every theory in a field should be able to explain: Benchmarks should be reproducible, generalize across materials and methodological variations, and be theoretically informative. We propose a set of benchmarks for theories and computational models of short-term and working memory. The benchmarks are described in as theory-neutral a way as possible, so that they can serve as empirical common ground for competing theoretical approaches. Benchmarks are rated on three levels according to their priority for explanation. Selection and ratings of the benchmarks is based on consensus among the authors, who jointly represent a broad range of theoretical perspectives on working memory, and they are supported by a survey among other experts on working memory. The article is accompanied by a web page providing an open forum for discussion; a site for submitting proposals for new benchmarks; and a repository for reference data sets for each benchmark.
LanguageEnglish
Pages885-958
JournalPsychological Bulletin
Volume144
Issue number9
DOIs
Publication statusPublished - Sep 2018

Fingerprint

Benchmarking
Short-Term Memory
Consensus
Psychology

Cite this

Oberaur, K., Lewandowsky, S., Awh, E., Brown, G., Conway, A., Cowan, N., ... Ward, G. (2018). Benchmarks for models of short-term and working memory. Psychological Bulletin, 144(9), 885-958. https://doi.org/10.1037/bul0000153
Oberaur, Klaus ; Lewandowsky, Stephan ; Awh, Edward ; Brown, Gordon ; Conway, Andrew ; Cowan, Nelson ; Donkin, Christopher ; Farrell, Simon ; Hitch, Graham ; Hurlstone, Mark John ; Ma, Wei Ji ; Morey, Candice ; Nee, Derek ; Schweppe, Judith ; Vergauwe, Evie ; Ward, Geoff. / Benchmarks for models of short-term and working memory. In: Psychological Bulletin. 2018 ; Vol. 144, No. 9. pp. 885-958.
@article{ff3408907144415fb97867be3aab844c,
title = "Benchmarks for models of short-term and working memory",
abstract = "Any mature field of research in psychology – such as short-term/working memory – is characterized by a wealth of empirical findings. It is currently unrealistic to expect a theory to explain them all; theorists must satisfice with explaining a subset of findings. The aim of the present article is to make the choice of that subset less arbitrary and idiosyncratic than is current practice. We propose criteria for identifying benchmark findings that every theory in a field should be able to explain: Benchmarks should be reproducible, generalize across materials and methodological variations, and be theoretically informative. We propose a set of benchmarks for theories and computational models of short-term and working memory. The benchmarks are described in as theory-neutral a way as possible, so that they can serve as empirical common ground for competing theoretical approaches. Benchmarks are rated on three levels according to their priority for explanation. Selection and ratings of the benchmarks is based on consensus among the authors, who jointly represent a broad range of theoretical perspectives on working memory, and they are supported by a survey among other experts on working memory. The article is accompanied by a web page providing an open forum for discussion; a site for submitting proposals for new benchmarks; and a repository for reference data sets for each benchmark.",
author = "Klaus Oberaur and Stephan Lewandowsky and Edward Awh and Gordon Brown and Andrew Conway and Nelson Cowan and Christopher Donkin and Simon Farrell and Graham Hitch and Hurlstone, {Mark John} and Ma, {Wei Ji} and Candice Morey and Derek Nee and Judith Schweppe and Evie Vergauwe and Geoff Ward",
year = "2018",
month = "9",
doi = "10.1037/bul0000153",
language = "English",
volume = "144",
pages = "885--958",
journal = "Psychological Bulletin",
issn = "0033-2909",
publisher = "American Psychological Association",
number = "9",

}

Oberaur, K, Lewandowsky, S, Awh, E, Brown, G, Conway, A, Cowan, N, Donkin, C, Farrell, S, Hitch, G, Hurlstone, MJ, Ma, WJ, Morey, C, Nee, D, Schweppe, J, Vergauwe, E & Ward, G 2018, 'Benchmarks for models of short-term and working memory' Psychological Bulletin, vol. 144, no. 9, pp. 885-958. https://doi.org/10.1037/bul0000153

Benchmarks for models of short-term and working memory. / Oberaur, Klaus; Lewandowsky, Stephan; Awh, Edward; Brown, Gordon; Conway, Andrew; Cowan, Nelson; Donkin, Christopher; Farrell, Simon; Hitch, Graham; Hurlstone, Mark John; Ma, Wei Ji; Morey, Candice; Nee, Derek; Schweppe, Judith; Vergauwe, Evie; Ward, Geoff.

In: Psychological Bulletin, Vol. 144, No. 9, 09.2018, p. 885-958.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Benchmarks for models of short-term and working memory

AU - Oberaur, Klaus

AU - Lewandowsky, Stephan

AU - Awh, Edward

AU - Brown, Gordon

AU - Conway, Andrew

AU - Cowan, Nelson

AU - Donkin, Christopher

AU - Farrell, Simon

AU - Hitch, Graham

AU - Hurlstone, Mark John

AU - Ma, Wei Ji

AU - Morey, Candice

AU - Nee, Derek

AU - Schweppe, Judith

AU - Vergauwe, Evie

AU - Ward, Geoff

PY - 2018/9

Y1 - 2018/9

N2 - Any mature field of research in psychology – such as short-term/working memory – is characterized by a wealth of empirical findings. It is currently unrealistic to expect a theory to explain them all; theorists must satisfice with explaining a subset of findings. The aim of the present article is to make the choice of that subset less arbitrary and idiosyncratic than is current practice. We propose criteria for identifying benchmark findings that every theory in a field should be able to explain: Benchmarks should be reproducible, generalize across materials and methodological variations, and be theoretically informative. We propose a set of benchmarks for theories and computational models of short-term and working memory. The benchmarks are described in as theory-neutral a way as possible, so that they can serve as empirical common ground for competing theoretical approaches. Benchmarks are rated on three levels according to their priority for explanation. Selection and ratings of the benchmarks is based on consensus among the authors, who jointly represent a broad range of theoretical perspectives on working memory, and they are supported by a survey among other experts on working memory. The article is accompanied by a web page providing an open forum for discussion; a site for submitting proposals for new benchmarks; and a repository for reference data sets for each benchmark.

AB - Any mature field of research in psychology – such as short-term/working memory – is characterized by a wealth of empirical findings. It is currently unrealistic to expect a theory to explain them all; theorists must satisfice with explaining a subset of findings. The aim of the present article is to make the choice of that subset less arbitrary and idiosyncratic than is current practice. We propose criteria for identifying benchmark findings that every theory in a field should be able to explain: Benchmarks should be reproducible, generalize across materials and methodological variations, and be theoretically informative. We propose a set of benchmarks for theories and computational models of short-term and working memory. The benchmarks are described in as theory-neutral a way as possible, so that they can serve as empirical common ground for competing theoretical approaches. Benchmarks are rated on three levels according to their priority for explanation. Selection and ratings of the benchmarks is based on consensus among the authors, who jointly represent a broad range of theoretical perspectives on working memory, and they are supported by a survey among other experts on working memory. The article is accompanied by a web page providing an open forum for discussion; a site for submitting proposals for new benchmarks; and a repository for reference data sets for each benchmark.

U2 - 10.1037/bul0000153

DO - 10.1037/bul0000153

M3 - Article

VL - 144

SP - 885

EP - 958

JO - Psychological Bulletin

T2 - Psychological Bulletin

JF - Psychological Bulletin

SN - 0033-2909

IS - 9

ER -