All models are wrong, some are useful, but are they reproducible? Commentary on Lee et al. (2019)

Michael Wilson, Russell Boag, Luke Strickland

Research output: Contribution to journalArticle

Abstract

Lee et al. (2019) make several practical recommendations for replicable and useful cognitive modeling. They also point out that the ultimate test of the usefulness of a cognitive model is its ability to solve practical problems. Solution-oriented modeling requires engaging practitioners who understand the relevantly applied domain but may lack extensive modeling expertise. In this commentary, we argue that for cognitive modeling to reach practitioners, there is a pressing need to move beyond providing the bare minimum information required for reproducibility and instead aim for an improved standard of transparency and reproducibility in cognitive modeling research. We discuss several mechanisms by which reproducible research can foster engagement with applied practitioners. Notably, reproducible materials provide a starting point for practitioners to experiment with cognitive models and evaluate whether they are suitable for their domain of expertise. This is essential because solving complex problems requires exploring a range of modeling approaches, and there may not be time to implement each possible approach from the ground up. Several specific recommendations for best practice are provided, including the application of containerization technologies. We also note the broader benefits of adopting gold standard reproducible practices within the field.
Original languageEnglish
JournalComputational Brain & Behavior
DOIs
Publication statusE-pub ahead of print - 12 Aug 2019

Fingerprint

modeling
containerization
transparency
gold
experiment
recommendation
need
material
test

Cite this

@article{96b600211dce4e4db7e3582a6cd65f6e,
title = "All models are wrong, some are useful, but are they reproducible? Commentary on Lee et al. (2019)",
abstract = "Lee et al. (2019) make several practical recommendations for replicable and useful cognitive modeling. They also point out that the ultimate test of the usefulness of a cognitive model is its ability to solve practical problems. Solution-oriented modeling requires engaging practitioners who understand the relevantly applied domain but may lack extensive modeling expertise. In this commentary, we argue that for cognitive modeling to reach practitioners, there is a pressing need to move beyond providing the bare minimum information required for reproducibility and instead aim for an improved standard of transparency and reproducibility in cognitive modeling research. We discuss several mechanisms by which reproducible research can foster engagement with applied practitioners. Notably, reproducible materials provide a starting point for practitioners to experiment with cognitive models and evaluate whether they are suitable for their domain of expertise. This is essential because solving complex problems requires exploring a range of modeling approaches, and there may not be time to implement each possible approach from the ground up. Several specific recommendations for best practice are provided, including the application of containerization technologies. We also note the broader benefits of adopting gold standard reproducible practices within the field.",
author = "Michael Wilson and Russell Boag and Luke Strickland",
year = "2019",
month = "8",
day = "12",
doi = "10.1007/s42113-019-00054-x",
language = "English",
journal = "Computational Brain & Behavior",
issn = "2522-0861",
publisher = "Springer Nature",

}

All models are wrong, some are useful, but are they reproducible? Commentary on Lee et al. (2019). / Wilson, Michael; Boag, Russell; Strickland, Luke.

In: Computational Brain & Behavior, 12.08.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - All models are wrong, some are useful, but are they reproducible? Commentary on Lee et al. (2019)

AU - Wilson, Michael

AU - Boag, Russell

AU - Strickland, Luke

PY - 2019/8/12

Y1 - 2019/8/12

N2 - Lee et al. (2019) make several practical recommendations for replicable and useful cognitive modeling. They also point out that the ultimate test of the usefulness of a cognitive model is its ability to solve practical problems. Solution-oriented modeling requires engaging practitioners who understand the relevantly applied domain but may lack extensive modeling expertise. In this commentary, we argue that for cognitive modeling to reach practitioners, there is a pressing need to move beyond providing the bare minimum information required for reproducibility and instead aim for an improved standard of transparency and reproducibility in cognitive modeling research. We discuss several mechanisms by which reproducible research can foster engagement with applied practitioners. Notably, reproducible materials provide a starting point for practitioners to experiment with cognitive models and evaluate whether they are suitable for their domain of expertise. This is essential because solving complex problems requires exploring a range of modeling approaches, and there may not be time to implement each possible approach from the ground up. Several specific recommendations for best practice are provided, including the application of containerization technologies. We also note the broader benefits of adopting gold standard reproducible practices within the field.

AB - Lee et al. (2019) make several practical recommendations for replicable and useful cognitive modeling. They also point out that the ultimate test of the usefulness of a cognitive model is its ability to solve practical problems. Solution-oriented modeling requires engaging practitioners who understand the relevantly applied domain but may lack extensive modeling expertise. In this commentary, we argue that for cognitive modeling to reach practitioners, there is a pressing need to move beyond providing the bare minimum information required for reproducibility and instead aim for an improved standard of transparency and reproducibility in cognitive modeling research. We discuss several mechanisms by which reproducible research can foster engagement with applied practitioners. Notably, reproducible materials provide a starting point for practitioners to experiment with cognitive models and evaluate whether they are suitable for their domain of expertise. This is essential because solving complex problems requires exploring a range of modeling approaches, and there may not be time to implement each possible approach from the ground up. Several specific recommendations for best practice are provided, including the application of containerization technologies. We also note the broader benefits of adopting gold standard reproducible practices within the field.

U2 - 10.1007/s42113-019-00054-x

DO - 10.1007/s42113-019-00054-x

M3 - Article

JO - Computational Brain & Behavior

JF - Computational Brain & Behavior

SN - 2522-0861

ER -