A systematic review of methodology used in the development of prediction models for future asthma exacerbation

Joshua Bridge, John D Blakey, Laura J Bonnett

Research output: Contribution to journalArticle

Abstract

BACKGROUND: Clinical prediction models are widely used to guide medical advice and therapeutic interventions. Asthma is one of the most common chronic diseases globally and is characterised by acute deteriorations. These exacerbations are largely preventable, so there is interest in using clinical prediction models in this area. The objective of this review was to identify studies which have developed such models, determine whether consistent and appropriate methodology was used and whether statistically reliable prognostic models exist.

METHODS: We searched online databases MEDLINE (1948 onwards), CINAHL Plus (1937 onwards), The Cochrane Library, Web of Science (1898 onwards) and ClinicalTrials.gov, using index terms relating to asthma and prognosis. Data was extracted and assessment of quality was based on GRADE and an early version of PROBAST (Prediction study Risk of Bias Assessment Tool). A meta-analysis of the discrimination and calibration measures was carried out to determine overall performance across models.

RESULTS: Ten unique prognostic models were identified. GRADE identified moderate risk of bias in two of the studies, but more detailed quality assessment via PROBAST highlighted that most models were developed using highly selected and small datasets, incompletely recorded predictors and outcomes, and incomplete methodology. None of the identified models modelled recurrent exacerbations, instead favouring either presence/absence of an event, or time to first or specified event. Preferred methodologies were logistic regression and Cox proportional hazards regression. The overall pooled c-statistic was 0.77 (95% confidence interval 0.73 to 0.80), though individually some models performed no better than chance. The meta-analysis had an I2 value of 99.75% indicating a high amount of heterogeneity between studies. The majority of studies were small and did not include internal or external validation, therefore the individual performance measures are likely to be optimistic.

CONCLUSIONS: Current prognostic models for asthma exacerbations are heterogeneous in methodology, but reported c-statistics suggest a clinically useful model could be created. Studies were consistent in lacking robust validation and in not modelling serial events. Further research is required with respect to incorporating recurrent events, and to externally validate tools in large representative populations to demonstrate the generalizability of published results.

Original languageEnglish
Pages (from-to)22
JournalBMC Medical Research Methodology
Volume20
Issue number1
DOIs
Publication statusPublished - 5 Feb 2020
Externally publishedYes

Fingerprint

Asthma
Meta-Analysis
Library Science
MEDLINE
Calibration
Chronic Disease
Logistic Models
Databases
Confidence Intervals
Research
Population
Therapeutics
Datasets

Bibliographical note

I confirm that I undertook the research leading to this publication in my capacity as a staff member or affiliated associate to the University of Western Australia

Cite this

@article{37683c5c7f9b4cffa3e8fa412e1c0a82,
title = "A systematic review of methodology used in the development of prediction models for future asthma exacerbation",
abstract = "BACKGROUND: Clinical prediction models are widely used to guide medical advice and therapeutic interventions. Asthma is one of the most common chronic diseases globally and is characterised by acute deteriorations. These exacerbations are largely preventable, so there is interest in using clinical prediction models in this area. The objective of this review was to identify studies which have developed such models, determine whether consistent and appropriate methodology was used and whether statistically reliable prognostic models exist.METHODS: We searched online databases MEDLINE (1948 onwards), CINAHL Plus (1937 onwards), The Cochrane Library, Web of Science (1898 onwards) and ClinicalTrials.gov, using index terms relating to asthma and prognosis. Data was extracted and assessment of quality was based on GRADE and an early version of PROBAST (Prediction study Risk of Bias Assessment Tool). A meta-analysis of the discrimination and calibration measures was carried out to determine overall performance across models.RESULTS: Ten unique prognostic models were identified. GRADE identified moderate risk of bias in two of the studies, but more detailed quality assessment via PROBAST highlighted that most models were developed using highly selected and small datasets, incompletely recorded predictors and outcomes, and incomplete methodology. None of the identified models modelled recurrent exacerbations, instead favouring either presence/absence of an event, or time to first or specified event. Preferred methodologies were logistic regression and Cox proportional hazards regression. The overall pooled c-statistic was 0.77 (95{\%} confidence interval 0.73 to 0.80), though individually some models performed no better than chance. The meta-analysis had an I2 value of 99.75{\%} indicating a high amount of heterogeneity between studies. The majority of studies were small and did not include internal or external validation, therefore the individual performance measures are likely to be optimistic.CONCLUSIONS: Current prognostic models for asthma exacerbations are heterogeneous in methodology, but reported c-statistics suggest a clinically useful model could be created. Studies were consistent in lacking robust validation and in not modelling serial events. Further research is required with respect to incorporating recurrent events, and to externally validate tools in large representative populations to demonstrate the generalizability of published results.",
author = "Joshua Bridge and Blakey, {John D} and Bonnett, {Laura J}",
note = "I confirm that I undertook the research leading to this publication in my capacity as a staff member or affiliated associate to the University of Western Australia",
year = "2020",
month = "2",
day = "5",
doi = "10.1186/s12874-020-0913-7",
language = "English",
volume = "20",
pages = "22",
journal = "BMC Medical Research Methodology",
issn = "1471-2288",
publisher = "BioMed Central",
number = "1",

}

A systematic review of methodology used in the development of prediction models for future asthma exacerbation. / Bridge, Joshua; Blakey, John D; Bonnett, Laura J.

In: BMC Medical Research Methodology, Vol. 20, No. 1, 05.02.2020, p. 22.

Research output: Contribution to journalArticle

TY - JOUR

T1 - A systematic review of methodology used in the development of prediction models for future asthma exacerbation

AU - Bridge, Joshua

AU - Blakey, John D

AU - Bonnett, Laura J

N1 - I confirm that I undertook the research leading to this publication in my capacity as a staff member or affiliated associate to the University of Western Australia

PY - 2020/2/5

Y1 - 2020/2/5

N2 - BACKGROUND: Clinical prediction models are widely used to guide medical advice and therapeutic interventions. Asthma is one of the most common chronic diseases globally and is characterised by acute deteriorations. These exacerbations are largely preventable, so there is interest in using clinical prediction models in this area. The objective of this review was to identify studies which have developed such models, determine whether consistent and appropriate methodology was used and whether statistically reliable prognostic models exist.METHODS: We searched online databases MEDLINE (1948 onwards), CINAHL Plus (1937 onwards), The Cochrane Library, Web of Science (1898 onwards) and ClinicalTrials.gov, using index terms relating to asthma and prognosis. Data was extracted and assessment of quality was based on GRADE and an early version of PROBAST (Prediction study Risk of Bias Assessment Tool). A meta-analysis of the discrimination and calibration measures was carried out to determine overall performance across models.RESULTS: Ten unique prognostic models were identified. GRADE identified moderate risk of bias in two of the studies, but more detailed quality assessment via PROBAST highlighted that most models were developed using highly selected and small datasets, incompletely recorded predictors and outcomes, and incomplete methodology. None of the identified models modelled recurrent exacerbations, instead favouring either presence/absence of an event, or time to first or specified event. Preferred methodologies were logistic regression and Cox proportional hazards regression. The overall pooled c-statistic was 0.77 (95% confidence interval 0.73 to 0.80), though individually some models performed no better than chance. The meta-analysis had an I2 value of 99.75% indicating a high amount of heterogeneity between studies. The majority of studies were small and did not include internal or external validation, therefore the individual performance measures are likely to be optimistic.CONCLUSIONS: Current prognostic models for asthma exacerbations are heterogeneous in methodology, but reported c-statistics suggest a clinically useful model could be created. Studies were consistent in lacking robust validation and in not modelling serial events. Further research is required with respect to incorporating recurrent events, and to externally validate tools in large representative populations to demonstrate the generalizability of published results.

AB - BACKGROUND: Clinical prediction models are widely used to guide medical advice and therapeutic interventions. Asthma is one of the most common chronic diseases globally and is characterised by acute deteriorations. These exacerbations are largely preventable, so there is interest in using clinical prediction models in this area. The objective of this review was to identify studies which have developed such models, determine whether consistent and appropriate methodology was used and whether statistically reliable prognostic models exist.METHODS: We searched online databases MEDLINE (1948 onwards), CINAHL Plus (1937 onwards), The Cochrane Library, Web of Science (1898 onwards) and ClinicalTrials.gov, using index terms relating to asthma and prognosis. Data was extracted and assessment of quality was based on GRADE and an early version of PROBAST (Prediction study Risk of Bias Assessment Tool). A meta-analysis of the discrimination and calibration measures was carried out to determine overall performance across models.RESULTS: Ten unique prognostic models were identified. GRADE identified moderate risk of bias in two of the studies, but more detailed quality assessment via PROBAST highlighted that most models were developed using highly selected and small datasets, incompletely recorded predictors and outcomes, and incomplete methodology. None of the identified models modelled recurrent exacerbations, instead favouring either presence/absence of an event, or time to first or specified event. Preferred methodologies were logistic regression and Cox proportional hazards regression. The overall pooled c-statistic was 0.77 (95% confidence interval 0.73 to 0.80), though individually some models performed no better than chance. The meta-analysis had an I2 value of 99.75% indicating a high amount of heterogeneity between studies. The majority of studies were small and did not include internal or external validation, therefore the individual performance measures are likely to be optimistic.CONCLUSIONS: Current prognostic models for asthma exacerbations are heterogeneous in methodology, but reported c-statistics suggest a clinically useful model could be created. Studies were consistent in lacking robust validation and in not modelling serial events. Further research is required with respect to incorporating recurrent events, and to externally validate tools in large representative populations to demonstrate the generalizability of published results.

U2 - 10.1186/s12874-020-0913-7

DO - 10.1186/s12874-020-0913-7

M3 - Article

VL - 20

SP - 22

JO - BMC Medical Research Methodology

JF - BMC Medical Research Methodology

SN - 1471-2288

IS - 1

ER -