Independent external validation of predictive models for urinary dysfunction following external beam radiotherapy of the prostate: Issues in model development and reporting

N. Yahya, Martin A. Ebert, M. Bulsara, A. Kennedy, David J. Joseph, J.W. Denham

    Research output: Contribution to journalArticle

    4 Citations (Scopus)
    149 Downloads (Pure)

    Abstract

    © 2016 Elsevier Ireland Ltd

    Background and purpose Most predictive models are not sufficiently validated for prospective use. We performed independent external validation of published predictive models for urinary dysfunctions following radiotherapy of the prostate. Materials/methods Multivariable models developed to predict atomised and generalised urinary symptoms, both acute and late, were considered for validation using a dataset representing 754 participants from the TROG 03.04-RADAR trial. Endpoints and features were harmonised to match the predictive models. The overall performance, calibration and discrimination were assessed. Results 14 models from four publications were validated. The discrimination of the predictive models in an independent external validation cohort, measured using the area under the receiver operating characteristic (ROC) curve, ranged from 0.473 to 0.695, generally lower than in internal validation. 4 models had ROC >0.6. Shrinkage was required for all predictive models’ coefficients ranging from -0.309 (prediction probability was inverse to observed proportion) to 0.823. Predictive models which include baseline symptoms as a feature produced the highest discrimination. Two models produced a predicted probability of 0 and 1 for all patients. Conclusions Predictive models vary in performance and transferability illustrating the need for improvements in model development and reporting. Several models showed reasonable potential but efforts should be increased to improve performance. Baseline symptoms should always be considered as potential features for predictive models.

    Original languageEnglish
    Pages (from-to)339-345
    JournalRadiotherapy and Oncology
    Volume120
    Issue number2
    Early online date28 Jun 2016
    DOIs
    Publication statusPublished - Aug 2016

    Fingerprint

    ROC Curve
    Prostate
    Radiotherapy
    Ireland
    Calibration
    Publications
    Datasets

    Cite this

    @article{ab705bfe52314f24b060cacfd5113871,
    title = "Independent external validation of predictive models for urinary dysfunction following external beam radiotherapy of the prostate: Issues in model development and reporting",
    abstract = "{\circledC} 2016 Elsevier Ireland LtdBackground and purpose Most predictive models are not sufficiently validated for prospective use. We performed independent external validation of published predictive models for urinary dysfunctions following radiotherapy of the prostate. Materials/methods Multivariable models developed to predict atomised and generalised urinary symptoms, both acute and late, were considered for validation using a dataset representing 754 participants from the TROG 03.04-RADAR trial. Endpoints and features were harmonised to match the predictive models. The overall performance, calibration and discrimination were assessed. Results 14 models from four publications were validated. The discrimination of the predictive models in an independent external validation cohort, measured using the area under the receiver operating characteristic (ROC) curve, ranged from 0.473 to 0.695, generally lower than in internal validation. 4 models had ROC >0.6. Shrinkage was required for all predictive models’ coefficients ranging from -0.309 (prediction probability was inverse to observed proportion) to 0.823. Predictive models which include baseline symptoms as a feature produced the highest discrimination. Two models produced a predicted probability of 0 and 1 for all patients. Conclusions Predictive models vary in performance and transferability illustrating the need for improvements in model development and reporting. Several models showed reasonable potential but efforts should be increased to improve performance. Baseline symptoms should always be considered as potential features for predictive models.",
    author = "N. Yahya and Ebert, {Martin A.} and M. Bulsara and A. Kennedy and Joseph, {David J.} and J.W. Denham",
    year = "2016",
    month = "8",
    doi = "10.1016/j.radonc.2016.05.010",
    language = "English",
    volume = "120",
    pages = "339--345",
    journal = "Radiotherapy & Oncology",
    issn = "0167-8140",
    publisher = "Elsevier",
    number = "2",

    }

    TY - JOUR

    T1 - Independent external validation of predictive models for urinary dysfunction following external beam radiotherapy of the prostate: Issues in model development and reporting

    AU - Yahya, N.

    AU - Ebert, Martin A.

    AU - Bulsara, M.

    AU - Kennedy, A.

    AU - Joseph, David J.

    AU - Denham, J.W.

    PY - 2016/8

    Y1 - 2016/8

    N2 - © 2016 Elsevier Ireland LtdBackground and purpose Most predictive models are not sufficiently validated for prospective use. We performed independent external validation of published predictive models for urinary dysfunctions following radiotherapy of the prostate. Materials/methods Multivariable models developed to predict atomised and generalised urinary symptoms, both acute and late, were considered for validation using a dataset representing 754 participants from the TROG 03.04-RADAR trial. Endpoints and features were harmonised to match the predictive models. The overall performance, calibration and discrimination were assessed. Results 14 models from four publications were validated. The discrimination of the predictive models in an independent external validation cohort, measured using the area under the receiver operating characteristic (ROC) curve, ranged from 0.473 to 0.695, generally lower than in internal validation. 4 models had ROC >0.6. Shrinkage was required for all predictive models’ coefficients ranging from -0.309 (prediction probability was inverse to observed proportion) to 0.823. Predictive models which include baseline symptoms as a feature produced the highest discrimination. Two models produced a predicted probability of 0 and 1 for all patients. Conclusions Predictive models vary in performance and transferability illustrating the need for improvements in model development and reporting. Several models showed reasonable potential but efforts should be increased to improve performance. Baseline symptoms should always be considered as potential features for predictive models.

    AB - © 2016 Elsevier Ireland LtdBackground and purpose Most predictive models are not sufficiently validated for prospective use. We performed independent external validation of published predictive models for urinary dysfunctions following radiotherapy of the prostate. Materials/methods Multivariable models developed to predict atomised and generalised urinary symptoms, both acute and late, were considered for validation using a dataset representing 754 participants from the TROG 03.04-RADAR trial. Endpoints and features were harmonised to match the predictive models. The overall performance, calibration and discrimination were assessed. Results 14 models from four publications were validated. The discrimination of the predictive models in an independent external validation cohort, measured using the area under the receiver operating characteristic (ROC) curve, ranged from 0.473 to 0.695, generally lower than in internal validation. 4 models had ROC >0.6. Shrinkage was required for all predictive models’ coefficients ranging from -0.309 (prediction probability was inverse to observed proportion) to 0.823. Predictive models which include baseline symptoms as a feature produced the highest discrimination. Two models produced a predicted probability of 0 and 1 for all patients. Conclusions Predictive models vary in performance and transferability illustrating the need for improvements in model development and reporting. Several models showed reasonable potential but efforts should be increased to improve performance. Baseline symptoms should always be considered as potential features for predictive models.

    U2 - 10.1016/j.radonc.2016.05.010

    DO - 10.1016/j.radonc.2016.05.010

    M3 - Article

    VL - 120

    SP - 339

    EP - 345

    JO - Radiotherapy & Oncology

    JF - Radiotherapy & Oncology

    SN - 0167-8140

    IS - 2

    ER -