Machine learning vs addiction therapists: A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication

Martyn Symons, Gerald F.X. Feeney, Marcus R. Gallagher, Ross Mc D. Young, Jason P. Connor

Research output: Contribution to journalArticle

Abstract

Background and objectives: Clinical staff providing addiction treatment predict patient outcome poorly. Prognoses based on linear statistics are rarely replicated. Addiction is a complex non-linear behavior. Incorporating non-linear models, Machine Learning (ML) has successfully predicted treatment outcome when applied in other areas of medicine. Using identical assessment data across the two groups, this study compares the accuracy of ML models versus clinical staff to predict alcohol dependence treatment outcome in behavior therapy using patient data only. Methods: Machine learning models (n = 28) were constructed (‘trained’) using demographic and psychometric assessment data from 780 previously treated patients who had undertaken a 12 week, abstinence-based Cognitive Behavioral Therapy program for alcohol dependence. Independent predictions applying assessment data for an additional 50 consecutive patients were obtained from 10 experienced addiction therapists and the 28 trained ML models. The predictive accuracy of the ML models and the addiction therapists was then compared with further investigation of the 10 best models selected by cross-validated accuracy on the training-set. Variables selected as important for prediction by staff and the most accurate ML model were examined. Results: The most accurate ML model (Fuzzy Unordered Rule Induction Algorithm, 74%) was significantly more accurate than the four least accurate clinical staff (51%–40%). However, the robustness of this finding may be limited by the moderate area under the receiver operator curve (AUC = 0.49). There was no significant difference in mean aggregate predictive accuracy between 10 clinical staff (56.1%) and the 28 best models (58.57%). Addiction therapists favoured demographic and consumption variables compared with the ML model using more questionnaire subscales. Conclusions: The majority of staff and ML models were not more accurate than suggested by chance. However, the best performing prediction models may provide useful adjunctive information to standard clinically available prognostic data to more effectively target treatment approaches in clinical settings.

Original languageEnglish
Pages (from-to)156-162
Number of pages7
JournalJournal of Substance Abuse Treatment
Volume99
DOIs
Publication statusPublished - 1 Apr 2019
Externally publishedYes

Fingerprint

Behavior Therapy
Alcoholism
Demography
Machine Learning
Nonlinear Dynamics
Cognitive Therapy
Psychometrics
Area Under Curve
Medicine
Therapeutics

Cite this

@article{8ddf9a0aa2c24cc9bcce4339b04eca27,
title = "Machine learning vs addiction therapists: A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication",
abstract = "Background and objectives: Clinical staff providing addiction treatment predict patient outcome poorly. Prognoses based on linear statistics are rarely replicated. Addiction is a complex non-linear behavior. Incorporating non-linear models, Machine Learning (ML) has successfully predicted treatment outcome when applied in other areas of medicine. Using identical assessment data across the two groups, this study compares the accuracy of ML models versus clinical staff to predict alcohol dependence treatment outcome in behavior therapy using patient data only. Methods: Machine learning models (n = 28) were constructed (‘trained’) using demographic and psychometric assessment data from 780 previously treated patients who had undertaken a 12 week, abstinence-based Cognitive Behavioral Therapy program for alcohol dependence. Independent predictions applying assessment data for an additional 50 consecutive patients were obtained from 10 experienced addiction therapists and the 28 trained ML models. The predictive accuracy of the ML models and the addiction therapists was then compared with further investigation of the 10 best models selected by cross-validated accuracy on the training-set. Variables selected as important for prediction by staff and the most accurate ML model were examined. Results: The most accurate ML model (Fuzzy Unordered Rule Induction Algorithm, 74{\%}) was significantly more accurate than the four least accurate clinical staff (51{\%}–40{\%}). However, the robustness of this finding may be limited by the moderate area under the receiver operator curve (AUC = 0.49). There was no significant difference in mean aggregate predictive accuracy between 10 clinical staff (56.1{\%}) and the 28 best models (58.57{\%}). Addiction therapists favoured demographic and consumption variables compared with the ML model using more questionnaire subscales. Conclusions: The majority of staff and ML models were not more accurate than suggested by chance. However, the best performing prediction models may provide useful adjunctive information to standard clinically available prognostic data to more effectively target treatment approaches in clinical settings.",
keywords = "Alcohol dependence, Cognitive behavioral therapy, Machine learning, Prediction, Treatment",
author = "Martyn Symons and Feeney, {Gerald F.X.} and Gallagher, {Marcus R.} and Young, {Ross Mc D.} and Connor, {Jason P.}",
year = "2019",
month = "4",
day = "1",
doi = "10.1016/j.jsat.2019.01.020",
language = "English",
volume = "99",
pages = "156--162",
journal = "Journal of Substance Abuse Treatment",
issn = "0740-5472",
publisher = "Elsevier",

}

Machine learning vs addiction therapists : A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication. / Symons, Martyn; Feeney, Gerald F.X.; Gallagher, Marcus R.; Young, Ross Mc D.; Connor, Jason P.

In: Journal of Substance Abuse Treatment, Vol. 99, 01.04.2019, p. 156-162.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Machine learning vs addiction therapists

T2 - A pilot study predicting alcohol dependence treatment outcome from patient data in behavior therapy with adjunctive medication

AU - Symons, Martyn

AU - Feeney, Gerald F.X.

AU - Gallagher, Marcus R.

AU - Young, Ross Mc D.

AU - Connor, Jason P.

PY - 2019/4/1

Y1 - 2019/4/1

N2 - Background and objectives: Clinical staff providing addiction treatment predict patient outcome poorly. Prognoses based on linear statistics are rarely replicated. Addiction is a complex non-linear behavior. Incorporating non-linear models, Machine Learning (ML) has successfully predicted treatment outcome when applied in other areas of medicine. Using identical assessment data across the two groups, this study compares the accuracy of ML models versus clinical staff to predict alcohol dependence treatment outcome in behavior therapy using patient data only. Methods: Machine learning models (n = 28) were constructed (‘trained’) using demographic and psychometric assessment data from 780 previously treated patients who had undertaken a 12 week, abstinence-based Cognitive Behavioral Therapy program for alcohol dependence. Independent predictions applying assessment data for an additional 50 consecutive patients were obtained from 10 experienced addiction therapists and the 28 trained ML models. The predictive accuracy of the ML models and the addiction therapists was then compared with further investigation of the 10 best models selected by cross-validated accuracy on the training-set. Variables selected as important for prediction by staff and the most accurate ML model were examined. Results: The most accurate ML model (Fuzzy Unordered Rule Induction Algorithm, 74%) was significantly more accurate than the four least accurate clinical staff (51%–40%). However, the robustness of this finding may be limited by the moderate area under the receiver operator curve (AUC = 0.49). There was no significant difference in mean aggregate predictive accuracy between 10 clinical staff (56.1%) and the 28 best models (58.57%). Addiction therapists favoured demographic and consumption variables compared with the ML model using more questionnaire subscales. Conclusions: The majority of staff and ML models were not more accurate than suggested by chance. However, the best performing prediction models may provide useful adjunctive information to standard clinically available prognostic data to more effectively target treatment approaches in clinical settings.

AB - Background and objectives: Clinical staff providing addiction treatment predict patient outcome poorly. Prognoses based on linear statistics are rarely replicated. Addiction is a complex non-linear behavior. Incorporating non-linear models, Machine Learning (ML) has successfully predicted treatment outcome when applied in other areas of medicine. Using identical assessment data across the two groups, this study compares the accuracy of ML models versus clinical staff to predict alcohol dependence treatment outcome in behavior therapy using patient data only. Methods: Machine learning models (n = 28) were constructed (‘trained’) using demographic and psychometric assessment data from 780 previously treated patients who had undertaken a 12 week, abstinence-based Cognitive Behavioral Therapy program for alcohol dependence. Independent predictions applying assessment data for an additional 50 consecutive patients were obtained from 10 experienced addiction therapists and the 28 trained ML models. The predictive accuracy of the ML models and the addiction therapists was then compared with further investigation of the 10 best models selected by cross-validated accuracy on the training-set. Variables selected as important for prediction by staff and the most accurate ML model were examined. Results: The most accurate ML model (Fuzzy Unordered Rule Induction Algorithm, 74%) was significantly more accurate than the four least accurate clinical staff (51%–40%). However, the robustness of this finding may be limited by the moderate area under the receiver operator curve (AUC = 0.49). There was no significant difference in mean aggregate predictive accuracy between 10 clinical staff (56.1%) and the 28 best models (58.57%). Addiction therapists favoured demographic and consumption variables compared with the ML model using more questionnaire subscales. Conclusions: The majority of staff and ML models were not more accurate than suggested by chance. However, the best performing prediction models may provide useful adjunctive information to standard clinically available prognostic data to more effectively target treatment approaches in clinical settings.

KW - Alcohol dependence

KW - Cognitive behavioral therapy

KW - Machine learning

KW - Prediction

KW - Treatment

UR - http://www.scopus.com/inward/record.url?scp=85061319305&partnerID=8YFLogxK

U2 - 10.1016/j.jsat.2019.01.020

DO - 10.1016/j.jsat.2019.01.020

M3 - Article

VL - 99

SP - 156

EP - 162

JO - Journal of Substance Abuse Treatment

JF - Journal of Substance Abuse Treatment

SN - 0740-5472

ER -