Multi-observer contouring of male pelvic anatomy: Highly variable agreement across conventional and emerging structures of interest

Dale Roach, Lois C. Holloway, Michael G. Jameson, Jason A. Dowling, Angel Kennedy, Peter B. Greer, Michele Krawiec, Robba Rai, Jim Denham, Jeremiah De Leon, Karen Lim, Megan E. Berry, Rohen T. White, Sean A. Bydder, Hendrick T. Tan, Jeremy D. Croker, Alycea McGrath, John Matthews, Robert J. Smeenk, Martin A. Ebert

Research output: Contribution to journalArticle

Abstract

Introduction: This study quantified inter-observer contouring variations for multiple male pelvic structures, many of which are of emerging relevance for prostate cancer radiotherapy progression and toxicity response studies. Methods: Five prostate cancer patient datasets (CT and T2-weighted MR) were distributed to 13 observers for contouring. CT structures contoured included the clinical target volume (CTV), seminal vesicles, rectum, colon, bowel bag, bladder and peri-rectal space (PRS). MR contours included CTV, trigone, membranous urethra, penile bulb, neurovascular bundle and multiple pelvic floor muscles. Contouring variations were assessed using the intraclass correlation coefficient (ICC), Dice similarity coefficient (DSC), and multiple additional metrics. Results: Clinical target volume (CT and MR), bladder, rectum and PRS contours showed excellent inter-observer agreement (median ICC = 0.97; 0.99; 1.00; 0.95; 0.90, DSC = 0.83 ± 0.05; 0.88 ± 0.05; 0.93 ± 0.03; 0.81 ± 0.07; 0.80 ± 0.06, respectively). Seminal vesicle contours were more variable (ICC = 0.75, DSC = 0.73 ± 0.14), while colon and bowel bag contoured volumes were consistent (ICC = 0.97; 0.97), but displayed poor overlap (DSC = 0.58 ± 0.22; 0.67 ± 0.21). Smaller MR structures showed significant inter-observer variations, with poor overlap for trigone, membranous urethra, penile bulb, and left and right neurovascular bundles (DSC = 0.44 ± 0.22; 0.41 ± 0.21; 0.66 ± 0.21; 0.16 ± 0.17; 0.15 ± 0.15). Pelvic floor muscles recorded moderate to strong inter-observer agreement (ICC = 0.50–0.97), although large outlier variations were observed. Conclusions: Inter-observer contouring variation was significant for multiple pelvic structures contoured on MR.

Original languageEnglish
Pages (from-to)264-271
Number of pages8
JournalJournal of Medical Imaging and Radiation Oncology
Volume63
Issue number2
Early online date4 Jan 2019
DOIs
Publication statusPublished - Apr 2019

Fingerprint

Observer Variation
Anatomy
Pelvic Floor
Seminal Vesicles
Urethra
Rectum
Prostatic Neoplasms
Colon
Urinary Bladder
Muscles
Cone-Beam Computed Tomography
Radiotherapy

Cite this

Roach, Dale ; Holloway, Lois C. ; Jameson, Michael G. ; Dowling, Jason A. ; Kennedy, Angel ; Greer, Peter B. ; Krawiec, Michele ; Rai, Robba ; Denham, Jim ; De Leon, Jeremiah ; Lim, Karen ; Berry, Megan E. ; White, Rohen T. ; Bydder, Sean A. ; Tan, Hendrick T. ; Croker, Jeremy D. ; McGrath, Alycea ; Matthews, John ; Smeenk, Robert J. ; Ebert, Martin A. / Multi-observer contouring of male pelvic anatomy : Highly variable agreement across conventional and emerging structures of interest. In: Journal of Medical Imaging and Radiation Oncology. 2019 ; Vol. 63, No. 2. pp. 264-271.
@article{71b144be510747f3a6dc873401acced7,
title = "Multi-observer contouring of male pelvic anatomy: Highly variable agreement across conventional and emerging structures of interest",
abstract = "Introduction: This study quantified inter-observer contouring variations for multiple male pelvic structures, many of which are of emerging relevance for prostate cancer radiotherapy progression and toxicity response studies. Methods: Five prostate cancer patient datasets (CT and T2-weighted MR) were distributed to 13 observers for contouring. CT structures contoured included the clinical target volume (CTV), seminal vesicles, rectum, colon, bowel bag, bladder and peri-rectal space (PRS). MR contours included CTV, trigone, membranous urethra, penile bulb, neurovascular bundle and multiple pelvic floor muscles. Contouring variations were assessed using the intraclass correlation coefficient (ICC), Dice similarity coefficient (DSC), and multiple additional metrics. Results: Clinical target volume (CT and MR), bladder, rectum and PRS contours showed excellent inter-observer agreement (median ICC = 0.97; 0.99; 1.00; 0.95; 0.90, DSC = 0.83 ± 0.05; 0.88 ± 0.05; 0.93 ± 0.03; 0.81 ± 0.07; 0.80 ± 0.06, respectively). Seminal vesicle contours were more variable (ICC = 0.75, DSC = 0.73 ± 0.14), while colon and bowel bag contoured volumes were consistent (ICC = 0.97; 0.97), but displayed poor overlap (DSC = 0.58 ± 0.22; 0.67 ± 0.21). Smaller MR structures showed significant inter-observer variations, with poor overlap for trigone, membranous urethra, penile bulb, and left and right neurovascular bundles (DSC = 0.44 ± 0.22; 0.41 ± 0.21; 0.66 ± 0.21; 0.16 ± 0.17; 0.15 ± 0.15). Pelvic floor muscles recorded moderate to strong inter-observer agreement (ICC = 0.50–0.97), although large outlier variations were observed. Conclusions: Inter-observer contouring variation was significant for multiple pelvic structures contoured on MR.",
keywords = "contouring, delineation, inter/intra-observer variability, prostate",
author = "Dale Roach and Holloway, {Lois C.} and Jameson, {Michael G.} and Dowling, {Jason A.} and Angel Kennedy and Greer, {Peter B.} and Michele Krawiec and Robba Rai and Jim Denham and {De Leon}, Jeremiah and Karen Lim and Berry, {Megan E.} and White, {Rohen T.} and Bydder, {Sean A.} and Tan, {Hendrick T.} and Croker, {Jeremy D.} and Alycea McGrath and John Matthews and Smeenk, {Robert J.} and Ebert, {Martin A.}",
year = "2019",
month = "4",
doi = "10.1111/1754-9485.12844",
language = "English",
volume = "63",
pages = "264--271",
journal = "Journal of Medical Imaging and Radiation Oncology",
issn = "0004-8461",
publisher = "John Wiley & Sons",
number = "2",

}

Roach, D, Holloway, LC, Jameson, MG, Dowling, JA, Kennedy, A, Greer, PB, Krawiec, M, Rai, R, Denham, J, De Leon, J, Lim, K, Berry, ME, White, RT, Bydder, SA, Tan, HT, Croker, JD, McGrath, A, Matthews, J, Smeenk, RJ & Ebert, MA 2019, 'Multi-observer contouring of male pelvic anatomy: Highly variable agreement across conventional and emerging structures of interest' Journal of Medical Imaging and Radiation Oncology, vol. 63, no. 2, pp. 264-271. https://doi.org/10.1111/1754-9485.12844

Multi-observer contouring of male pelvic anatomy : Highly variable agreement across conventional and emerging structures of interest. / Roach, Dale; Holloway, Lois C.; Jameson, Michael G.; Dowling, Jason A.; Kennedy, Angel; Greer, Peter B.; Krawiec, Michele; Rai, Robba; Denham, Jim; De Leon, Jeremiah; Lim, Karen; Berry, Megan E.; White, Rohen T.; Bydder, Sean A.; Tan, Hendrick T.; Croker, Jeremy D.; McGrath, Alycea; Matthews, John; Smeenk, Robert J.; Ebert, Martin A.

In: Journal of Medical Imaging and Radiation Oncology, Vol. 63, No. 2, 04.2019, p. 264-271.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Multi-observer contouring of male pelvic anatomy

T2 - Highly variable agreement across conventional and emerging structures of interest

AU - Roach, Dale

AU - Holloway, Lois C.

AU - Jameson, Michael G.

AU - Dowling, Jason A.

AU - Kennedy, Angel

AU - Greer, Peter B.

AU - Krawiec, Michele

AU - Rai, Robba

AU - Denham, Jim

AU - De Leon, Jeremiah

AU - Lim, Karen

AU - Berry, Megan E.

AU - White, Rohen T.

AU - Bydder, Sean A.

AU - Tan, Hendrick T.

AU - Croker, Jeremy D.

AU - McGrath, Alycea

AU - Matthews, John

AU - Smeenk, Robert J.

AU - Ebert, Martin A.

PY - 2019/4

Y1 - 2019/4

N2 - Introduction: This study quantified inter-observer contouring variations for multiple male pelvic structures, many of which are of emerging relevance for prostate cancer radiotherapy progression and toxicity response studies. Methods: Five prostate cancer patient datasets (CT and T2-weighted MR) were distributed to 13 observers for contouring. CT structures contoured included the clinical target volume (CTV), seminal vesicles, rectum, colon, bowel bag, bladder and peri-rectal space (PRS). MR contours included CTV, trigone, membranous urethra, penile bulb, neurovascular bundle and multiple pelvic floor muscles. Contouring variations were assessed using the intraclass correlation coefficient (ICC), Dice similarity coefficient (DSC), and multiple additional metrics. Results: Clinical target volume (CT and MR), bladder, rectum and PRS contours showed excellent inter-observer agreement (median ICC = 0.97; 0.99; 1.00; 0.95; 0.90, DSC = 0.83 ± 0.05; 0.88 ± 0.05; 0.93 ± 0.03; 0.81 ± 0.07; 0.80 ± 0.06, respectively). Seminal vesicle contours were more variable (ICC = 0.75, DSC = 0.73 ± 0.14), while colon and bowel bag contoured volumes were consistent (ICC = 0.97; 0.97), but displayed poor overlap (DSC = 0.58 ± 0.22; 0.67 ± 0.21). Smaller MR structures showed significant inter-observer variations, with poor overlap for trigone, membranous urethra, penile bulb, and left and right neurovascular bundles (DSC = 0.44 ± 0.22; 0.41 ± 0.21; 0.66 ± 0.21; 0.16 ± 0.17; 0.15 ± 0.15). Pelvic floor muscles recorded moderate to strong inter-observer agreement (ICC = 0.50–0.97), although large outlier variations were observed. Conclusions: Inter-observer contouring variation was significant for multiple pelvic structures contoured on MR.

AB - Introduction: This study quantified inter-observer contouring variations for multiple male pelvic structures, many of which are of emerging relevance for prostate cancer radiotherapy progression and toxicity response studies. Methods: Five prostate cancer patient datasets (CT and T2-weighted MR) were distributed to 13 observers for contouring. CT structures contoured included the clinical target volume (CTV), seminal vesicles, rectum, colon, bowel bag, bladder and peri-rectal space (PRS). MR contours included CTV, trigone, membranous urethra, penile bulb, neurovascular bundle and multiple pelvic floor muscles. Contouring variations were assessed using the intraclass correlation coefficient (ICC), Dice similarity coefficient (DSC), and multiple additional metrics. Results: Clinical target volume (CT and MR), bladder, rectum and PRS contours showed excellent inter-observer agreement (median ICC = 0.97; 0.99; 1.00; 0.95; 0.90, DSC = 0.83 ± 0.05; 0.88 ± 0.05; 0.93 ± 0.03; 0.81 ± 0.07; 0.80 ± 0.06, respectively). Seminal vesicle contours were more variable (ICC = 0.75, DSC = 0.73 ± 0.14), while colon and bowel bag contoured volumes were consistent (ICC = 0.97; 0.97), but displayed poor overlap (DSC = 0.58 ± 0.22; 0.67 ± 0.21). Smaller MR structures showed significant inter-observer variations, with poor overlap for trigone, membranous urethra, penile bulb, and left and right neurovascular bundles (DSC = 0.44 ± 0.22; 0.41 ± 0.21; 0.66 ± 0.21; 0.16 ± 0.17; 0.15 ± 0.15). Pelvic floor muscles recorded moderate to strong inter-observer agreement (ICC = 0.50–0.97), although large outlier variations were observed. Conclusions: Inter-observer contouring variation was significant for multiple pelvic structures contoured on MR.

KW - contouring

KW - delineation

KW - inter/intra-observer variability

KW - prostate

UR - http://www.scopus.com/inward/record.url?scp=85059553176&partnerID=8YFLogxK

U2 - 10.1111/1754-9485.12844

DO - 10.1111/1754-9485.12844

M3 - Article

VL - 63

SP - 264

EP - 271

JO - Journal of Medical Imaging and Radiation Oncology

JF - Journal of Medical Imaging and Radiation Oncology

SN - 0004-8461

IS - 2

ER -