Hostname: page-component-78c5997874-dh8gc Total loading time: 0 Render date: 2024-11-13T09:24:17.783Z Has data issue: false hasContentIssue false

Evaluation of the impact of teaching on delineation variation during a virtual stereotactic ablative radiotherapy contouring workshop

Published online by Cambridge University Press:  10 December 2021

Finbar Slevin*
Affiliation:
Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, Leeds, UK University of Leeds, Leeds, UK
Romélie Rieu
Affiliation:
The Royal Marsden NHS Foundation Trust, London, UK
Matthew Beasley
Affiliation:
Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, Leeds, UK
Richard Speight
Affiliation:
Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, Leeds, UK
Katharine Aitken
Affiliation:
The Royal Marsden NHS Foundation Trust, London, UK The Institute of Cancer Research, London, UK
James Good
Affiliation:
University Hospitals Birmingham NHS Foundation Trust, Birmingham, UK
Fiona McDonald
Affiliation:
The Royal Marsden NHS Foundation Trust, London, UK The Institute of Cancer Research, London, UK
Thomas Rackley
Affiliation:
Velindre Cancer Centre, Cardiff, UK
Ganesh Radhakrishna
Affiliation:
The Christie NHS Foundation Trust, Manchester, UK
Anoop Haridass
Affiliation:
The Clatterbridge Cancer Centre NHS Foundation Trust, Liverpool, UK
Louise J. Murray
Affiliation:
Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, Leeds, UK University of Leeds, Leeds, UK
Alison C. Tree
Affiliation:
The Royal Marsden NHS Foundation Trust, London, UK The Institute of Cancer Research, London, UK
Ann M. Henry
Affiliation:
Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, Leeds, UK University of Leeds, Leeds, UK
*
Author for correspondence: Finbar Slevin, Leeds Cancer Centre, Leeds Teaching Hospitals NHS Trust, Beckett Street, Leeds LS9 7TF, UK. E-mail: finbarslevin@nhs.net
Rights & Permissions [Opens in a new window]

Abstract

Introduction:

Variation in delineation of target volumes/organs at risk (OARs) is well recognised in radiotherapy and may be reduced by several methods including teaching. We evaluated the impact of teaching on contouring variation for thoracic/pelvic stereotactic ablative radiotherapy (SABR) during a virtual contouring workshop.

Materials and methods:

Target volume/OAR contours produced by workshop participants for three cases were evaluated against reference contours using DICE similarity coefficient (DSC) and line domain error (LDE) metrics. Pre- and post-workshop DSC results were compared using Wilcoxon signed ranks test to determine the impact of teaching during the workshop.

Results:

Of 50 workshop participants, paired pre- and post-workshop contours were available for 21 (42%), 20 (40%) and 22 (44%) participants for primary lung cancer, pelvic bone metastasis and pelvic node metastasis cases, respectively. Statistically significant improvements post-workshop in median DSC and LDE results were observed for 6 (50%) and 7 (58%) of 12 structures, respectively, although the magnitude of DSC/LDE improvement was modest in most cases. An increase in median DSC post-workshop ≥0·05 was only observed for GTVbone, IGTVlung and SacralPlex, and reduction in median LDE > 1 mm was only observed for GTVbone, CTVbone and SacralPlex. Post-workshop, median DSC values were >0·7 for 75% of structures. For 92% of the structures, post-workshop contours were considered to be acceptable or within acceptable variation following review by the workshop faculty.

Conclusions:

This study has demonstrated that virtual SABR contouring training is feasible and was associated with some improvements in contouring variation for multiple target volumes/OARs.

Type
Original Article
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Introduction

Delineation of target volumes and organs at risk (OARs) is a key component of radiotherapy planning, but inter/intra-observer variation in contouring is well recognised and is a significant source of error within treatment workflows. Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Min, Jameson and Holloway2 Potential reasons for this variation may include the influence of disease site experience/expertise and skills in cross-sectional image interpretation. Reference Vinod, Min, Jameson and Holloway2Reference Njeh4 The consequences of contouring variation may be profound; incorrect delineation is associated with inferior survival outcomes in clinical trials. Reference Peters, O’Sullivan and Giralt5,Reference Weber, Tomsej, Melidis and Hurkmans6

Various methods exist to minimise contouring variation including delineation protocols, atlases, auto-contours, peer review and teaching. Reference Vinod, Min, Jameson and Holloway2,Reference Gwynne, Gilson, Dickson, McAleer and Radhakrishna3,7,Reference Weiss and Hess8 Radiotherapy is a craft specialty, necessitating the acquisition and refinement of contouring skills during clinical practice. Reference Walls, Hanna and McAleer9 To mitigate the potential impact on training of the reduction in junior doctor working hours, smarter and more efficient methods of delivering training are required. Reference Datta and Davies10 Dedicated contouring workshops may be a valuable source of experiential learning especially concerning new radiotherapy techniques. Reference De Bari, Dahele, Palmu, Kaylor, Schiappacasse and Guckenberger11Reference Khoo, Schick and Plank13

Following changes to the commissioning of stereotactic ablative radiotherapy (SABR) in the UK, the Royal College of Radiologists (RCR) and UK SABR Consortium organised a workshop which focused on SABR contouring for lung cancer and bone and nodal oligometastatic disease. 14 The aim of the workshop was to share expertise and experience in SABR techniques and improve participants’ contouring skills. Given the COVID-19 pandemic, the workshop took place in virtual format. In this study, we evaluated the impact of teaching during the workshop on contouring variation for multiple target volumes/OARs in the thorax and pelvis.

Methods and Materials

Format of the workshop

The workshop took place on 19 and 22 October 2020; each session lasted two hours in duration and was delivered using Adobe® Connect (Adobe, San Jose, CA, USA). Participants were UK-based consultants in clinical oncology, and the workshop was aimed at those without prior expertise in SABR, although relative baseline experience was not assessed/recorded prior to the workshop. Participants were asked to delineate target volumes/OARs for three cases prior to the workshop using the web-based platform EduCase™ (RadOnc eLearning Centre, Inc., Fremont, CA, USA). A video tutorial was provided, which explained how to use EduCase™.

The target volumes/OARs for the three cases were as follows:

Right upper lobe primary lung cancer

  • IGTVlung (internal target volume)

  • BrachialPlex

  • BronchusProx

  • Oesophagus

  • Spinal_Canal

Left pelvic bone metastasis secondary to breast cancer

  • GTVbone (gross tumour volume)

  • CTVbone (clinical target volume)

  • Femur_Head_Left

  • Rectum

Right common iliac lymph node secondary to prostate cancer

  • GTVnode

  • Bowel_Large

  • SacralPlex

Each case was accompanied by a clinical vignette (history, diagnosis, investigations and intended treatment) and instructions detailing which structures were to be delineated and on which axial computed tomography (CT) slices. CT axial slice thickness was 3 mm for the lung cancer case and 1 mm for the bone/node cases. Image co-registration performed in EduCase™ between CT and positron emission tomography-computed tomography (PET-CT) was available for all cases, magnetic resonance imaging (MRI) was available for the nodal and pelvic bone cases, and 4DCT was available for the primary lung case. For the lung cancer case, IGTVlung could be defined on the maximum intensity projection (MIP) scan with reference to the average intensity projection, 0% and 30% respiratory phases.

Pre-workshop contours, anonymised to clinician, were reviewed across the two workshops, and teaching was provided for each case including demonstration of a reference contour produced by the workshop faculty of UK consultant clinical oncologists with a combined total of consultant experience of approximately 50 years. Relevant published contouring guidance and atlases were identified during both sessions. Teaching included clinical cases to illustrate the general principles of patient selection, planning and treatment delivery of SABR for primary lung cancer and oligometastatic disease and a dedicated session for target volume/OAR contouring.

Following each workshop, participants were invited to review/adjust their contours based on the teaching. Final attempts could be submitted up to two weeks after the second workshop session, although no further contours eligible for inclusion in the study were submitted more than 3 days after the final workshop session. The faculty provided individual written feedback to participants on their post-workshop contours (this information was not provided for pre-workshop contours).

Participants were asked to provide feedback for individual speaker sessions and the overall workshop experience using a 5-point Likert scale and free text responses.

Analysis of participant contours

Each participant’s contours were compared against a reference contour, which was produced by the clinician who led each case discussion during the workshop and peer-reviewed by a second faculty member. For each structure, the specific axial CT slices to be contoured was specified; these were non-contiguous, and therefore, a volume was not obtained. Some participants had delineated contours on slices other than those specified in the case. Therefore, to ensure a fair comparison for all participants, only contours on those pre-specified slices were considered. Participants with only one set of contours (e.g., only pre-workshop contours) were excluded. Participants with two sets of submitted contours but where no changes were made to the post-workshop contours were included.

EduCase™ provides 2-dimensional (i.e., area) Dice similarity coefficient (DSC) and line domain error (LDE) values for individual slices for participant contours compared with the reference contour. DSC is an overlap measure, which measures the intersection of two contours relative to the union and ranges from 0 (zero overlap) to 1 (perfect overlap). Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vaassen, Hazelaar and Vaniqui15,Reference Vinod, Jameson, Min and Holloway16 DSC can be calculated by the following formula: Reference Dice17, Reference Rivin del Campo, Rivera and Martínez-Paredes18

$$\matrix{ {{\rm{DSC}}{\mkern 1mu} = {\mkern 1mu} {\mkern 1mu} {\rm{2}} \times \left( {{\rm{Are}}{{\rm{a}}_{{\rm{reference}}}} \cap {\rm{Are}}{{\rm{a}}_{{\rm{participant}}}}} \right)} \cr {/\left( {{\rm{Are}}{{\rm{a}}_{{\rm{reference}}}} + {\rm{Are}}{{\rm{a}}_{{\rm{participant}}}}} \right)} \cr } $$

where Areareference∩Areaparticipant is the intersecting overlap of the two areas and Areareference+Areaparticipant is the union of the two areas.

LDE is a distance metric within EduCase™, which measures the average absolute Euclidean distance in millimetres between corresponding points on the reference and participant contours.

Since each structure was not a volume but instead a series of individual slices, a summary measure per structure for each participant was produced. The median value of DSC/LDE for each of these slices was calculated for each of the structures contoured by each participant. These median structure DSC/LDE values for participants with both pre- and post-workshop contours were exported into IBM-SPSS Statistics for Windows version 26 (IBM Corp., Armonk, NY, USA). Each of the included contours was reviewed by two of the authors (Finbar Slevin and Romélie Rieu) to identify potential reasons for low DSC/high LDE values.

Following the workshop, the faculty reviewed participants’ post-workshop contours and provided a score (acceptable, within acceptable variation or unacceptable) and written feedback.

Statistical considerations

The median DSC/LDE and interquartile range (IQR) are presented as summary statistics for all the participants’ median structure DSC/LDE values pre- and post-workshop, since a normal distribution of data could not be assumed and also to minimise the influence of outlying values. Box and whisker plots were produced by importing data into R 3·6·1 (R Core Team, R Foundation for Statistical Computing, Vienna, Austria) using the ggplot2 library. Reference Wickham19 A statistical comparison of the median DSC/LDE for each participant’s structures pre- and post-workshop was undertaken using the Wilcoxon signed ranks test in SPSS, since this was paired data. A p value of <0·05 was taken to indicate a statistically significant difference.

Results

Fifty participants registered for the workshop and 43 submitted at least one set of contours for each of the cases. Of these 43 participants, 21 (49%), 20 (47%) and 22 (51%) participants produced pre- and post-workshop contours for the lung cancer, pelvic bone metastasis and pelvic node metastasis cases, respectively. A summary of the DSC/LDE values pre- and post-workshop and results of statistical comparisons are shown in Table 1. The spread of the median DSC/LDE values for each structure across all of the participants is illustrated in Figure 1.

Table 1. Summary of median Dice similarity coefficient (DSC) and line domain error (LDE) measurements before/after teaching for each structure

LDE, line domain error; DSC, Dice similarity coefficient.

* Indicates statistically significant result.

Figure 1. Box and whisker plots for the target volume/organs at risk structures for the lung cancer, pelvic bone metastasis and common iliac nodal metastasis cases. The top row represents Dice similarity coefficient (DSC) results, and the bottom row represents line domain error (LDE) results. The box represents the middle 50% of the data and is bounded by the upper (Q3) and lower (Q1) quartiles, and the horizontal line indicates the median value. The upper whiskers represent Q3 + 1.5 × IQR and the lower whiskers represent Q1–1.5 × IQR. Any outliers beyond these ranges are indicated as dots.

Statistically significant improvements in DSC post-workshop were observed for each structure except for IGTVlung, Spinal_Canal, Oesophagus and BronchusProx. Only BronchusProx was associated with a worsening in median DSC post-workshop, but this difference was not statistically significant. The magnitude of increase in DSC post-workshop was often small; only GTVbone (0·08), IGTVlung (0·05) and SacralPlex (0·37) were associated with a ≥ 0·05 increase in median DSC. A median value of DSC > 0·7 and >0·8 post-workshop was observed for nine (75%) and five (42%) of the 12 structures, respectively; no median DSC value was >0·9.

Statistically significant improvements in LDE post-workshop were observed for each structure except for BronchusProx, Oesophagus and Spinal_Canal. Similar to DSC results, BronchusProx was associated with a worsening in median LDE post-workshop although this difference was not statistically significant. Again, the magnitude of improvement was often small; only GTVbone (1·7 mm), CTVbone (1·2 mm) and SacralPlex (42 mm) were associated with >1 mm reduction in median LDE post-workshop.

Some post-workshop contours were unchanged from pre-workshop: GTVnode (5 participants, 24%), Bowel_Large (10 participants, 46%), GTVbone (2 participants, 10%), CTVbone (2 participants, 10%), Rectum (8 participants, 40%), IGTVlung (8 participants, 40%), Spinal_Canal (11 participants, 52%), Oesophagus (7 participants, 33%), BronchusProx (7 participants, 33%). When the data were re-analysed without these unchanged structures, no significant differences were observed.

Regarding BrachialPlex, the case instructions did not specify that only the ipsilateral structure was to be delineated and some participants contoured bilateral structures. Similarly for Femur_Head_Left, the femoral head (i.e., excluding the femoral neck) was to be delineated but several participants delineated both the femoral head and neck and/or produced bilateral structures. Therefore, these two structures were omitted from statistical comparisons.

Regarding post-workshop contours, a summary of the feedback provided to participants is shown in Table 2. Ninety-two per cent of post-workshop contours were considered to be acceptable or within acceptable variation.

Table 2. Summary of qualitative feedback on participants’ post-workshop contours

Eighty-four per cent of participants provided feedback on the workshop; of these, feedback regarding the overall workshop experience and each of the individual speakers was considered to be ‘good’ or ‘very good’ in 82% and 99% of responses, respectively. Ten per cent of feedback concerned technical issues during the workshop (e.g., sound quality).

Discussion

This study has evaluated the impact of teaching during a SABR contouring workshop for a relatively large number of participants and multiple target volume/OARs in the thorax and pelvis. The positive feedback provided by participants about the workshop suggests that it is feasible to deliver contouring teaching in a virtual capacity. This is important since, accelerated by the changes adopted during the COVID-19 pandemic, it is likely that medical training and teaching will increasingly be delivered using virtual methods. Another important message is that it is possible to continue to provide contouring training during the COVID-19 pandemic when in-person meetings are restricted. Virtual training also has the potential to reach a larger audience, without geographical restrictions, than could typically be achieved in person. We demonstrated that median DSC/LDE values for participants who completed pre- and post-workshop contours for most of the target volume/OARs were similar to the reference contour, with DSC > 0·7 for 75% of structures and LDE < 5 mm for 83% of structures. While statistically significant improvements post-workshop in DSC and LDE were observed for 50% and 58% of structures, respectively, the magnitude of improvement was small in most cases and the clinical significance of such modest improvements remains uncertain.

Although multiple studies on the effect of teaching on contouring variation have been reported, several factors make a direct comparison between these and our study challenging. Reference Vinod, Jameson, Min and Holloway16 Heterogeneity exists between studies concerning the numbers of participants, types of teaching, the structures for which contouring variation is evaluated and the types of metrics used to evaluate this variation and the use of statistical tests. Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Min, Jameson and Holloway2,Reference Vinod, Jameson, Min and Holloway16 However, systematic reviews of such studies have demonstrated that an improvement in contouring variation through teaching can be achieved. Reference Vinod, Min, Jameson and Holloway2,Reference Cacicedo, Navarro-Martin, Gonzalez-Larragan, De Bari, Salem and Dahele20 We did not observe a large increase in DSC/reduction in LDE post-workshop, and a number of limitations of our work may explain this. While participants were asked to review their pre-workshop contours after teaching and produce a post-workshop submission, only approximately half of participants did so which reduced the number for which an analysis of teaching impact could be performed. Furthermore, even for those who did re-submit a second set of contours in some cases no changes were made. Possible reasons for this could include satisfaction with pre-workshop contours, insufficient time to re-contour every structure and a lack of hands-on time during the workshop to practise/fully compare contours with the reference contour. The latter point may be particularly relevant since it has been previously suggested that active participation is more likely to improve learning during contouring workshops. Reference Eriksen, Salembier and Rivera21 Insufficient provision of practical experience was raised as a potential explanation for failure to observe improved contouring post-teaching in a previous study of a head and neck contouring programme, although there may be time/resource challenges to effectively deliver this especially for larger audiences and during the COVID-19 pandemic where face-to-face meetings are restricted. Reference D’Souza, Jaswal and Chan22 Residual differences in knowledge/ability between participants despite teaching were also suggested as a possible reason why significant improvements in prostate/rectal contouring were not observed in a previous evaluation of the impact of teaching. Reference Szumacher, Harnett and Warner23

Low DSC/high LDE values for certain structures in our study could be related to interpretation of the case instructions, especially for BrachialPlex and Femur_Head_Left. The latter structure was also only to be delineated on a single axial CT slice at the very inferior aspect of the femoral head. Different methods for contouring BrachialPlex exist, and there remains variation in practice. Reference Hall, Guiou and Lee2426 Given the high dose per fraction used with SABR and variable reliance on MRI across different treatment centres, the UK SABR Consortium Guidance recommends contouring the subclavian/axillary vessels as a surrogate for BrachialPlex. 26 National consensus is needed, and future iterations of the recently published OAR harmonisation guidance will support this. Reference Mir, Kelly and Xiao25 For SacralPlex, some participants delineated the visible nerve using the MRI while others delineated a larger surrogate structure using the CT. Both of these may be legitimate approaches, although contouring as per the Yi et al guidance does not rely on expert MRI interpretation of nerve position and may therefore be simpler for those learning. Reference Yi, Mak and Yang27 However, unfamiliarity with the contouring of certain OARs might have contributed to low DSC/high LDE results. Although not statistically significant, the median DSC/LDE for BrochusProx appeared slightly worse post-workshop. The reason for this was not clearly apparent, and the magnitude of difference was small, but it could possibly be related to delineation uncertainties regarding the distal extent of the lobar bronchi. A visual guide to delineation of BrachialPlex, BronchusProx and SacralPlex is illustrated in Figure 2 while recommended contouring guidance/atlases are collated in Table 3. Reference Mir, Kelly and Xiao25Reference Wright, Yom and Awan30

Figure 2. Visual guide to delineation of SacralPlex, BrachialPlex and BronchusProx. In (a–c), SacralPlex; iliacus muscle (green), L5 vertebral body (dark blue), obturator internus muscle (orange), psoas muscle (light blue), SacralPlex (purple), vessels (yellow) are shown. SacralPlex is contoured using a 5 mm diameter roller ball. In 2A, superior border of SacralPlex is shown at L4/5 vertebral interspace; SacralPlex is shown bordered by (ilio)psoas muscle anteriorly and vertebral body posteriorly. In 2B, at the sacroiliac foramen, SacralPlex is shown bordered by vessels anteriorly, iliacus muscle laterally and sacral ala posteriorly. In 2C, inferior border of SacralPlex is shown at the level of the superior femoral neck bordered by obturator internus muscle anteriorly, gluteus maximus muscle posteriorly. In (d–f), BrachialPlex contoured as suggested by UK SABR Consortium Guidelines 26 ; anterior scalene muscle (orange), BrachialPlex (light blue), common carotid artery (red), internal jugular vein (posterior scalene muscle (brown), subclavian artery (pink) and subclavian vein (dark blue) are shown. Intravenous contrast is helpful, and BrachialPlex is contoured using a 5 mm diameter roller ball. 2D shows a proximal slice: the superior border of BrachialPlex is at the bifurcation of the brachiocephalic trunk into the jugular/subclavian veins (or carotid/subclavian arteries). In 2E, a middle section of BrachialPlex is shown; the plexus sits between anterior and middle scalene muscles. In 2F, the neurovascular complex including the subclavian and axillary vessels is contoured as a surrogate for the brachial plexus, ending after the neurovascular structures cross the second rib. In (g–i), BronchusProx; BronchusProx (purple) is shown. In 2G, superior border of BronchusProx is the distal 2 cm of trachea including carina. In 2·2H, the mid-section of BronchusProx is shown and includes right/left upper lobe bronchi, bronchus intermedius, right middle lobe bronchus, lingular bronchus and right/left lower lobe bronchi. In 2I, contouring of lobar bronchi stops immediately at the site of a segmental bifurcation.

Table 3. A summary of resources to support target volume/organ at risk delineation

The metric thresholds that correlate to a minimum expected standard of contouring are uncertain but it has been previously suggested that DSC > 0·7 indicates a good level of agreement. Reference Vinod, Min, Jameson and Holloway2 However, previous studies have demonstrated discrepancies between contours considered to be acceptable based on expert review and the results of overlap measure comparisons. Reference Duke, Tan and Jensen31 In this study, 92% of the post-workshop contours were considered to be acceptable/within acceptable variation while 75% of structures had a DSC > 0·7. A range of comparison metrics exists, and each provides different information about the relationship between two contours and each has its limitations. Reference Vinod, Jameson, Min and Holloway16 A summary of commonly used metrics for contour comparison is shown in Supplementary Table; it is unclear which is the optimum metric to use. Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Min, Jameson and Holloway2,Reference Vinod, Jameson, Min and Holloway16,Reference Rivin del Campo, Rivera and Martínez-Paredes18,Reference Ackerly, Andrews, Ball, Guerrieri, Healy and Williams32Reference Taha and Hanbury37 For this reason, it has previously been recommended that multiple metrics ideally be reported including measures of volume, overlap and distance. Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Jameson, Min and Holloway16 In this study, we only reported DSC and LDE since we did not have volumetric contouring data. It should be emphasised that DSC may provide less reliable results when applied to very small contours and it may lack discrimination for very large volumes. Reference Rivin del Campo, Rivera and Martínez-Paredes18 However, it does provide some insight into both the volumetric and spatial relationship between two contours and it is frequently reported in contouring studies. Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference De Bari, Dahele, Palmu, Kaylor, Schiappacasse and Guckenberger11

Quantitative concordance in target volume/OAR delineation does not necessarily equate to a clinically acceptable contour; incorrect delineation of even a small proportion of a target volume or an OAR could have profound clinical consequences, especially for SABR where tight margins, steep dose gradients and ablative doses are used. Reference Vinod, Min, Jameson and Holloway2,Reference Murray, Franks and Hanna38,Reference Trakul, Koong and Chang39 This risk means that quantitative metrics should ideally be accompanied by visual review of contours and provision of qualitative feedback, analogous to the peer review process used in clinical practice and recommended by the RCR. 7 This approach is used in clinical trials for pre-trial approval for participation or on-trial individual case evaluation. Qualitative feedback can be provided detailing acceptable/unacceptable variation from the protocol, and a similar process was used in this study for feedback on post-workshop contours. Reference Gwynne, Gilson, Dickson, McAleer and Radhakrishna3,Reference Cox, Cleves, Clementel, Miles, Staffurth and Gwynne40Reference Gwynne, Jones and Maggs42 However, this approach may be time-consuming and an efficient/reliable method of assessment which can identify clinically relevant discrepancies is needed. Reference Jameson, Holloway, Vial, Vinod and Metcalfe1,Reference Vinod, Jameson, Min and Holloway16,Reference Duke, Tan and Jensen31

The practice of clinical oncology takes place against an increasingly complex backdrop of developments in imaging and novel methods of treatment delivery. Alongside ever-increasing pressures in healthcare services, considerable challenges exist for training and continuous professional development of trainees and consultants, respectively. Reference Walls, Hanna and McAleer9 Formal training initiatives have been established to deliver the acquisition, and maintenance, of contouring competences in an attempt to improve target volume/OAR delineation beyond what could be achieved by a single workshop in isolation. The Fellowship in Anatomic deLineation and CONtouring (FALCON) programme is a European Society of Radiation Oncology (ESTRO) initiative that provides access to e-learning contouring resources in addition to its use within dedicated workshops. Reference Rivin del Campo, Rivera and Martínez-Paredes18,Reference Eriksen, Salembier and Rivera21 The RCR ARENA and Clinical Oncology Planning Project (COPP) are some example of initiatives to increase access to expert/peer-led structured outlining training to promote consistency in target volume and OAR outlining, and facilitate robust assessment of outlining practice for all grades of Clinical Oncologists. Reference Evans, Radhakrishna and Gilson43

This study has a number of additional limitations. The workshop was limited in its time/level of interactivity because of restrictions imposed during the pandemic, and this could have impacted on the educational experience/DSC and LDE results that we observed (although participant feedback for the workshop remained positive). The same cases were used for both pre- and post-workshop contouring; while this enabled the analysis of paired data, it meant that post-workshop contour performance could have been influenced by familiarity with the case and thus extrapolation of similar levels of performance to other cases would not necessarily be guaranteed. We did not stratify by prior experience when undertaking our analysis; this was because this information was not available to the authors but it could have influenced the results that were obtained. The workshop was aimed at those without prior experience in SABR but experience with OAR delineation would have varied depending on disease site expertise. We also did not evaluate longer-term maintenance of contouring competences by provision of further cases for contouring as part of this workshop, although response rates for such interventions for a single workshop in isolation may be limited. Reference Cacicedo, Navarro-Martin, Gonzalez-Larragan, De Bari, Salem and Dahele20 It might be expected that initial educational gains could progressively negate over time, meaning that ongoing evaluation of performance during training programmes/as part of consultant continuous professional development will be required. Finally, feedback on post-workshop contours was only available for approximately half of participants included in our analyses; this affected the conclusions that can be drawn regarding the qualitative feedback but does reflect the challenge of providing such information in a timely manner.

When planning a contouring workshop, the following considerations may be relevant based on prior recommendations/the authors’ experience Reference Gwynne, Gilson, Dickson, McAleer and Radhakrishna3,Reference Vinod, Jameson, Min and Holloway16,Reference Cacicedo, Navarro-Martin, Gonzalez-Larragan, De Bari, Salem and Dahele20,Reference Duke, Tan and Jensen31,Reference Roques44 :

  • Workshop format; incorporation of time to practise contouring/re-contouring is recommended in addition to didactic teaching (the duration of the workshop should be considered in relation to this)

  • Clarity of instructions for cases to be contoured including detailed delineation guidance and specification of laterality, where relevant

  • Timely access to relevant target volume/OAR guidance/atlases

  • Provision of co-registered imaging

  • Target audience; disease sites, numbers of target volume/OARs, number/complexity of cases

  • Choice of assessment; quantitative metrics (such as volume, distance and overlap metrics) should ideally be used in conjunction with qualitative feedback. Be realistic about how much qualitative feedback can be provided in a timely manner to each participant

  • Post-workshop, provision of expert contour (where available) for participant comparison

  • Where a reference contour is used; discussion regarding variation that may occur between even ‘expert’ outliners. One approach could be to use three expert contours and demonstrate the union and overlap as the maximum and minimum acceptable contours

  • Identification of common errors/sources of variation for particular target volume/OARs

  • Highlight available e-learning resources for self-directed learning

  • Design of workshop feedback to evaluate participant confidence in contouring before/after the workshop

  • Audiovisual/technological considerations; including undertaking a ‘trial run’ of the equipment prior to the workshop, including a method of quality assurance for displayed imaging, provision for participants with disabilities and recommendations that participants ensure they have a stable Internet connection and adequate audiovisual equipment to fully participate

Conclusion

This study has demonstrated that virtual contouring training is feasible and that teaching during a virtual SABR contouring workshop for multiple target volumes/OARs was associated with some improvements in contouring variation. Virtual contouring workshops could play an important role in aiding the acquisition of contouring competences alongside formal training initiatives.

Supplementary material

For supplementary material accompanying this paper visit https://doi.org/10.1017/S1460396921000583

Acknowledgements

The authors would like to acknowledge Scott Kaylor and EduCase™ for their support of the workshop/analysis of participant contouring data. We thank Paul Elbourn and team at Profile Productions, as well as the representatives at the Royal College of Radiologists and UK SABR Consortium for their assistance in developing and organising the workshop.

Financial Support

Finbar Slevin is a Clinical Research Fellow supported by a Cancer Research UK Centres Network Accelerator Award to the ART-NET consortium (grant number A21993). Matthew Beasley and Richard Speight are supported by a Cancer Research UK Centres Network Accelerator Award to the ART-NET consortium (grant number A21993). Louise J Murray is an associate professor supported by a Yorkshire Cancer Research funded by Yorkshire Cancer Research (award number L389LM). Alison C Tree acknowledges support from Cancer Research UK (C33589/A28284) and the CRUK ICR/RMH RadNet centre. This project represents independent research supported by the National Institute for Health research (NIHR) Biomedical Research Centre at The Royal Marsden NHS Foundation Trust and the Institute of Cancer Research, London. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. Ann M Henry is supported by grants from Cancer Research UK (award number 108,036), National Institute for Health Research (NIHR) (award number 111,218), Medical Research Council (MRC) (award number 107,154) and Sir John Fisher Foundation (charity, no award number).

Conflicts of Interests

James Good reports honoraria from ViewRay and GenesisCare. Fiona McDonald reports consulting for Accuracy and AstraZeneca, speaker fees from AstraZeneca and Elekta and a research grant from Merck Sharp & Dohme. Alison C Tree declares research funding from Elekta, Accuracy and Varian, and honoraria from Elekta and Genesis healthcare.

Footnotes

Joint first authors

Joint senior authors

References

Jameson, M G, Holloway, L C, Vial, P J, Vinod, S K, Metcalfe, P E. A review of methods of analysis in contouring studies for radiation oncology. J Med Imaging Radiat Oncol 2010; 54 (5): 401410.CrossRefGoogle ScholarPubMed
Vinod, S K, Min, M, Jameson, M G, Holloway, L C. A review of interventions to reduce inter-observer variability in volume delineation in radiation oncology. J Med Imaging Radiat Oncol 2016; 60 (3): 393406.CrossRefGoogle ScholarPubMed
Gwynne, S, Gilson, D, Dickson, J, McAleer, S, Radhakrishna, G. Evaluating target volume delineation in the era of precision radiotherapy: FRCR, revalidation and beyond. Clin Oncol 2017; 29 (7): 436438.CrossRefGoogle ScholarPubMed
Njeh, C F. Tumor delineation: the weakest link in the search for accuracy in radiotherapy. J Med Phys 2008; 33 (4): 136140.CrossRefGoogle ScholarPubMed
Peters, L J, O’Sullivan, B, Giralt, J et al. Critical impact of radiotherapy protocol compliance and quality in the treatment of advanced head and neck cancer: results from TROG 02.02. J Clin Oncol 2010; 28 (18): 29963001.CrossRefGoogle ScholarPubMed
Weber, D C, Tomsej, M, Melidis, C, Hurkmans, C W. QA makes a clinical trial stronger: evidence-based medicine in radiation therapy. Radiother Oncol 2012; 105 (1): 48.CrossRefGoogle ScholarPubMed
Royal College of Radiologists. Radiotherapy target volume definition and peer review – RCR guidance. 2017. https://www.rcr.ac.uk/publication/radiotherapy-target-volume-definition-and-peer-review. Accessed on 20th January 2021.Google Scholar
Weiss, E, Hess, C F. The Impact of Gross Tumor Volume (GTV) and Clinical Target Volume (CTV) definition on the total accuracy in radiotherapy. Strahlenther Onkol 2003; 179 (1): 2130.CrossRefGoogle ScholarPubMed
Walls, G M, Hanna, G G, McAleer, J J. Learning radiotherapy: the state of the art. BMC Med Educ 2020; 20 (1): 150.CrossRefGoogle ScholarPubMed
Datta, S T, Davies, S J. Training for the future NHS: training junior doctors in the United Kingdom within the 48-hour European working time directive. BMC Med Educ 2014; 14 (Suppl 1): S12.CrossRefGoogle ScholarPubMed
De Bari, B, Dahele, M, Palmu, M, Kaylor, S, Schiappacasse, L, Guckenberger, M. Short interactive workshops reduce variability in contouring treatment volumes for spine stereotactic body radiation therapy: experience with the ESTRO FALCON programme and EduCase™ training tool. Radiother Oncol 2018; 127 (1): 150153.CrossRefGoogle ScholarPubMed
Dewas, S, Bibault, J E, Blanchard, P et al. Delineation in thoracic oncology: a prospective study of the effect of training on contour variability and dosimetric consequences. Radiat Oncol 2011; 6: 118.CrossRefGoogle ScholarPubMed
Khoo, E L H, Schick, K, Plank, A W et al. Prostate contouring variation: can it be fixed? Int J Radiat Oncol Biol Phys 2012; 82 (5): 19231929.CrossRefGoogle ScholarPubMed
NHS England. Stereotactic ablative radiotherapy (SABR) for patients with metachronous extracranial oligometastatic cancer (all ages). 2020. https://www.england.nhs.uk/publication/stereotactic-ablative-radiotherapy-sabr-for-patients-with-metachronous-extracranial-oligometastatic-cancer-all-ages/. Accessed on 20th January 2021.Google Scholar
Vaassen, F, Hazelaar, C, Vaniqui, A et al. Evaluation of measures for assessing time-saving of automatic organ-at-risk segmentation in radiotherapy. Phys Imaging Radiat Oncol 2020; 13: 16.CrossRefGoogle ScholarPubMed
Vinod, S K, Jameson, M G, Min, M, Holloway, L C. Uncertainties in volume delineation in radiation oncology: a systematic review and recommendations for future studies. Radiother Oncol 2016; 121 (2): 169179.CrossRefGoogle ScholarPubMed
Dice, L R. Measures of the amount of ecologic association between species. Ecology 1945; 26 (3): 297302.CrossRefGoogle Scholar
Rivin del Campo, E, Rivera, S, Martínez-Paredes, M et al. Assessment of the novel online delineation workshop dummy run approach using FALCON within a European multicentre trial in cervical cancer (RAIDs). Radiother Oncol 2017; 124 (1): 130138.CrossRefGoogle ScholarPubMed
Wickham, H. ggplot2: elegant graphics for data analysis. 2016. https://ggplot2.tidyverse.org. Accessed on 26th January 2021.CrossRefGoogle Scholar
Cacicedo, J, Navarro-Martin, A, Gonzalez-Larragan, S, De Bari, B, Salem, A, Dahele, M. Systematic review of educational interventions to improve contouring in radiotherapy. Radiother Oncol 2020; 144: 8692.CrossRefGoogle ScholarPubMed
Eriksen, J G, Salembier, C., Rivera, S et al. Four years with FALCON – an ESTRO educational project: achievements and perspectives. Radiother Oncol 2014; 112 (1): 145149.CrossRefGoogle Scholar
D’Souza, L., Jaswal, J, Chan, F et al. Evaluating the impact of an integrated multidisciplinary head & neck competency-based anatomy & radiology teaching approach in radiation oncology: a prospective cohort study. BMC Med Educ 2014; 14: 124.CrossRefGoogle ScholarPubMed
Szumacher, E, Harnett, N, Warner, S et al. Effectiveness of educational intervention on the congruence of prostate and rectal contouring as compared with a gold standard in three-dimensional radiotherapy for prostate. Int J Radiat Oncol Biology Phys 2010; 76 (2): 379385.CrossRefGoogle Scholar
Hall, W H, Guiou, M, Lee, N Y et al. Development and validation of a standardized method for contouring the Brachial Plexus: preliminary dosimetric analysis among patients treated with IMRT for Head-and-Neck Cancer. Int J Radiat Oncol Biol Phys 2008; 72 (5): 13621367.CrossRefGoogle ScholarPubMed
Mir, R, Kelly, S M, Xiao, Y et al. Organ at risk delineation for radiation therapy clinical trials: global Harmonization Group consensus guidelines. Radiother Oncol 2020; 150: 3039.CrossRefGoogle ScholarPubMed
UK SABR Consortium. Stereotactic Ablative Radiotherapy (SABR): a resource (version 6.1). 2019. https://www.sabr.org.uk/wp-content/uploads/2019/04/SABRconsortium-guidelines-2019-v6.1.0.pdf. Accessed on 8th February 2021.Google Scholar
Yi, S K, Mak, W, Yang, C C et al. Development of a standardized method for contouring the Lumbosacral Plexus: a preliminary dosimetric analysis of this organ at risk among 15 patients treated with intensity-modulated radiotherapy for lower gastrointestinal cancers and the incidence of radiation-induced Lumbosacral Plexopathy. Int J Radiat Oncol Biol Phys 2012; 84 (2): 376382.CrossRefGoogle ScholarPubMed
American Association of Physicists in Medicine. Report No. 263 – Standardizing Nomenclatures in Radiation Oncology. 2018. https://www.aapm.org/pubs/reports/detail.asp?docid=171. Accessed on 8th February 2021.Google Scholar
De la Pinta, C. SBRT in non-spine bone metastases: a literature review. Med Oncol 2020; 37 (12): 119.CrossRefGoogle ScholarPubMed
Wright, J L, Yom, S S, Awan, M J et al. Standardizing normal tissue contouring for radiation therapy treatment planning: an ASTRO consensus paper. Pract Radiat Oncol 2019; 9 (2): 6572.CrossRefGoogle ScholarPubMed
Duke, S L, Tan, L T, Jensen, N B K et al. Implementing an online radiotherapy quality assurance programme with supporting continuous medical education – report from the EMBRACE-II evaluation of cervix cancer IMRT contouring. Radiother Oncol 2020; 147: 2229.CrossRefGoogle ScholarPubMed
Ackerly, T, Andrews, J, Ball, D, Guerrieri, M, Healy, B, Williams, I. Discrepancies in volume calculations between different radiotherapy treatment planning systems. Australas Phys Eng Sci Med 2003; 26 (2): 9193.CrossRefGoogle ScholarPubMed
Brock, K K, Dawson, L A, Sharpe, M B, Moseley, D J, Jaffray, D A. Feasibility of a novel deformable image registration technique to facilitate classification, targeting, and monitoring of tumor and normal tissue. Int J Radiat Oncol Biol Phys 2006; 64 (4): 12451254.CrossRefGoogle ScholarPubMed
Fotina, I, Lütgendorf-Caucig, C, Stock, M, Pötter, R, Georg, D. Critical discussion of evaluation parameters for inter-observer variability in target definition for radiation therapy. Strahlenther Onkol 2012; 188 (2): 160167.CrossRefGoogle ScholarPubMed
Hanna, G G, Hounsell, A R, O’Sullivan, J M. Geometrical analysis of radiotherapy target volume delineation: a systematic review of reported comparison methods. Clin Oncol 2010; 22 (7): 515525.CrossRefGoogle ScholarPubMed
Maurer, C R, Rensheng, Q, Raghavan, V. A linear time algorithm for computing exact Euclidean distance transforms of binary images in arbitrary dimensions. IEEE Trans Pattern Anal Mach Intell 2003; 25 (2): 265270.CrossRefGoogle Scholar
Taha, A A, Hanbury, A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging 2015; 15: 29.CrossRefGoogle ScholarPubMed
Murray, P, Franks, K, Hanna, G G. A systematic review of outcomes following stereotactic ablative radiotherapy in the treatment of early-stage primary lung cancer. Br J Radiol 2017; 90 (1071): 20160732.CrossRefGoogle ScholarPubMed
Trakul, N, Koong, A C, Chang, D T. Stereotactic body radiotherapy in the treatment of pancreatic cancer. Semin Radiat Oncol 2014; 24 (2): 140147.CrossRefGoogle ScholarPubMed
Cox, S., Cleves, A, Clementel, E, Miles, E, Staffurth, J, Gwynne, S. Impact of deviations in target volume delineation – time for a new RTQA approach? Radiother Oncol 2019; 137: 18.CrossRefGoogle ScholarPubMed
Gwynne, S, Spezi, E, Sebag-Montefiore, D et al. Improving radiotherapy quality assurance in clinical trials: assessment of target volume delineation of the pre-accrual benchmark case. Br J Radiol 2013; 86 (1024): 20120398.CrossRefGoogle ScholarPubMed
Gwynne, S, Jones, G, Maggs, R et al. Prospective review of radiotherapy trials through implementation of standardized multicentre workflow and IT infrastructure. Br J Radiol 2016; 89 (1064): 20160020.CrossRefGoogle ScholarPubMed
Evans, E, Radhakrishna, G, Gilson, D et al. Target volume delineation training for clinical oncology trainees: the Role of ARENA and COPP. Clin Oncol (R Coll Radiol) 2019; 31 (6): 341343.CrossRefGoogle ScholarPubMed
Roques, T W. Patient selection and radiotherapy volume definition — can we improve the weakest links in the treatment chain? Clin Oncol 2014; 26 (6): 353355.CrossRefGoogle ScholarPubMed
Figure 0

Table 1. Summary of median Dice similarity coefficient (DSC) and line domain error (LDE) measurements before/after teaching for each structure

Figure 1

Figure 1. Box and whisker plots for the target volume/organs at risk structures for the lung cancer, pelvic bone metastasis and common iliac nodal metastasis cases. The top row represents Dice similarity coefficient (DSC) results, and the bottom row represents line domain error (LDE) results. The box represents the middle 50% of the data and is bounded by the upper (Q3) and lower (Q1) quartiles, and the horizontal line indicates the median value. The upper whiskers represent Q3 + 1.5 × IQR and the lower whiskers represent Q1–1.5 × IQR. Any outliers beyond these ranges are indicated as dots.

Figure 2

Table 2. Summary of qualitative feedback on participants’ post-workshop contours

Figure 3

Figure 2. Visual guide to delineation of SacralPlex, BrachialPlex and BronchusProx. In (a–c), SacralPlex; iliacus muscle (green), L5 vertebral body (dark blue), obturator internus muscle (orange), psoas muscle (light blue), SacralPlex (purple), vessels (yellow) are shown. SacralPlex is contoured using a 5 mm diameter roller ball. In 2A, superior border of SacralPlex is shown at L4/5 vertebral interspace; SacralPlex is shown bordered by (ilio)psoas muscle anteriorly and vertebral body posteriorly. In 2B, at the sacroiliac foramen, SacralPlex is shown bordered by vessels anteriorly, iliacus muscle laterally and sacral ala posteriorly. In 2C, inferior border of SacralPlex is shown at the level of the superior femoral neck bordered by obturator internus muscle anteriorly, gluteus maximus muscle posteriorly. In (d–f), BrachialPlex contoured as suggested by UK SABR Consortium Guidelines26; anterior scalene muscle (orange), BrachialPlex (light blue), common carotid artery (red), internal jugular vein (posterior scalene muscle (brown), subclavian artery (pink) and subclavian vein (dark blue) are shown. Intravenous contrast is helpful, and BrachialPlex is contoured using a 5 mm diameter roller ball. 2D shows a proximal slice: the superior border of BrachialPlex is at the bifurcation of the brachiocephalic trunk into the jugular/subclavian veins (or carotid/subclavian arteries). In 2E, a middle section of BrachialPlex is shown; the plexus sits between anterior and middle scalene muscles. In 2F, the neurovascular complex including the subclavian and axillary vessels is contoured as a surrogate for the brachial plexus, ending after the neurovascular structures cross the second rib. In (g–i), BronchusProx; BronchusProx (purple) is shown. In 2G, superior border of BronchusProx is the distal 2 cm of trachea including carina. In 2·2H, the mid-section of BronchusProx is shown and includes right/left upper lobe bronchi, bronchus intermedius, right middle lobe bronchus, lingular bronchus and right/left lower lobe bronchi. In 2I, contouring of lobar bronchi stops immediately at the site of a segmental bifurcation.

Figure 4

Table 3. A summary of resources to support target volume/organ at risk delineation

Supplementary material: File

Slevin et al. supplementary material

Slevin et al. supplementary material

Download Slevin et al. supplementary material(File)
File 96.7 KB