Results and analysis

The Medline search generated 444 results. After screening and removal of duplicates, 61 papers were included in the review. Most available simulators on the market utilise high-resolution computed tomography (CT) scans to generate 3D models of temporal bones that are visualised by 3D glasses. Learners interact with these models using haptic feedback devices.

The Voxel-Man™ Simulator (Voxel-Man Group, Hamburg, Germany; Figure 1) is the most widely validated platform in the literature and was the first commercially available temporal bone simulator. It has a viewing station, haptic feedback device, foot control pedal and central processing system onto which several pre-programmed training scenarios are loaded. Bespoke CT scans also can be input to allow case-specific surgical rehearsal.⁵ Melbourne University's Virtual Reality Surgical Simulator was first described using a ‘haptic workbench’ platform, which can be shared by the instructor and student, however it also can be used with 3D glasses (Figure 2).^{Reference Wijewickrema, Piromchai, Zhou, Ioannou, Bailey and Kennedy6} CardinalSim™ (Stanford University) is a free-access software offering models based on CT scans and using 3D glasses.^{7,Reference Compton, Agrawal, Ladak, Chan, Hoy and Nakoneshny8} SurgiSim, which was developed from CardinalSim (Calgary, Western and Stanford Universities), utilises the commercially available Oculus™ headset and is compatible with various commercially available haptic feedback devices. SurgiSim includes a virtual operating-room microscope, instrument tray, mayo stand, ceiling-mounted surgical lights and a television screen to make the experience more realistic.^{Reference de Lotbiniere-Bassett, Volpato Batista, Lai, El Chemaly, Dort and Blevins9} Ohio State University (OSU) also has a free-access temporal bone simulator software with data from CT scans and uses a binocular display device (Figure 3).^{Reference Wiet, Stredney, Kerwin, Hittle, Fernandez and Abdel-Rasoul10,Reference Wiet, Bryan, Dodson, Sessanna, Stredney and Schmalbrock11} Finally, Visible Ear Simulator (Rigshospitalet and Alexandra Institute, Denmark) is a free-access software^{12,Reference Sorensen, Mosegaard and Trier13} that differs from other platforms as it uses cryosections rather than CT scans to generate models.^{Reference Sieber, Andersen, Sørensen and Mikkelsen14}

Figure 1. Voxel-Man™ Temporal Bone Simulator, image by Christian Stelling.

Figure 2. Melbourne University's Virtual Reality Surgical Simulator, image by Dr Sudanthi Wijewickrema.

Figure 3. Ohio State University Temporal Bone Simulator.

Validation, both subjective and objective, is a crucial aspect of deployment of simulation in training. Subjective validity can be divided into face validity (i.e. assessing the realism of the simulator) and content validity (i.e. assessing the training effectiveness and educational value). Objective validity describes the simulator's ability to differentiate between different surgeon experience levels.^{Reference Van Nortwick, Lendvay, Jensen, Wright, Horvath and Kim15} The literature on validity assessments of the available platforms is summarised in Table I of the supplementary material, available online.^{Reference Compton, Agrawal, Ladak, Chan, Hoy and Nakoneshny8–Reference Wiet, Stredney, Kerwin, Hittle, Fernandez and Abdel-Rasoul10,Reference Khemani, Arora, Singh, Tolley and Darzi16–Reference Fang, Wang, Liu, Su and Yeh29}

Despite the validation work in the literature, it can be difficult to determine the overall effect of virtual reality simulation on training outcomes in the real world. Indeed, a Cochrane review in 2015, which concluded there was limited evidence to support inclusion of virtual reality simulation into surgical training programmes in ENT, recommended further study.^{Reference Piromchai, Avery, Laopaiboon, Kennedy and O'Leary30} A subsequent meta-analysis showed improvement in overall performance following training on the simulator.^{Reference Lui and Hoy31}

There is good evidence that the Voxel-Man simulator improves surgical training. Nash et al. assessed novice trainees on specific surgical tasks and found improvement in the time taken, overall score and structural damage over the course of the study.^{Reference Nash, Sykes, Majithia, Arora, Singh and Khemani32} Francis et al. assessed residents completing two key steps in a cortical mastoidectomy^{Reference Francis, Malik, Diaz Voss Varela, Barffour, Chien and Carey33} and found improvement in overall score, time taken and number of injuries. Al-Noury assessed four residents performing mastoidectomies on two patients: one case was completed with no simulation training; the second case was performed after practising on the Voxel-Man simulator.^{Reference Al-Noury34} Evaluations were conducted by a senior surgeon and nurse. The residents who had used the simulator had a higher global rating score and task-based checklist, as well as a shorter operation time. Furthermore, they felt more confident after utilising the Voxel-Man simulator.^{Reference Al-Noury34}

The Voxel-Man simulator has been compared with cadaveric training.^{Reference Reddy-Kolanu and Alderson35} Trainees on a two-day mastoid surgery course performed cortical mastoidectomy, atticotomy and posterior tympanotomy on both formats. Trainees judged that cadaveric training was better in terms of resembling real operating conditions and for feedback on their learning. They judged the Voxel-Man simulator to be superior in several domains: providing repetition of a skill, allowing regularity of training, titrating task difficulty, adaptability of teaching method, meeting learning needs and defining clearer goals and outcomes.

Groups have designed curricula around virtual reality simulators. Arora et al. developed a Voxel-Man simulator programme for cortical mastoidectomy.^{Reference Arora, Hall, Kotecha, Burgess, Khemani and Darzi36} This included two familiarisation tasks (skeletonisation of sinodural angle and identification of lateral semicircular canal) and four procedural tasks (short process of incus, delineation of facial nerve and chorda tympani, and extended cortical mastoidectomy). Trainees could not progress to the next step of the curriculum until meeting minimum requirements of measure, such as bone volume removed correctly, maximum number of injuries, having the burr tip visible and completing the task in a timely manner. A Likert tool showed that trainees found the simulator programme helped them develop hand-eye co-ordination, instrument navigation, drilling technique, surgical anatomy and surgical skills.

The Voxel-Man simulator also has been demonstrated to be useful in avoiding distraction and being able to multi-task.^{Reference Ahmed, Ahmad, Stewart, Francis and Bhatti37} The practical utility of the imported CT scan feature was assessed by Arora et al. who found gains in confidence and facilitation of planning and training. Overall, trainees found this function more useful than trainers.^{Reference Arora, Swords, Khemani, Awad, Darzi and Singh38}

Zhao et al. evaluated the usefulness of the Melbourne Temporal Bone Simulator in teaching novice trainees and found that the virtual reality group performed significantly better on cadaveric bones after a self-directed two-hour simulator session as compared to trainees who received traditional teaching.^{Reference Zhao, Kennedy, Yukawa, Pyman and O'Leary39} These findings were seen in another study that employed supervised virtual reality teaching.^{Reference Zhao, Kennedy, Yukawa, Pyman and O'Leary40} The addition of automated guidance to this simulator was found to further improve performance.^{Reference Wijewickrema, Zhou, Ioannou, Copson, Piromchai and Yu41}

In terms of improving real-world performance, 10 physicians were given five sessions on the Melbourne simulator.^{Reference Gawęcki, Węgrzyniak, Mickiewicz, Gawłowska, Talar and Wierzbicka42} They assisted in real cochlear implant surgery following each session. After completing this programme, the participants were scored using a validated tool^{Reference Talks, Lamtara, Wijewickrema, Gerard, Mitchell-Innes and O'Leary43} during supervised surgery and a significant positive association was found between the results of their fifth virtual reality session and their supervised surgery, demonstrating that the simulator improves real world operative performance.^{Reference Gawęcki, Węgrzyniak, Mickiewicz, Gawłowska, Talar and Wierzbicka42}

Copson et al. assessed otolaryngology registrars performing simulated cochlear implant surgery before and after training on the Melbourne simulator with automated feedback. Performances were assessed using a validated tool and demonstrated significant improvement in total performance scores.^{Reference Copson, Wijewickrema, Zhou, Piromchai, Briggs and Bailey44}

CardiSim was evaluated by Locketz et al.^{Reference Locketz, Lui, Chan, Salisbury, Dort and Youngblood45} Sixteen residents’ performances and confidence levels in dissecting a cadaveric temporal bone were evaluated before and after simulated practice. Confidence levels were significantly higher following practice and correlated with increased performance scores.^{Reference Locketz, Lui, Chan, Salisbury, Dort and Youngblood45}

The Ohio State University Temporal Bone Simulator was evaluated and compared to cadaveric simulation.^{Reference Wiet, Stredney, Kerwin, Hittle, Fernandez and Abdel-Rasoul10} There was no significant difference in terms of outcomes when evaluated after two weeks of practice, meaning that, at the least, the temporal bone simulator is not inferior to cadaveric practice.

The Visible Ear Simulator has been evaluated extensively regarding its impact on training. Andersen et al. found that final performance of residents was comparable between simulator and cadaveric training. They suggested that there is not a significant training benefit of using cadavers over simulation at a junior level, thereby allowing cadaveric materials to be reserved for more-advanced training.^{Reference Andersen, Cayé-Thomasen and Sørensen46} Another potential advantage of simulators is the self-directed learning opportunity, potentially as preparatory work before attending a dissection course. Andersen et al. demonstrated that self-directed training is effective, especially when distributed rather than massed together.^{Reference Andersen, Konge, Cayé-Thomasen and Sørensen47} Indeed, automated summative feedback has been shown to improve performance and retention.^{Reference Frithioff, Frendø, von Buchwald, Trier Mikkelsen, Sørensen and Andersen48} When utilising virtual reality prior to cadaveric dissection, residents displayed an improvement in performance.^{Reference Andersen, Foghsgaard, Konge, Cayé-Thomasen and Sørensen49} This could be related to the lower cognitive load identified in virtual reality training compared to cadaveric training.^{Reference Andersen, Mikkelsen, Konge, Cayé-Thomasen and Sørensen50} In addition, regular virtual reality usage prior to a cadaveric course further improved outcomes with cadaveric dissection.^{Reference Andersen, Foghsgaard, Cayé-Thomasen and Sørensen51} Another advantage of simulation is the possibility for decentralised training at home without the need for dedicated dissection labs, which has been shown to improve cadaveric dissection performance.^{Reference Frendø, Thingaard, Konge, Sørensen and Andersen52,Reference Frendø, Konge, Cayé-Thomasen, Sørensen and Andersen53} Creating a structured self-assessment in a self-regulated training curriculum has been shown to promote cognitive engagement and motivation to learn the task.^{Reference Andersen, Frendø, Guldager and Sørensen54}

However, some studies do present conflicting results. When the Visible Ear Simulator was used by novices, there was no benefit in varying the anatomy.^{Reference Arnesen, Frithioff, Sørensen, Andersen and Frendø55} This may simply imply that novices’ learning needs are at a more basic level and that nuances in anatomy do not enhance their learning at this level. In another study, ultrahigh fidelity graphics when used by novices seemed to heighten the cognitive load and worsen outcome measures.^{Reference Frithioff, Frendø, Mikkelsen, Sørensen and Andersen56}

The benefit of simulation training also may be procedure dependent. For example, when evaluating the Visible Ear Simulator for cochlear implant procedures, there did not appear to be a benefit when applied to cadaveric training.^{Reference Frendø, Frithioff, Konge, Cayé-Thomasen, Sørensen and Andersen57} The effect of repetition in training with the Visible Ear Simulator was evaluated and showed mixed results, motivation alongside supervision and testing are required in addition to individual simulator use.^{Reference Fartoussi, Sørensen and Andersen58}

West et al. identified a ceiling effect of the benefit in novices when using the virtual reality simulator and this tended to occur before the 60-minute time limit.^{Reference West, Konge, Cayé-Thomasen, Sørensen and Andersen59} This could be improved by improving the tutor function of the simulation.^{Reference Andersen, Konge, Mikkelsen, Cayé-Thomasen and Sørensen60} It might also be improved by implementing a structured self-assessment,^{Reference Andersen, Guldager, Mikkelsen and Sørensen61} which has been shown to improve cognitive engagement and motivation^{Reference Andersen, Mikkelsen and Sørensen62} as well as cadaveric dissection performance.^{Reference Andersen, Frithioff, von Buchwald, Sørensen and Frendø63}

A systematic review of mastoidectomy training by Al-Shahrestani et al. found insufficient automatic feedback from temporal bone simulators for them to be accepted for certification.^{Reference Al-Shahrestani, Sørensen and Andersen64} Temporal bone simulators are widely used across the world in surgical training.^{Reference Favier, Ayad, Blanc, Fakhry and Andersen65–Reference Lui, Compton, Ryu and Hoy67}