Emergency medical services (EMS) agencies are at the forefront of disaster and mass casualty incidents (MCI) response. Reference Catlett, Jenkins and Millin1 EMS can face a wide variety of different mass casualties and disasters. As stated in a recent work by Gamberini et al. over the last 6 decades, Italy has suffered a number of natural, anthropic, and technological MCI, as such an all-hard approach is paramount when designing educational disaster medicine programs. Reference Gamberini, Imbriaco and Ingrassia2 Experience and training are well-recognized features of teams responding to disasters and MCI. Reference Oldenburger, Baumann and Banfield3 Limited research has been published to establish objectives and format for MCI education for involved prehospital health care personnel. Reference Clawson, Menachemi, Kim and Brooks4 In the United States, since September 11, 2001, several authors have worked to design and propose national standardized all-hazards disaster core competencies for acute care physicians, nurses, and prehospital personnel. Reference Subbarao, Lyznicki and Hsu5,Reference Schultz, Koenig, Whiteside and Murray6 The authors conclude in their paper that training based on standardized competencies will ensure that acute care medical professionals practice, demonstrate, and maintain the skills needed for effective disaster-related medical response. Data from surveying US nationally certified EMS providers found that almost 91% reported receiving individual-level preparedness training. Reference Fernandez, Studnek and Margolis7 In Germany, where ambulances are also staffed with prehospital physicians and nurses like Italy, providers have specific training requirements for disaster relief operations. Reference Fischer, Wafaisade and Bail8 Recent reports suggest that formal mass casualty training is being now delivered at the postgraduate training level, and this is usually based on general principles rather than operational procedures. Reference Sarin, Biddinger and Brown9,Reference Carenzo, Bazurro and Colombo10 It appears reasonable for a prehospital management agency to design and run a dedicated prehospital management training program for its personnel. In 2008, Regione Lombardia, one of Italy’s 20 regions, established a centralized prehospital management agency called Agenzia Regionale Emergenza Urgenza (AREU). In 2012, AREU decided to develop a standardized and centralized core curriculum to train all of its operators in the principles of medical management of mass casualties and specifically context-specific protocols and operating procedures. This is a descriptive paper that reports the development, implementation, and results of a sustainable regionwide, mass-casualty management prehospital training program implemented by the Regione Lombardia EMS agency AREU in Italy. We hope that sharing our experiences and results will encourage discussion of good practice and will help managers and decision makers deliver high-quality training to respond to MCI situations.
Methods
A case study presenting the development, delivery, and assessment of a regionwide, all-hazards prehospital training program. The course was designed in 2013 with the train-the-trainers program during the same year and the provider program still ongoing as of today.
Setting
AREU serves an area of 24.864 km2 and a population of approximately 10 million, ranging from very high-density urban areas (4 million in the urban area of Milan) to very rural and remote locations, and approximately 40% of the region is mountainous.
AREU directly manages, in a centralized fashion, training, operations, and governance of all prehospital emergency medical services operating in its territory. At any given day, AREU operates a fleet of 5 HEMS/SAR helicopters (2 operating during 24 hours a day, and 3 during daylight), 51 critical care physician-led response cars (staffed by a prehospital physician, a critical care nurse, and a driver technician) using a system similar to that described by Rehn in London, Reference Rehn, Davies, Smith and Lockey11 50 nurse-led cars (staffed by a critical care prehospital nurse and a driver technician), and more than 200 basic life support ambulances staffed by a team of 3 emergency medical technician equivalent operators, many of which are being operated by volunteers. Overall, these resources respond to more than 1 million calls every year. Clinicians (doctors and nurses) can choose to work a variable amount of their overall monthly clinical time in the prehospital system. Most physicians staffing ground emergency vehicles will operate an 80/20 ratio between hospital and prehospital time. HEMS, dispatch center clinicians, and managerial roles are full-time. The resources are managed by 4 interconnected dispatch centers (SOREU) spread around the region.
Curriculum Development and Didactic Tools
The content of training was based on core competencies required for disaster response published in the United States and internationally, Reference Subbarao, Lyznicki and Hsu5,Reference Schultz, Koenig, Whiteside and Murray6 along with national and regional laws and internal standard operating procedures. The core of the training was the internal mass casualty management standard operating procedures.
The course was designed using a blended-learning approach. Course participants could access online course materials, including learning modules and the regional prehospital mass casualty plan, in a dedicated online platform designed using the Moodle Environment. Moodle (moodle.org) is an online Learning Management System (LMS) developed on pedagogical principles and used internationally by schools and universities for blended-learning and distance education. It allows educators to create private websites with online courses to achieve predetermined learning goals. Each participant received individual credentials allowing them to access the learning modules, interact with the faculty via dedicated forums, and complete the pretests and posttests.
The program learning objectives are: (1) principles of command and control, communications, and resource management; (2) principles of mass casualty triage and the START triage protocol; (3) on-scene roles, identification, and management; (4) Regione Lombardia and AREU Mass Casualty management standard operating procedures; and (5) inter-agency communications and relations (principles of fire-brigade and police operations). In detail, at distance online training included all theoretical knowledge, including law and ethics, triage, roles and responsibilities, including the first on scene, casualty clearing, medical posts, communications, and media management. Students could also choose up to 3 elective modules: hazardous materials (HazMat), urban search and rescue (USAR), and psychology aspects of disasters and mass casualty. During the live course scene, assessment and roles and responsibilities were covered in the form of a lecture, triage protocols were presented with a tabletop exercise, and 2 iterations of the ISEE exercise were run following a brief familiarization with the tool. The overall scientific program of the course is presented in Table 1.
Table 1. Scientific educational program of the provider course
A, assessment; CS, computerizes simulation; Deb, debriefing; Dem, demonstration; L, lecture; ONF, online forum; PD, plenary discussion.
The initial curriculum and the train-the-trainer program were designed and delivered by the AREU mass casualty management office personnel, along with guest faculty from the Research Center in Emergency and Disaster Medicine (CRIMEDIM) and the European Master in Disaster Medicine. Reference Della Corte, Hubloue and Ripoll Gallardo12
Train-the-Trainer (Instructor) Course
A cohort of experienced prehospital providers from different areas of the region was selected as the initial instructor class. The selection was based on formal qualifications and professional experience and was managed in a centralized fashion. Each class member participated in an edition of the train-the-trainers workshop during which they were exposed to instruction on theories about adult learning, training methodologies, collaborative content development, and delivery and mentoring from experienced trainers. Instructors in training also familiarized with the new teaching tools (simulations) while being supported by expert external facilitators. Candidate instructors had to pass a final exam to qualify for the role.
Provider Training Event
The provider course is open to a mixed classroom of physicians, nurses, technicians, and drivers. Before attending the class, course participants are required to complete a 4-hour-long, distance-learning didactical module that exposes them to principles of mass casualty management and to the regional disaster response plan. Each edition of the live course lasts a working day (8 hours) and includes formal lectures and case-based discussions in the morning and computer-based simulation in the afternoon. Each course includes 4 to 5 instructors and circa 15 participants. Simulation sessions were designed with the following goals: training and exercising the command and control chain as well as communication and resource management skills. Each simulation session was run using the ISEE simulation software (ISEE Support, Wemmel, Belgium). Reference Delooz, Debacker, Moens and Johannik13 The software allows users to re-create virtual worlds, where virtual resources can be portrayed and used to respond to a virtual emergency involving casualties. For this course, a specific virtual world was created and named “Lombardialand” (Figure 1). Starting from high definition regional cartography, all prehospital and hospital resources were introduced in the virtual world, including the exact number of ambulances and cars, helicopters, fire departments, law enforcements, and hospitals. Based on historical data, a number of possible MCI scenarios were designed (major highway car crash, railway accident, mass gathering, and industrial event), each with approximately 50 to 120 virtual casualties. During the simulation, course participants were assigned response roles as they would in real life and were expected to declare the major incident (METHANE), assess the scene, set up the incident command system, communicate, triage and care for the victims, plan the evacuation strategy, and utilize the available resources efficiently.
Figure 1. The Lombardialand interactive virtual world. The base layer for the simulation exercises during the training course. It contains an accurate reproduction of the actual emergency resources of Regione Lombardia, Italy.
Summative Assessment
Each course participant was asked to complete a multiple-choice pretest before taking part in the course. At 30 days following the end of the live training, participants were asked to complete a 45-item multiple-choice summative assessment. This had a predetermined pass mark. Those who failed were allowed a re-test in the following 30 days.
Participant Feedback
After completion of the course, a 5-item Likert scale-based survey was used to obtain participant feedback on the training experience. The questions investigated the general domains and specific items: (1) learning objectives (course met the predetermined learning objectives); (2) faculty (the quality of teaching); (3) course organization (course logistics and organizational aspects such as technology, venue, and ancillary services); (4) perceived relevance (subjective perception of whether the overall training program has a direct transferability to the operational capacity of the trainee); and (5) overall assessment (global rating of the quality of the educational event). A free text section allowed them to provide positive and negative comments and improvement suggestions, and a final qualitative thematic analysis was performed. The free text feedback section was analyzed by themes, and items were grouped by section as positive feedback, negative feedback, and areas for improvement.
Economic Analysis
A cost analysis of the training program has been conducted using Levin’s framework for educational cost-effectiveness. First, all resources needed to run the training program, also called “ingredients,” are identified. Second, monetary values are placed on each ingredient. Third, the total cost of each ingredient is summed and expressed as a cost-per-learner. Reference Levin and McEwan14,Reference Levin15 Cost categories were divided into: (1) equipment and learning materials, (2) personnel costs, (3) facility costs, (4) required client inputs, and (5) other program inputs. Ingredients are presented as suggested in the systematic review of cost in simulation-based medical education by Zendejas et al. Reference Zendejas, Wang and Brydges16
Statistical Analysis and Ethical Approval
Data were collected and analyzed in Excel spreadsheets (version 14.4; Microsoft, Redmond, WA). Data distributions are reported by average and standard deviation (SD) or median and interquartile ranges (IQRs). The AREU medical directorate considered the project protocol. It met local criteria for, and was considered as, an agency service evaluation and quality improvement project (Protocol number 0010223). No additional interventions were performed, and the study recorded only the frequency of events in normal training programs, with a view to service improvement. Therefore, ethical approval was not required. All the authors confirm that the standards of the Declaration of Helsinki were maintained during the delivery of the educational programs, and confidentiality was maintained.
Results
Between May 14, 2013, and June 26, 2013, a total of 3 train-the-trainer courses were run, preparing a total of 43 instructors. Between June 26, 2013, and December 31, 2020, a total of 84 editions of the provider training event were delivered, training overall 1329 prehospital providers; 510 (38%) were prehospital physicians, 543 (41%) were prehospital nurses, and 276 (21%) were emergency medical technicians and drivers. Over the whole study period, out of the total number of provider participants, 1239 (93%) passed the summative assessment and were qualified as being operationally mass casualty “ready”; 71 (6%) of them had to take the test more than once before passing, and 29 were unreachable and never took the summative test (and declined further participation). Table 2 presents details by year and job qualification.
Table 2. Editions and participants by year: Participants by job type, pass overall (passing rate within parentheses)
* Year 2020 was suspended due to the COVID-19 pandemic.
Regarding participants’ feedback following their educational experience, the overall program was rated 4.4 ± 0.7 out of 5. The highest overall ratings were for Faculty and Perceived Relevance. The lowest overall score regarded the logistic aspects within the course organization. Details of participant feedback by category and training year are presented in Table 3. Recurrent themes in the positive feedback section regarded the faculty and the interactivity of the course. Most common comments were about the competency of the faculty, clear exposure, accessibility, and professional credibility, followed by the methodology of simulation, the ability to practice, and the constructive climate of the course. Recurrent themes in the negative feedback section regarded technical issues accessing the online platform before the course, the spaces where the simulation was conducted, and the length of the simulation itself (too short). Most commonly, participants reported wanting to run more scenarios under more roles. Recurrent themes in the improvement area suggested more feedback, the presence of refresher courses, and the need for real-size training, at this stage offered on an ad-hoc basis in the system. Finally, many participants asked for formal case-based studies on local events and their management rather than standardized but hypothetical scenarios.
Table 3. Participants’ satisfaction with the training event by year. All ratings from an ordinal assessment scale of 5 items
The overall cost of running the provider program during the study period was €321 510 (circa US $382 000). The average cost per edition was €3828 and €242 per participant. Essential costs according to Levin’s cost categories are presented in Table 4.
Table 4. Essential costs to run the provider training program in the study period
Since September 2017, there was a 50% increase in instructor’s hourly fee. Required client input and other program inputs were not available in our cost analysis. Our cost analysis excludes the opportunity costs: costs that are incurred from taking time to learn or teach (ie, lost clinical revenue from staff when taking time to teach).
* Year 2020 was suspended due to the COVID-19 pandemic.
Discussion
In this paper, we present a sustainable regionwide, mass-casualty educational program, successfully training more than 1200 providers to an operational readiness level since its inception in 2014. To the best of our knowledge, this is the first report of an EMS-led, agency-wide, large-scale multidisciplinary mass-casualty training program. We designed, implemented, and delivered a comprehensive, competency-driven training program, intended to provide a large audience of prehospital clinicians and staff with the skills necessary for the effective management of mass-casualty management on the job. The course was designed with a combination of online and classroom-based lectures and simulations. Lectures were designed to be short and to deliver essential reference concepts to be used in the subsequent practical exercises. Simulation was the key educational strategy to teach technical and attitudinal skills during this training. Reference Rodriguez-Paz, Kennedy and Salas17
Satisfactory pass rates are reported for all professions, and the number of participants not passing the final assessment is around 2%. Effectiveness of a model relatively similar to ours, although shorter, has been reported in smaller studies with shorter training times and smaller sample sizes. In the work by Risavi et al., Reference Risavi, McLaughlin and Stuart18 a convenience sample of 141 licensed EMS providers was subject to a 4-hour interactive training program consisting of both lectures and interactive exercises and demonstrated a significant improvement in knowledge after the training program in the fields of scene size-up, incident command system setup, and medical management. In our study, we did not specifically investigate the relative improvement between the pretest and the summative posttest, as, for operational purposes, each professional was required to reach a minimum level of proficiency, defined by the summative test passing rate, to be considered operationally ready. This is why we were mostly interested in the overall number of providers who could be trained effectively over time to reach this standard.
Participant feedback was consistently high with scores equal to or above 4 out of 5 for all the assessed items. Course organization was constantly rated lower than faculty and perceived applicability of the training program in real-life operations. The latter is particularly relevant in the design and monitoring of the training program as it ensures participants are motivated in perceiving their training as having a significant impact in their daily work, and the time they are spending in the classroom as well as simulations has a meaningful effect on their clinical practice and performance. Reference Kogan and Shea19
A cost-effectiveness analysis in this field is difficult due to the limited ability of measuring clinically relevant outcomes. Reference Walsh, Levin, Jaye and Gazzard20,Reference Lin, Cheng and Hecker21 Return on investment analysis in the field of disaster medicine has been attempted regarding the mitigation phase, Reference Shreve and Kelman22 specifically in the context of community preparedness in natural disaster risk reduction. Reference Kousky, Ritchie, Tierney and Lingle23 Direct return on investment of this training program can be estimated by the total number of participants trained over the years, the optimal passing rate, and the durability of the program over a relatively affordable cost per participant, although this is limited by not having measured the opportunity cost. The pass rate is very satisfactory and suggests that the agency is not wasting money in holding courses that result with not certifying its own providers. Some clinical effects of the training program can be indirectly assessed in the real world by looking at performance matrixes published in a recent study about a railway accident that happened in the region in 2018. Reference Faccincani, Stucchi and Carlucci24 The authors describe the disaster response to a 5-car train derailment analyzing the performance for both the EMS involved and the hospitals involved. The findings show that EMS performed highly in declaring the major incident and setting up the incident command system, as well as scene management and triage. Long transport time for high priority codes was reported but, according to the authors, this was due to the likelihood of complex entrapment. Another indirect outcome of this standardized training program can also be found in the rapid internal reorganization that AREU performed to effectively respond to the COVID-19 emergency and its interaction with emergency departments. Reference Spina, Marrazzo and Migliari25–Reference Sechi, Migliori and Dassi29
A number of limitations of this study should be acknowledged. The major limitation is probably the absence of available follow-up of knowledge retention in trained providers. It is known from literature that infrequently used skills tend to fade and regular refresher training is recommended. Reference Risavi, Terrell, Lee and Holsten30,Reference Dittmar, Wolf and Bigalke31
Second, our study did not specifically investigate the effectiveness of the training initiative with the pretest versus posttest comparison since, for operational purposes, each professional was required to reach a minimum level of proficiency, defined by the summative test passing rate. Third, as with any online tests (without a proctor), there is no way of guaranteeing that participants did not cheat. Some mitigation strategies were put into place, specifically that participants accessing the web page accept a student honor code stating that by starting the exam they acknowledged that they were the assigned student taking the quiz and the work was entirely their own; moreover, exams included shuffling questions and a time limit.
Fourth, this study is limited as it only quantifies the associated costs and does not necessarily analyze the benefits of our MCI education curriculum. If improved patient outcomes can be demonstrated as a result of the training initiative, the associated significant expenses with its implementation and maintenance may prove worth the investment. Interestingly, recently published reports about MCI management, which unavoidably involved AREU staff, can be considered indirect outcomes and effects of the training.
Conclusion
This is the first report describing a system-wide, large-scale, all-hazards educational training program for EMS personnel. We have described a simple, yet interactive simulation and blended-learning approach, which has yielded good passing rates, good participant satisfaction, and contained costs to systematically train EMS personnel to mass-casualty management in a long-term and economically affordable way. We hope this work will be of help to other services wishing to implement, update, or evaluate their system-wide mass-casualty training.
Acknowledgments
We would like to thank the AREU Maxi Emergenza instructors team for invaluable assistance and support in delivering the training program over the years.
Funding statement
No funding was received for this study.
Conflict(s) of interest
LC, PLI, and DC received honoraria for lecturing and consulting in the field of disaster medicine training. LC and PLI are accredited consultants for Agenzia Regionale Emergenza Urgenza (AREU). The other authors declare no competing interests.
