Retention of study participants is of serious methodological concern, given the importance of follow-up to the validity of research findings. If attrition is too high, additional resources must be committed to compensate for participants who are lost to follow-up and to avoid loss of statistical power. Reference Gul and Ali1 While some attrition is anticipated in all studies, benchmarks of 70 or 80 percent retention per study arm are common criteria for affirming the validity of a behavioral intervention. Reference Gul and Ali1
Attrition is particularly threatening to the validity and reliability of study outcomes in longitudinal research. Variations in retention rates within and between research sites may lead to distorted results. Thus, factors that may lead to greater site-specific attrition are of particular interest for researchers. Sudden events, such as natural disasters, that affect an entire city or country add particular challenges to overall retention. For example, the disruption may make clinic visits or other procedures more difficult and favor retention at 1 site over another.
When major natural disaster-level disruptions (eg, hurricanes) occur, researchers typically face extraordinary difficulties just to maintain contact with study volunteers and to sustain research activities. Reference Packenham, Rosselli and Ramsey2 In these critical situations, effective communication is key to continued engagement, but how to retain participants in the immediate aftermath of a natural disaster is not well known. Little attention has been paid to the relative merits of different communication and disaster preparedness strategies to address these unique obstacles. Reference Packenham, Rosselli and Ramsey2-Reference Rodríguez-Díaz4
For a study on the use of human immunodeficiency virus (HIV) self-testing to screen sexual partners, researchers designed a daily sexual diary for research participants in Puerto Rico (PR) to record information on their sexual encounters via text messaging using short message service computer-assisted self-interview (SMS-CASI). When Hurricanes Irma and Maria, 2 of the most intense tropical cyclones worldwide of 2017, struck the island, both communication lines and travel infrastructure suffered widespread damage. Reference Rodríguez-Díaz4,Reference Zorrilla5 The unexpected events provided valuable lessons for engagement, re-engagement, retention, and communication, with participants in the event of catastrophic natural disasters.
METHODS
This study was a 5-year, randomized controlled trial exploring the effectiveness of HIV self-testing (HIVST) as a risk reduction tool for high-risk populations. Participants were recruited in New York City and PR between March 2014 and June 2018, and participant follow-up was completed in December 2018. Participants were recruited through mixed-methods using social media and traditional strategies, detailed elsewhere. Reference Iribarren, Ghazzawi and Sheinfil6
Participants’ eligibility criteria were HIV-negative, 18 years of age or older, identifying as a man or transgender woman who has sex with men, reporting 3 or more occasions of condomless anal intercourse (CAI) with serodiscordant or unknown status partners in prior 3 months, 2 or more sexual partners in previous 3 months, and not on oral pre-exposure prophylaxis (PrEP) at the time of recruitment. Participants responded to a brief pre-screening survey. Those who qualified were invited to an in-person screening visit (Visit 1).
In Visit 1, participants completed a computer-assisted self-interview (CASI) to determine their eligibility for enrollment. The CASI assessed demographic information, sexual risk behaviors in the past 3 months, HIV knowledge, alcohol and substance use history, and willingness to use a self-test with partners. Furthermore, participants self-administered an oral HIV antibody test using the OraQuick® In-Home HIV Test while monitored by a researcher. Those who fulfilled the eligibility criteria were invited to enroll in the study and asked to return for an enrollment visit (Visit 2).
In Visit 2, participants were randomized into either control or intervention and enrolled in our SMS-CASI system for daily reporting of sexual behavior and HIVST kit use. Participants were asked to return for a follow-up visit in 3 months (Visit 3), while those in the intervention were also asked to return for an additional visit in 6 months (Visit 4). In subsequent follow-up visits (Visits 3 and 4), participants were tested for HIV and completed a CASI on-site.
Puerto Rico Research Site Participant Tracking Protocol
To maintain contact during the study participation, participants were asked to provide their personal cell phone number, the cell phone number of a friend or family member, an e-mail address, a mailing address, and their social media (eg, Facebook Messenger) contact information if they had one and were willing to provide it. Participants were asked to choose contact preferences (methods and time of day). Participants were routinely called 1 week before their site visit appointments as a reminder and again the day before the visit to confirm. If necessary, the appointment date and time could be changed as needed by the participant.
Puerto Rico Research Site Disaster/Hurricane Preparedness Protocol and Aftermath of Hurricanes
During the hurricane season, the PR research site had a protocol in place to protect the safety and property of patients, research participants, and personnel. Once alerted of a hurricane by the local weather service, all clinic appointments were canceled until staff could return safely to the site. Researchers called all participants who had appointments that week to cancel appointments and secured participants’ contact information for the next week. Participants were instructed that research staff would contact them as soon as operations resumed.
During September 2017, 2 hurricanes landed in PR. On September 6, Hurricane Irma passed over the northeastern shore of the island, causing flooding, wind damage, and loss of power on a significant part of the island. Reference Zorrilla5 The PR research team was able to return to normal activities during the week of September 11, 2017. Researchers called participants to confirm their visits for that week and to reschedule visits missed due to Hurricane Irma. On September 18, the team received a warning alerting them of a major hurricane, so that day the clinic staff prepared for closure of the research offices, and appointments were again canceled.
The morning of September 20, 2017, Hurricane Maria made landfall as a Category 4 hurricane causing catastrophic damage to the whole island. The entire power grid was destroyed, and cell phone communication was almost completely cut off. In addition, because most antennas were down, there also was no Internet service, no functioning TV stations, and no radio broadcasting, except for 1 station that remained on the air for 72 hours. Running water was cut off because the water pumps had no power. All roads were either flooded or closed due to debris. A night curfew was put in place by emergency authorities to avoid robbery and looting (from discussions with the PR team). Reference Rodríguez-Díaz4,Reference Morabia and Benjamin7
SMS-CASI System
Per protocol, participants received daily text messages at a time of their choosing until the end of their participation in the study. Participants were first sent a reminder that asked about their readiness to report sexual behavior and HIVST kit use to the SMS-CASI system. They were then asked to report daily on the number of CAI occasions, the number of CAI occasions with a negative partner, the number of occasions in which they saw a partner’s negative test results, and the number of unused test kits remaining in their possession. Participants received monetary compensation of US $1 for each completed SMS-CASI session, plus a 50% bonus if they reported at least an average of 6 out of 7 sessions per week.
All procedures were approved by the Institutional Review Boards at the New York State Psychiatric Institute and the University of Puerto Rico Medical Sciences Campus.
MEASURES AND DATA ANALYSIS
We used Hurricane Maria as the time index cut-point for pre-post hurricane analysis because of the significantly greater devastation that it caused to the island of PR as compared with Hurricane Irma. Descriptive statistics for key variables were used to examine and test for frequencies related to participant communication, engagement, retention, and re-engagement. In addition to the daily SMS-CASI, participants completed a baseline assessment on-site via CASIs. The survey included a demographics section with questions regarding age, race, ethnicity, employment/student status, and income. Descriptive data were generated using SPSS Statistics, Version 24.0, 2016 (IBM Corp, Armonk, NY).
Operationalization of Categories and Measurements
Using sent, received, error, and meta-data, we measured 4 categories: engagement, re-engagement, retention, and communication. Engagement, operationalized as maintaining active participation, was assessed by looking at all messages received from the participant during the period of analysis. Re-engagement was operationalized as the participants’ ability to re-connect with the SMS-CASI system after the hurricane and was assessed by measuring the number of days elapsed until participants started to report again. In this way, we were able to measure participants’ capacity to continue participation in the study, despite the difficulties created by the hurricane. Retention was measured according to how many participants remained in the study and included data on participants’ return for their next study visit. It also included instances where participants were actively engaged with the SMS-CASI system within the 30 days following Hurricane Maria. Communication was defined as the receipt of any type of information and was measured using descriptive statistics of text messages that were received from participants, analyzed by context and content. We excluded messages sent to participants from the SMS-CASI system and focused on all instances where the participants tried to communicate with or through our SMS-CASI. This included responses to reminder prompts, general questions sent, and communicating sexual behavior during and after the hurricane.
RESULTS
Characteristics of the Participant Sample
The mean age of the sample was 31.67 (range 18-51, SD = 10.27) years. Most participants identified as male (N = 11; 91.7%) and 1 participant identified as transgender female. Similarly, most participants identified as gay or homosexual (N = 10; 83.3%), with the remainder of participants identifying as bisexual (N = 1; 8.3%) or other (eg, heterosexual transexual female; N = 1; 8.3%). Employment among participants was split, with most participants either working full-time (N = 5; 41.7%) or part-time (N = 5; 41.7%), and a minority unemployed (N = 2; 16.7%). Most participants were not currently in school (N = 10; 83.3%), though a few were (N = 2; 16.7%). Finally, participants’ annual income had a wide range (US $0–$98 000). Four participants reported income of < US $20 000. Of them, 2 reported earning US $0. Two participants reported annual earnings of > US $30 000 (1 reported US $31 000 and the other US $98 000). Two participants didn’t provide data for annual income (Table 1).
TABLE 1 Sample Characteristics
SMS-CASI Functionality
SMS-CASI Performance Data
The SMS-CASI system continued to send out reminder messages from a server in New York during both hurricanes and thereafter. For the 30 days prior to Hurricane Maria, the SMS-CASI system processed 6315 messages from our site in PR (~211 messages/day), of which 3972 were sent by the system, and 1 of which failed. On the day of the hurricane, September 20, 2017, the SMS-CASI system processed 161 messages, of which 110 were sent by the system, with zero failed messages. Finally, for the 29 days following the hurricane, the SMS-CASI system sent 3419 messages, 1 of which failed. It processed a total of 5380 sent and received messages (~185 messages/day). A total of 46 errors (explained below) were recorded before the hurricane, only 1 during the hurricane, and 78 after the hurricane, for a total of 125 (1.97% of total messages) errors detected within the period of analysis. Messages received from the PR participants were classified as engagement and are further discussed later in the “Engagement” section.
SMS-CASI and Participant-Generated Errors
Internal SMS-CASI system assessment tools indicated that, in the period before the hurricane, there were 40 errors generated by participants and 6 errors generated by the CASI system. In contrast, there were 77 errors made by participants and 1 error generated by the SMS-CASI system in the period after the hurricane. On the day of the hurricane, the system generated only 1 error, and there were no errors generated by participants. Participant errors included sending the wrong response type, providing cumulative responses rather than discrete responses, sending the wrong password, and sending invalid responses. The 1 system error was due to a failed data transmission to the SMS-CASI service provider. According to the assessment metrics of our SMS-CASI service provider (Nexmo), all messages received by our service provider were successfully delivered to our SMS-CASI system. However, local reports of damaged infrastructure suggested that there could have been challenges, delays, or failures from the participants’ phones due to issues with their local cellular service providers. An exact number is unknown due to lack of access to the SMS data curated by the participants’ individual local cellular service providers.
Engagement
Total Messages Received From Puerto Rico’s Participants Before and/or After Hurricane Maria
Engagement was measured by the cumulative number of messages received from PR’s participants before, during, and after the hurricane. All participants who were participating in the study during the 30-day periods before and after the hurricane (N = 9) continued to respond to the SMS-CASI system during the time of the hurricane and/or thereafter. Most participants consistently responded to the SMS-CASI system at the time of the hurricane and thereafter, with some decreases and fluctuations (Table 2). For the 30 days prior to Hurricane Maria, participants in PR sent 2343 messages (~78/day). On the day of the hurricane, participants sent 51 messages, and for the 29 days after the hurricane participants sent 1960 messages (~68/day).
TABLE 2 SMS-CASI Use Before and After Hurricane Maria
Re-Engagement
Participant Re-Engagement With the SMS-CASI System Measured as Time to First Response Post-Hurricane
Re-engagement was characterized and measured by the participants’ ability to resume communication with the system after disruption. Three participants had no comparison data because they completed the study before the hurricane. The remaining 9 participants re-engaged with the system at various times during and after the hurricane. The range of time that it took participants to re-engage with the study via the SMS-CASI system was 0 to 16 days; 89% of participants re-engaged with the system within 2 weeks, and 100% of participants re-engaged within 3 weeks. The average number of days needed for re-engagement was 6.4.
Challenges to Re-Engagement After Catastrophic Disaster: A Narrative
When contact occurred with participants, many shared with site staff their traumatic experiences of the consequences of this event in their lives (see section, Communication). Some expressed being anxious during and after the event. There were many simultaneous stressors: lack of power, water, housing, communication, transportation, and health services. However, the greatest burden was the delay in returning to a normal way of life.
Research team members in PR described experiencing back-to-back hurricanes as a catastrophic event and fearing for the safety of their family members, friends, co-workers, and clinic participants. For the first few days following Hurricane Maria, people sought shelter, clean water and food, along with fuel for their cars and, for those who had them, their backup power generators. Despite the catastrophe, staff members described how people helped each other, and workers, doctors, nurses, police, and emergency personnel worked hard to restore some sense of normality. Within each community, in both urban and rural areas, people helped each other and worked day-by-day to stay well. Although the response to the emergency by both local government leaders and the federal government was suboptimal, the Puerto Rican people were resilient and willing to resume a normal way-of-life in the midst of chaos. Clinic staff strove to help the participants re-engage in the study and resume the SMS, which helped assure 100% retention.
Retention
Participants Who Started Before and Returned After the Hurricane
We defined retention as the total number of participants who started before Hurricane Maria hit and continued to report, or re-engaged, with the system after Hurricane Maria (see Table 2). We also included data from study visits (Table 3). Overall, 100% of participants continued participating in the study within the 30 days after Hurricane Maria. Eight (66.6%) of the participants were actively reporting to the SMS-CASI system before, during, and after Hurricane Maria. Three participants (25%) were active before Hurricane Maria but had completed their final study visit before the hurricane. These participants were not included in the retention analysis.
TABLE 3 Participant Follow-Up Visits to Study Site
* First visit to the study site after Hurricane Maria.
Communication
Communication of Study Protocol Behavior
All 6 data elements continued to be communicated from the participant (readiness to report, daily number of unprotected sexual encounters, knowledge of partners’ HIV test results, number of confirmation of partners’ HIV test results, number of test kits left [intervention only], and other general participant-to-researcher communications) post Hurricane Maria.
Reports of Sexual Activity, by Participant
Participants consistently reported whether they were having sexual intercourse without a condom (Figure 1 and Table 4), predominantly reporting that no sexual intercourse without a condom occurred. If their answer to P1 was equal to 0, then P2 was skipped, so P1 – P2 = Instances of no sexual behavior (see Figure 1). Though the data suggest decreased sexual activity in the month after the hurricane, there isn’t enough parallel data for each participant before the hurricane to reliably compare (see Table 4). For instance, participant w1173 (see Tables 2 and 3) began the study after the hurricane and thus has no comparison data and does not show up in Table 4.
FIGURE 1 SMS-CASI Questions in Spanish (With English Translations)
TABLE 4 Counts of Reports by Question Before and After Hurricane Maria
Q1-Q3 = Questions 1-3; w = intervention group; x = control group; (-) = no data available.
Disaster Preparedness Narratives
During the week of October 2, 2017, the research team re-started clinic activities and began to contact participants through a mix of contact modalities (phone calls, text messages, e-mail, and Facebook messages). Direct phone calling was not successful; however, text messages did get through. Researchers contacted all participants in PR who were enrolled at that time to be sure they were safe and to let them know that the clinic was open. Researchers began to reschedule missed appointments and to keep scheduled ones. The participants slowly renewed contact, and, ultimately, no participants were lost to follow-up.
DISCUSSION
Hurricanes Irma and Maria, 2 of the most intense tropical cyclones worldwide of 2017, Reference Zorrilla5 were regarded as the worst natural disasters on record in PR. Reference Rodríguez-Díaz4,Reference Morabia and Benjamin7 Retaining participants is 1 of the most critical challenges of any study. Retaining participants in the wake of a catastrophic natural disaster greatly increases that challenge. The literature on participant retention and re-engagement in the aftermath of a hurricane is sparse. However, Packenham et al. suggest that having a call-in system in place for researchers and participants to communicate is key. Reference Packenham, Rosselli and Ramsey2 Our research on using text messaging confirms that direct lines of communication are important to reach and retain participants and to share critical information. Despite that loss of electricity and damaged telecommunication poles, wires, and antennas hindered communication for both the participants and the local research site, backup generators allowed people to charge their phones, and Google provided cellular service using Loon balloons, which is why cellular communication was the most stable.
In addition, we found that having a mobile, real-time, automated, low-burden communication system, such as SMS-CASI text messaging, greatly aided our participants’ ability to stay connected and engaged in the study. It also aided our ability to observe participant resilience in resuming SMS-CASI reporting, communicate important information, and continue to monitor their sexual and HIV testing behavior. Daily SMS-CASI text reminders and SMS-CASI reports acted as a “heartbeat” system. Some participants resumed responding to the SMS-CASI within 1 to 2 days, which helped staff to know that they were still willing and, more importantly, able and safe enough to participate.
The SMS-CASI system also acted as a direct line of communication from researchers to participants. Though the SMS-CASI reminders and SMS-CASI were automated, SMS-CASI text messages could also be sent manually. We used the system to communicate with participants in cases where they had not responded in 3 or more days. This method was effective in making sure that participants were not lost to follow-up, communicating information to help them troubleshoot issues with the system, and making sure that the participants had a direct line of communication to site staff in real-time. Participants would often confirm that they received the message, using complex or simple responses or an emoji (ie, thumbs-up or smiley face).
Physical and psychological stresses and the decrease in basic resources led participants to shift priorities away from study participation. Reference Packenham, Rosselli and Ramsey2,Reference Dzau, Lurie, Tuckson and Harvey8 Thus, the high level of participant engagement with the SMS-CASI system after Hurricane Maria was unexpected and had 3 very important effects. First, it reduced anxiety and tension among the research staff because it indicated that no participants had been killed or critically injured. Second, it showed that participants cared about the study and wished to stay engaged with the research. Third, it exemplified that Puerto Ricans immediately strove to reclaim their normal lives, even as early as the next day, returning to their routines however they could. Hence, participants’ continued engagement with the study and the SMS-CASI system following Hurricane Maria demonstrated that their level of re-engagement was very high.
Last, the breakdown of civic infrastructure made it difficult for participants to travel to study sites to complete testing, interviews, questionnaires, and other protocols. Both participants and research staff in PR were affected by damaged infrastructure. During the period immediately after Hurricane Maria, there were no participants who needed to travel to the study site. Nevertheless, our experience demonstrated the importance of having protocols in place in future studies for communicating, engaging, and retaining participants in the event of a natural disaster.
FUTURE WORK
Future research should look at the mechanisms that aid participant re-engagement after a natural disaster. Research should also measure the impact of incentives and compensation on participant engagement and retention after a natural disaster. Much of the literature suggests the need to develop emergency protocols for both participant and site staff. Reference Packenham, Rosselli and Ramsey2,Reference Rodríguez-Díaz4,Reference Zorrilla5,Reference Dzau, Lurie, Tuckson and Harvey8 Surveying site staff and participants, including using posttraumatic stress disorder and harm reduction interviewing techniques, could provide insight on those procedures that are critical to participant and researcher safety and re-engagement in the wake of a catastrophic natural disaster.
Limitations
This study was not designed or intended in any way to collect data relevant to Hurricane Maria or any natural disaster. However, use of an SMS-CASI for daily automated reporting provided a rare opportunity to analyze participant behavior in real-time before, during, and after Hurricane Maria. Our sample size of 12 participants limits the statistical strength and generalizability of our results. Unfortunately, only 9 participants were still in the phase of reporting sexual behavior via SMS-CASI immediately before, at the time the hurricane hit, and/or immediately after. Nevertheless, given that we were collecting daily reports via SMS-CASI text messaging, those 9 participants generated 5490 data points during the time period of interest and analysis, which gave us sufficient data to conduct an analysis of the effects of the hurricane on texting behavioral data in a research study.
CONCLUSIONS
The devastation of Hurricane Maria continues to negatively impact the livelihood of Puerto Rican citizens and critical research studies being conducted there. Lack of participant retention is detrimental to the thousands of important and groundbreaking studies in PR that have contributed to its economy and population welfare for generations. Daily SMS-CASI helps participants stay engaged in research studies after hurricanes or similar natural disasters. SMS-CASI text message responses from participants indicated to researchers that participants were not critically injured during Hurricane Maria. Participants were able to stay engaged with research staff in real-time, which had a positive impact on participant participation. Similarly, when participants reported to the SMS-CASI system, it helped curb the psychological stress of research staff. Thus, disaster preparedness protocols are critical for hurricane prone research environments, and daily SMS-CASI text messaging as a reporting protocol may have positive effects in the wake of a natural disaster.
Supplementary material
To view supplementary material for this article, please visit https://doi.org/10.1017/dmp.2020.25
Financial Support
This work is funded by the Eunice Kennedy Shriver Institute of Child Health and Human Development (NICHD) and the National Institutes of Health (NIH) [R01 HD076636] PI: Carballo-Diéguez]. The HIV Center for Clinical and Behavioral Studies is funded by a National Institute of Mental Health (NIMH) center grant [P30-MH43520 PI: Remien]. William Brown III was supported by the National Library of Medicine (NLM) [grant numbers R01-LM012355 PI: Schillinger, T15-LM007079 PI: Hripcsak, R01-LM013045 PI: Bibbins-Domingo] the National Institute on Minority Health and Health Disparities (NIMHD) [grant number P60-MD006902 PI: Bibbins-Domingo], the Agency for Healthcare Research and Quality (AHRQ) [grant number K12-HS026383], and the National Center for Advancing Translational Sciences (NCATS) of the NIH [UCSF-CTSI grant number KL2-TR001870] during various stages of the research and/or preparation of the article. The content is solely the responsibility of the authors and does not necessarily represent the official views of NICHD, NIMH, NLM, NIMHD, AHRQ, NCATS, or the NIH.
