Teachable moments

Prior research on “teachable moments” has found that the success of interventions to change behavior can depend on the timing of the intervention's delivery (see Lawson and Flocke (Reference Lawson and Flocke2009) and McBride et al. (Reference McBride, Emmons and Lipkus2003) for a conceptual review). For example, hospitalization has been identified as a propitious time to implement interventions aimed at smoking cessation (Emmons and Goldstein, Reference Emmons and Goldstein1992), alcohol use disorder (Gentilello et al., Reference Gentilello, Villaveces, Ries, Nason, Daranciang, Donovan, Copass, Jurkovich and Rivara1999) and suicide (O'Connor et al., Reference O'Connor, Mcclay, Choudhry, Shields, Carlson, Alonso, Lavin, Venanzi, Comtois, Wilson and Nicolson2020). Some research suggests, moreover, that the length of time during which a teachable moment can be exploited is brief, emphasizing the importance of timing. One prior study found that the impact on the future behavior of attendance at a counselling appointment by patients who abused alcohol was twice as great when appointments occurred less than two days following an emergency department admission compared to those delayed by two days or longer (Williams et al., Reference Williams, Brown, Patton, Crawford and Touquet2005). The authors conclude that the “half-life” of the teachable moment in this situation was two days.

Just-in-time feedback

A recent movement, mainly in employee management but also in health and education behavior change, involves giving people real-time performance feedback. The assumption underlying just-in-time feedback is that it increases the likelihood that individuals learn associations between an action and its effect. Such feedback can have a greater impact on behavior than the delayed, summarized, feedback that is typically given in, e.g., annual performance reviews. In the domain of behavior change (of which preventing succumbing to phishing would be an example), two papers, both focusing on diet, have shown that just-in-time feedback is effective in changing behavior. Researchers in one study observed a substantial reduction in food intake when dieters were given immediate feedback from a wearable sensor that tracked their ‘chew count’ and were given the goal of reducing chew counts by 25% (Farooq et al., Reference Farooq, McCrory and Sazonov2017). They found that just-in-time feedback reduced the amount of food consumed, with no changes in perceived satiety.

The second paper is a review of the methods and results of 31 different just-in-time interventions targeting diet and physical activity. Conclusions from these collective findings were that the most successful interventions involved feedback that was continuously available, personalized, actionable and coupled with a well-defined behavioral objective (Schembre et al., Reference Schembre, Liao, Robertson, Dunton, Kerr, Haffey, Burnett, Basen-Engquist and Hicklen2018).

Papers addressing the modality and timing of feedback extend beyond health behaviors and management into other realms of human behavior. For instance, in the context of energy consumption, Zangheri et al. (Reference Zangheri, Serrenho and Bertoldi2019) conducted a comprehensive analysis of over 70 studies and concluded that customizing feedback to energy users can result in changes in energy consumption behavior and drive investments in sustainable energy use. However, the success of feedback was found to be dependent on various factors including the medium of feedback delivery, the geographical context and the timeframe in which feedback was provided. In a different paper, Schwartz and Loewenstein (Reference Schwartz and Loewenstein2017) documented the transient impact of emotions produced by affect-inducing stimuli on behaviors intended to combat climate change. While sadness-evoking messages initially led to increased pro-environmental behaviors, such as more engagement with an energy-footprint calculator and heightened donations, these behaviors diminished as the emotional arousal subsided. In contrast, non-emotion-evoking interventions produced much lower initial behavioral responses, but these persisted more over time. There was greater behavioral persistence in response to the emotion-inducing stimulus; however, in a treatment reported in a follow-up study (Study 4), in which participants could make non-binding behavioral commitments immediately following exposure to the emotion-evoking stimuli. These results point to the importance of feedback timing not only in relation to an initial behavior but also in relation to some desired action or commitment to change behavior in the future. We review relevant work on just-in-time feedback in phishing detection and cybersecurity in the next section.

Teaching

Perhaps the most common type of anti-phishing intervention involves teaching and training. Many organizations offer mandatory or voluntary programs to alert employees to the threat of phishing and instruct them about how to respond. One study examined the impact of a mandatory anti-phishing training program provided to 5,416 employees, at a U.S. healthcare institution, who were classified as “offenders” and “nonoffenders” based on their click rates in response to a series of 20 sham phishing attacks (Gordon et al., Reference Gordon, Wright, Glynn, Kadakia, Mazzone, Leinbach and Landman2019). Click rates declined over the course of the study for both groups, most likely as a result of cumulative exposure to the sham attacks. However, there was no evidence that the training program produced a more rapid decline in desired (non)responses to phishing.

Gamified simulations

A specific form of teaching involves computer-based games intended to train individuals to spot phishing attacks (Arachchilage and Love, Reference Arachchilage and Love2013). One group of researchers describes the design and evaluation of an online game, “Anti-Phishing Phil,” that teaches users to identify phishing sites and techniques to avoid phishing attacks (Sheng et al., Reference Sheng, Magnien, Kumaraguru, Acquisti, Cranor, Hong and Nunge2007). Evaluating the game in a small study, and comparing it to other interventions, they found that users were better able to identify fraudulent websites immediately after spending 15 min playing the game. All of the interventions led to greater success in resisting phishing, but the game was no better than the alternative interventions that were tested, and the immediacy of the testing raises questions about the persistence, in real-world settings, of any of the interventions.

Warnings

A somewhat weaker and less-expensive intervention involves simply warning individuals about the threat of phishing. Not surprisingly, given the common failure of more elaborate training, such warnings have not proved very effective. In one experiment, for example, 290 visitors to a shopping district in the Netherlands were randomly assigned to receive a warning, a priming intervention or neither (Junger et al., Reference Junger, Montoya and Overink2017). They were then asked for their email address and nine digits from their 18-digit bank account number. The warning consisted of a leaflet which began with the bold text “Beware of Phishing!” and then briefly explained what a phishing attack is and what it is designed to do. It also included an admonition: “Never share personal or banking information with anyone!” (emphases from the source). In the priming condition, shoppers were asked, before the sensitive information was solicited, a series of questions gauging their familiarity with phishing and internet security. Relatively high rates of disclosing sensitive information were observed across all conditions: 79.1% of the subjects filled in their email address and 43.5% provided bank account information. However, despite the short time interval between the intervention and the measured behavior, neither the priming questions nor the warning influenced revelation rates.

Related work has investigated the optimal timing of warnings to encourage users to set stronger passwords. Qu et al. (Reference Qu, Xiao and Shi2023) investigated the optimal timing of security warnings in online games and found that warnings presented after, rather than before, a profile set-up activity effectively shifted participants toward stronger security practices in one of two framing conditions tested.

Punishments

In some environments in which phishing is especially damaging, more heavy-handed interventions involving punishments for succumbing to a phishing attempt may be effective. In one study illustrating this potential (Kim et al., Reference Kim, Lee and Kim2020), a sham phishing email was sent to employees at a medium-sized manufacturing company. Employees who succumbed to the phishing scam were then randomly assigned to either a punishment condition or a no-punishment control group. The punishment was heavy-handed: employees who succumbed to the phishing attack were visited by a member of the employer's security team, temporarily lost access to the intranet, had to submit a note explaining their misbehavior and were informed that they might receive a negative annual performance evaluation. This punitive intervention significantly reduced susceptibility to a second pseudo-phishing attack by an impressive 25 percentage points. However, even after this reduction, the rate of succumbing to the second phishing scam by the punished group was still roughly twice as high as the succumbing rates of employees who had not succumbed to the initial phishing scam. Moreover, the penalizing nature of the intervention would seem unlikely to be implemented in a broad range of workplace settings.

Just-in-time feedback

Several papers report on the implementation of just-in-time feedback delivered to individuals who are exposed to, and fall prey to, simulated phishing attempts (Dodge et al., Reference Dodge, Carver and Ferguson2007; Jakobsson et al., Reference Jakobsson, Johnson and Finn2008), although research to evaluate the efficacy of this approach is less common. Moreover, much of the research that has been done to evaluate the efficacy of feedback following a simulated phishing attempt suffers from methodological limitations. For example, Jansson and von Solms (Reference Jansson and von Solms2013) used a pre-post designFootnote ² (with no control group), in which phishing emails were sent to 25,579 faculty users of an academic email system, followed by instant warning messages to the 1,304 users who succumbed to the attack, followed by a second set of sham emails to the original 25,579 recipients. Although there was a decline in adverse responses to the second test-phishing email, there was also a decline in active users of the email system (faculty who actually read their email), which made it difficult to interpret this result, a problem that could have been mitigated by the inclusion of a no-warning control group.

Two studies in the prior research on phishing prevention are closest in design to our work. In one study (Caputo et al., Reference Caputo, Pfleeger, Freeman and Johnson2013), researchers sent three sham phishing emails to 1,359 employees at a large organization. Employees who succumbed to at least one of them were randomized to either a control group, which received no offer of training, or four treatment arms, each of which offered training that varied in the framing of the message offering the training. Specifically, the offer message varied gain vs loss frame language and whether it focused on the risk for the individual vs their coworkers. The intervention did not have a statistically significant effect; there were no statistically significant differences in reporting or click rates between the differently framed warning messages or, more importantly, between the average of the warning message groups and the no feedback control.

In a later, but not as methodologically strong study (in part, because it lacked a no-message control condition), Burns et al. (Reference Burns, Johnson and Caputo2019) sent a sham phishing email to 260 participants enrolled in a Master of Business Administration (MBA) program. Subsequently, to the 70% of students who succumbed to the attack, they provided links to one of five brief training modules. All of the experimental treatments informed participants that they had been spear phished, and four also provided messages about why it is bad to succumb to a spear phishing attack. Finally, a second phishing email was sent to all 260 individuals who received the first message plus a new no-randomly assigned sample of 140 individuals who had not received the original email. Although no significant differences were found between the randomized treatment groups, given the small sample and the lack of a comparable no-intervention control group, the lack of positive results should be treated with caution.

We build upon this existing work by testing the efficacy of an intervention that provides just-in-time feedback at a teachable moment on phishing susceptibility with a large sample and a no-intervention control condition. Although our primary contribution is to test the efficacy of this intervention in the realm of online security, in so doing we also provide additional evidence on the impact of behavioral interventions that leverage just-in-time feedback and teachable moments in a field setting where the impact of such an intervention can be clearly evaluated.