How effective is HVPT for improving speech production accuracy?

In answer to RQ1, small-to-medium effects of HVPT for production gains were found (g = 0.49 for pretest-posttest comparison, g = 0.66 for treatment and control comparison). These findings are consistent with the previous meta-analysis by Sakai and Moorman (Reference Sakai and Moorman2018) focusing on studies of general perception training (d = 0.54 for pretest-posttest comparison; d = 0.89 for treatment and control comparison). The relatively smaller gains for production were observed compared to perception (g = 0.96 for pretest-posttest comparison; g = 0.97 for treatment and control comparison) according to a recent meta-analysis of HVPT for perception learning (Uchihara et al., Reference Uchihara, Karas and Thomson2021).

In answer to RQ2 and RQ3 regarding retention and generalization, the analyses indicate that the effect of perception-to-production transfer may not be long-lasting, and production gains may not be generalizable to untrained stimuli. Although no clear difference was found between posttest and delayed posttest accuracy (g = –0.10), the long-term improvement from pretest to delayed posttest was considered very small (g = 0.26) and appeared to be on a trajectory of regression towards the learners’ starting point prior to training.

As for generalization to new phonetic contexts/words, the medium effect (g = 0.61) was observed for learning old items (i.e., stimuli that appeared during training), whereas a marginally significant and small effect (g = 0.20, p = .091) was found for learning new items (i.e., stimuli that did not occur during training). These findings contrast with previous meta-analysis findings for perception learning (Uchihara et al., Reference Uchihara, Karas and Thomson2021) that found, for retention, perception at the delayed posttest was much more accurate than perception at the pretest (g = 0.98) and a negligible difference was observed between posttest and delayed posttest (g = –0.08, p = .058). For generalization, perception gains for old items and old talkers (g = 0.91) and for new items and new talkers (g = 0.96) were almost comparable.

In answer to RQ4 regarding the correlational link between perception and production gains, the results of correlation analyses between perception and production were not consistent across levels of analysis. At the participant level, a nonsignificant and negligible relationship was found (r = .09). At the study level, a significant and medium-sized relationship (r = .45) was found for perception and production gains (comparable to the finding of Sakai & Moorman, Reference Sakai and Moorman2018: rho = .39), and a larger correlation was observed for retention data (rho = .78). Although the relatively large correlations observed at the study level seem to suggest the strong link between speech perception and production, the results need to be interpreted with caution. The study-level results may simply indicate that the studies with the higher quality of training setup and implementation are likely to improve both perception and production accuracy (Thomson, Reference Thomson2012b), not necessarily suggesting a causal relationship between perception and production.

The findings can be summarized as follows: (a) perception-only training led to moderate production gains; (b) strong support for the retention of production gains was lacking; (c) production learning was not generalized to new items; (d) the correlation between perception and production gains at the participant level was negligible, although moderate-to-large correlations were found at the study level. These findings collectively do not provide strong evidence for perception-to-production transfer as a result of exposure to high variability input with corrective feedback.

The failure of transfer may be largely because of the time-lagged nature of the relationship between perception and production development (Nagle, Reference Nagle2018, Reference Nagle2021; Nagle & Baese-Berk, Reference Nagle and Baese-Berk2022). According to the SLM (Flege, Reference Flege and Strange1995a) and SLM-r (Flege & Bohn, Reference Flege, Bohn and Wayland2021), the mechanisms underlying L1 speech learning are still available to L2 learners, but our findings suggest that accessing production mechanisms, while still possible, is more difficult than accessing perceptual mechanisms. This is likely because the two mechanisms are distinct despite being connected. Perceptual changes are cognitive, while articulatory processes include both cognitive and physical components. On the articulation side, motor movements in a learner’s L1 are so automatic that initiating a new articulatory command is difficult after a certain age (Scovel, Reference Scovel1969). This is true of motor skills that are unrelated to speech, such as athletes attempting to change an established running gait or golfers modifying their swings, etc. Relatedly, the time lag between improvement in L2 perception and production may be accentuated to a greater extent, given that learners’ own productions may become the primary input to their new L2 system. L2 listeners without articulatory control fine-tuned to the L2 system may bring their own perceptual distortions to the production task (Thomson, Reference Thomson, Derwing, Munro and Thomson2022a). Conversely, the current findings did not provide strong support for the gesturalist claim that perception of L2 speech relies on knowing how to produce the sound using articulatory gestures (PAM-L2: Best & Tyler, Reference Best, Tyler, Munro and Bohn2007). If perception learning required the perceiver to fully understand how to use the articulatory gestures, a stronger perception-to-production transfer in terms of immediate gains, retention, and generalization would have been observed.

Alternatively, the correlational data at the participant level suggest a stronger link between speech perception and production at a fixed point in time. Despite the absence of a significant correlation for gain scores, significant and moderate overall correlations were found for pretest (r = 0.41) and posttest (r = 0.33) data. Because these correlations do not reflect perception and production gains as a result of training, the findings indicate that learners who can accurately perceive target sounds tend to produce the sounds accurately at a fixed point in time. These findings, especially the fact that the correlation at the pretest is slightly higher than that at the posttest, imply that the perception-production link may emerge on the robust and stable representations developed through past L2 experience for a long term. In order to see a stronger link in training data, a rigorous longitudinal study tracking the progress in perception and production over an extended period of time is warranted (e.g., Nagle, Reference Nagle2018).

What factors moderate the transfer of HVPT to L2 production?

To answer RQ5, we explored the effects of nine categorical and four continuous variables on production improvement. L2 proficiency and target phones were not found as significant variables, indicating that HVPT seems to be beneficial in production learning for learners with different proficiency levels and for various target sounds including segmental (i.e., vowels and consonants) and suprasegmental features (i.e., lexical tones and syllable structures). This aligns with an exemplar view of speech learning, or Flege (Reference Flege and Strange1995a)’s emphasis on the role of relative experience with oral language. While proficiency in English varies across learners, this is often rooted in differences in written language proficiency (especially in FL contexts), which will not provide the sort of auditory input necessary for the development of L2 speech categories. As such, an L2 learner may develop advanced proficiency in reading, vocabulary, and grammar without much experience in listening. Such advanced proficiency learners appear to benefit from HVPT as much as lower-proficiency learners.

Rater experience did not significantly affect perceived intelligibility of L2 production. A slightly larger effect size for experienced raters (g = .44) compared to inexperienced raters (g = 0.36) may point to a higher consistency for experienced raters (Isaacs & Thomson, Reference Isaacs and Thomson2013). However, the small and nonsignificant difference suggests that the choice of experienced or inexperienced raters does not appear to substantially affect the evaluation of L2 production accuracy. Both experienced and less experienced raters were equally reliable. The number of talkers during training was not a significant predictor, suggesting that increasing or decreasing talker variability within the range of 3 to 30 may not affect the transfer of HVPT to production. A minimum of 3 talkers may be sufficient for significant production gains to be observed. However, the majority of studies limited the number of talkers from 3 to 6 (k = 37); thus, talker effects with greater numbers need to be explored in future research before the optimal number of talkers is determined. On the other hand, other variables related to learner profiles (learning context and AOT), training features (phonetic information, duration, and training time per session) were found to have a major impact on production gains. Further, choice of assessment for production tests (outcome measures, elicitation tasks, and prompt modality) resulted in differential findings. In what follows, findings of these key variables (learning context, AOT, phonetic information, training duration, training time per session, and assessment) will be discussed in greater detail.

Learner features

Regarding the context of learning, the relatively larger effect for FL (g = 0.60) compared to SL (g = 0.34) indicates that HVPT was likely to be less impactful on L2 production for SL learners than FL learners. The smaller benefit of perception training for SL learners may be attributed to their higher degree of L2 experience developed through immersion in naturalistic settings (Iverson et al., Reference Iverson, Pinet and Evans2012). Eleven studies reporting information about the mean length of residence showed that SL learners in our meta-analysis were considered relatively experienced in terms of the length of residence in the target language country (M = 18 months, range = 1–49 months). According to the hypothesis of a window of maximal opportunity (Derwing & Munro, Reference Derwing and Munro2015), substantial phonetic learning occurs during the early period of residence in the L2-speaking country. Thus, it is possible that SL learners’ perception begins to plateau after a period of immersion, or at least further increases are so modest (e.g., Derwing & Munro, Reference Derwing and Munro2013) that it would take much longer to detect change than the short durations afforded by the studies considered here. Perceptual categories that have already changed and stabilized may not improve as much from perception training when compared with less developed representations of FL learners, which may cause perception training to be less impactful on SL learners’ production (Nagle & Baese-Berk, Reference Nagle and Baese-Berk2022).

The negative relationship between AOT and production gains indicated that older adult learners tended to benefit less from HVPT than younger adult learners in the current data of participants whose age ranged from 12 to 37. One possible reason for this finding is that it may be more challenging for learners with established L1 perceptual knowledge to improve L2 production accuracy. Given that older learners are assumed to receive a greater amount of L1 input and develop a more robust L1 categorical knowledge (Flege, Reference Flege and Strange1995a), the interference from L1 knowledge with L2 category learning can be considered larger. As the L1 becomes more robust, older learners may have more difficulty discriminating phonetic differences that exist crosslinguistically, which might reduce the beneficial impact of training on production outcomes. It is important to note, however, that the mean effect of age does not mean that older learners cannot benefit. For example, the oldest learner (fifty years old) in Thomson (Reference Thomson2011) evidenced among the greatest improvement in production of the participants in that study.

Perception training features

Provision of phonetic information had a positive impact on the training effect on production gains (g = 0.94) compared to no phonetic information (g = 0.45). These findings support the meta-analysis of general perception training on production improvement (Sakai & Moorman, Reference Sakai and Moorman2018) showing that a large effect was observed for learners presented with phonetic information (d = 0.99) compared to those without such information (d = 0.40). Explicit attention to and knowledge of target forms are likely to contribute to the development of L2 production accuracy (Saito, Reference Saito2019), and drawing learners’ attention to target forms increases the efficacy of L2 phonetic training (Guion & Pederson, Reference Hazan, Sennema, Iba and Faulkner2007). The robustness of these findings suggests an important pedagogical implication for L2 speech learning as presenting phonetic information about target sounds prior to having learners complete perception training can optimize the transfer of HVPT to production. This may be due to their ability to apply explicit knowledge to production, particularly in highly controlled speaking tasks, which predominated the studies included in this meta-analysis.

Regarding training duration, training that lasted for one month or longer (g = 0.32) was less effective than training that lasted for shorter than one month (g = 0.61). Scheduling training programs for a longer period of time may not necessarily bring about the most benefit for production learning (Wong, Reference Wong2013). A recent meta-analysis by Kim and Webb (Reference Kim and Webb2022) suggests that a shorter spacing interval during learning is particularly beneficial, in light of the high degree of complexity involved in the learning of pronunciation. Given that studies scheduling a longer-term training program tend to provide longer intervals between training (e.g., 8 sessions within 2 weeks in Lee & Lyster, Reference Lee and Lyster2017 vs. 5 sessions within 5 weeks in Carlet, Reference Carlet2017), in such studies, learners might have been exposed to training stimuli with a longer spacing interval between training sessions. When the spacing is longer, learners may have difficulty accessing phonological information during subsequent exposures to auditory input, which might have reduced the efficacy of perception training.

Training time per session was negatively associated with production learning, indicating that the longer the time per training session was, the lower the production gain was. Despite the significant association, the scatterplot for this relationship (see Figure 6) showed variability with several studies deviating from the regression line. This figure appears to reveal that studies around 25 minutes consistently produce relatively higher effect sizes (g = 0.44 to 1.62) after which effect sizes seemed to drop at around 30 minutes (g = 0.17 to 0.96) and afterward with a few exceptions. The data tentatively suggest that perhaps 20 to 30 minutes may be appropriate for the amount of training time per session in order to maximize the training benefit for production learning. Given that phonetic training tends to be tedious without any incentive (e.g., giving monetary rewards for correct responses in Bradlow et al., Reference Bradlow, Akahane-Yamada, Pisoni and Tohkura1999), training for 30 minutes or longer may not necessarily increase the effect of training (Logan & Pruitt, Reference Logan, Pruitt and Strange1995).

Figure 6. Scatterplot of training time per session (min) and production gains (Hedges’ g).

Features of production tests

The moderator analysis of outcome measures showed a larger training effect, in the order of transcription (g = 1.14), identification (g = 0.45), human rating (g = 0.28), and acoustic analysis (g = 0.18). The nonsignificant and small effect for acoustic analysis is not consistent with previous meta-analyses of L2 speech learning (Sakai & Moorman, Reference Sakai and Moorman2018; Saito & Plonsky, Reference Saito and Plonsky2019), but the result needs to be interpreted with caution due to small sample size (k = 3). A relatively large effect size for transcription from small sample size (k = 4) also needs to be interpreted with caution. The result may be confounded by other moderator variables. Particularly, the large effect was mainly attributed to two effect sizes (g > 2.0) from a single study (Wong, Reference Wong2013), which focused on younger learners (M = 16.3 and 16.4 years old vs. overall mean across all studies = 23.9 years old) in the FL context and the phonetic training lasted for 15 minutes, shorter than the overall mean across all studies (M = 35.8 minutes).

A smaller effect was observed for human ratings compared to intelligibility measured by native speaker forced-choice identification. To further examine the small effect for human ratings, we examined the rating scales used in 11 relevant studies adopting scalar rating measures. As summarized in Table 2, researchers measured three constructs of L2 pronunciation proficiency in terms of general accuracy (e.g., correct vs. incorrect), nativelikeness (i.e., how different L2 speech is from an L1 variety), and comprehensibility (i.e., how easily L2 speech is identified). When general accuracy was further specified in the manuscript or rating labels (e.g., Macdonald, Reference Macdonald2012 for nativelikeness), such studies were assigned to the specific construct (nativelikeness or comprehensibility). In one study (Lopez-Soto & Kewley-Port, Reference Lopez-Soto and Kewley-Port2009), the scale descriptors focused on both nativelikeness and comprehensibility in the same scale, which was thus labeled as nativelikeness & comprehensibility. The analysis of average effect size descriptively demonstrates that the training effect tends to be smaller when nativelikeness rating was used (g = 0.09) compared to comprehensibility (g = 0.36) or accuracy (g = 0.39). The relatively smaller effect of training for nativelikeness compared to comprehensibility and intelligibility is consistent with earlier studies showing that L2 oral production becomes more comprehensible and intelligible with continued exposure to L2 spoken input (Derwing & Munro, Reference Derwing and Munro2013; Uchihara et al., Reference Uchihara, Webb, Saito and Trofimovich2022) and explicit instruction (Derwing et al., Reference Derwing, Munro and Wiebe1998), whereas development of native-like pronunciation tends to be slow and gradual (Saito, Reference Saito2015). Accordingly, perhaps the relatively smaller effect for human rating in this meta-analysis may have been attributed to the inclusion of studies focusing on nativelikeness.

Table 2. Summary of 11 studies using human rating to measure production accuracy of L2 sounds

Note: Multiple effect sizes were averaged to yield a single effect size per study. Some researchers stated they measured “intelligibility” but it seems to conflate it with nativelikeness (e.g., Dong et al., Reference Dong, Clayards, Brown and Wonnacott2019; Trakantalerngsak, Reference Trakantalerngsak2016).

Elicitation tasks and prompt modality

Regarding the production elicitation tasks, inconsistency in the effect of training depending on the assessment task used to measure production indicates that the tasks are incommensurate and likely measuring different phenomena. A much larger effect of training was observed in the order of word reading (g = 0.72), recalling (g = 0.58), sentence reading (g = 0.45), and repetition (g = 0.28). These findings can be explained in terms of varied cognitive demands present across tasks. Word reading is the least cognitively demanding and allows learners to direct their explicit attention to target sounds; hence, production gains as evaluated by a word reading task might be tapping into changes in learners’ declarative knowledge rather than procedural knowledge (Thomson, Reference Thomson, Derwing, Munro and Thomson2022a; Thomson & Derwing, Reference Thomson and Derwing2015). In contrast, a word recall task may involve a slightly higher degree of retrieval effort and pronunciation may be at least partially influenced by procedural knowledge. Similarly, sentence reading tasks also provide greater distraction than word reading tasks, allowing less reference to declarative knowledge. In contrast, a delayed repetition task forces learners to move phonetic information obtained from the prompt into long-term memory and then retrieve it after an interruption. This means that the resulting production is more likely to be an indication of their developing interlanguage system. The effect of differential demands on attention on production accuracy was confirmed by the result that prompts providing only auditory input led to a smaller effect (g = 0.19) than when orthographic input is provided (g = 0.62 for written input; g = 0.56 for auditory + written input). Recalling and repetition tasks are similar in many respects except for the provision of auditory input. In terms of cognitive load, a similar degree of cognitive demands may be required from both recalling (e.g., presented with a card on which a test word is printed, then turned over, and asked to recall and pronounce the word in Hardison, Reference Hardison2003) and delayed repetition (e.g., pronounce after white noise in Dong et al., Reference Dong, Clayards, Brown and Wonnacott2019 or after 3000ms in Aliaga-García, Reference Aliaga-García2017). Despite this similarity, a much smaller effect was found for repetition (g = 0.28) compared to recalling (g = 0.58), suggesting that the exposure to auditory stimuli, regardless of efforts to mitigate the influence of spoken input, may induce a high degree of phonological involvement with target sounds (Llompart & Reinisch, Reference Llompart and Reinisch2019) and impact production outcomes.

Ultimately, we take the view that the choice of assessment should be determined by the goal of the assessment. If the goal is to test declarative knowledge, then reading tasks and other highly controlled tasks will work well. If, however, the goal is to evaluate the extent to which changes in perception manifest in changes in procedural knowledge in production, then less controlled tasks, such as delayed repetition or even spontaneous speaking tasks should be used. The latter should be considered the gold standard, while the former’s usefulness may be primarily limited to formative assessment. This task hierarchy is evident in Thomson and Derwing’s (Reference Thomson, Derwing, Levis, Le, Lucic, Simpson and Vo2016) HVPT perceptual training study in which they found improvement in production in a delayed imitation task, but not in a more spontaneous production task which elicited the same targets in sentences that the learners were asked to create on the fly.

Conclusion

The primary goal of the current meta-analysis was to examine to what extent HVPT brings about perception-to-production transfer in terms of gains, retention, and generalization, and which variables related to learner, training, and outcome measures modulate the overall effectiveness of HVPT. The present meta-analysis provided partial support for a relationship between L2 speech perception and production. It provides evidence for the transfer of HVPT to production with small-to-medium effects and further supports the significant relationship between perception and production learning and retention at the study level. However, we did not find strong support for long-term retention of production learning and generalization to untrained stimuli, nor a significant perception-production link for gain scores at the participant level. In addition, several learner-related variables were found to influence production gains, and methodological choices regarding how to assess production also impacted the magnitude and/or type of gain evidenced. These findings provided insights into various factors contributing to the efficacy of HVPT for L2 speech production learning and its measurement.

With the aim of optimizing the effectiveness of HVPT in improving L2 production, the findings of this meta-analysis have several pedagogical implications. First, the weak evidence for the production transfer in this meta-analysis suggests that delaying the introduction of explicit production training may be worth considering. It is possible that without more finely tuned perceptual categories, attempts to match production against emergent perceptual categories break down. The learners’ imprecise productions would become perceptual input, leading to the erosion of perceptual gains (Thomson, Reference Thomson, Levis, Derwing and Sonsaat Hegelheimer2022b). As such, there could be a benefit of delaying production practice until the perceptual system is capable of self-monitoring production. However, it should be noted that the current meta-analysis exclusively focused on perception training without production training; thus, the extent to which an additional component of production practice promotes (or hinders) the transfer remains unknown. We hope that, as the studies increase in number, a future meta-analysis will focus on production HVPT studies while considering a variety of training features revealed by some successful cases, such as combinational effects (perception+production vs. production; Herd et al., Reference Herd, Jongman and Sereno2013), training task (e.g., imitation task; Aliaga-García, Reference Aliaga-García2017), visual input (waveform inspection; Herd et al., Reference Herd, Jongman and Sereno2013), and stimuli status (nonwords vs. real words; Mora et al., Reference Mora, Ortega, Mora-Plaza and Aliaga-García2022).

The current findings also offer practical guidance with the goal of optimizing the production transfer. Practitioners should note that production learning is enhanced when HVPT involves (a) a focused and intensive training program (e.g., 20 to 30 min sessions for less than one month), (b) provision of explicit phonetic information that promotes production learning, (c) younger L2 learners, and (d) learners studying L2 in foreign language contexts. It is also important to note that HVPT is better suited to improve explicit knowledge related to L2 speech production (elicited via word reading), while whether it improves procedural knowledge needed for spontaneous speech production remains uncertain.

Lastly, there are several suggestions for future HVPT studies with a view to improving methodological soundness and rigor in the investigation of the perception-production link. First, based on the finding that the average sample size was low (N = 17.06), future HVPT research should endeavor to have more participants (for the same call, see also Sakai & Moorman, Reference Sakai and Moorman2018). Given the great amount of time and cost required for the implementation of HVPT experiments, researchers may find online experiment builders such as Gorilla (Anwyl-Irvine et al., Reference Anwyl-Irvine, Massonnié, Flitton, Kirkham and Evershed2020) or online training platforms (e.g., English Accent Coach, Thomson, Reference Thomson2012a) useful. Second, based on the findings that interrater reliability was not reported in many studies (unreported rates: 57% for native speaker identification, 38% for scalar ratings, 50% for transcription), we urge that future studies should report the pretest and posttest interrater agreement and evaluate the impact of test reliability on their results. Third, spontaneous tasks should be used to elicit L2 speech production. In this meta-analysis, all studies predominantly relied on using controlled production tasks (reading aloud, recalling, and repetition), reflecting the same trend in the L2 pronunciation literature in general (Thomson & Derwing, Reference Thomson and Derwing2015). Using spontaneous tasks such as picture naming (e.g., Nagle, Reference Nagle2021) is important from a perspective of ecological validity and will provide additional insights into the multidimensional view of the perception-production link. Fourth, the construct definition of production accuracy needs to be specified in adopting scalar rating measurements. In some studies, both features of nativelikeness and comprehensibility or intelligibility were mentioned in a single descriptor or rating label. Given that the constructs of nativelikeness, comprehensibility, and intelligibility are confirmed to be independent at the sentence (Derwing & Munro, Reference Derwing and Munro2009) or word (Uchihara, Reference Uchihara2022) level, it is advisable to treat them separately in measuring global constructs of L2 pronunciation proficiency. Further, general accuracy (i.e., correct vs. incorrect distinctions) may need to be defined more clearly. Such a broad description of L2 pronunciation accuracy may cause variations in the way listeners interpret what constitutes accuracy, given that the intended message may not be clear—“was the intention ‘close enough to be recognized as a particular phoneme’ or was it ‘native-like versus non-native-like’?” (Thomson & Derwing, Reference Thomson and Derwing2015, p. 337).