Research Question 1: What variety/varieties of English does the CET use?

The study echoes Davies et al.'s (Reference Davies, Hamp–Lyons and Kemp2003) main finding that the CET relies, although not consistently, on the Inner Circle standard. While topics and accents in the selected-response items are dependent on the Inner Circle context, global and local varieties have also been included, especially in the constructed-response tasks.

Inner Circle varieties

Inner Circle topics in Listening and Reading

As shown in Figure 1 and 2, most of the Listening and Reading materials in the CET could be related to the Inner Circle, especially US and UK. About 50% in Listening and 70% in Reading were identified as Inner Circle topics, among which US topics accounted dominantly for about 70% and 80% in Listening and Reading, respectively. UK topics ranked the second but accounted for a much smaller proportion. A few were relevant to other Inner Circle countries such as Canada, Australia, and New Zealand, but the percentage was rather negligible.

Figure 1. Topics in listening materials by region (N = 180)

Figure 2. Topics in reading materials by region (N = 144)

The Specifications (2016) designates ‘original English-language materials’ (p. 1) as the sources, which could partly explain the dominant percentage of Inner Circle topics. In fact, most sources of the reading texts could be traced to Inner Circle newspapers, magazines, and websites (see Figure 3), with the top sources being NPR News, Interesting Engineering, BBC, The Guardian, and The New York Times. Of course, some of the sources target a more global audience (e.g. Interesting Engineering), but no local media sources were found.

Figure 3. Sources of reading materialsFootnote ⁵ (N = 144)

The test's adaptation to the original sources mainly focused on text length and difficulty at the linguistic level, preserving the content and tone as addressing mainly the Inner Circle audience. Therefore, much content information could assume background knowledge. For instance, a reading passage (CET4_12_19[1]) opens with ‘a polar wind brought bitter cold to the Midwest’ without mentioning the ‘Midwest’ of which region, apparently composing for a local audience (in the US, which could be decided based on later message that contained regions and businesses in the country, e.g. Chicago, USPS, etc.). Many topics are hardly relevant to the Chinese culture and can be unfamiliar to Chinese test takers. For instance, a reading passage (CET4_12_18[1]) about healthy lifestyle discussed only Western dishes, such as cereal, frozen oranges and apples, and macaroni-and-cheese, which are rarely seen in the Chinese diet. Another example (CET4_12_19[2]) discusses the expensive E-textbook industry in the US, which can sound strange to the Chinese students who typically purchase paper books that are rarely found to be expensive. This unselective adoption of original materials can lead to validity and bias concerns, which will be further discussed under Research Question 3.

Inner Circle accents in Listening

The Specifications (2016) openly states that the listening assessment ‘uses standard American and British accents’ (p. 6). This statement may not be easy to interpret, because in China, American and British English varieties are ‘often mixed without distinction’ (Davies et al., Reference Davies, Hamp–Lyons and Kemp2003: 577). Even when they are used in distinction, the definition of such standard accents can differ. In a narrow sense, they refer to the American and British accents spoken by educated NSs in the U.S. and U.K. In a broad sense, each may also contain other varieties in the Inner Circle; for instance, ‘standard American accent’ can also refer to educated Canadian accent and ‘standard British accent’ to educated Australian and New Zealand accents. Nevertheless, based on the test specification, it is obvious that the test is intended to use Inner Circle accents in Listening. An examination of the sample and real tests revealed that the listening test relied exclusively on varieties of Inner Circle accents, and no Chinese or other accents were identified.

Global and local varieties

Listening and Reading

Although relying on Inner Circle sources, the CET listening and reading passages also included topics related to global contexts (see Figures 1 and 2), particularly in CET-4. Specifically, 50% (n = 90) of the listening materials could be identified as global, with 22 passages situated in European settings (e.g. Italy and France), 5 in AsiaFootnote ⁶, and others undefined; 42% of the reading materials could be identified as global, with 8 passages situated in Europe, 5 in AsiaFootnote ⁷, and others unidentified. Although a small portion, it is worth noting that two topics involving Chinese culture were found in Reading – one (CET4_12_18[3]) discusses the writer's experience of having a Chinese medicine treatment in a China Town; the other (CET4_06_18[3]) is about Neon lights in Hong Kong. The statistics indicate that global topics in general account for a large percentage of the topics in Listening and Reading. Even though topic selection within the global category relies predominantly on European and other settings, the inclusion of global cultures that have more contact with China (e.g. Asian topics) as well as the local Chinese topics can serve as a good starting point and example for further diversifying topic selection.

Writing

Most of the writing prompts are, as found in Davies et al. (Reference Davies, Hamp–Lyons and Kemp2003), similar with the TOEFL writing test (p. 578), which are mostly argumentative as shown in the following example (CET6_12_19[1]):

1. (1) Write an essay on the importance of having a sense of social responsibility.

However, I disagree with Davies et al. (Reference Davies, Hamp–Lyons and Kemp2003) that the similarity signals dependence on the Inner Circle standard. Rather, such prompts do not assume any background knowledge related to any particular contexts and therefore show that the test is doing as fair a job as the TOEFL.

More importantly, a few prompts in more recent tests have attempted to include local elements, which has not been discussed in Davies et al. (Reference Davies, Hamp–Lyons and Kemp2003), although, again, such prompts concentrated in the CET-4 tests. For instance, the three prompts in the CET4_12_19 set asked the examinees to recommend a place/city/university to a foreign friend. In CET4_06_19(2), the prompt states:

1. (2) Write a news report to your campus newspaper on a visit to a Hope elementary school organized by your Student Union.

The term ‘Hope elementary school’ refers to an elementary school built and supported by charitable contributions, which has strong Chinese characteristics and is representative of the local language variety, China English.

Translation

Translation does not typically occur in ISLPTs or even LSLPTs and therefore has rarely been discussed in prior literature. Interestingly, with the purpose to ‘introduce the Chinese cultural, historical, and social development [to a foreign audience]’ (Specifications, 2016: 4), the translation topics in both CET-4 and 6 were exclusively local. For example, many original Chinese texts were about the social development in China, such as Chinese family values (CET-4, Dec 2019), the use of mobile payment in China (CET-4, Dec 2018), the museums in China (CET-6, Dec 2018), etc. Prompts focusing on Chinese history and culture covered topics such as Chinese Lion Dance (CET4_06_19), famous mountains, rivers, and lakes (e.g. Mount Tai, CET4_12_17; Yellow River, CET4_06_17), and famous dynasties in history (e.g. Ming, Song, and Tang, CET6_06_17), to name just a few.

Some translation prompts also contained ‘commonly used’ (Specifications, 2016) Chinese idioms, such as ‘ào rán zhàn fang’ (proudly bloom ,meaning flowers blooming vibrantly) (CET6_12_19[1]) and ‘chū wū ní ér bù rǎn’ (out dirty mud but not polluted，meaning emerging pure and clean from the murky water, typically used to describe the characteristics of the lotus flower) (CET6_12_19[3]), and not surprisingly, culture-loaded names of historic figures, places, and events. Translating these idioms and terms requires a mastery of the educated local English variety, China English (Honna, Reference Honna2020).

Research question 2: How does the CET define language proficiency? Specifically, to what extent does the test assess NS linguistic forms and accuracy?

The study examined the assessment goals, required skills, question types, scoring rubrics, and sample answers in the Specifications (2016) to gain an understanding of how the test defines language proficiency to be assessed. The examination reveals that the CET's assessment goal does not focus on NS linguistic forms and accuracy; rather, it emphasizes gauging overall communicative competence, regarding the local variety as acceptable and appropriate.

Item types

The CET does not contain discrete-point grammar items, which have been criticized by WEs scholars for focusing solely on NS linguistic forms and accuracy (e.g. Lowenberg, Reference Lowenberg2002; Matsuda, Reference Matsuda2003; Canagarajah, Reference Canagarajah2006). Remarkably, Listening abandoned the task of ‘Compound Dictation’ (i.e. filling in blanks of a listening passage with words or phrases) (Li & Zhao, Reference Li and Zhao2016), or a listening cloze item, which has been questioned for being unable to assess high-order language abilities (e.g. Cohen, Reference Cohen1980; Buck, Reference Buck2001; Cai, Reference Cai2013). Avoiding such discrete-point grammar questions reflects the test's possible intention to deemphasize a specific standard or variety of English as the goal of assessment.

It is also worth noting that the SET contains paired interactions between Chinese test takers at different proficiency levels, which enables the elicitation of the examinees’ interactive and negotiation skills through NNS-NNS communication (Canagarajah, Reference Canagarajah2006; Taylor, Reference Taylor2006; Harding & McNamara, Reference Harding and McNamara2018). This reflects a strong move toward assessing WEs (Hu, Reference Hu, Aslagoff, McKay, Hu and Renandya2012), especially when considering the possibility that, as Davies et al. (Reference Davies, Hamp–Lyons and Kemp2003) indicated, raters of the CET are proficient English speakers with Chinese as their native language. Admittedly, it is challenging to conclude whose norms are actually referred to by the Chinese raters, and it needs further research to confirm whether such NNS rater recruitment still remains nowadays; however, the scoring criteria specified in the Specifications (2016) can speak to this issue and will be discussed in the following section.

Scoring criteria

An analysis of the scoring criteria (see Appendix III) of the constructed-response items (Writing, Speaking, and Translation) in the Specifications (2016) also suggests what the CET assesses is not centered on linguistic accuracy, but communicative competence at all linguistic levels (Berns, Reference Berns, Nelson, Proshina and Davis2020). For instance, the rubric of the SET covers not only ‘Accuracy and Range’ at the grammatical level, but also Length and Coherence and ‘Flexibility and Appropriateness’ at the textual and pragmatic level. Specifically, the criteria of ‘Flexibility and Appropriateness’ refers to the abilities to ‘respond to different situations and topics’, ‘participate in discussions actively’, and ‘adapt language use to different situations, functions, and purposes’ (p.11–12), emphasizing the evaluation of candidates’ competence in terms of functional effectiveness (Matsuda, Reference Matsuda2003; Hu, Reference Hu, Aslagoff, McKay, Hu and Renandya2012) and strategic competence (Jenkins, Reference Jenkins2006; Jenkins et al., Reference Jenkins, Cogo and Dewey2011, Hu, Reference Hu, Aslagoff, McKay, Hu and Renandya2012), thus decentering the assessment of NS linguistic forms. As Elder and Davies (Reference Elder and Davies2006) pointed out, focusing the assessment on meaning making and functional effectiveness instead of language form enables a test to avoid the necessity for a description of which language norm(s) is the target, thus being considered a possible way to assess WEs or ELF.

Meanwhile, Elder and Davies (Reference Elder and Davies2006) raised the concern about what counts as an intelligible and successful conversation. Berns (Reference Berns, Nelson, Proshina and Davis2020) also claimed that communicative competence is an indispensable topic to WEs studies, and the key questions center around whose norms are considered acceptable, intelligible, and appropriate in different social contexts. An examination of the rubric descriptors and the translation and writing benchmarks in the Specifications (2016) indicates that there is no emphasis on the NS norms. For example, no reference was made to the ‘native speaker’ norms (Harding & McNamara, Reference Harding and McNamara2018); instead, ‘certain levels of native (Chinese) accent that don't affect intelligibility’ (p .4) are not penalized in the SET; additionally, the rating of all constructed items allows for ‘occasional minor errors’ (p. 5), suggesting a likely intention to differentiate ‘errors’ from ‘deviations’ and acknowledgment of the local variety. The following CET-4 benchmark translation about traditional Chinese hospitality can help us better understand how the test defines ‘minor errors’ and thus ‘intelligibility’ and ‘appropriateness’:

• (3) The traditional Chinese way of treating guests requires hosts to prepare abundant and various dishes, and make the guests unable to finish them all. The typical menu for a Chinese feast consists of a set of cold dishes, which are served at the beginning and some hot dishes after that, such as meat, chicken, ducks, and vegetables. In most feasts, a complete fish is considered necessary unless various kinds of seafood have been served. Nowadays, Chinese people like to mix western special dishes with traditional Chinese cuisine, so it is not rare to find steak on the table. In addition, salad has gained its popularity constantly, even though Chinese people are not likely to eat dishes that have not been cooked in tradition. There is generally a soup in a feast, which can be served at the beginning or the end of the meal. Besides, desserts and fruits often mark the end of a feast. (p. 200)

Although the underlined expressions are not idiomatic or ‘correct’ under the Inner Circle standard, the response is used as the benchmark for the highest level (score 14) of writing, which means the ‘deviations’ were treated as merely differences. This suggests that the test treats the local variety as, if not the norm, at least acceptable and intelligible. The same applies to the Writing sample response, where the essay receiving the perfect score (score 14) also contains expressions that may sound strange or incorrect to NSs, such as ‘to hear such argument’ and ‘harmful for following reasons’ (p. 197). Of course, more research needs to be conducted on how raters rate in real tests to deepen our understanding of their interpretation of intelligibility, appropriateness, and acceptability in actuality.

Research Question 3: How can the CET better assess WEs?

What has been done

Following the lead of ISLPTs, the CET has taken active moves, essentially the weak approach, to assess WEs by diversifying the sampling sources within the Inner Circle. Generally, the CET fits fairly into the context of local testing in China. As a norm-dependent Expanding Circle country, China has relied on the NS-model (e.g. Kirkpatrick, Reference Kirkpatrick, Rubdy and Saraceni2006) in English education at the tertiary level (He & Zhang, Reference He and Zhang2010), which largely explains its dependence on the Inner Circle standards and echoes Hu's (Reference Hu, Aslagoff, McKay, Hu and Renandya2012) advocacy to ‘make allowances for individual aspirations to Inner Circle Norms’ (p. 138). Meanwhile, the CET has taken actions to meet its purpose of assessing general English abilities in a wide range of communication contexts (Specifications, 2016). For instance, it also includes global and local elements in its input texts and item prompts and decenters the evaluation of the NS linguistic forms and accuracy by emphasizing intelligibility and communicative effectiveness. Some stronger approaches were also utilized, such as the paired interaction in the SET which has the potential to assess NNS–NNS communicative and strategic competence. The accommodations respond to the changing role of English from a way to ‘[interact] with native speakers with a focus on understanding the customs, the cultural achievements’ (Berns, Reference Berns2005: 86) to a tool for a mixture of international and intra-national purposes. The incorporation of global and local elements to the Inner Circle standards, or the Standard English plus method (Li, Reference Li2006), speaks to the test's local validity and reliability. The bias issue is also addressed when the assessment acknowledges the acceptability of the local variety and recognizes certain levels of ‘differences’ and ‘deviations’.

Recommendations

The nature of WEs tends to favor pluralism, instead of a certain, single variety or group of varieties; therefore, it is important to make ways for diversity in assessment, to prepare the English learners in the Expanding Circle for real-life communications (Canagarajah, Reference Canagarajah2006; Hu, Reference Hu, Aslagoff, McKay, Hu and Renandya2012). The following aspects of the CET should be further diversified.

The topics in Listening and Reading, as noted earlier, have been too restricted to the Inner Circle context. The local validity and authenticity of the test can be questioned when the topics and contents relate little to the culture the candidates are familiar with and do not assess students’ use of English in real life (Lowenberg, Reference Lowenberg1993; Canagarajah, Reference Canagarajah2006). Besides, topic familiarity has been suggested to play a crucial role in L2 listening and reading comprehension (Markham & Latham, Reference Markham and Latham1987; Leeser, Reference Leeser2004), since background knowledge enables the audience to connect new information to existing knowledge (Anderson & Lynch, Reference Anderson and Lynch1988) and make inferences needed for a coherent mental representation of a text's content (e.g. Kintsch, Reference Kintsch1988; van den Broek et al., Reference Van den Broek, Young, Tzeng, Linderholm, van Oostendorp and Goldman1999). Therefore, irrelevant topics can bias against the Chinese examinees who have limited exposure to the corresponding culture- or context-specific background knowledge. To better assess WEs, the test should not only diversify sampling within the Inner Circle, but it should also address the global and especially the local contexts. As mentioned earlier, among the 36 tests delivered within the three years, only two reading passages and no listening passages were situated specifically in the Chinese context. Creating space for one or two texts from the local setting in one test could be a good start to assessing WEs. Regarding global topics, test developers can also consider cultures that have more contact with China (e.g. South Korea, Japan, India, Thailand) rather than relying predominantly on European contexts.

Topic familiarity has also been proved to relate strongly with writing performance (e.g. Hamp–Lyons & Mathias, Reference Hamp–Lyons and Mathias1994; Magno, Reference Magno2008; Mahdavirad, Reference Mahdavirad2016). However, the majority of writing prompts do not relate to any local topics. Besides, the very few topics involving local varieties are not updated (e.g. the ‘Hope elementary school’ topic). Therefore, the test needs to incorporate more local and updated topics to the writing prompts.

The accents used in Listening, although not confined to a single variety, have been restricted to the varieties in the Inner Circle. The test should expose the examinees to more varieties to ‘foster their sociolinguistic awareness and sensitivity’ (Hu, Reference Hu, Aslagoff, McKay, Hu and Renandya2012: 136; Brown, Reference Brown2004; Kachru, Reference Kachru and Hinkel2011). The local variety, Chinese accent, can be a fair candidate. Studies (e.g. Harding, Reference Harding2012) have indicated that Chinese students could be advantaged when taking listening assessment recorded by proficient Chinese-accented speakers. Indeed, the CET test-takers typically have much and even more exposure to the Chinese accent (e.g. learning English with teachers who share their L1) than other accents (He & Zhang, Reference He and Zhang2010).

A recommended way to diversify the topic and accent selection discussed above is sample more from local English media, such as China Central Television (CCTV)-News (a local English TV channel), and China Daily and Beijing Today Weekly (local English newspapers). Take China Daily (see http://global.chinadaily.com.cn/) as an example. It covers a wide range of global (including the Inner Circle) and local topics (e.g. World, China, Technology, Business, Culture, Travel, and Sports) that are closely relevant to Chinese people. One of the latest articles from ChinaDaily-Opinion discussing a popular topic in China – ‘Food waste is a shameful chronic disease’ – can be a potential Reading material. Another hot topic – ‘Should mukbanger, or Chibo be banned?’ – could be adapted for the writing or speaking test, with a slight modification by adding a brief explanation of the term ‘Mukbanger’.Footnote ⁸ In addition, the newspaper uses a combination of NNS and NS journalists that compose for both global and local audiences and is a source of the local variety, China English.