General information for retrofitting the traditional BPD Questionnaire

As mentioned, early self-reported assessments did not provide any further insights into the specific areas of BPD. In order to address this problem, the present study aimed to retrofit the traditional BPD questionnaire under the CDM framework. The improved BPD assessment with cognitive diagnostic properties (CDA-BPD) can measure and evaluate BPD using the symptoms in the DSM-5, and offer cognitive diagnosis information beyond what a single test score could provide using these criterion profiles.

The CDA-BPD was initially administered using the traditional BPD instrument – the Borderline Personality Questionnaire (BPQ; Poreh et al., Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006) – an 80-item true/false self-report scale that comprises nine subscales corresponding to the nine BPD criteria of the Diagnostic and Statistical Manual of Mental Disorders (4^th ed., text-rev.; DSM-IV-TR; American Psychiatric Association, 2000). This study chose the BPQ questionnaire for the following reasons: (1) The BPQ is a representative and widely used questionnaire, and researchers have confirmed that it is a reliable and valid tool to assess borderline personality features (Fonseca-Pedrero, Paino, Lemos-Giráldez, Sierra-Baigrie, & Muñiz, Reference Fonseca-Pedrero, Paino, Lemos-Giráldez, Sierra-Baigrie and Muñiz2011); (2) The BPQ is a true/false self-reported instrument that exactly meets the requirement of the CDM for dichotomic data. Although the BPQ is based on DSM-IV-TR criteria, many studies report that there are no differences between the definitions of the nine symptoms of BPD in the DSM-IV-TR and DSM-5 (Conway, Hammen, & Brennan, Reference Conway, Hammen and Brennan2012; Iacono, Reference Iacono2013), so the BPQ was still of significance for the current measurement of BPD.

Additionally, a crucial step to conduct the CDA-BPD was to construct a Q-matrix, which played an important role because it describes what attributes or symptom criteria are measured by each item. Let q_jk denote the element in row j and column k of a J × K Q-matrix, where J and K represented the numbers of items and attributes/symptom criteria respectively. The element q_jk was specified to be 1 if the kth attribute was required to answer item j correctly, and zero otherwise. The quality of the Q-matrix determines the quality of the estimated diagnostic model, and a poorly created Q-matrix provides less informative diagnostic or classification indices (Tatsuoka, Reference Tatsuoka, Frederiksen, Glaser, Lesgold and Shafto1990). In the present study, in order to guarantee the highest quality, the Q-matrix of the CDA-BPD was directly based on the primal BPQ scale (Poreh et al., Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006), which describes the relationship between 80 items and nine attributes/symptom criteria. Appendix A shows the initial Q-matrix of the BPQ scale used in this article. Each row of the Q-matrix pertains to an item, and each column to a symptom. For example, the first row denotes that item 1, “I often do things without thinking them through”, measures “Impulsivity in at least two areas that are potentially self-damaging (e.g. spending, sex, substance abuse, reckless driving, binge eating)” (Symptom 4), while item 2, “I often become depressed or anxious out of blue”, measures “Affective instability due to a marked reactivity of mood (e.g. intense episodic dysphoria, irritability, or anxiety usually lasting a few hours and only rarely more than a few days)” (Symptom 6). It is critical that all CDMs are equivalent when all items have a simple structure.

Cognitive diagnosis model: G-DINA model

The aim of CDMs is to build close connections between individuals’ item responses and their attribute patterns or symptom profiles. In the last few years, several CDMs have been proposed (Templin & Henson, Reference Templin and Henson2010). Those vary in the way that attributes are combined and formalized to estimate the probability of item responses. Most of the applications of CDMs in psychological assessment in particular have involved deterministic inputs, the noisy “and” gate model (DINA; Junker & Sijtsma, Reference Junker and Sijtsma2001), the addictive cognitive model (ACDM; de la Torre, Reference de la Torre2011), and the reduced reparametrized unified model (RRUM; Hartz, Reference Hartz2002). To synthesize the various CDMs, de la Torre (Reference de la Torre2011) proposed a general model, the G-DINA (generalized deterministic inputs, noisy “and” gate) model, which allowed reformulating many of the existing CDMs as special cases of this general model.

In the G-DINA model, the required attributes for item could be represented by the reduced vector $\alpha _{lj}^* =({\alpha _{l1}},...,{\alpha _{lK_j^*}})'$ , where $l = 1,...,{2^{K_j^*}}$ and ${2^{K_j^*}}$ represented the number of unique attribute patterns. The probabilities that examinees with reduced attribute vector $\alpha _{lj}^*$ would answer item j correctly was expressed as $P({X_j} = 1|\alpha _{lj}^*)$ . The formulation of the G-DINA model in its saturated form (i.e. no restrictions were made) was

$$P({X_j} = 1|\alpha _{lj}^*) = {\delta _{j0}} + \sum\limits_{k = 1}^{K_j^*} {{\delta _{jk}}} {\alpha _{lk}} + \sum\limits_{k' = k + 1}^{K_j^*} {\sum\limits_{k = 1}^{K_j^* - 1} {{\delta _{jkk'}}} {\alpha _{lk}}} {\alpha _{lk'}}... + {\delta _{j12...K_j^*}}\prod\limits_{k = 1}^{K_j^*} {{\alpha _{lk}}} .$$ (1)

The function above was the sum of the effects based on the presence of specific attributes and their interactions. Specifically, δ_{j 0} represented the baseline probability of a correct response when none of the required attributes was mastered; δ_jk was the main effect due to attribute α_k , or the change in the probability of a correct (or most effective) response when a single attribute (i.e. α_k ) was mastered; δ_{jkk ’}, a first-order interaction effect, represented the change in the probability of a correct response when both α_k and α_{k ’} were mastered; and ${\delta _{j12...K_j^*}}$ was the interaction effect of all required attributes, or the change in the probability of a correct response when all required attributes were needed (de la Torre, Reference de la Torre2011). The parameters of the G-DINA model could be estimated using the marginal maximum likelihood estimation (MMLE) algorithm.

The saturated model based on the above function covered all possible item effects (e.g. intercept, main, and interaction effects). By constraining some item effects of the saturated forms, the G-DINA model could be transformed into different reduced CDMs. For example, the DINA model (Junker & Sijtsma, Reference Junker and Sijtsma2001) only included the intercept and final interaction effects, and it was expressed as

$$P({X_j} = 1|\alpha _{lj}^*) = {\delta _{j0}} + {\delta _{j12...K_j^*}}\prod\limits_{k = 1}^{K_j^*} {{\alpha _{lk}}} .$$ (2)

The ACDM model (de la Torre, Reference de la Torre2011) is a special case of the G-DINA model by supposing no interaction effects, and it was formulated as

$$P({X_j} = 1|\alpha _{lj}^*) = {\delta _{j0}} + \prod\limits_{k = 1}^{K_j^*} {{\delta _{jk}}{\alpha _{lk}}} .$$ (3)

The RRUM model (Hartz et al., Reference Hartz2002) is the log-link G-DINA model without interaction terms, and it was defined as

$$\log \left[ {P({X_j} = 1|\alpha _{lj}^*) = {\delta _{j0}} + \prod\limits_{k = 1}^{K_j^*} {{\delta _{jk}}{\alpha _{lk}}} } \right].$$ (4)

More technical details about the saturated and reduced CDMs can be found in de la Torre (Reference de la Torre2011) and de la Torre and Chen (Reference de la Torre and Chen2011).

Symptom criteria of BPD in the DSM-5

The DSM-5 defines the main features of BPD, and for an individual to be classified as BPD, he or she must satisfy five or more symptom criteria in Table 1. It should be emphasized that the DSM-5 diagnostic criteria for BPD has been widely used in academic research and clinical diagnostic practice (Bach, Sellbom, Bo, & Simonsen, Reference Bach, Sellbom, Bo and Simonsen2016; Ferrer et al., Reference Ferrer, Andión, Calvo, Hörz, Fischer-Kern and Kapusta2018; Miller, Morse, Nolf, Stepp, & Pilkonis, Reference Miller, Morse, Nolf, Stepp and Pilkonis2012).

Table 1. Symptom criteria of BPD defined in the DSM-5

In this context, with the response data, CDM could estimate two kinds of probabilities for each individual, specifying the posterior probability of satisfying each criterion according to the DSM-5 (Templin & Henson, Reference Templin and Henson2006), and the posterior probability of meeting five or more criteria of borderline personality disorder (PPBPD), which was also the probability of BPD in the DSM-5. Specifically, if an individual had a PPBPD greater than .5, then the individual could be classified as BPD. It should be noted that the posterior probability of satisfying each criterion for each individual showed differences in why an individual received a diagnosis of BPD; for example, two individuals who were both diagnosed as BPD may have diverse pathogeny paths for the disorder because they had different probabilities of symptom criteria. Specifically, except when several other criteria of BPD are met, the first individual may be very sensitive to environmental circumstances, and they may experience intense abandonment fears and inappropriate anger even when faced with a realistic time-limited separation or when there are unavoidable changes in plans (S1). The first individual also may display recurrent suicidal behavior, gestures, threats, or self-mutilating behavior (S5). The second individual may differ from the first individual in that he or she may be troubled by chronic feelings of emptiness (S7) and may frequently express inappropriate, intense anger or have difficulty controlling their anger (S8). Both individuals may be diagnosed as BPD, but the expressions of their disorders are the result of distinct behaviors. The probability of any individual meeting any defined criterion could be estimated by the CDM.

A cognitive diagnosis structure of symptom criteria for the CDA-BPD

The present study not only focused on the evaluation of BPD, but also investigated the potential structure of psychological traits. The underlying structure of the CDA-BPD was based on one explored by Poreh et al. (Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006). According to Poreh et al., BPD has two underlying traits: identity/interpersonal and impulsivity. The identity/interpersonal factor includes abandonment (S1), relationships (S2), self-image (S3), affective instability (S6), emptiness (S7), and quasi-psychotic states (S9). The impulsivity factor includes impulsivity (S4), suicide (S5), and intense anger (S8). This study applied a CDM to verify the structure between the two factors and nine symptom criteria. Because the CDM could estimate the presence or absence of a set of criteria, it was possible to model the association between these criteria and verify the structure suggested by Poreh et al. (Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006). According to Templin and Henson (Reference Templin and Henson2006), a method to model the probability of class membership is the concept of tetrachoric correlations to place the structure on the joint distribution of the attributes. A tetrachoric correlation is the correlation between two underlying normally distributed variables that have both been dichotomized by respective cut-point parameters. This study dichotomized each criterion by using the marginal probability of all participants for each criterion as a cut-point. Therefore, a set of cut-point parameters was estimated. The cut-point parameters were essential because they could be converted to a proportion (or percentage) representing the marginal level of presence for each attribute (or, in the study, satisfaction of each criterion). Such parameters provided information regarding the base rate of each attribute in the population represented by the data set (Templin & Henson, Reference Templin and Henson2006).

Measures

Borderline Personality Questionnaire (BPQ; Poreh et al., Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006) is an 80-item true/false self-report measure that assesses borderline personality traits with nine subscales. Results (Poreh et al., Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006) show that the BPQ has high internal consistency (Kuder-Richardson coefficient = 0.94) and good discriminant validity.

CDA-BPD. The initial item pool for the CDA-BPD contained all 80 items from the BPQ (Poreh et al., Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006). The CDA-BPD items were first translated into Chinese, called the first draft. It was then given to four assessors, including two experts in English and two experts in psychology/psychometrics. Their task was to judge the accuracy of translation and relevance/suitability of each item for measuring BPD in the socio-cultural context of China, and this formed the second draft. Then the study applied 30 Chinese college students to verify the quality and accuracy of the second draft by cognitive interview. Finally, the Chinese version of the CDA-BPD was formed. The study used the G-DINA model to analyze the psychometric characteristics of each item in the CDA-BPD. Furthermore, the study selected high-quality items to develop the final CDA-BPD, which included 55 items. The reliability and validity of the final CDA-BPD will be introduced in detail in the results section of this article.

McLean Screening Instrument for Borderline Personality Disorder (MSI-BPD; Zanarini et al., Reference Zanarini, Vujanovic, Parachini, Boulanger, Frankenburg and Hennen2003) is a 10-item, self-report screening measure. According to Zanarini et al. (Reference Zanarini, Vujanovic, Parachini, Boulanger, Frankenburg and Hennen2003), the MSI-BPD has good test–retest reliability (r = .72), high internal consistency (α = .74), and acceptable item-total correlation (range .45–.63). A score of 7 or more is the optimal clinical cut-off (Zanarini et al., Reference Zanarini, Vujanovic, Parachini, Boulanger, Frankenburg and Hennen2003). Wang, Liang, and Zhong (Reference Wang, Liang and Zhong2008) revised the MSI-BPD in Chinese and the results showed that the Chinese version of MIS-BPD had good reliability and validity. In this study, the Chinese version of MIS-BPD was administered and it had a Cronbach’s alpha of .73 and a split-half of .72 in the current study.

Personality Diagnostic Questionnaire-Fourth Edition-BPD Scale (PDQ-4-BPD; Hyler, Reference Hyler1994) contained nine true/false items targeting the BPD criteria. As mentioned by Hyler (Reference Hyler1994), a cut-off score of ≥5 indicates the presence of BPD traits. Yang et al. (Reference Yang, McCrae, Costa, Yao, Dai, Cai and Gao2000) developed the Chinese version of Personality Diagnostic Questionnaire (PDQ-4) and the results showed that the reliability and validity were acceptable. In this study, the Chinese version of PDQ-4-BPD was applied and it had a Cronbach’s alpha of .71 and a split-half of .68 in the current study.

The present study used the Chinese version of the MIS-BPD (Wang et al., Reference Wang, Liang and Zhong2008) and PDQ-4-BPD (Yang et al., Reference Yang, McCrae, Costa, Yao, Dai, Cai and Gao2000) as an evidence to support the convergent validity of the CDA-BPD for two reasons: (1) they are both self-reported scales of BPD based on the DSM diagnostic system and are widely used in Chinese BPD research; (2) the Chinese version of the MIS-BPD (Wang et al., Reference Wang, Liang and Zhong2008) and the PDQ-4-BPD (Yang et al., Reference Yang, McCrae, Costa, Yao, Dai, Cai and Gao2000) have good reliability and validity that have been demonstrated in many studies (H. Chen, Zhong, & Liu, Reference Chen, Zhong and Liu2011; Huang et al., Reference Huang, Napolitano Li, Wu, Yang, Xi, Li and Li2014).

Participants

The total sample came from 20 Chinese universities and consisted of 1,097 participants who agreed to take part in the study after being informed that their personal information would be kept secret and the test would occupy them for about half an hour. The sample had a mean age of 20.63 years (SD = 1.524, range = 18–24) and included 431 males (39.3%) and 666 females (60.7%). The total sample included two parts: one part had 1,029 participants who were used to calibrate the item parameters of the CDA-BPD via CDM; the other part was comprised of 68 participants who were used as a validation sample to assess the sensitivity and specificity of the CDA-BPD.

The validation sample included two paired groups according to the demographic characteristics: BPD group (N ₁ = 34) and health group (N ₂ = 34).

BPD group. The BPD group had a score ≥7 for the MIS-BPD and a score ≥5 for the PDQ-4-BPD. Participants also reported that their emotional and behavioral instability seriously affected their daily lives and activities. The BPD group also met the exclusion criteria: (1) ever satisfied the DSM-5 criteria for schizophrenia or schizoaffective disorder; (2) ever had a history of psychiatric treatment, drinking, or drug abuse; or (3) ever been diagnosed with any brain disease caused by infection, trauma, tumor, inheritance or vascular disease. The BPD group comprised 41.2% males (58.8% females), with 43.2% from city (56.8% from country); 44.1% was a freshman at university (11.8% sophomore, 17.6% junior, 26.5% senior), and had a mean age of 20.62 years (SD = 1.60, range =18–24).

Healthy group. The healthy group had a score of MIS-BPD less than 7 and a score of PDQ-4-BPD less than 5. Moreover, each individual in the healthy group must be paired one by one with the BPD group according to age, sex, grade, and where they live. For example, if there was a participant in the BPD group who was 20 years old, male, a freshman in college and from city, the participant in the healthy group must have the same demographic variables (20 years old, male, a freshman in college and from the city). The healthy group was recruited according to the following exclusion criteria: (1) ever been diagnosed with severe somatic diseases (e.g. chronic diseases, diabetes, cancer); (2) ever had treatment for psychiatric illness over the past year. Written informed consent was obtained from all participants in this study.

Statistical analysis

The statistical analysis consisted of five steps as follows.

Step 1: Use the G-DINA model to analyze the psychometric characteristics of each item in the CDA-BPD. Analyzing psychometric characteristics is very important in assessing BPD. The psychometric characteristics of each item based on the CDM in this study included item-fit, differential item functioning (DIF), and item discrimination. The study used the S-X² item fit statistic (Orlando & Thissen, Reference Orlando and Thissen2003) to examine item fit, the Wald test statistic (Hou, de la Torre, & Nandakumar, Reference Hou, de la Torre and Nandakumar2014) to detect DIF in different groups (e.g. female and male; rural and urban), and applied the Disc_j index to analyze the item discrimination. For the CDMs, two discrimination indexes were proposed: one was suggested by de la Torre (Reference de la Torre2008) and the other by de la Torre and Chiu (Reference de la Torre and Chiu2016). Each item in the CDA-BPD measured one attribute; therefore, the above two indexes were equivalent. Formula (5) was the index of de la Torre (Reference de la Torre2008), shown as,

$$Dis{c_j} = P({X_j} = 1|\alpha _{lj}^* = 1) - P({X_j} = 1|\alpha _{lj}^* = 0),$$ (5)

where

$$P({X_j} = 1|\alpha _{lj}^* = 1) = {\rm{1}} - P({X_j} = 0|\alpha _{lj}^* = 1) = 1 - {s_j},$$ (6)

and

$$P({X_j} = 1|\alpha _{lj}^* = 0) = {g_j}.$$ (7)

In the above function, s_j (slip parameters) referred to the probability that an individual with all required attributes provided an incorrect response, g_j (guessing parameters) referred to the probability that an individual without all required attributes provided a correct response. $P({X_j} = 1|\alpha _{lj}^* = 1)$ referred to the response probability for individuals who present with all the symptom criteria measured by item j, and $P({X_j} = 1|\alpha _{lj}^* = 0)$ was the response probability for individuals who did not present any symptom criteria measured by item j. Disc_j was a comprehensive index of slip and guessing parameters. The higher value of Disc_j indicated that the quality of the item was better and could discriminate different individuals.

Step 2: Select high-quality items to develop the final CDA-BPD. The selection of high-quality items was based on the statistical indexes, including discrimination, model-fit in item level, and DIF in Step 1. Any item with low discrimination (Disc_j < 0.2), poor item fit (p < .01) or having DIF (p < .01) was excluded. To obtain the final CDA-BPD with high-quality items, this procedure was repeated until no item was excluded. All the analyses of the CDM were based on the GDINA R package (Version 3.5.1 64-bit; Ma & de la Torre, Reference Ma and de la Torre2016).

Step 3: Examine the reliability and validity of the final CDA-BPD. To examine the reliability of the CDA-BPD, the study used the coefficients of Cronbach’s alpha and Guttman split-half reliability, which were based on the CTT framework; under the framework of the CDM, the symptom-level classification consistency and classification accuracy reliability indices (Cui, Gierl, & Chang, Reference Cui, Gierl and Chang2012), and Templin’s attribute reliability, which was proposed by Templin and Bradshaw (Reference Templin and Bradshaw2013), were also evaluated for the CDA-BPD. These reliability indicators could estimate how accurately a CDM classified subjects into correct attribute profiles. Besides, the convergent validity of the CDA-BPD was quantified by the correlation between the CDA-BPD and the Chinese version of the MIS-BPD and PDQ-4-BPD. The content validity of CDA-BPD was also investigated by checking whether the CDA-BPD contained all domains of BPD.

Step 4: Estimate the sensitivity and specificity of the final CDA-BPD. The accuracy of an instrument was commonly assessed by calculating the instrument’s sensitivity and specificity. In the present study, the sensitivity of the CDA-BPD was defined as the percentage of those who truly had BPD that was correctly identified by the assessment. It was also sometimes called the true positive rate. The specificity of the CDA-BPD was defined as the percentage of those who truly did not have BPD that was correctly identified by the assessment. The specificity was called the true negative rate. In addition, the study estimated the PPBPD, which was based on the CDA-BPD and DSM-5 via the CDM, and it meant that individuals who had over 0.5 PPBPD were defined as BPD. Clearly, if an assessment was perfect for its domain, it would be expected to have both the sensitivity and specificity equal to 100%. Moreover, ROC curve analyses were used in the study to evaluate the predictive effectiveness of the CDA-BPD. The perfect assessment had the area under the ROC curve (area under the curve [AUC]) of 1.00, while an assessment with no predictive validity had an AUC of .50 and a linear ROC curve that paralleled the diagonal. Although considered rather arbitrary, AUC values of .60 to .70, .70 to .90, and those above .90 were generally associated with an acceptable or fair, good, and excellent discriminant assessment respectively (Swets, Reference Swets1988). Sensitivity and specificity evidences of CDA-BPD were collected by using a validation sample in the current study.

Step 5: Analyze the latent structure of symptom criteria for the final CDA-BPD. The estimated structure of the CDA-BPD was based on the model developed by Poreh et al (Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006). In their development, the authors used the unidimensional sum scores of items to represent latent factor scores for each criterion. The hypothesized structure mapped the latent criteria onto identity/interpersonal (symptom criterion 1, 2, 3, 6, 7 and 9) and impulsivity (symptom criterion 4, 5, and 8).

In the current study, the method used to analyze the structure of the CDA-BPD was different from Poreh et al (Reference Poreh, Rawlings, Claridge, Freeman, Faulkner and Shelton2006). Specifically, the nine criteria were treated as dichotomous variables by nine respective cut-point parameters (the study dichotomized each criterion by using the marginal probability of all participants for each criterion as a cut-point). These cut-point parameters were able to be converted to a proportion representing the marginal level of presence each criterion. Then the tetrachoric correlations among pairs of nine criteria were analyzed and this was used to verify the structure of symptom criteria for the CDA-BPD. And the method mentioned above was referred to Templin and Henson (Reference Templin and Henson2006).

The ratio of chi-square to the degrees of freedom (df) was calculated in addition to chi-square to evaluate model fitting, with ratios between 1.5 and 5.0 indicating acceptable fit (Byrne, Reference Byrne2012). Additional parameters for fit estimation were: the normal fit index (NFI), the non-normal fit index (NNFI), the comparative fit index (CFI), the root mean square error of approximation (RMSEA), the goodness of fit index (GFI), the adjusted goodness of fit index (AGFI). RMSEA values of 0.08 or lower and NFI, NNFI, CFI, GFI, AGFI values of 0.90 or higher were considered acceptable (Browne & Cudeck, Reference Browne and Cudeck1993; Hu & Bentler, Reference Hu and Bentler1999). Criterion level confirmatory factor analysis (CFA) was carried out with LISREL 8.0 (Joreskog & Sorbom, Reference Joreskog and Sorbom1996) in the current study.