2. Specification of the Hierarchical Model

Let us by $X$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {X}}$$\end{document} denote an $N\times n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\times n$$\end{document} matrix of responses of N persons to n items taking values of 1 if the response is correct and 0 otherwise, and by $T$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {T}}$$\end{document} an $N\times n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N\times n$$\end{document} matrix of the corresponding response times. It may be noted that unlike what is typical for experimental psychology, in educational and psychological measurement one observation is usually available for a combination of a person with an item, and the number of persons is typically much larger than the number of items.

The hierarchical model for response times and accuracy is (van der Linden, Reference van der Linden2007):

(1) $\begin{array}{c}f(X,T|\theta ,\tau )=\prod _p\prod _{i}f\left(t_{pi}\right|{\tau }_p,{\gamma }_i)f\left(x_{pi}\right|{\theta }_p,{\delta }_i),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f({\mathbf {X}},{\mathbf {T}}\,|\, {\varvec{\theta }}, {\varvec{\tau }})=\prod _p\prod _i f(t_{pi}\,|\, \tau _p,{\varvec{\gamma }}_i)f(x_{pi}\,|\, \theta _p, {\varvec{\delta }}_i), \end{aligned}$$\end{document}

where $t_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}$$\end{document} and $x_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{pi}$$\end{document} are the response time and accuracy of person p on item i, ${\tau }_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _p$$\end{document} and ${\theta }_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _p$$\end{document} are the speed and the ability parameters of person p, and ${\gamma }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\gamma }}_i$$\end{document} and ${\delta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\delta }}_i$$\end{document} are the vectors of item parameters of item i related to time and accuracy, respectively. At the lower level of the hierarchical model both the model for response accuracy and the model for response time need to be specified. At the higher level the models for the relationship between the person parameters ( ${\theta }_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta _p$$\end{document} and ${\tau }_p$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau _p$$\end{document} ) and for the relationship between the item parameters ( ${\delta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\delta }}_i$$\end{document} and ${\gamma }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\gamma }}_i$$\end{document} ) need to be specified. The hierarchical framework provides users with a plug-and-play approach with a flexible choice of models at the lower and the higher level. Below we describe the full specification of the hierarchical model assuming conditional independence which is considered in this study (note that this specification slightly deviates from the original article of van der Linden (Reference van der Linden2007)).

The model for the response accuracy is the following (Birnbaum, Reference Birnbaum, Lord and Novick1968):Footnote ¹

(2) $\begin{array}{c}Pr(X_{pi}=1|{\theta }_p)=\frac{exp({\alpha }_i{\theta }_p+{\beta }_i)}{1+exp({\alpha }_i{\theta }_p+{\beta }_i)},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Pr (X_{pi}=1\,|\, \theta _p)=\frac{\exp (\alpha _i\theta _p+\beta _i)}{1+\exp (\alpha _i\theta _p+\beta _i)}, \end{aligned}$$\end{document}

that is, ${\delta }_i=\{{\alpha }_i,{\beta }_i\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\delta }}_i=\{\alpha _i,\beta _i\}$$\end{document} , where ${\alpha }_i>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _i>0$$\end{document} and ${\beta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _i$$\end{document} are the slope and the intercept of the ICC of item i which relates the ability of the person to the probability of a correct response to the item. The slope ${\alpha }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _i$$\end{document} reflects the discriminative power of the item, since it specifies the strength of the relationship between the latent ability and the response to the item, and the intercept ${\beta }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _i$$\end{document} reflects item easiness.

The response times are distributed according to the log-normal distribution (denoted by $lnN$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ln \mathcal {N}$$\end{document} )

(3) $\begin{array}{c}{T}_{pi}|{\tau }_p\sim lnN({\xi }_i-{\tau }_p,{\sigma }_i^2),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} T_{pi}\,|\, \tau _p \sim \ln \mathcal {N}(\xi _{i}-\tau _p,\sigma ^2_{i}), \end{aligned}$$\end{document}

that is ${\gamma }_i=\{{\xi }_i,{\sigma }_i^2\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\gamma }}_i=\{\xi _i,\sigma ^2_i\}$$\end{document} . The mean of the logarithm of response time of person p to item i depends on the item time intensity ( ${\xi }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\xi _{i}$$\end{document} ) and the person speed, and the variance parameter depends on the item. The residual variance of log-response time ${\sigma }_i^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_i$$\end{document} can be interpreted as the inverse of item time discrimination (van der Linden, Reference van der Linden2006), that is, the larger $\frac{1}{{\sigma }_i^2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\frac{1}{\sigma ^2_i}$$\end{document} is, the larger the proportion of variance of response time explained by the variation of speed across persons is.Footnote ²

At the higher level, a multivariate normal distribution for the item parameters $\{{\xi }_i,ln\left({\sigma }_i^2\right),ln\left({\alpha }_i\right),{\beta }_i\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{\xi _i,\ln (\sigma ^2_i),\ln (\alpha _i),\beta _i\}$$\end{document} and a multivariate normal distribution for the person parameters $\{{\theta }_p,{\tau }_p\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{\theta _p,\tau _p\}$$\end{document} are assumed with the identifiability restrictions of ${\mu }_{\theta }={\mu }_{\tau }=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _{\theta }=\mu _{\tau }=0$$\end{document} and ${\sigma }_{\theta }^2=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_{\theta }=1$$\end{document} (for the explanation of sufficiency of these constraints, see van der Linden, Reference van der Linden2007).

3. Motivating Example: A Violation of Conditional Independence

We present an analysis of a data set of the Major Field Test for the Bachelor’s Degree in BusinessFootnote ³, which is a low-stakes test used to assess the mastery of concepts, principles, and knowledge of graduating bachelor students in business education. The test is not used for making individual-level decisions, but for evaluating educational programmes. It is a computerised test in which responses and response times are recorded automatically. The test consists of 120 multiple-choice (with four response options) items separated into two parts. Only the first part of the test was analysed in this study. The time limit for this part was one hour, while the average time used by the respondents was 42 minutes. Some items in the test are based on diagrams, charts and data tables. The test items cover a wide range of difficulties and are aimed at the evaluation of both depth and breadth of business knowledge. From the original sample with the responses of 1000 persons to 60 items,Footnote ⁴ 11 items were removed due to low item-rest score correlations ( $<.1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$<\!.1$$\end{document} ). The responses for which recorded response time was equal to 0 were treated as missing values.

To test the assumption of conditional independence between response times and accuracy given speed and ability we used the Lagrange Multiplier test of van der Linden and Glas (Reference van der Linden and Glas2010). In this test for each item the hierarchical model assuming conditional independence (van der Linden, Reference van der Linden2007) is tested against a model that allows for differences in the expected log-response time for correct and incorrect responses by including an extra item parameter in the model for the response times:

(4) $\begin{array}{c}{t}_{pi}\sim lnN\left({\xi }_i,+,{\lambda }_i,(1-x_{pi}),-,{\tau }_p,,,{\sigma }_i^2\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} t_{pi}\sim \ln \mathcal {N}\left( \xi _i+\lambda _i(1-x_{pi})-\tau _p,\sigma ^2_i\right) . \end{aligned}$$\end{document}

Figure 1 shows the distribution of the p values for the item-level conditional independence test. For more than half of the items, conditional independence is violated (using $\alpha =.05$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha =.05$$\end{document} for each test). These results indicate that the assumption of conditional independence cannot be maintained for these data. However, as has been demonstrated in simulation studies (Bolsinova & Tijmstra, Reference Bolsinova and Tijmstra2016), the Lagrange multiplier test of van der Linden and Glas (Reference van der Linden and Glas2010) also may pick up violations of conditional independence that are of a different type than what is specified in Eq. 4. Therefore, this test does not yet tell us in what way exactly conditional independence is violated.

Figure 1.

Distribution of the p values of the Lagrange Multiplier test for conditional independence between response time and accuracy. Most of the p values are below .05, indicating that conditional independence is violated.

To investigate in which way conditional independence is violated we performed two posterior predictive checks (Meng, Reference Meng1994; Gelman, Meng, & Stern, Reference Gelman, Meng and Stern1996; Sinharay, Johnson, & Stern, Reference Sinharay, Johnson and Stern2006) that focus on differences in the behaviour of the items with respect to response accuracy between slow and fast responses, following the approach proposed by Bolsinova and Tijmstra (Reference Bolsinova and Tijmstra2016). Here, we defined slow and fast responses by a median split by defining a transformed time variable:

(5) $\begin{array}{c}{t}_{pi}^{\ast }=\left(\begin{array}{c}1\text{if}{t}_{pi}\ge t_{med,i},\\ 0\text{if}{t}_{pi}<t_{med,i},\end{array}\right)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} t^*_{pi}={\left\{ \begin{array}{ll}1\text { if } \quad t_{pi}\ge t_{med,i},\\ 0 \text { if } \quad t_{pi}< t_{med,i},\end{array}\right. } \end{aligned}$$\end{document}

where $t_{med,i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{med,i}$$\end{document} is the median response time for item i. Our goal is to investigate whether observed differences between the slow and the fast responses with respect to the difficulty of the items and their discriminatory power are unlikely to be observed under the hierarchical model assuming conditional independence.

When posterior predictive checks are implemented, the measures of interest have to be repeatedly computed for replicated datasets. Therefore, for reasons of computational convenience we decided not to estimate IRT difficulty and discrimination parameters separately for the slow and for the fast responses, but to compute simple classical test theory statistics which can be viewed as proxies for the difficulty and discrimination, namely the proportion of correct responses and the item-rest correlation, respectively. For each item two discrepancy measures were computed: the difference between the proportion of correct responses to the item among slow responses and among fast responses,

(6) $\begin{array}{c}{D}_{1i}=\frac{{\sum }_p{x}_{pi}{t}_{pi}^{\ast }}{{\sum }_p{t}_{pi}^{\ast }}-\frac{{\sum }_p{x}_{pi}(1-t_{pi}^{\ast })}{{\sum }_p(1-t_{pi}^{\ast })},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{1i}=\frac{\sum _p x_{pi}t^*_{pi}}{\sum _p t^*_{pi}}-\frac{\sum _p x_{pi}(1-t^*_{pi})}{\sum _p (1-t^*_{pi})}, \end{aligned}$$\end{document}

and the difference between item-rest correlations of the item among slow and fast responses,

(7) $\begin{array}{c}{D}_{2i}=Cor\left(x_{i,slow},,,x_{+,slow}^{\left(i\right)}\right)-Cor\left(x_{i,fast},,,x_{+,fast}^{\left(i\right)}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{2i}=Cor\left( {\mathbf {x}}_{i,slow},{\mathbf {x}}^{(i)}_{+,slow}\right) -Cor\left( {\mathbf {x}}_{i,fast},{\mathbf {x}}^{(i)}_{+,fast}\right) , \end{aligned}$$\end{document}

where $x_{i,slow}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {x}}_{i,slow}$$\end{document} and $x_{i,fast}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {x}}_{i,fast}$$\end{document} are vectors of responses of all persons such that $t_{pi}^{\ast }=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}^*=1$$\end{document} or $t_{pi}^{\ast }=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}^*=0$$\end{document} , respectively; $x_{+,slow}^{\left(i\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {x}}_{+,slow}^{(i)}$$\end{document} and $x_{+,fast}^{\left(i\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {x}}_{+,fast}^{(i)}$$\end{document} are vectors of the numbers of correct responses to all the items excluding item i of all persons such that $t_{pi}^{\ast }=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}^*=1$$\end{document} or $t_{pi}^{\ast }=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}^*=0$$\end{document} , respectively.

To assess whether the observed values $D_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{1i}$$\end{document} and $D_{2i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{2i}$$\end{document} are plausible under conditional independence they can be compared to values drawn from the posterior predictive distribution of these measures given the data and the hierarchical model, which can be obtained using draws from the posterior distribution of the model parameters. The conditional independence model was estimated using a Gibbs Sampler. The prior distributions and the sampling procedure were specified following the specification of Bolsinova and Tijmstra (Reference Bolsinova and Tijmstra2016), which means that independent vague priors were used for the hyper-parameters (mean vector and covariance matrix of the item parameters, variance of speed, and correlation between speed and accuracy). Two independent chains with 10,000 iterations each (5000 iterations burn-in, 5 thinning) were used. Using each of these resulting 2000 samples a new replicated dataset was simulated according to the hierarchical model: $X_{rep}^{\left(g\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {X}}^{(g)}_{rep}$$\end{document} , $T_{rep}^{\left(g\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {T}}^{(g)}_{rep}$$\end{document} , where superscript g denotes g-th sample from the posterior distribution. In each replicated dataset $D_{1i}^{\left(g\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{1i}^{(g)}$$\end{document} and $D_{2i}^{\left(g\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{2i}^{(g)}$$\end{document} were computed for each item. For each item two posterior predictive p values were computed:

(8) $\begin{array}{cc}{p}_{1i}=& \frac{{\sum }_{g}I(D_{1i}>D_{1i}^{\left(g\right)})}{2000},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p_{1i}= & {} \frac{\sum _g \mathcal {I}(D_{1i}>D_{1i}^{(g)})}{2000}, \end{aligned}$$\end{document}

(9) $\begin{array}{cc}{p}_{2i}=& \frac{{\sum }_{g}I(D_{2i}>D_{2i}^{\left(g\right)})}{2000}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} p_{2i}= & {} \frac{\sum _g \mathcal {I}(D_{2i}>D_{2i}^{(g)})}{2000}. \end{aligned}$$\end{document}

If $p_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{1i}$$\end{document} is close to 0 or close to 1, it means that the difference between the proportion of correct responses among the slow and the fast responses observed in the empirical data is not likely under the model. Similarly, if $p_{2i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_{2i}$$\end{document} is close to 0 or is close to 1, the observed difference between the item-rest correlations is unlikely under the model. Figure 2 shows the histograms of the posterior predictive p values of the items for the model assuming conditional independence. The large number of extreme p values indicates that the model does not capture an important aspect of the data, namely that the items behave differently for the slow and the fast responses.

Figure 2.

Posterior predictive p values for the hierarchical model assuming conditional independence: a difference between the proportion of correct responses to an item for slow and fast responses; b difference between the item-rest correlations for slow and fast responses.

Next, we investigated whether the observed deviation from conditional independence can be explained by the extended hierarchical model, in which the model for response time is extended (see Eq. 4), while the same model for response accuracy (see Eq. 2) is used as in the hierarchical conditional independence model. In this extension of the hierarchical model, conditional independence as specified in Eq. 1 is violated, and this violation is taken to be fully explained using the additional parameter ${\lambda }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda _{i}$$\end{document} . This way of modelling conditional dependence is in line with the first approach described in the Introduction, that is, the distribution of response time is modelled conditional on whether the response is correct or not.

To determine whether this extension of the hierarchical model is able to fully explain the observed deviation from conditional independence in the data, we analysed to what extent the observed $D_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{1i}$$\end{document} and $D_{2i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{2i}$$\end{document} would be plausible under this model. Figure 3 shows the histograms of the posterior predictive p values for the extended hierarchical model. The situation has improved with respect to the difference in the proportion of correct responses, but not with respect to the difference in the item-rest correlations. The finding that there was only an improvement with respect to the differences in the proportion of correct responses could be expected, since the extended hierarchical model only allows for a shift in the mean conditional response time to correct the incorrect responses. That is, such a shift cannot account for any observed variation in the discriminative power of an item as a function of response time, as reflected in differences in the item-rest correlations. Based on these results, we suggest extending the hierarchical model such that it takes into account that both the proportion of correct responses to the items and their discriminative power might change as a function of response time.

Figure 3.

Posterior predictive p values for the model with an extra parameter for the difference in response times distributions of the correct and incorrect responses: a difference between the proportion of correct responses to an item for slow and fast responses; b difference between the item-rest correlations for slow and fast responses.

4. Residual Log-Response Time as a Covariate for the Parameters of the ICC

4.1. Model Specification

When considering the possible effects of having relatively fast or slow responses, we want to disentangle the particular response time from the overall speed of a person and the overall time intensity of an item. That is, it is not $t_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}$$\end{document} in isolation that informs us whether a response is relatively slow or fast, but rather the difference between $t_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}$$\end{document} and the expected response time for person p on item i. Two identical response times might differ with regard to whether they are fast or slow, depending on the speed of the corresponding persons and the time intensity of the items. Because of this, it may be reasonable to use a standardised residual of the log-response time (derived from Eq. 3) to capture the extent to which a response should be considered to be fast or slow. Let us denote the standardised residual log-response time of person p to item i by $z_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}$$\end{document} :

(10) $\begin{array}{c}{z}_{pi}=\frac{ln{t}_{pi}-({\xi }_i-{\tau }_p)}{{\sigma }_i}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} z_{pi}=\frac{\ln t_{pi}-(\xi _{i}-\tau _p)}{\sigma _i}. \end{aligned}$$\end{document}

If $z_{pi}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}>0$$\end{document} , it means that the response of person p to item i is relatively slow, while if $z_{pi}<0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}<0$$\end{document} it is relatively fast. The residuals are standardised in order to make the regression coefficients specifying the effects of residual time on accuracy comparable across items by taking into account the differences in ${\sigma }_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma _i$$\end{document} . If conditional independence between response times and accuracy given ability and speed holds, then the probability of a correct response to item i does not depend on whether the response is relatively slow or relatively fast, given ability and speed.

We suggest to use a time-related covariate both for the intercept and for the slope of the ICC, such that it can also capture the difference between the discriminative power of the items for the slow and fast responses as observed in the empirical data. Let the slope and the intercept in Eq. 2 depend on the standardised residual of the log-response time:

(11) $\begin{array}{cc}& \begin{array}{ccc}{\alpha }_{pi}& =& {\alpha }_{0i}{\alpha }_{1i}^{z_{pi}},\text{or equivalently}\\ ln\left({\alpha }_{pi}\right)& =& ln\left({\alpha }_{0i}\right)+ln\left({\alpha }_{1i}\right)z_{pi},\text{and}\end{array}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\begin{array}{rlll} \alpha _{pi} &{}=&{} \alpha _{0i}\alpha _{1i}^{z_{pi}}, \quad \text { or equivalently}\\ \ln (\alpha _{pi}) &{}=&{} \ln (\alpha _{0i}) + \ln (\alpha _{1i})z_{pi}, \quad \text { and }\\ \end{array} \end{aligned}$$\end{document}

(12) $\begin{array}{cc}& {\beta }_{pi}={\beta }_{0i}+{\beta }_{1i}{z}_{pi}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned}&\beta _{pi}=\beta _{0i}+\beta _{1i}z_{pi}. \end{aligned}$$\end{document}

Since the slope parameter in the two-parameter logistic model is restricted to positive values, a linear model for $ln\left({\alpha }_{pi}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ln (\alpha _{pi})$$\end{document} rather than for ${\alpha }_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{pi}$$\end{document} is used. Another reason for using a multiplicative effect for the slope instead of the linear effect is that the slope parameter is itself a multiplicative parameter. The parameters ${\alpha }_{0i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{0i}$$\end{document} and ${\beta }_{0i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{0i}$$\end{document} are the baseline slope and the baseline intercept of the item response function of item i, which refer to the responses $x_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$x_{pi}$$\end{document} which are given as fast as expected for person p on item i (i.e., $z_{pi}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}=0$$\end{document} ). The parameters ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} and ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} are the effects of $z_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}$$\end{document} on the slope and the intercept of the ICC. If ${\alpha }_{1i}=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}=1$$\end{document} or ${\beta }_{1i}=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}=0$$\end{document} it means that there is no effect of residual log-response time of the slope or on the intercept, respectively. The sign of ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} indicates whether relatively fast responses are more ( ${\beta }_{1i}<0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i} < 0$$\end{document} ) or less ( ${\beta }_{1i}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i} > 0$$\end{document} ) often correct than relatively slow responses. Likewise, depending on the value of ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} , relatively fast responses either contain more ( ${\alpha }_{1i}<1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i} < 1$$\end{document} ) or less ( ${\alpha }_{1i}>1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i} > 1$$\end{document} ) information about ability. As with any IRT model, these inferences should only be taken to hold generally, and may break down when very extreme or aberrant response processes are considered (e.g., with very extreme residual response times).

If one would assume that persons keep a constant speed across items (which is usually assumed within the hierarchical modelling framework), then ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} would be closely related to the conditional accuracy function (van Breukelen, Reference van Breukelen2005), since the residual response time does not reflect a change in effective speed as a latent variable. It may also be noted that the marginal model for response accuracy (i.e., after integrating out response times) is no longer a two-parameter logistic model. Although theoretically conditional independence between response time and accuracy can be checked by testing the fit of the two-parameter logistic model for response accuracy, this kind of test would have low power compared to tests designed specifically to detect conditional dependence such as the posterior predictive checks that are developed in this paper.

The full model allows the effects of the covariate to vary across the items. However, one might assume that the effect of responding relatively fast or slow is the same for all the items, choosing one of the constrained models: equal ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} and equal ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} for all items, equal ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} but varying ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} , or equal ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} but varying ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} . It may be noted that if one chooses to model only the effects on the intercept (i.e., varying ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} while ${\alpha }_1=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _1=1$$\end{document} for all items) then the model is similar in structure to the model of Ranger and Ortner (Reference Ranger and Ortner2012) with the exception that we consider a logistic model for response accuracy instead of a normal ogive model.

As in the hierarchical model that assumes conditional independence, we need to specify the higher-level models for the person parameters and for the item parameters. The dependence between the item parameters of individual items is modelled by a multivariate normal distribution for the vector $\{{\xi }_i,ln\left({\sigma }_i^2\right),ln\left({\alpha }_{0i}\right),ln\left({\alpha }_{1i}\right),{\beta }_{0i},{\beta }_{1i}\}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\{\xi _i,\ln (\sigma _i^2),\ln (\alpha _{0i}),\ln (\alpha _{1i}),\beta _{0i},\beta _{1i}\}$$\end{document} with a mean vector ${\mu }_I$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\mu }}_{\mathcal {I}}$$\end{document} and a covariance matrix ${\Sigma }_I$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{\mathcal {I}}$$\end{document} . Logarithmic transformation is used for the parameters which are restricted to positive numbers. A multivariate normal distribution is used for speed and ability with the same identifiability restrictions as in the conditional independence model: ${\mu }_{\theta }={\mu }_{\tau }=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mu _{\theta }=\mu _{\tau }=0$$\end{document} and ${\sigma }_{\theta }^2=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_{\theta }=1$$\end{document} . Hence, two person population parameters are freely estimated, namely the correlation between speed and ability (denoted by ${\rho }_{\theta \tau }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _{\theta \tau }$$\end{document} ) and the variance of speed (denoted by ${\sigma }_{\tau }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_{\tau }$$\end{document} ).

4.2. Estimation

For the estimation of the model we developed a Gibbs Sampler (Geman & Geman, Reference Geman and Geman1984; Casella & George, Reference Casella and George1992) implemented in the R programming language (R Development Core Team, 2006) to obtain samples from the joint distribution of the model parameters:

(13) $\begin{array}{c}f\left({\alpha }_0,,,{\alpha }_1,,,{\beta }_0,,,{\beta }_1,,,\theta ,,,\xi ,,,{\sigma }^2,,,\tau ,,,{\mu }_I,,,{\Sigma }_I,,,{\sigma }_{\tau }^2,,,{\rho }_{\theta \tau },|X,T\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} f\left( \varvec{\alpha _0},\varvec{\alpha }_{1},\varvec{\beta }_{0},\varvec{\beta }_1,{\varvec{\theta }},\varvec{\xi },\varvec{\sigma }^{2},{\varvec{\tau }},{\varvec{\mu }}_{\mathcal {I}},{\varvec{\Sigma }}_{\mathcal {I}},\sigma ^2_{\tau },\rho _{\theta \tau }\,|\, {\mathbf {X}},{\mathbf {T}}\right) , \end{aligned}$$\end{document}

which includes both the parameters of the individual persons and items and the hyper-parameters of the person population distribution and the item population distribution.

Although the variance of $\theta$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document} is constrained to 1, to improve the convergence of the model at each iteration of the Gibbs Sampler the full covariance matrix ${\Sigma }_P$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{\mathcal {P}}$$\end{document} is sampled and at the end of each iteration all parameters are transformed to fit the scale defined by ${\sigma }_{\theta }^2=1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_{\theta }=1$$\end{document} (see Appendix for details). We choose independent vague prior distributions for the item and the person hyper-parameters: $N(0,100)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathcal {N}(0,100)$$\end{document} for the means of the item parameters, half t-distributions with $\nu =2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nu =2$$\end{document} degrees of freedom and a scale parameter $A=2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$A=2$$\end{document} for the standard deviations of the item parameters, a marginally uniform joint distribution for the correlations between the item parameters (Huang & Wand, Reference Huang and Wand2013), and an inverse-Wishart distribution with 4 degrees of freedom and identity matrix $I_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {I}_2$$\end{document} as the scale parameter for ${\Sigma }_P$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{\mathcal {P}}$$\end{document}  (Hoff, Reference Hoff2009). Results are not sensitive to the specification of the prior scale parameter, because the posterior distribution is dominated by the data when the sample size is large (Hoff, Reference Hoff2009, p.110).

The estimation algorithm includes Metropolis–Hastings steps  (Metropolis, Rosenbluth, Rosenbluth, Teller, & Teller, Reference Metropolis, Rosenbluth, Rosenbluth, Teller and Teller1953) and a modification of the composition algorithm by Marsman et al. (Reference Marsman, Maris, Bechger and Glas2014). In the Gibbs Sampler, the model parameters are subsequently sampled from their full conditional posterior distributions given the current values of all other parameter (see Appendix for details).

For model comparison purposes, modifications of the algorithm have also been developed to estimate the constrained models (equal ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} and ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} , equal ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} but varying ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} , or equal ${\beta }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{1i}$$\end{document} but varying ${\alpha }_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha _{1i}$$\end{document} ), models with different time-related covariates (one might be interested in the effect of $t_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t_{pi}$$\end{document} , $t_{pi}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$t^*_{pi}$$\end{document} , as defined in Eq. 5, and $ln{t}_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ln t_{pi}$$\end{document} on the IRT parameters instead of $z_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}$$\end{document} ), the hierarchical model assuming conditional independence, and the modified hierarchical model with an extra parameter for the difference in the location parameters of the distribution of the response times given a correct and an incorrect response (see Eq. 4).

4.3. Model Selection and Goodness-of-Fit

To select the best model the deviance information criterion [DIC] can be used, because it adequately takes the complexity of hierarchical models into account (Spiegelhalter, Best, Carlin, & van der Linde, Reference Spiegelhalter, Best, Carlin and van der Linde2002). The DIC can be computed using the output of the Gibbs Sampler. First, at each iteration (after discarding the burn-in and thinning) the deviance is computed. For example, for the full model the deviance is:

(14) $\begin{array}{c}{D}^{\left(g\right)}=-2ln\left(f,\left(X,T|,{\alpha }_0^{\left(g\right)},,,{\alpha }_1^{\left(g\right)},,,{\beta }_0^{\left(g\right)},,,{\beta }_1^{\left(g\right)},,,{\theta }^{\left(g\right)},,,{\xi }^{\left(g\right)},,,{\sigma }^{2\left(g\right)},,,{\tau }^{\left(g\right)}\right)\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D^{(g)}=-2\ln \left( f\left( {\mathbf {X}},{\mathbf {T}}\,|\,\varvec{\alpha }_0^{(g)},\varvec{\alpha }_{1}^{(g)},\varvec{\beta }_{0}^{(g)},\varvec{\beta }_1^{(g)},{\varvec{\theta }}^{(g)},\varvec{\xi }^{(g)},\varvec{\sigma }^{2(g)},{\varvec{\tau }}^{(g)}\right) \right) . \end{aligned}$$\end{document}

This expression does not include the hyper-parameters ${\sigma }_{\tau }^2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2_{\tau }$$\end{document} , ${\rho }_{\theta \tau }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _{\theta \tau }$$\end{document} , ${\mu }_I$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\mu }}_{\mathcal {I}}$$\end{document} and ${\Sigma }_I$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{\mathcal {I}}$$\end{document} , because the distribution of the data is independent of the hyper-parameters given the individual item and person parameters. Second, the deviance is computed for the posterior means of the model parameters:

(15) $\begin{array}{c}\hat{D}=-2ln\left(f,\left(X,T|,{\hat{\alpha }}_0,,,{\hat{\alpha }}_1,,,{\hat{\beta }}_0,,,{\hat{\beta }}_1,,,\hat{\theta },,,\hat{\xi },,,{\hat{\sigma }}^2,,,\hat{\tau }\right)\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\hat{D}}=-2\ln \left( f\left( {\mathbf {X}},{\mathbf {T}}\,|\,\hat{\varvec{\alpha }}_0,\hat{\varvec{\alpha }}_{1},\hat{\varvec{\beta }}_{0},\hat{\varvec{\beta }}_1,\hat{{\varvec{\theta }}},\hat{\varvec{\xi }},\hat{\varvec{\sigma }}^{2},\hat{{\varvec{\tau }}}\right) \right) . \end{aligned}$$\end{document}

The DIC is equal to:

(16) $\begin{array}{c}DIC=\frac{{\sum }_g{D}^{\left(g\right)}}{G}+p_D,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} DIC=\frac{\sum _gD^{(g)}}{G}+p_D, \end{aligned}$$\end{document}

where G is the total number of iterations which are taken into account when computing the DIC and $p_D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$p_D$$\end{document} is the number of effective parameters which is equal to the difference $\left(\frac{{\sum }_g{D}^{\left(g\right)}}{G},-,\hat{D}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\left( \frac{\sum _gD^{(g)}}{G}-\hat{D}\right) $$\end{document} .

To evaluate the absolute fit of the best fitting model, posterior predictive checks for a global discrepancy measure between the data and the model can be used, for example the log-likelihood of the data under the model. For each g-th sample from the posterior distribution of the model parameters given the observed data, a replicated dataset ( $X_{rep}^{\left(g\right)},T_{rep}^{\left(g\right)}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbf {X}}_{rep}^{(g)},{\mathbf {T}}_{rep}^{(g)}$$\end{document} ) is simulated under the model and the log-likelihood is computed both for the observed data and the replicated data:

(17) $\begin{array}{cc}L{L}_{obs}^{\left(g\right)}=& ln\left(f,\left(X,T|,{{\alpha }_0}^{\left(g\right)},,,{\alpha }_1^{\left(g\right)},,,{\beta }_0^{\left(g\right)},,,{\beta }_1^{\left(g\right)},,,{\theta }^{\left(g\right)},,,{\xi }^{\left(g\right)},,,{\sigma }^{2\left(g\right)},,,{\tau }^{\left(g\right)}\right)\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} LL_{obs}^{(g)}= & {} \ln \left( f\left( {\mathbf {X}},{\mathbf {T}}\,|\,\varvec{\alpha _0}^{(g)},\varvec{\alpha }_{1}^{(g)},\varvec{\beta }_{0}^{(g)},\varvec{\beta }_1^{(g)},{\varvec{\theta }}^{(g)},\varvec{\xi }^{(g)},\varvec{\sigma }^{2(g)},{\varvec{\tau }}^{(g)}\right) \right) ,\end{aligned}$$\end{document}

(18) $\begin{array}{cc}L{L}_{rep}^{\left(g\right)}=& ln\left(f,\left(X_{rep}^{\left(g\right)},,,T_{rep}^{\left(g\right)},|,{{\alpha }_0}^{\left(g\right)},,,{\alpha }_1^{\left(g\right)},,,{\beta }_0^{\left(g\right)},,,{\beta }_1^{\left(g\right)},,,{\theta }^{\left(g\right)},,,{\xi }^{\left(g\right)},,,{\sigma }^{2\left(g\right)},,,{\tau }^{\left(g\right)}\right)\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} LL_{rep}^{(g)}= & {} \ln \left( f\left( {\mathbf {X}}_{rep}^{(g)},{\mathbf {T}}_{rep}^{(g)}\,|\,\varvec{\alpha _0}^{(g)},\varvec{\alpha }_{1}^{(g)},\varvec{\beta }_{0}^{(g)},\varvec{\beta }_1^{(g)},{\varvec{\theta }}^{(g)},\varvec{\xi }^{(g)},\varvec{\sigma }^{2(g)},{\varvec{\tau }}^{(g)}\right) \right) . \end{aligned}$$\end{document}

The posterior predictive p value is the proportion of samples in which the observed data are less likely under the model than the replicated data. If the posterior predictive p value is small, then the data are unlikely under the model. The goodness-of-fit can be further evaluated using posterior predictive checks based on $D_{1i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{1i}$$\end{document} and $D_{2i}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_{2i}$$\end{document} statistics (see Eqs. 6, 7).

6. Simulation Study

To assess parameter recovery of the model a simulation study based on the empirical example was performed. In this applied paper we are not aiming at showing the performance of the model for various combinations of item hyper-parameters, person hyper-parameters, test and sample sizes, but rather mainly at the specific combination of those factors from the empirical data that we are dealing with. Therefore, in the simulation study we used the estimates of the item and the person hyper-parameters to simulate replicated datasets of the same sample size (1000 persons) and the same number of items (49). To investigate how parameter recovery is affected by a decrease in sample size and number of items and to evaluate the applicability of the model in a wider range in conditions, three more conditions were considered: $N=500,n=49$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=500, n=49$$\end{document} ; $N=1000,n=25$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=1000, n=25$$\end{document} ; $N=500,n=25$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=500, n=25$$\end{document} . For each condition, 100 datasets were simulated under the full model with $z_{pi}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$z_{pi}$$\end{document} as a covariate for the parameters of the ICC. In each replication the model was fitted using the Gibbs Sampler with one chain of 10,000 iterations (including 5000 iterations of burn-in).

Table 4 shows the simulation results: the average expected a posteriori estimates of the hyper-parameters and the number of the replications (out of 100) in which the true value was within the 95 $\%$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\,\%$$\end{document} credible interval. First, let us consider the results obtained when the same sample size and the number of items as in the empirical example were used. The mean vector of the item parameters, the standard deviation of speed and the correlation between speed and ability were correctly recovered. The correlations between the item parameters are estimated to be closer to zero than the true values, and the variances of the item parameters are slightly overestimated. However, this bias is relatively small and does not influence the substantive interpretation of the relations between the item parameters.

Table 4.

Results of the simulation study: the expected a posteriori (EAP) estimates of the hyper-parameters averaged across 100 replications and the number of replications in each the true value was within the $95\%$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$95\,\%$$\end{document} credible interval.

When the sample size was reduced (500 instead of 1000), the results were not seriously affected. However, when the number of items was reduced (25 compared to 49), the bias of the variances of item parameters and of the correlations between them increased. This is likely due to the fact that these hyper-parameters were estimated based on a relatively small sample of items. In the condition with $N=1000$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=1000$$\end{document} and $n=25$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=25$$\end{document} , the number of $95\%$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$95\,\%$$\end{document} credible intervals which contained the true value is smaller than when the sample size was smaller. This can be explained by the fact that the posterior variance is smaller when the sample size is larger. Overall these results indicate that for accurate recovery of the item hyper-parameters test size should not be too small.