THE PRECISE definition and understanding of a task is crucial for its use as a measurement tool. In the previous chapter we have seen how tasks can be characterised in terms of their policies and their difficulty viewed as a search problem. Setting the perspective on the policies is helpful, but we need to pursue this direction further. A test is much more than just a task. In what follows we will introduce tools to examine how the performance of an agent changes for a range of difficulties. When building a test, we will focus on how to sample from the whole set of tasks and instances to obtain an appropriate pool of items, which maximise the information they take about the subject. This choice of items can be performed by diversity, difficulty and also by discriminating power, a challenging issue that is crucial for the design of adaptive tests.

AGENT CHARACTERISTIC CURVES

In our eagerness for quantification, we usually summarise a phenomenon by a single number. For instance, when we want to evaluate an agent for a given task, we sample from the task and record its aggregate response. This value can then be compared with those of other agents. Should not this be enough for a cognitive test and the comparison of agents? Unfortunately, much information is lost whenever we summarise the results of many task instances into a single number. The study of the responses for single instances can provide valuable information about how the agent performs but, most importantly, it can give us ways of designing better tests.