The state of the art in any science includes the criteria for evaluating research. Like every other aspect of science, it has to be developed. The criteria for evaluating AI research are not in very good shape. If we had better standards for evaluating research results in AI the field would progress faster.

One problem we have yet to overcome might be called the “Look, ma, no hands” syndrome. A paper reports that a computer has been programmed to do what no computer program has previously done, and that constitutes the report. How science has been advanced by this work or other people are aided in their research may not be apparent.

Some people put the problem in moral terms and accuse others of trying to fool the funding agencies and the public. However, there is no reason to suppose that people in AI are less motivated than other scientists to do good work. Indeed I have no information that the average quality of work in AI is less than that in other fields. I have grumbled about there being insufficient basic research, but one of the reasons for this is the difficulty of evaluating whether a piece of research has made basic progress.

It seems that evaluation should be based on the kind of advance the research purports to be. I haven't been able to develop a complete set of criteria, but here are some considerations.