THE bacterium Myxococcus xanthus, which accomplishes so much with a few signals, moves by gliding on slime trails. The exact mechanism of this action is unknown. But it is clear that using and following an existing slime trail is easier than laying one down from scratch, and the more that bacteria follow a trail, the more slime is laid down. It is observed that a bacterium approaching an existing slime trail at an angle will change direction to follow the trail. If the trail is not used, it dries up. Ants and tropical termites, and even some bees, use odor trails to guide their fellows to food sources. Trails that are used successfully to find food are reinforced on the return trip. If a trail is not used, the pheromones quickly evaporate and the trail vanishes. The mechanism is of such value that it has evolved independently by different routes: from “tandem running” in ants and from an “alarm trail” to a damaged portion of the nest in termites.

These organisms have evolved a remarkable implementation of reinforcement learning. The qualitative foundations of reinforcement learning were laid by the American psychologist William L. Thorndike at the beginning of the twentieth century. He was an undergraduate English major, but he changed direction upon reading William James's Principles of Psychology. Thorndike then went to Harvard to study with James, and James even let Thorndike raise chickens for animal learning experiments in the basement of his house.