Introduction

Our knowledge of the primary visual pathway is voluminous but fragmentary. We know a great deal about the behavior of cells in the major way stations of this pathway – the retina, lateral geniculate nucleus (LGN), and striate cortex – but know rather little about how they function together to process the visual image. At each level, we know about various classes of cells, and about the spatial, temporal, and chromatic aspects of their receptive fields. But we know rather little about how cells at one level converge upon those at the next, and we know still less about the function of these receptive fields, and why particular receptive fields are employed at each level. An answer to these questions constitutes a theory of the algorithmic architecture (algotecture) of the visual pathways. This theory must address data from many sources, and must adhere to numerous mathematical constraints. In what follows, I will enumerate some of the considerations that must guide the attack upon this fundamental problem in vision and brain science. I will then sketch the beginnings of a candidate theory for one part of cortex. The theory raises as many questions as it answers, but that is not necessarily a bad thing, and at least it provides a concrete hypothesis that can be tested, modified, or replaced by a better theory.

I will restrict my focus to one portion of the visual pathway: the oriented simple cells of primate striate cortex. This system, while much less than the whole primary visual pathway, is nonetheless immense and fundamental.