In “The puzzle of ideography” Morin argues that ideographic systems of pictorial signs are limited in their communicative capacities because “Graphic codes can only be standardized for a limited number of meaning–symbol mappings” (target article, abstract). Although we agree that ideographies are limited systems, we disagree with Morin's reasons why, especially the notion that the graphic modality itself is limited in its semiotic capacities. Specifically, ideographies attempt to artificially force a graphic system to behave like a writing system, which itself is an adaptation of the vocal modality into the graphic modality.

First, it is important to recognize that “standardization” or “conventionality” is orthogonal to symbolicity (Peirce, Reference Peirce and Buchler1940). These notions are often conflated (de Saussure, Reference de Saussure1972 [1916]), but standardization or conventionality is how much a signal is patterned and recognized across individuals. These signals (idiosyncratic or patterned) correspond to meanings through various interfaces characterizing their signification (i.e., iconicity, indexicality, and symbolicity). To clarify: A signal in sounds or graphics can be standardized or not, and how that signal corresponds to conceptual structures characterizes its signification(s).

Despite their stereotype as “arbitrary symbols,” spoken languages display all types of signification, as in Figure 1a (Clark, Reference Clark1996; Ferrara & Hodge, Reference Ferrara and Hodge2018), and so does the graphic modality (Fig. 1b). Because all modalities use all types of signification, symbolicity is not the issue, but rather the question is about standardization.

Figure 1. (a) Vocal and (b) graphic signification, and standardized (c) components of drawings, (d) combinatorial signs, and scene templates (e) without and (f) with reference to other graphics.

Contrary to Morin's statements, the graphic modality displays voluminous standardization at multiple levels of complexity. Although small “pictographs” like hearts, stars, or peace or radiation signs are easy to recognize as standardized (as in ideographies), all graphics use standardized building blocks (Arts & Schilperoord, Reference Arts, Schilperoord and Fernandes2016; Cohn, Reference Cohn2013; Wilson & Wilson, Reference Wilson and Wilson1977). Drawings are constructed from low-level visual patterns, such as how people draw eyes, headshapes, houses, and flowers, as in the conventionalized hands by three comic artists in Figure 1c. Graphics also use classes of combinatorial signs, such as the inventory of elements that float above characters heads or replace eyes, which use systematic and symbolic meaning-making (Fig. 1d), in addition to visual vocabulary like motion lines, impact stars, speech balloons, and other highly standardized, culturally variable, graphic codes. Although these small visual “morphemes” can combine into novel pictures, larger units can also be systematized, whether as templates of abstract scenes (Fig. 1e) or templates in reference to other scenes, which often invoke symbolic meanings (Fig. 1f). These observations contrast with the phenomenologically based idea that drawing and graphics are about articulating one's idiosyncratic vision of what one sees, despite this notion being unsupported by cognitive, cultural, and developmental research (Wilkins, Reference Wilkins and Cohn2016; Wilson, Reference Wilson, Hardiman and Zernich1988) and having erroneous origins (Willats, Reference Willats2005).

Visual representations also allow a range of conventionalized sequencing. Many patterns of two-unit sequences persist to show causative before–after relations, contrasts, or analogies (Schilperoord & Cohn, Reference Schilperoord and Cohn2022). Longer sequences often provide visual lists of related images, such as what is allowed in a park or on an airplane (Cohn & Schilperoord, Reference Cohn and Schilperoord2022). Visual narratives also use recursive combinatorial structures for sequential images displaying structural features of linguistic grammars, but operating at a higher-level information structure than the organization of nouns and verbs (Cohn, Reference Cohn2013; Cohn & Schilperoord, Reference Cohn and Schilperoord2022). These visual narrative sequences are natural productions of sequential images, and the specific sequencing constructions they use have been shown to vary across cultures' comics (Cohn, Reference Cohn, Grishakova and Poulaki2019), again indicating culturally relative standardization, not universality. In addition, their processing invokes the same neural responses as linguistic syntax and semantics, and the understanding of these visual sequences requires proficiency that is acquired through exposure to and practice with those cultural graphic systems (Cohn, Reference Cohn2020).

In contrast to these natural graphic systems, ideographies use a basic lexicon of simple graphics attempting to have “word” levels of information, often created top-down by individuals, rather than emerging from a language community. These graphics are intended to be sequenced through a syntax, but because graphics do not naturally afford sentence-level combinatorics, they end up parasitic to spoken languages. Writing systems themselves are adaptations of the spoken into the graphic modality, but ideographies then assume graphics should behave like writing systems – forced into sentence-level sequences – to take on “linguistic” properties. This is why ideographies are largely invented by specific people (e.g., Blissymbolics, as in Morin's Fig. 4), because they do not proliferate instinctively through human history or cultures.

These invented ideographies are thus systems that attempt to mimic the structures of speech/writing which serve as people's reference point for what “linguistic graphics” should be like. However, this denies the affordances and linguistic properties already displayed by natural graphics in the first place. It should be no wonder then that ideographies don't work.

There is a clear analogue to this in the bodily modality. Sign languages are natural linguistic systems that optimize the affordances of the bodily modality and thereby do things in ways that differ from the structure of speech (Liddell, Reference Liddell2003). Yet attempts persist to force sign languages to have the properties of spoken languages, like Manually Coded English, which maps the lexicon and grammar of spoken English onto the body (Supalla, Reference Supalla, Fischer and Siple1991). As adaptations of one modality to another, these systems deny the bodily affordances that natural sign languages display, just like ideographies deny the affordances apparent in natural graphic systems.

To conclude, ideographies are limited because they attempt to make the natural expressive graphic modality behave like writing, itself a conversion of the spoken modality into graphics. This quality undermines Morin's claim that “understanding why ideography has not worked in the past may help us understand how technology could make it work in the future” (target article, sect. 1, para. 8), because ideographies' unnaturalness will never “make it work” (target article, sect. 1, para. 8). Rather, this discussion raises the importance of investigating the affordances of all our modalities and their meaning-making capacities, and especially a greater integration of graphics into the study of the mind and cognition.