3.1. Real-time narrative of sound

In A Reflection in the Brook, the live performance of the violin plays an important role. On the sound level, it is the source of the material for the real-time sampling and morphing of the max audio program, which is the ‘morphing base’ for all the electronic music sound in the work. On the other hand, some of the parameters output by the real-time sound sampling and analysis module are also transmitted to the real-time video processing computer (program) via a wireless network. These data are used to synchronise the live audio with the live video and to visualise some of the audio signals, in short, the violin is also used to trigger and control the video changes in real time.

The opening of the work is introduced by the interplay of several sets of figures that using special bowings for the violin and real-time audio. After a brief atmospheric buildup, the solo violin, like a storyteller, begins to play the lyrical, long melody of the first section a free rhythmic E–F♯–B–E motive of the theme (Figure 3). The Max real-time audio program samples the melodic theme in real time and then adds multiple delays and reverbs at different time positions to make the distorted electronic music sound like echoes in the mountains. The music in the introductory passage focuses on the heroine, Xiaoqing, living alone in the lonely mountains. The composer has effectively created the lonely situation of Xiaoqing, through the melody of the song, the multilayered overlapping delayed echoes, the changes in the surround sound field and the simplicity of the layering of volumes, which sets the scene for the conflict that follows.

Figure 3. Core materials of the melodic theme E–F♯ –B–E in violin part.

The ‘Mirror’ section continues the same musical introduction as the introductory section, except that the music begins to take on a more ‘dissonant’ character. Here the composer focus on the dialogue between Xiaoqing and herself in the mirror. This ‘mirroring’ is multifaceted, with the violin playing representing the real Xiaoqing, a series of thematic figures consisting of special bowings – including ricochet on the bridge with the hair of the bow, ricochet with the wood of the bow, and continual pizzicato – creating special sound effects that represent a mirror image of Xiaoqing (Figure 4), and a completely abstract sound that has been distorted by various computerised real-time audio processing techniques to symbolise another, more distant mirror image of reality. The violin playing in this section is highly improvised – the violinist plays the first thematic figure and then, based on real-time feedback from the computer, randomly selects a subsequent figure from a number of pre-designed figures. The computer analyses the data on the pitches played to realise four feedback modes: separation of the performance fragments into finely divided and dense figures; change the pitch of phrases and regroup them; simulated pizzicato strings; and random generation of harmonic sounds. The violin and the computer are constantly ‘listening’ and ‘answering’, creating a dialogue and interaction between the human and the computer. The sound also alternates between ‘reality’ and ‘distortion’, with layers of metamorphosis and increasing tempo, foreshadowing the conflicting and complex feelings of the protagonist as she constantly asks questions of her ‘mirror’ self in her search for identity but cannot find the answers.

Figure 4. Core materials of the ‘mirror theme’ in violin part.

‘Death of Xiaoqing’ is the climax of the work and is completed by the ‘Death’ and the ‘Epilogue’, which symbolises the sublimity of the soul. In the process of expression of the protagonist’s death, Xiaoqing’s struggle is intense – the music develops in a highly dissonant manner: the dissonance of the timbre, the density of texture, the rhythmic complexity and the loudness all increase. The Max audio program is overlaid in real time with delay effects with a certain rhythm and highly irregular delayed voices, creating a disordered, dense rhythm. Later in the ‘Death’ section, some MSP formant processing is added to the real-time sound processing, intensifying the tension of the timbres changes, making the sound increasingly torn, thick and discordant, presenting an orchestral, group effect, and with a few rapid violin glissandos pushing the discordant sound to the extreme, heralding the arrival of the “Death” moment. The extremely dark, dissonant sound energy quickly diminishes as the violin and the computer’s real-time audio section begin to play long hymn tunes together, swirling around the entire sound field, diminishing in volume and fading away, as if Xiaoqing’s soul is leaving the body.

After a moment of silence, a lighter motive which is symbolic of the ‘Soul Sublimation’ of the protagonist leads into the ‘Epilogue’ section. The E–F♯–B–E scale, with E as the dominant note, is in fact a variation on the ‘Introduction’ theme. And there are many similarities between the musical processing of the entire ‘Epilogue’ section and the ‘Introduction’ section, both introduce and create a mutual echo between the beginning and the end. The tone of the epilogue is bright, warm and harmonious, in contrast to the highly dissonant ‘Death’ section that precedes it, revealing a certain sense of ‘purification’. The violin is dominated by a melodic and functional harmonic sound, and the sound processing in the Max audio program continues the method of adding multiple time delays to real-time audio samples as in the ‘Introduction’ section (Figure 5). The difference here is that the delayed part gradually increases with each sample node and are carefully designed in advance in terms of rhythms, which are added in real time to vary the pitch of the different ranges based on the pitch frequency of the live performance, thus constantly gathering the power of the part and creating a group-like sound effect. The music of the ‘Epilogue’ is significantly brighter and more powerful than that of the ‘Introduction’. The music here is hopeful and makes the ‘Epilogue’ the most glorious section of the piece, forming two juxtaposed sections of climax with the preceding ‘Death’, the contrast between them breaking the conventional ‘arch’ structure. It is easy to see the composer’s approach to the ‘Death of Xiaoqing’ in the arrangement of the music – death is not the end, but the beginning of a new life, and this beginning is open-ended. It gives the audience space for expectation and imagination of Xiaoqing’s ‘rebirth’.

Figure 5. Adding multiple time delays to real-time audio samples in Max/Msp (Wang Reference Wang2021: 88)

3.2. Real-time narrative of images

The artistic presentation of A Reflection in the Brook is multilinear. Each medium is different in its approach and presentation, and the mode of interaction between the mediums changes according to the needs of the performance, thus expanding the expressive space of the work.

As a highly dramatic work, the image presentation of A Reflection in the Brook is particularly important. It not only provides the thread of the story, moves the plot forward and switches scenes, but also helps the music to convey the character’s inner struggles and intense dramatic conflicts to the audience. All the images in A Reflection in the Brook are based on real-time generation and real-time interaction. Some of them are triggered by the sound signals from the scene, others are generated by capturing the actors’ movements. The ‘real time’ makes it difficult to create and perform, requiring a precise connection between sound and image and a rhythmic synchronisation. From a creative conceptual point of view, this narrative approach also gives the work the character of new media art, reflecting the composer’s aesthetic awareness of reshaping traditional national culture with a modern mindset.

The four main visual modules of the work’s images correspond to the ‘Introduction’, ‘Mirror’, ‘Death’ and ‘Epilogue’. The narrative themes in the four sections, symbolising the different psychological states of the protagonist in the four periods. The author has briefly summarised the basic logic of the meaning and technical realisation of the visual themes in the work, as shown in Table 2.

Table 2. The meaning and basic technical logic of visual themes

As can be seen from Table 2, the four visual modules with clear symbolic meanings are all completed by the jitter programming system, with each thematic module corresponding to a different programming that reflects the changing fortunes of the protagonist through different visual representations.

The ‘line drawing’ theme in the first visual module tends to an atmospheric description, where the composer does not aim for complete synchronisation of sound and image, but keeps the two together in keynote. The music here is simple, reflecting a mood of solitude. The vision uses a simple presentation – a camera captures the image of the dancer’s performance and triggers the jit.qt.grab object in the vision program to convert the image into matrix data in real time, where the parameters inside the cell are monitored and analysed. Since the difference between the cell parameters is a direct result of the visual difference between light and dark. The lower the cell parameters, the greater the number of lines generated. The jit.gl.mesh object dynamically generates line patterns in low parameters areas, creating an effect similar to the ink lines in traditional Chinese painting. These black and white lines are intertwined with light and dark, setting off a cold and bleak visual tone, indicating the loneliness and anguish of the characters.

The second section focuses on the creation of the ‘Mirror’ effect, which is an important plot in the piece. Historical interpretations of Feng Xiaoqing’s ‘shadow love’ are based on her particular fondness for looking in the mirror – her obsession with the water reflection and herself in it. In contrast to previous scholars’ views, Jin Ping offers an alternative interpretation of this act. The ‘mirror image’ passage reflects Xiaoqing’s continuous search for herself – in the face of the unchangeable reality of her time, she tries to discover herself by ‘asking herself questions’ in the mirror. The music and images here present a sense of exploration, symbolising the awakening of the protagonist’s feminine consciousness. Here the inner conflict of the characters is revealed, but not to the extent of an intense conflict. In the visual process, the author has designed two different ‘mirroring’ effects for Xiaoqing’s ‘self-searching’. The first is a real-time delay of the image: the ‘jit.qt.grab’ object acquires the image of the actor’s performance from the camera. The 30 frames are pre-stored into a ‘buffer’ using the ‘jit.matrixset’ object, the five ‘jit.matrixset’ objects are used to play the 30 frames at staggered times, adjusting the speed of the delay in real time to create a ‘staggered’ delay effect. It is a gentler mirroring effect, as if to speak of the inner confusion of the character. The second is a more complex ‘multi-person dance’ effect, showing Xiaoqing’s constant search for ‘who am I?’ in multiple mirror images. In terms of technical means, a combination of camera-delayed recording playback and real-time performance by the actors is used. The camera records the first theme performed by the dancers in its entirety, then gradually fades into the previously recorded first theme as the dancers perform the second theme and combines it with the second theme video being recorded in real time, thus achieving a real-time interactive effect between the dancer’s performance and the multiple digital images.

In ‘Death of Xiaoqing’, the composer uses a completely abstract ‘illuminated geometry’ to express the intense inner struggle of the character. The image begins to interact with the sound in a highly synchronised, real-time manner. The position of the light is driven by the sound signal, the iPad remote control and the kinect joint position. The switching speed of the graphics is directly controlled by the bow speed of the violin. It continues through to the epilogue. The bright, dazzling graphic tones contrast with the darkness of the visual backdrop, as if plunging into an abyss, making the ‘Death’ the darkest part of the piece. The abstract graphics here twist, split, distort and switch in rapid succession to the extremely incongruous music. A dazzling light constantly shines against the dark and depressing backdrop, as if the protagonist is in fierce confrontation with the hideous forces of reality. Ultimately, these struggles seem in vain – Xiaoqing is in a frenzied emotion as she dies. The violin’s fierce bowing shifts the digital image rapidly between eight still images and a black screen, matching the extremely tense body movements of the dance, until the node symbolising the death of Xiaoqing comes to an abrupt end. The theme of ‘Particle Eruption’ is presented, with large, black, dynamic particles rising in an eruption behind the ‘dead’ Xiaoqing, a metaphor for the gushing of blood, representing the ultimate release of emotion and catharsis (Figure 6). Here the composer uses kinectFootnote ⁴ to position the dancer’s body and uses the body coordinates to control the coordinates of the radiation source, thus ensuring that the particles are always emitted from the dancer’s back as she moves.

Figure 6. The actress playing Xiaoqing moving with particles erupting from her back in the performance.

The ‘Epilogue’ section is the brightest of the piece, and one that is rich in the spirit of ‘romanticism’. After a moment of silence, the dancer rises slowly from the stage. Abstract shapes made up of luminous dots and lines appear, accompanied by music and dance body movements, and become geometric halos of ‘hope’ (Figure 7). For the first time, these halos of light appear in bright colours other than black and white, symbolising the rising of Xiaoqin’s soul in hope (Figure 8). In the visual processing of Xiaoqing’s death, the composer breaks away from the traditional impression of an oppressed woman dying of depression in a feudal society. Her extreme calmness and beauty express a feminine bravery that beyond reality and embodies her silent yet powerful resistance. From the ‘dancer’s resurrection from the dead’ to the ‘bright abstract figure of rebirth’, it is clear that the meaning of Xiaoqing’s death is more of a yearning for rebirth and light. The author’s anticipation of Xiaoqing’s rebirth is full of positive expectations. This expectation goes beyond life and death in the secular sense and explores the continuation of spiritual forces beyond the material and physical.

Figure 7. In the finale section of the performance, Xiaoqing interacts with a bright abstract pattern, signifying ‘new life’.

Figure 8. Xiaoqing’s rebirth.

4.1. Visu-audition

In multimedia works, the relationship between the visual and auditory senses is commonly perceived as a superposition of the two senses, but this perception is still based on the idea of the visual and auditory senses as separate entities. Michel Chion, a leading French sound theorist, has argued that ‘audiovisual’ is a total perception, not a simple addition of two different perceptions, sound and image. Like a chord in music, it is not a superposition of three different notes, but a holistic form, defined by the intervals between the notes (Chion Reference Chion2014: 1). In order to demonstrate that the two senses are not simply additive, equal or symmetrical, Chion cites two concepts: ‘audio-vision’ and ‘visu-audition’ (Chion Reference Chion2014: 198-99),Footnote ⁵ indicating two different audiovisual subordination relationships in the phenomenon of ‘audiovisual association’.

From previous listening experiences, most audiovisual representations in multimedia electronic music tend to be more in favour of the latter (visu-audition), that is, ‘the type of perception deliberately focuses attention on the sense of hearing, and our listening is accompanied, emphasized and aided by an image content, which influences what we hear and may lead us to project a particular perception onto it’ (Chion Reference Chion2014: 199). To be more precisely, unlike many works that begin with a visual presentation and then create a sound for the visual, the subject of multimedia electronic music is sound, and the visual presentation is an extension of the sound presentation, a visual representation of the meaning of the sound in the mind of the creator. And this audiovisual relationship applies to A Reflection in the Brook as well, even though it is a fully real-time multimedia production.

Although the sounds and images in A Reflection in the Brook occur ‘simultaneously’ in terms of expression, there is a sequence of first sound first and then an image in the composition. The visual motifs that most embody the conflict and inner struggle – the ‘glowing abstract geometry’ – are based on a computer program that visualise the audio signal in real time (Visualisation Routine), thus achieving a highly synchronisation and connection with the musical content. The images in the ‘Introduction’ and ‘Mirror’ section, although not directly generated from the sound, are based on the visual design of the sound theme and remain consistent with the tone and image of the sound. In the climax, Xiaoqing’s ‘Death’ section, a large number of abstract shapes twist, split, distort and switch in the darkness of the backdrop, reflecting not only the intense inner torment and struggle of the protagonist on an emotional level, but also giving the audience a visual guidance on a perceptual level – a directional and spatial guide to the movement of sound. As Chion explains, ‘Our listening is accompanied, emphasized, and assisted by an image content’ (Chion Reference Chion2014: 199). Thus, the images in A Reflection in the Brook are still an extension of sound, an image of sound, but in a mature work this ‘sequential’ order of audiovisual elements is intentionally fused or balanced, and the audience’s experience is an artistic whole in which the audiovisual adds value to each other.

4.2. Imitation vs counterpoint

The idea that sound is not an isolated auditory art has been extensively researched, and this is clearly evident in the theory and practice of film sound. In many cases, there is an interaction between the auditory and visual senses and even between the other senses, a phenomenon known as ‘interoception’, which is based on the ‘transmodal perception’Footnote ⁶ of the human being. Thus, people’s understanding of sound imagery can often rely on the experience of vision or even other senses. Conversely, sound can also be acted upon by other perceptions – influencing the perception of form, speed, time, intensity and space. This multidimensional sensory ‘association’ can effectively help to match sound imagery with visual images, making the ‘invisible’ sound ‘tangible’.

Therefore, in the creation of multimedia electronic music, two more common logics of audiovisual integration have developed. One is the association of a figurative, clearly symbolic sound theme with an equally figurative visual theme that directly represents the content conveyed by the sound. Similar to the reverse application of sound to the direct reproduction of visual images in film (e.g., hearing a car – seeing a car.) This is one of the more common techniques used by most Chinese composers in multimedia electronic music composition, where the establishment of visual images relies on previous visual experience of sound, which is mostly derived from cultural and historical factors, and the meaning it conveys is more intuitive. The other is the multiplication of abstract sound and abstract visual images in terms of colour, space, form and texture, correlating the morphological features of the sound with the stylistic elements of the visuals, such as ‘linear’, ‘granular’ and ‘fragmented’. This approach is more focused on the combination of the modalities.

In terms of composition, these two approaches are similar to the ‘mimetic’ approach to music composition, where one medium imitates another in terms of image, or form. In A Reflection in the Brook, however, the combination of audio and visual takes a different approach. Sound and image are like two separate ‘parts’, which, in terms of their horizontal temporal development, are presented as narrative clues, each presenting four thematic sections: ‘Introduction’, ‘Mirror’, ‘Death’ and ‘Epilogue’. In vertical audiovisual relationships, there are points of close integration – although not in a figurative way. For example, at the point of Xiaoqing’s death, the sound is expressed as a fading, ‘dissipating’ multivocal long tone, signifying that Xiaoqing’s soul is leaving her body. The image uses a large eruption of black particles to symbolise the gushing of blood and the release of emotion. The long tones and the particles are not unified or even contradictory in terms of visual image or form, but they constitute a kind of narrative complement here, and together they emphasise a moment, create a rhythm, co-construct time and space for the imagination. If the same analogy of musical composition is used, this approach is similar to the ‘counterpoint’ of polyphonic music. The vocal parts are both closely linked and varied, independent but also forming a vertical correspondence in terms of important beats.