腾讯视频链接 (video link for visitors in China): http://v.qq.com/page/o/i/u/o0304j7xjiu.html
The Humanistic Movement (THM) is a generative music system. It collaborates with a human dancer, and improvises music using the dancer’s body movements as its inspiration source.
THM is not a body-instrument. It hates one-to-one mapping from gesture to sound, which awkwardly limits the dancer’s movements, making her, rather than the system, responsible for the composition. THM wants the dancer to dance, with confidence that the system will take good care of the music.
Master’s Spirit in Markov Chains
And the dancer need not worry that without her direct control, the system would generate ugly sounds. In fact, THM’s musical style comes from Wolfgang Amadeus Mozart. It has calculated second-order Markov Chains1 of note progression of the first movement of Mozart’s Piano Sonata No. 11 in A major, so it really has Mozart’s style in mind. For every two-note sequence in this work, THM knows the frequencies of all possible following notes. For example, it knows that after the note sequence E4 B3, the frequencies of the following notes are:
With this knowledge, when generating a new note, THM looks back for the last two notes it has generated, and looks them up in the Markov chains. It can then follow the frequency table of the following notes, so that it plays in the style of the great master. Because of the randomness built in this process, the music is new in every performance, yet the system has a consistent style all the time, just like a real musician with her own personality.
Movement-Influenced Melody and Tempo
While THM has its own music ideas, the dancer still has influence on the composition, with the direction and acceleration of her arm captured by the Myo armband in real time. THM always bases its work on the current state of the dancer, making the music in tune with the dance.
Whenever a new note comes, the system first examines whether the dancer’s arm is pointing higher or lower than the its direction at the last note, and will accordingly look for relatively higher or lower notes in its style reference. In this way, the dancer has influence on the melody with her dance movement. Meanwhile, she has not to be overstressed, since the responsibility of deciding the exact notes are on the shoulder of THM. So the dancer can move freely, and feel that the melody flows in accordance with her movements.
The relation between arm direction and note progression is most perceivable when the music has a slow tempo. When the music goes faster, the link becomes harder to perceive. Furthermore, this is still a low level mapping which cannot represent higher level states, such as emotion, of the dancer. In order to improve its intelligence, THM introduces tempo alteration in accordance with the intensity of the dancer’s movements. In some parts of the music, the system examines the acceleration the dancer’s arm, and generates fast (e.g. eighth notes) or slow (e.g. quarter notes) notes according to the reading. The acceleration indicates the speed and complexity of the dancer’s movements, and therefore is a good representation of her emotion. By mapping it to the intensity of the music, THM receives a higher level of influence from the dancer.
In Pursuit of Rhythm
The rhythm of time-based artwork is a complex notion, and THM seeks to achieve some extent of it through the organization of musical structure. Besides the above-mentioned tempo alteration, several other strategies are employed in this work.
Repetition and variation of previously generated music occurs on a bar scale. Although the work of THM is not melodic enough for the audience to memorize a long segment of it, the reoccurrence of a bar seconds after its first appearance is readily recognizable. When the audience realize that the music is repeating it self (and that the system has memory of its previous work), they are more willing to enjoy the piece as a planned work rather than totally random hits on the keyboard.
The musical form of THM was also carefully planned. In order to build up the mood of the work effectively, the music has a predefined gradually developing structure:
The music starts slowly with whole notes whose pitch proceed in tune with the dancer’s movements. It speeds up across time, gradually switching through half, quarter, eighth and sixteenth notes. When different note-lengths coexist in one bar, there will be tempo alteration according to the dancer’s movements. When a bar has fixed-length notes which are fast enough to form a sub-melody, repetition and variation will be employed to enhance the rhythm. Chords on a lower octave are gradually introduced to further enrich the sound. After eight bars of sixteenth notes, which is the fastest part of the piece, the music slows down and finally ends with a whole note on A4, which is its key note.
This article mainly covers the composition logic of the THM system. The whole architecture of the system is shown in the graph above. Before the composer can make use of the dancer’s movement data, the data has to be captured by the Myo armband and pre-processed by the corresponding infrastructure, which was discussed in The Humanistic Movement: Bodily Data Gathering and Cross-application Interoperability. After the notes get composed, they are sent to the sound engine, which was implemented in Ableton Live with Max for Live, and to the visual engine, which was implement in C++ based on openFrameworks, to produce sound and corresponding visuals.
The concept behind THM was further discussed in The Humanistic Movement: Proposal.
The order of Markov chains decides the extent of simulation of the master’s note choices. With the order of zero, the system chooses the notes based on the master’s overall distribution of note choice frequencies, with no knowledge of its previous composition. The higher the order is, the more previously composed notes the system will look back for. Second order Markov chains can already support significantly accurate simulation of the master’s style, and is reasonably simple to implement. ↩