The Humanistic Movement

Read More

腾讯视频链接 (video link for visitors in China):

The Humanistic Movement (THM) is a generative music system. It collaborates with a human dancer, and improvises music using the dancer’s body movements as its inspiration source.

THM is not a body-instrument. It hates one-to-one mapping from gesture to sound, which awkwardly limits the dancer’s movements, making her, rather than the system, responsible for the composition. THM wants the dancer to dance, with confidence that the system will take good care of the music.

Master’s Spirit in Markov Chains

And the dancer need not worry that without her direct control, the system would generate ugly sounds. In fact, THM’s musical style comes from Wolfgang Amadeus Mozart. It has calculated second-order Markov Chains1 of note progression of the first movement of Mozart’s Piano Sonata No. 11 in A major, so it really has Mozart’s style in mind. For every two-note sequence in this work, THM knows the frequencies of all possible following notes. For example, it knows that after the note sequence E4 B3, the frequencies of the following notes are:

With this knowledge, when generating a new note, THM looks back for the last two notes it has generated, and looks them up in the Markov chains. It can then follow the frequency table of the following notes, so that it plays in the style of the great master. Because of the randomness built in this process, the music is new in every performance, yet the system has a consistent style all the time, just like a real musician with her own personality.

Movement-Influenced Melody and Tempo

While THM has its own music ideas, the dancer still has influence on the composition, with the direction and acceleration of her arm captured by the Myo armband in real time. THM always bases its work on the current state of the dancer, making the music in tune with the dance.

Whenever a new note comes, the system first examines whether the dancer’s arm is pointing higher or lower than the its direction at the last note, and will accordingly look for relatively higher or lower notes in its style reference. In this way, the dancer has influence on the melody with her dance movement. Meanwhile, she has not to be overstressed, since the responsibility of deciding the exact notes are on the shoulder of THM. So the dancer can move freely, and feel that the melody flows in accordance with her movements.

The relation between arm direction and note progression is most perceivable when the music has a slow tempo. When the music goes faster, the link becomes harder to perceive. Furthermore, this is still a low level mapping which cannot represent higher level states, such as emotion, of the dancer. In order to improve its intelligence, THM introduces tempo alteration in accordance with the intensity of the dancer’s movements. In some parts of the music, the system examines the acceleration the dancer’s arm, and generates fast (e.g. eighth notes) or slow (e.g. quarter notes) notes according to the reading. The acceleration indicates the speed and complexity of the dancer’s movements, and therefore is a good representation of her emotion. By mapping it to the intensity of the music, THM receives a higher level of influence from the dancer.

In Pursuit of Rhythm

The rhythm of time-based artwork is a complex notion, and THM seeks to achieve some extent of it through the organization of musical structure. Besides the above-mentioned tempo alteration, several other strategies are employed in this work.

Repetition and variation of previously generated music occurs on a bar scale. Although the work of THM is not melodic enough for the audience to memorize a long segment of it, the reoccurrence of a bar seconds after its first appearance is readily recognizable. When the audience realize that the music is repeating it self (and that the system has memory of its previous work), they are more willing to enjoy the piece as a planned work rather than totally random hits on the keyboard.

The musical form of THM was also carefully planned. In order to build up the mood of the work effectively, the music has a predefined gradually developing structure:

The music starts slowly with whole notes whose pitch proceed in tune with the dancer’s movements. It speeds up across time, gradually switching through half, quarter, eighth and sixteenth notes. When different note-lengths coexist in one bar, there will be tempo alteration according to the dancer’s movements. When a bar has fixed-length notes which are fast enough to form a sub-melody, repetition and variation will be employed to enhance the rhythm. Chords on a lower octave are gradually introduced to further enrich the sound. After eight bars of sixteenth notes, which is the fastest part of the piece, the music slows down and finally ends with a whole note on A4, which is its key note.

System Architecture

This article mainly covers the composition logic of the THM system. The whole architecture of the system is shown in the graph above. Before the composer can make use of the dancer’s movement data, the data has to be captured by the Myo armband and pre-processed by the corresponding infrastructure, which was discussed in The Humanistic Movement: Bodily Data Gathering and Cross-application Interoperability. After the notes get composed, they are sent to the sound engine, which was implemented in Ableton Live with Max for Live, and to the visual engine, which was implement in C++ based on openFrameworks, to produce sound and corresponding visuals.

The concept behind THM was further discussed in The Humanistic Movement: Proposal.


  1. The order of Markov chains decides the extent of simulation of the master’s note choices. With the order of zero, the system chooses the notes based on the master’s overall distribution of note choice frequencies, with no knowledge of its previous composition. The higher the order is, the more previously composed notes the system will look back for. Second order Markov chains can already support significantly accurate simulation of the master’s style, and is reasonably simple to implement.

The Humanistic Movement: Bodily Data Gathering and Cross-application Interoperability

Read More

The first step of the THM project is the preparation of infrastructures—in order to base the generative art on bodily data, we need to get the data; in order to implement the core logic and sound module in separate platforms, we need to enable the communication between the two.

After building these infrastructures with the iteration of the first two prototypes, the architecture of the THM system is fixed like this:

  • It senses bodily data with two Myo armbands.
  • The data is then gathered and analyzed by the core logic implemented in C++ programming language based on openFrameworks.
  • The core logic composes the music, and sends directions to the sound engine implemented in Max/MSP for audio output.
  • With the foundation of openFrameworks, the core logic can compose and render visuals by itself if needed.

Bodily Data Gathering

The screenshot above demonstrates all the bodily data that can be fetched with the Myo SDK. Multiple armbands can be connected to the system at the same time, which enables data capture from both arms.

Two types of bodily data are gathered. The majority of them are spatial data, which contains

  • orientation data,
  • accelerometer output, and
  • gyroscope output.

Indicating the direction of the forearm, the orientation data is calculated by the Myo SDK from accelerometer data and gyroscope data, and is originally provided in the form of a quaternion. The THM core logic then translates it into its equivalent Euler angles, which directly correspond to the

  • pitch (vertical angle),
  • yaw (horizontal angle), and
  • roll (rotational angle)

of the arm. The Euler angles are represented both numerically and graphically in the screenshot.

According to my experience, the pitch data is very reliable. Referring to the horizontal plane as origin, the number ranges between -π/2 (arm towards ground) and π/2 (arm towards sky), and is unrelated to the frontal direction of the body, and unrelated to whether the arm is pointing to the front or the back. After using the armband in a drama performance, I was surprised by how effective the pitch data indicates the emotional state of the performer. It might be one of the most exploited input data in THM as well.

The reference coordinate system of the yaw and roll data are determined by the position of the armband at the point of connection establishment. In consequence, when the user turns his or her body, the reading will be shifted. Another issue of these data is that the reference coordinate tends to drift over time, which also affects the usefulness of the reading. These data range from -π to π (representing a whole circle).

The raw data from the accelerometer and gyroscope can also be accessed via the Myo SDK. They measure the linear and rotational acceleration of the armband in g (the gravitational constant) or °/s (degree per second) respectively. By calculating the SRSS (square root of the sum of the squares) of its components, we get the magnitude of the linear acceleration. When the armband is placed still, this number stays near 1, representing the influence of gravity. When the armband is in free fall, the number approaches 0. It is apparent that acceleration of the arm contains rich rhythmic information. Therefore, it is promising to take advantage of the accelerometer and gyroscope output.

Another type of bodily data that can be acquired via the Myo SDK is gesture data. As indicated by the lower right part of the screenshot, when the armband is worn on arm and the sync gesture is performed and correctly recognized, the SDK provides additional information, including

  • which arm the armband is on,
  • the direction of the x-axis of the armband, and most importantly,
  • the gesture of the hand, which in its turn includes
    • rest
    • fist
    • wave in
    • wave out
    • fingers spread
    • thumb to pinky

While it sounds promising, the gesture data is actually hard to make use of in this project. One reason is that it requires synchronization every time the armband is worn, which is inconvenient and sometimes frustrating, since it is not always easy to make the sync gesture recognized by the Myo SDK. The essential reason, however, is that the gesture data is calculated from the electricity measured on the skin of the arm, which is actually a side effect of the muscle movement itself. Therefore, the calculated hand gesture cannot be perfectly accurate, and is sensitive to exterior influences such as tight clothes on the upper arm, or extreme arm positions. Based on the above considerations, I currently have few intention to make use of the gesture data in THM.

Cross-application Interoperability

Because the core logic and the sound engine of THM are implemented as separate applications base on different platforms (openFrameworks and Max/MSP, respectively), it is necessary to build the communication mechanism between them. The Open Sound Control (OSC) protocol nicely serves this purpose, enabling sending data and commands from the core logic to the sound engine in the form of UDP datagrams. In the core logic, the openFrameworks addon ofxOsc is used to form and send OSC messages. In the sound engine, the udpreceive patch is used to receive the messages, and then the route patch is used to categorize the data and commands, as demonstrated in the screenshot above.

The sound engine represented by the screenshot is only a demonstration of the interoperability between it and the core logic. It simply feeds the bodily data to two groups of oscillators to generate sound, with the frequencies related to arm pitches in the past few seconds, and the amplitudes to arm yaws. Together with the core logic, these primitive prototypes demonstrate the architecture of the THM system described in the beginning of this article.

The next step of the THM project is to explain the bodily data on higher levels (such as tempo, rhythm and emotion), and generate more artistic output.

The Humanistic Movement: Proposal

Read More

I’m passionate about body movements and its interaction with space. For the final project of my Major Studio 1, I propose The Humanistic Movement (THM) as a 8-week exploration of generative art from bodily data.

A Humanistic View on Generative Art

Perhaps the broadest definition of generative art is like this:

Generative art is art created with the use of an autonomous system which generates data or interprets data input, and maps it to human-perceivable scales of visual, audio and/or other sensory data as output.

This kind of definition is open-minded as well as meaningless. Perceivable is not equal to affective. Merely mapping data to the human scale doesn’t guarantee good art.

Instead, artists should look for a humanistic scale of output in order to infuse their generative work with rhythm and verve, and therefore provoke emotional and reflective reaction from the audience.

The Model: Generating Art from Body Movements

The humanistic scale itself, however, is a mystery. Not only new media artists, but also traditional artists in all artistic domains, have been struggling for a touch with the human heart. Hopefully, in the making of generative art we have a shortcut—to wire in human-being themselves as data source, generating art from human for human.

On the other hand, if the generative system merely translates its input mechanically, making one-to-one mapping from some sensory data stream to a perceivable data set, the result will be by no means exciting from the viewpoint of generative art—the system doesn’t has a genius for art. The best possible outcome is a new piece of “instrument”, which has a high requirement on its “player” for the creation of good artwork.

THM will be of higher intelligence. Rather than relying on the data source to generate art, it only uses it as a reference. The way it works can be imagined as a musician collaborating with a dancer in an improvisatory work. Each one is autonomous. The dancer moves her body on her own initiative, as well as the musician plays her instrument. They don’t base their every next step exactly on each other’s latest action. Rather, they communicate on higher levels of artistic elements such as tempo and mood, in this way exchanging inspirations in the process of the performance, and seeking to achieve harmony in their collaborative artwork.

THM, as a computational generative art system, will function as the above-mentioned improvisatory musician. It captures data from body movements, and generate music and/or visuals in tune with the input, as an attempt to achieve the level of artistry which usually belongs to human masters.

If fruitful, THM is going to be my long-term research and making theme, and the outcome will be a systematic approach to humanistic generative music/visuals. For this Major Studio project, after several explorative prototypes, the final outcome will be one art piece demonstrating the concept.

Data-gathering Technologies

In order to capture data from body movements, sensory technologies need to be considered and evaluated. THM doesn’t not mean to hinder its human collaborator. Rather, it would like her to move as free as possible; it would like her to dance.

Several data-gathering technologies have been considered:

1. Myo (Preferred)

Myo is a gesture control armband developed by ThalmicLabs. With a 9-axis inertial measurement unit (IMU), it calculates spatial data about the orientation and movement of the user's arm. In addition, by measuring the electrical activity generated by arm muscles, it recognizes hand gestures. It is wireless and lightweight, not a big hindrance to body movements. Two armbands can be paired to the same computer, enabling movement capture of both arms.

The biggest issue with Myo is that the data it captured is inaccurate. Since the skin electricity measured by Myo is only the side effect of muscle movement, the measurement can be interfered by exterior factors such as muscle-binding clothes (even on the upper arm). Furthermore, when the arm goes to extreme positions, the measured data tends to jump the opposite extreme. As a loosely coupled generative system, THM does not have strict requirements on data accuracy. Nevertheless, captured data need to be pre-processed to reduce unreasonable behaviors of the system.

2. Kinect (Back-up)

Being developed and sold by Microsoft by years, Kinect is a sophisticated body tracking solution which captures the position of the whole human body through a set of cameras. No body attachment of sensor is needed. There are few reasons not to try it out if time is sufficient. My only concern for now is that it requires the person to be in front of the cameras in order to be captured, therefore limits her movement to some extent.

3. Brainwave Gathering Technologies (Alterative)

Besides gathering bodily data, an alternative data-sources is the mind. Brain-Computer Interface has been being researched for decades, and various ready-to-use data gathering technologies, such as EPOC, Muse and MindWave, are being shipped to the market. A large part of the data gathered by these devices (EEG, Attention/Meditation level, etc) are on subconscious level. For our goal of gathering humanistic data input, the brainwave approach is also very promising. I might try this alternative in future steps.

Supportive Projects

For the better outcome of this Major Studio 1 project, I plan to merge the final projects of some other courses to support it. This will allow me more time to work on it, and resources from these supportive courses.

  1. Theoretical Research on Generative Music/Visuals in Independent Study: Symphony + Technology, to provide theoretical basis for this project.
  2. Final project with Max/MSP in Dynamic Sound and Performance, as the sound-generating engine for THM.
  3. (Speculative) final project with openFrameworks in Creativity and Computation: Lab if required by the course and/or time-permitted, as the visual-generating engine for THM.


After this proposal, there will be three prototypes and a final presentation of the project. The date and expected outcome of each phase is listed below:

  • Oct. 27/29 Prototype 1: getting data from body movement
  • Nov. 03/05 Prototype 2: generating sound/visuals
  • Nov. 17/19 Prototype 3: art directions; finalizing design
  • Dec. 08/10 Final Presentation

5 in 5

Read More

In this Major Studio 1 assignment, I was supposed to create five projects consecutively, with each project conceptualized, produced and documented in a single day. There is no restriction on theme or medium, and experimental attempts are highly encouraged.

#1: Meditation for 1H

The first three projects are related to my body. I named this serious Half Familiar, Half New, for that in each project, I started with a topic I'm familiar with, yet ended up with an experience I've never had before.

In this very first project, I used time-lapse video to record myself meditating for one hour. I've been doing Zen meditation in recent years, which helps me to get closer to a mental state of mindfulness. I also treat meditation as a practice of spiritual strength. However, I've never done a meditation as long as one hour.

In addition to the extension of meditation time, I've also changed its location to an outdoor environment. People usually meditate in a quiet room. However, since during meditation one's senses tend to be more acute, it might make sense to do so in an open area.

The outcome of my meditation is a mixture of different feelings. First of all, it is indeed physically demanding to meditate on such a time scale. As time passed on, it was gradually harder to concentrate my mind on my breath, and easier to lose my mind into a mental state like dreaming. I could feel the weakness of my body, and I was even unconfident of completing this challenge. Fortunately, I endured this one hour without problem. When I opened my eyes, the world turned blue, because I have been closing my eyes under sunlight for so long time.

The good news is that, in long-time meditation, it is relatively easy to fall into a "deeper" mental state even if you are not of an optimum physical strength. The experience reinforced my thoughts that people's physical and mental states are interrelated with each other. Being still for a long time really helped my mind to quiet down, and this explains the reason to practice mindfulness in this kind of body set.

During my meditation in the community park, I was able to recognize various kinds of surrounding sounds, such as the splash of the fountain, and children playing around me. Making myself quiet helped me to better sense the environment, and even the mental state of my self. However, without the help from a camera, I can never know the way my body looks in the meditation. Although predictable, the time-lapse record still surprised me on how motionless I was during the whole process. What's more, when reviewing the record, I found out that there are many more things that I hadn't notice during the process, such as squirrels playing around me in a very short distance, and the change of lightness of the sunlight. These interesting discoveries showed the limit of human senses, even in a highly mindful state.

#2: Fifth Avenue—From The New School to Central Park

In the second project, I used a mobile app called Hyperlapse to record what I have seen in a jogging along the fifth avenue. I've been a runner for years, so that the distance from The New School to Central Park is not a problem to me. The amazing part of this experience is the video captured during the process.

The team of Instagram have done such a wonderful job in making Hyperlapse that its stabilization algorithm enables people to shoot time-lapse video with a hand-held phone, even when they are walking. Before, shooting time-lapse video requires a tripod in order to make the image stable, which largly limits its usage. Now, even a jogging can be recorded and condensed into a two-minute video with ease.

To me, the outcome is stunning. The buzzy night of midtown Manhattan was captured and highlighted by these video clips. They are rough but vivid, just like what people tend to make with the first grasp of a cutting-edge technology. An interesting thing is that since the recording of video gave me so much fun, I almost lost my mind into it, experimenting ways of camera placement and movement, without remembering that I was there originally for a jogging.

(Music credit: U2, The Miracle)

#3: A Moment of Butoh

My third and last project dealing in the Half Familiar, Half New serious, I tried to recollect my body memories of Butoh movements. As a genre of contemporary dance, Butoh is a mixture of modern dance and Japanese traditional culture. It tends to build a dark and deep atmosphere using restrained body movements.

I had learned some modern dance and Butoh during the spring, but since I began to travel in the summer, I never practiced dance again. This project is an attempt to awaken the dancer in me, and get myself more prepared to continue my learn dancing in New York. I set up the room, and used a fixed camera to record myself doing random movements, in hope of producing a montage of me dancing.

The process was quite disappointing at first. I found out that my body has almost forgotten the way it used to dance, and I cannot achieve a proper tenseness of it, in order to make clean movements. Every piece of footage was ugly when I first looked at it. I didn't know what can be done with them.

Then, after collecting an hour of video clips, I stopped dancing, sat down, and looked at them for a second time. I began to notice interesting movements in the footage. It might be not that bad if I only collect and arrange this moments. This idea resulted in the following piece. I named it A Moment of Butoh, hoping that in all my random search, I've found at least one moment that matches the state of a Butoh dancer.

(Music credit: Nocturnal Emissions, 01 from Music For Butoh)

#4: How to Do Chicken Right, Seriously

When I say "seriously," I really mean it. Eating in America makes me homesick, since cooks here don't give chicken the extent of care it deserves, which results in a bland and coarse taste. The Chinese people treat chicken as precious food; they carefully prepare chicken for their New Year's Eve—that's the way I was going to cook my chicken.

The slideshow above demonstrates the process of cooking a whole chicken. The basic idea is to remove the blood and body fluid from the meat, and bring out the flavor using various cooking methods. The whole process took me three hours, and the outcome definitely cured a nostalgic stomach.

#5: Pushing the Beats

At the beginning of this semester, I bought myself these pieces of MIDI controllers:

I daydreamed with them, but have had no time to play around with them during the first three weeks. When dealing with my very last 5 in 5 project, I decided that I just want to do something with them. Simple things should suffice.

The video below demonstrates me making a sequence of beats, using a MIDI controller named Push, which is a new type of musical instruments which lets the musician to modify the arrangements of a musical piece on the fly. What I have made is not some masterpiece, apparently. Nevertheless, I had fun in making it.

Final Thoughts

Doing 5 in 5 is my first experience of formally documenting the process when doing projects. The video or photo documentary enabled me to reflect on my process, which is full of interesting findings. The documentary also functions as a effective showcase of what has been done. Therefore, it is a good design practice to document.

On the other hand, adding an observer affects the process, even if the observer is a camera. To some extent, keeping the observer in mind makes people act like performing. This shifts their focus from the object to the subject themselves, which is equivalent to the objectification of the subject. Furthermore, the documentary itself becomes another form of outcome, which is sometimes more interesting than the direct product of the project, and will last longer than the latter, especially so if the project is action-taking (e.g., performance art) rather than object-making.

The presentation of 5-in-5 projects in class is eye-opening, since everyone demonstrated his or her unique area of interest and way of doing things. It becomes even more mind-revealing when people are asked to do choose their topic and approach in a serious of projects. This process is also a great opportunity for me to reflect on myself, which hopefully leads to a better understanding of the relationship between me and the world I'm experiencing and affecting.

Dragon Nest · 龙之谷

Read More

Excerpt, performed in Sino-Japanese Butoh Festival 2014

Date: April 27th, 2014
Venue: BeijingDance / LDTX

Dance Group: White Fox Butoh
Choreography: KATSURA Kan
Dancers: KATSURA Kan, SHI Weili, ZHAO Zhiyong, LIU Chao, LIU Wei