Willow and Flowers · 柳暗花明

Read More

(The game can only be played on a computer with a web browser other than Chrome.)

Willow and Flowers is a poetic interactive experience of reaching one's goal after hitting dead ends, turning back and find alternative paths.

The name of the project comes from a famous line by the great Chinese poet Lu You, which translates as: "After endless mountains and rivers that leave doubt whether there is a path out, suddenly one encounters the shade of a willow, bright flowers and a lovely village."

Willow and Flowers is a web game made with Unity.


  • Click the game scene to have control.
  • Use the mouse the look around.
  • Use the arrow keys to move.
  • Try to find a way to the open end of the scene (indicated by the floating red arrow).

(Music Credit: Lost by Capozio.)

Little Universe · 小宇宙

Read More


腾讯视频链接 (video link for visitors in China): http://v.qq.com/page/y/9/n/y0304z44l9n.html

There they are, poetic objects in the Little Universe, revolving around themselves, whispering to each other, regardless of whether or not they are visited by you—their exotic guest.

Colorful the objects are, and they are not mindless clouds of stardust. Each object has its own temperament— a tendency to get close to someone, and to run away from another. In this way, the Little Universe reveals its rich dynamics.


This interactive installation is designed to bring about a serene and meditative atmosphere. It has an intimate physical form, inviting the audience to interact with it, to touch and move it with their hands. Without interfering with the dialogue between the objects, the audience can still enjoy the poetic scene by watching, listening and contemplating their behaviors and relationships.

While the audience is not explicitly informed of the logic behind the behaviors, the five-element theory (五行) of Chinese philosophy inspires the distinct characteristics and relationships between the objects. The colors of the five objects represent the five elements—metal (金), wood (木), water (水), fire (火), and earth (土). There are two orders of presentation within the five-element system: in the order of mutual generation (相生), one element can give birth to another, while in the order of mutual overcoming (相克), the element can suppress yet another. Represented by the attracting and repelling forces between the particle systems surrounding the objects, the relationships between the poetic objects enrich the contemplative intricacy of the Little Universe.


The Little Universe installation is built to integrate four components:

  1. poetic objects with which the users interact,
  2. projection system,
  3. sound generation system, and
  4. position detection/tracking system.

The poetic objects are made with foam balls implanted with microcontrollers and infrared LEDs facing upwards, atop a flat surface as the cosmic background. The infrared light from the objects is captured by a Microsoft Kinect motion sensor, and analyzed by an openFrameworks application in order to locate the objects. A particle system is projected onto the objects, and based on the position of the objects and their relationships, the behaviors of the particle systems are updated and rendered by the openFrameworks application. The image is then processed by MadMapper and sent to the projector, which has been calibrated so that the projection will be aligned with the physical objects. Control messages are sent from the openFrameworks application to Ableton Live via the Open Sound Control (OSC) protocol, generating the ambient music and sound effects.

The position detection system precisely locates the objects on the table and tracks them throughout the installation. The projection on each object behaves with distinct characteristics depending on the viewer’s interaction, which requires the system to be able to distinguish the objects from each other. Thus, the application has to be always aware of the exact location of the individual objects.

The tracking system comprises of camera vision alongside infrared LEDs. Utilizing blob detection on Kinect’s infrared stream, the application will precisely locate the objects according to their implanted infrared LEDs. A signal processing procedure enables the system to distinguish the objects within 300 ms post detection.

Kinect’s infrared stream can afford up to 30 FPS, which translates to a 30 Hz sampling rate. Hence, theoretically, we can reconstruct the any input signals with a bandwidth less than 15 Hz. The infrared LEDs on the objects are constantly blinking, which generate patterns that are perceived as square wave signals by the camera vision system. All the LEDs in the objects generate periodic square waves with a period equal to one third of a second. However, each LED generates a signal with a distinct duty cycle. The application tells the objects apart by measuring the duty cycles of the LEDs through the Kinect’s data frames. After detection, the application starts tracking the objects, and in the event of tracking loss or confusion between objects, a precise identification will be performed again through frequency analysis.

(Little Universe is a collaboration among SHI Weili, Miri PARK, and Saman TEHRANI, with the guidance of Marko TANDEFELT.)

inSight: The Prehistory of Homo Interscient

Read More


While the daily routine of archaeologists is dealing with antiques, we are nonetheless amazed by the genius of our forefathers from time to time, and are shocked by the social impacts of their inventions. After all, what we treat as artifacts were once the most cutting-edge technologies. Since the dust raised by them has already settled, we may learn from what has happened in order to predict how technology will affect human beings in the future.

Today we are super excited to introduce our newest discovery, which will fill a long-existing gap in the prehistory. As we all know, the continuously recorded history starts after the Total War, and before the war, there existed two kinds of human beings—homo interscient, which is us, and homo sapiens, our ancestor. It might not be easy for us to imagine that the homo sapiens cannot really know each other's mind. For us, thinking together with other people equals to exchanging ideas with them. For the homo sapiens, however, thinking is one thing, while informing other people is another. In order to exchange ideas, it seems that they mainly relied on very primitive methods, such as modulate their voice to carry information with data rates as low as several bits per second. Not only did they have inefficient ways to communicate within their species, but they also lacked the solidarity which might have helped us in defeating them in the war.

Nonetheless, it was the homo sapiens who invented us, the homo interscient. We already know that mind communication was treated by them as a scientific and technological breakthrough. And we also know that once there was a first community of people like us, they immediately saw themselves a different kind of human than the old species. So did homo sapiens. Accumulated enmity between the two species eventually led to the outbreak of the Total War, and apparently, we won it, admittedly with a heavy loss. The destruction was so severe that most of the historical records before our current era got lost, so that people have no way to know the details of the prehistory, and can only rely on archaeological discoveries.

The video clip we recovered recently is an significant breakthrough in knowing the early days of the homo interscient. It suggests an answer to an eye-catching question—how was the interscient technology adopted in its early age? As the video (which is a commercial advertisement) shows, the technology was first promoted as a consumer product named inSight. The manufacturer carefully chose a minimal set of interscient applications (or maybe that actually is how much they were able to offer on that stage), and advertised the technology as an augmentation to people's lifestyle. This seeming harmlessness might facilitated the early adoption. And once there was a solid community of adopters, the trend could never be stopped.

—Weili, Tyler, and Lama

(inSight: The Prehistory of Homo Interscient is a collaboration among SHI Weili, Tyler HENRY, and Lama SHEHADEH.)

The Humanistic Movement

Read More


腾讯视频链接 (video link for visitors in China): http://v.qq.com/page/o/i/u/o0304j7xjiu.html

The Humanistic Movement (THM) is a generative music system. It collaborates with a human dancer, and improvises music using the dancer’s body movements as its inspiration source.

THM is not a body-instrument. It hates one-to-one mapping from gesture to sound, which awkwardly limits the dancer’s movements, making her, rather than the system, responsible for the composition. THM wants the dancer to dance, with confidence that the system will take good care of the music.

Master’s Spirit in Markov Chains

And the dancer need not worry that without her direct control, the system would generate ugly sounds. In fact, THM’s musical style comes from Wolfgang Amadeus Mozart. It has calculated second-order Markov Chains1 of note progression of the first movement of Mozart’s Piano Sonata No. 11 in A major, so it really has Mozart’s style in mind. For every two-note sequence in this work, THM knows the frequencies of all possible following notes. For example, it knows that after the note sequence E4 B3, the frequencies of the following notes are:

With this knowledge, when generating a new note, THM looks back for the last two notes it has generated, and looks them up in the Markov chains. It can then follow the frequency table of the following notes, so that it plays in the style of the great master. Because of the randomness built in this process, the music is new in every performance, yet the system has a consistent style all the time, just like a real musician with her own personality.

Movement-Influenced Melody and Tempo

While THM has its own music ideas, the dancer still has influence on the composition, with the direction and acceleration of her arm captured by the Myo armband in real time. THM always bases its work on the current state of the dancer, making the music in tune with the dance.

Whenever a new note comes, the system first examines whether the dancer’s arm is pointing higher or lower than the its direction at the last note, and will accordingly look for relatively higher or lower notes in its style reference. In this way, the dancer has influence on the melody with her dance movement. Meanwhile, she has not to be overstressed, since the responsibility of deciding the exact notes are on the shoulder of THM. So the dancer can move freely, and feel that the melody flows in accordance with her movements.

The relation between arm direction and note progression is most perceivable when the music has a slow tempo. When the music goes faster, the link becomes harder to perceive. Furthermore, this is still a low level mapping which cannot represent higher level states, such as emotion, of the dancer. In order to improve its intelligence, THM introduces tempo alteration in accordance with the intensity of the dancer’s movements. In some parts of the music, the system examines the acceleration the dancer’s arm, and generates fast (e.g. eighth notes) or slow (e.g. quarter notes) notes according to the reading. The acceleration indicates the speed and complexity of the dancer’s movements, and therefore is a good representation of her emotion. By mapping it to the intensity of the music, THM receives a higher level of influence from the dancer.

In Pursuit of Rhythm

The rhythm of time-based artwork is a complex notion, and THM seeks to achieve some extent of it through the organization of musical structure. Besides the above-mentioned tempo alteration, several other strategies are employed in this work.

Repetition and variation of previously generated music occurs on a bar scale. Although the work of THM is not melodic enough for the audience to memorize a long segment of it, the reoccurrence of a bar seconds after its first appearance is readily recognizable. When the audience realize that the music is repeating it self (and that the system has memory of its previous work), they are more willing to enjoy the piece as a planned work rather than totally random hits on the keyboard.

The musical form of THM was also carefully planned. In order to build up the mood of the work effectively, the music has a predefined gradually developing structure:

The music starts slowly with whole notes whose pitch proceed in tune with the dancer’s movements. It speeds up across time, gradually switching through half, quarter, eighth and sixteenth notes. When different note-lengths coexist in one bar, there will be tempo alteration according to the dancer’s movements. When a bar has fixed-length notes which are fast enough to form a sub-melody, repetition and variation will be employed to enhance the rhythm. Chords on a lower octave are gradually introduced to further enrich the sound. After eight bars of sixteenth notes, which is the fastest part of the piece, the music slows down and finally ends with a whole note on A4, which is its key note.

System Architecture

This article mainly covers the composition logic of the THM system. The whole architecture of the system is shown in the graph above. Before the composer can make use of the dancer’s movement data, the data has to be captured by the Myo armband and pre-processed by the corresponding infrastructure, which was discussed in The Humanistic Movement: Bodily Data Gathering and Cross-application Interoperability. After the notes get composed, they are sent to the sound engine, which was implemented in Ableton Live with Max for Live, and to the visual engine, which was implement in C++ based on openFrameworks, to produce sound and corresponding visuals.

The concept behind THM was further discussed in The Humanistic Movement: Proposal.


  1. The order of Markov chains decides the extent of simulation of the master’s note choices. With the order of zero, the system chooses the notes based on the master’s overall distribution of note choice frequencies, with no knowledge of its previous composition. The higher the order is, the more previously composed notes the system will look back for. Second order Markov chains can already support significantly accurate simulation of the master’s style, and is reasonably simple to implement.

You & You

Read More


You & You is an interactive music program which performs a whole song based on 3 seconds of the user's voice input. A one-man chorus is built through repetition and tonal modification.

This project was implemented in Max/MSP. For Mac users, an OS X build can be downloaded and experienced.

The Critique of Poetic Interaction

Read More

This article seeks to corroborate Jon Kolko's argument on poetic interaction, as a desirable higher-level goal of interaction design.

Chapter 1: The Critique of Poetic Interaction

Interaction Design Ought to Be More than Usable

In his insightful book Thoughts on Interaction Design, Jon Kolko sets the tone for this discipline. He leaps forward "beyond style" (as Don Norman praises the book), grasps the substance of interaction design, and through a high-level overview of the way interaction designers work, distinguishes them from artists, graphical designers, industrial designers, interactive designers, GUI designers, information architects, UX managers and marketers. With better self-awareness, interaction designers should find their rightful place in the industry, and their voice should be properly heard.

Kolko states more than that. Good interaction designers should be more than dutiful, and good interaction design is more than making things usable. In the second half of the book, higher responsibility is put on designers' shoulders. On the social side, designers should imbue their works with judgment, and use their influential power to shape people's behavior; and beyond that, designers ought to aim at those large-scale social or cultural wicked problems, employ a whole new set of design methods with the participation of citizens themselves, in order to wake up their sense of social responsibility.

On the aesthetic side, Kolko insists that designers are far beyond stylists of transient and cheap feelings. Instead, they are responsible for the authenticity of overall experiences; they are poets of interactions. In the fifth chapter, Poetry, Spirit, and Soul, the concept of poetic interaction is proposed as designers' goal.

Ends and Means of the Poetic Pursuit

Poetic interaction is defined by its poetic effect1:

  • resonates immediately
  • continues to inform later

In order to achieve this effect, the poetic elements of interaction are claimed to be:

  • honesty
  • mindfulness
  • vivid and refined attention to sensory detail

The poetic pursuit has rich connotations. It stresses on immediate engagement, yet is at the same time foresighted. One the surface it quests for feeling, yet it is endowed with an analytical body. The formulation of poetic effect grasps the sequential manifestation of emotion and reason. A design with this poetic manner is calm, yet without loss of humanity.

However, the ends and means of poetic interaction are not in perfect coherence.

First, the three poetic elements are not subordinate to the same agent. As the means to attain the poetic effect on the user's side, it behooves these elements to be applicable on the designer's. While honesty and attention to sensory detail can be seen as qualities of a design, the second element, mindfulness, is an effect on the user–itself being an end to pursuit. I actually like this symmetry of agents, and it will be better revealed in a refined definition of poetic interaction later on.

Second, the poetic elements don't share an equal position.

In his diction, Kolko seems to treat honesty (and its synonyms) as an all-encompassing concept. To him, honesty implies integrity, and virtually consists of integrity to the business vision, integrity to the consumer, and integrity to materials. This resembles the concept of authenticity he has introduced in the book's previous chapter, Experience and Authenticity. Authenticity is said to depend entirely on designers' "craftsmanship and their intent and ability to evoke emotion," and in my opinion, is to a large extent interchangeable with honesty.

Vivid and refined attention to sensory detail, however, is dwarfed by the concepts above, and well falls into their region. Isn't this attention a kind of integrity–integrity to sensory detail? There is a need to sort out all these concepts. They are all aesthetic characteristics of a design, and can only be achieved by a wholehearted designer.

Third, probably due to the brevity of the book chapter, how the means support the ends are not fully reasoned out. Via reading one vaguely feels that aesthetic meticulousness may contribute to user's immediate resonance in usage, and a state of mindfulness may help better reflection afterwards. But these are not explicitly discussed. The chapter did discuss the approach to mindfulness. Influenced by the concept of mental state known as flow, Kolko claims that in order to encourage a mindful state, the interaction need to be "appropriately complicated." This is as insincere as the term planned obsolescence which he had attacked. And this is evil–the complexity of an activity is determined by the activity itself, and should not be artificially convoluted by its form.

In spite of the imperfections mentioned above, the book chapter is still marvelous and thought-provoking. The definition of poetic effect shows an excellent taste of interaction design, and precisely points out a worthwhile goal for designers to achieve. The discussion on poetic elements is a nice start point to form a framework of methodology. The idea of poetic interaction deserves study, corroboration, being fully understood, and better spread.

Chapter 2: Use Drama Theory to Guide Interaction Design

Why Theater is More (Or No More) Suitable Than Other Humanities

It is spontaneous, yet it is rehearsed. It is participatory, yet it is presented.
It is real, yet it is simulated.
It is understandable, yet it is obscure.
It is unique to the moment, yet it is repeatable. The actors are themselves, yet they are characters. The audience believes, yet it does not believe.
The audience is involved, yet it remains apart.

Robert Cohen, Theatre2

Tracing the source of the word "poetic", we find Aristotle's Poetics. Surprisingly or not, under this title is actually the world's earliest treatise on drama theory. Indeed, in the majority of times in history, most of the plays were written in verse, and their writers were called poets. Theater originated from festive ceremonies and performances of the bards. The theoretical framework of this discipline has been beautifully set up by one of the greatest philosophers of humankind more than two thousand years ago, and has been practiced and enriched generation after generation. Confident enough about using drama theory to guide our reflection on poetic interaction?

Wait, why not other humanities?

One reason is drama's resemblance to human-computer interaction. As stressed by Kolko, interaction designers differ from other designers in that they work with the fourth dimension (time) in mind. Humanities like painting and sculpture lack this property. Among other disciplines, literature, music and movie are basically non-interactive, leaving theater as the only one that has direct interaction between performers and audience. Though theater audience cannot direct a play like a user manipulates his computer, it is indeed one salient feature of theater that during the play, the performers and the audience both have influence on each other. Given these similarities, we may be hopeful that interaction designers can learn something from dramatists.

Yet the real surprise comes from Poetics itself–it talks about poetic effect (Chapter 26)! The poetic effect, as Aristotle writes it, consists of "not any or every pleasure, but the very special kind." This proper pleasure is the goal of tragedy, and its named catharsis.

Catharsis is achieved by arousing and purgation of certain emotions:

  • pity, occasioned by undeserved misfortune
  • fear, occasioned by undeserved misfortune of one like ourselves

Can we say that there exists perfect match between Aristotle's catharsis and Kolko's poetic effect? Pity resonates immediately, and fear is aroused by reflection. Theater and interaction design, we can say, seek the same ends! The pursuit of poetic interaction is now well-grounded and worthwhile. Theater, especially that of ancient Greek, as claimed by Nietzsche, is the greatest production by the combination of Dionysian emotion and Apollonian reason. So why not learn the methodology from Aristotle and other masters after him?

Very promising. Just do it with caution. Theater and interaction design are different disciplines, anyway. Like any other literary and artistic works, a drama is always a finished, self-contained one with content in itself. In other words, drama is representation of its content. But interaction design is largely about form. The content of an interaction is essentially produced by the user. An interaction design is never finished before the interaction. In this sense, as Kolko states, interaction designers have much less control of their works. Interaction designers and dramatist work in different ways, and have different emphases.

So much roundabout. Let's get on with the meat of Poetics.

Tragic Elements: A Comparison with Poetic Interaction

In the early 1990s, Brenda Laurel wrote a seminal book, Computers as Theatre, to apply her dramatic expertise to the field of human-computer interaction. Her stress is on the representative essence of theater, and the book has a focus on the field of interactive fantasy, the automated production of plots. Nonetheless, when she introduces dramatic theories, the juxtaposition between theater and human-computer interaction is illuminating. I suggest a reading of this book for interaction designers with even a minor interest on theater. It will be informative on both topics.

The Aristotelian drama theory is built upon a structure of the six elements of drama. There is a precedence among these elements. Laurel believes that the same elements are applicable to the analysis of human-computer interaction, which I consider is a little far- fetched. But her explanation of their hierarchical structure is instructive for us to examine the poetic elements of interaction.

The tragic elements are listed below, in a descending order of significance:

  • plot
  • character
  • thought
  • diction
  • melody
  • spectacle

Laurel explains that, each element is the formal cause of all those below it, and the material cause of all those above it. Or we can says that these elements are ends and means of each other. The lower elements serve to enhance the upper ones, function as materials in the building of the latter. And ultimately, all these tragic elements serve one goal: to achieve catharsis in the audience.

Of the six tragic elements, the lower four are less significant. They are more of technical means than substance. Plot and character form the core of tragedy. Plot is the whole action represented in the performance. Each character is a set of traits, forming an agent of the action. There exist different voices on priorities of the two. Many dramaturges maintain that character is what a play seeks to represent and depict. They insist that character is the carrier of emotions, and character depth is what gives a play its vitality. This argument is chiefly against the so-called well-made play, in which the mechanical perfectionism of plot stifles the naturalness and freshness of a play. Aristotle insists, on the other side, that performers do not act in order to portray the characters, instead, they impersonate characters for the sake of plot.

The debate can go on forever. From it we can have a feel for the complementary relationship between tragic plot and character. The plot is human plot; and the characters are created in order to present the story. And it is natural for us to make an analogy between these two tragic elements with our poetic elements. Mindfulness is reached when the user is totally immersed in the current action. And if we merge attention to sensory detail into honesty, it will be clear that the combined concept represents the characteristics of a design. Since the subject of mindfulness is the user rather than the designer, it is fair to say that mindfulness is one step closer than honesty to the poetic effect, whose subject is also the user. That is to say, mindfulness is the formal cause of honesty, and honesty is the material cause of mindfulness. Like plot and character, only when mindfulness and honesty are perfectly achieved on each side of a design, poetic effect, the catharsis of interaction, can be accomplished.

We are ready to give a refined definition of poetic interaction with the support of drama theory.

A Refined Definition of Poetic Interaction

A poetic interaction is one that has poetic effect:

  • resonates immediately
  • continues to inform later

In order to achieve this effect, the poetic elements of interaction are:

  • mindfulness, of the user
  • honesty, of the design(er)

The poetic element honesty, then, can be analyzed in detail. It includes, but should not be limited to:

  • integrity to the business vision
  • integrity to the consumer
  • integrity to materials
  • integrity to sensory detail

Here we have finished the definition of poetic interaction. It has only been slightly modified. But we are now more confident of it. We know that the poetic effect of interaction design is well-reasoned, worthwhile, and shares the same spiritual pursuit of other forms of human culture. The poetic elements are well-structured, and although the description is concise, we have a sea of theories and methodologies to explore from this start point.

Don't stop here. Don't let this discussion on definition misguide you, to consider that we have already know enough about poetic interaction. What interaction design can learn from theater is way beyond definition of terms. Just as Laurel asked, who better understands human interaction than the dramatists? They are the experts on how to keep people's attention engaged, how to reveal information on the perfect spot, what is the best scope of one interaction in order to form a holistic experience, how to exploit multisensory representation to make an authentic feel, and many, many more. The poetic effect is a goal of both interaction designers and dramatists. We feel lucky that we have them as companies.


  1. The terms poetic effect and poetic elements are actually named by myself, but their descriptions are given by Jon Kolko.

  2. Note that in this article, the words theater (theatre), drama, tragdy, etc. are used interchangeably.


  • Aristotle. Poetics. Athens. c.335 BCE.
  • Csikszentmihalyi, Mihaly. Flow: The Psychology of Optimal Experience. New York: Harper & Row, 1990.
  • Cohen, Robert. Theatre. 6th ed. Boston: McGraw-Hill, 2003.
  • Kolko, Jon. Thoughts on Interaction Design. Burlington, Massachusetts: Morgan Kaufmann, 2010.
  • Laurel, Brenda. Computers as Theatre. Reading, Massachusetts: Addison-Wesley, 1991.

(Title image credit: Jerzy Strzelecki, The Greek theatre in Syracuse, Sicily, Italy)