AI: Mind the Gap

Client: MIT Museum
Studio: Bluecadet
Role: Lead developer, Black Box and Collaborative Poetry
Technologies: Unity, HLSL, Tensorflow, GPT-3, GPT-2, TouchDesigner, Three.js, P5.js, React, Strapi
Awards: The Webby Awards, Metaverse, Immersive & Virtual, Best Experiential Design, Webby Winner, 2023; MUSE Design Awards, gold winner in Interior Design - Exhibits, Pavilions & Exhibitions; blooloop Innovation Awards, third place in Experiential Technology

MIT Museum invites visitors to take part in ongoing research and demonstrates how science and innovation will shape the future of society. The museum moved to a new location in 2022 with fully re-imagined exhibitions. Bluecadet is the museum's media partner, and has provided services from concept development to production. I participated in the whole process during 2020—2022, and led the development of two core experiences in the AI gallery.

Black Box invites visitors to draw a face, and uses holographic spatial projection to visualize how a custom-trained neural network processes the face drawing layer by layer to identify its expression. Collaborative Poetry invites visitors to collaborate with an AI in writing a poem to show AI as a tool to unlock human creative expression.

Black Box

The Black Box installation tries to demystify AI, to demonstrate how machine learning and neural networks function by revealing the inner workings.

To build the human connection between AI and visitors, we used face drawings as the content. We created a web app that prompts users to draw specific facial expressions, and collected over 7,000 labeled face drawings from the MIT community. With this data, we trained a neural network model that recognizes facial expressions. The model structure is simple enough to be visualized. It has over 80% accuracy on test data, but is naive enough so that visitors can test its boundaries by drawing fuzzy expressions.

We implemented the neural network in Unity with parallel computing and shader programming, so that the internal status of the neural network can be visualized in real time while it is computing.

To achieve an engaging visual experience, we engineered a 3-layer holographic projection system. We precisely placed the virtual cameras in Unity to match the exhibition space, so that when visitors stand in front of the kiosk, they will see holographic visuals of the neural network without the need of a wearable device.

Collaborative Poetry

The Collaborative Poetry installation presents AI as a tool to unlock human creative expression. It welcomes visitors with a curved overhead canvas with poems afloat—all of them are collaborative work between humans and AI. It invites the visitors to write a poem together. And once the poem is finished, it will be published onto the canvas, joining the stream of human-AI collaborations.

With thoughtful interaction design and careful engineering, we turned OpenAI's GPT-3 model into our poetry AI. The AI suggests subjects and moods to inspire the poet within the visitors, takes turns with the visitors when writing, and suggests a title for the poem when it's finished.

We are mindful of keeping the museum as a healthy platform, and it's possible for both humans and AIs to write improper words. We implemented a moderation system with automated tools to support visitors in writing properly, as well as a reviewing tool for the museum to remove bad content.