1 / 6
Traditionally, animation has been produced through laborious, frame-by-frame processes or pre-rendered computational pipelines. Embodied Animation introduces a different paradigm: live-time animation in which the body itself becomes the creative engine. Here, the hand motion is transformed into a living brush, with gestures serving as the catalyst for real-time augmented animation.
Drawing on traditions such as hand shadow puppetry and sand painting, forms where body movement and ephemerality are central, the installation bridges ancient performative practices with contemporary generative technologies. Using motion capture, LoRA-trained diffusion models, live prompt engineering and coding, this installation translates hand gestures into dynamic visual transformations and a responsive soundscape that fuses motion, image, and music into one immersive environment.
The project reimagines human augmentation not as a mechanical extension, but as an expressive interface. Unlike conventional pipelines, Embodied Animation generates imagery, guided simultaneously by human intention and machine intelligence. The result is a multisensorial hybrid, an animation, an installation, a theater, and a performance, improvised, embodied, and augmented.
Technically, the system is built as a real-time interactive pipeline integrating motion capture, generative models, and audiovisual synthesis. Hand gestures are captured using a Leap Motion sensor, providing continuous spatial and kinematic data as primary input. A secondary iPad-based control interface enables live prompt engineering and parameter modulation through TouchOSC. Within TouchDesigner, the captured motion and control data are processed and connected to a LoRA-trained diffusion model optimized through StreamDiffusion. This pipeline builds on recent developments in latent diffusion image synthesis and real-time perception-based interaction systems.
Anna Borou Yu is a media artist, dancer and researcher. She is the cofounder of MYStudio, researcher at MIT Theater Arts and MIT Media Lab, and PhD Candidate at University of California Santa Barbara. Working across AI-mediated performance, interactive installation, dance, XR, and digital cultural heritage, she designs responsive media environments that investigate embodiment, attention, agency, and mediated selfhood. Her works have been recognized internationally at the Venice Biennale, Ars Electronica, Lumen Prize, Arte Laguna Prize, ACM SIGGRAPH Asia, CVPR Art Gallery, IEEE AIART Gallery, MIT Artfinity Art Festival, Common Senses Festival, and numerous art and film festivals.
Jiajian Min is a media artist, curator, educator and creative entrepreneur. He is cofounder and executive director of MYStudio, a collaborative researcher affiliated with Harvard University and MIT, chair of the MIT Global AI Film Hack, cofounder of HarvardXR conference. Recognized as one of the Forbes China 100 Most Influential Chinese, his projects have been exhibited at leading venues such as Venice Biennale and Ars Electronica Art Gallery. He lectures internationally at SXSW, Cinequest Film Festival, ACM Siggraph Asia and top universities like Yale, Brown, UChicago, USC, etc.
