top of page
Dec-07-2022 02-29-59.gif
title tex.png

An interactive motion-capture installation

About this project

This interactive motion-capture installation creates dynamic visual output by capturing every visitor’s facial movement data secretly when watching and reacting to a premade video on different screens, visualizing the complicated relationship between humans and machines.

Keywords:

Motion-capture Touchdesigner

Interactive exhibition 

Group project

10/2022 - 12/2022

​My role

Conceptual design, filmmaking, and installation design.

Key features

As the digital world tempts us to obtain an ever-growing amount of information online, digital media is becoming mainstream all around the world and the information carrier is changed to all kinds of devices, including phones, tablets, computers, and LEDs. No one could imagine what life would be like if there were no screens to show us the world.


As a generally acknowledged fact, humans have dominance over machines: we manipulate devices and decide what we want to see. But what if the fact is totally the opposite? What if the machine has its own consciousness to manipulate and decide what to show us? Every time when we browse the latest news it keeps pushing lots of tragedies which could aggravate vicarious trauma in our minds, or it just keeps selling anxiety by showing others’ the effortless life you are thirsty for. It is as if the machines make the decision to present information thus manipulating our bodies and emotions through our daily lives.


Somehow in the future, humans and machines may exist as equal individuals without distinction, the relationship could be far more complicated than just served and manipulated.

Prewords

In my other project "Be Framed", I focused on the mechanical motion of people. For example, workers in the assembly line, master copy fakers in oil painting village, and art examination students in China. They are poeple in machienry mode. I would like to further extend the concept in this project. The role of people and machines are reversed. Machines seem to have the ability of deep thinking, instead of people.

 

Somehow, people and machines are all in a formless frame. People are restricted in the frame of life and pressure. Machines are following some kind of rule all the time. Therefore, I create a fenced-look installation in the project "Be Framed". In this project, I will explore machine consciousness in a visual way through screens. 

section1.png
section3.png

Installation of project "Be framed"

Installation

Installation

Research

The format and shape of the final visual output convey the original intention that we humans are actually controlled by screens unconsciously. To achieve this goal, we decided to abstract the dynamic facial data as intertwined wires to visualize the complicated relationship between humans and machines.

Vernissage day

IMG_2703.PNG
IMG_2699.PNG
IMG_2701.PNG
IMG_2654_edited.jpg
IMG_2696.JPG
IMG_2695.JPG
IMG_2700.PNG

Research

Artistic Research

1085999_102_1160_MTM4MjcyMjYxMDIwNzUyNDg5Mjc.jpeg
future-you-og.jpeg
universaleverything-futureyou-1.jpeg
universaleverything-futureyou-3-1600x892.jpeg

INSIGHTS: Translate human movement information into machine’s language; real-time data transfer; human’s actions with consciousness; human control the machine.

Data

The data in this project is a medium connecting the input of human reactions to the screen in the digital world and the final visual output. By capturing, processing, and transforming real-time data, the influences of digital information on humans in real life can be clearly explained and visualized. However, every visitor’s reactions to the video are unpredictable, so data is random to generate various outputs.

System

Armed with digital media, we get information as diverse as possible which has brought us unprecedented satisfaction. In the Internet era, the constraints of space and time have been broken through thus humans are actually dependent and small in this huge system. The machine is connected by a network to show us a vast volume of information, at the same time, humans are connected with each other by various mediums of information presented by the machine system.

Machine’s Consciousness

Machine’s Consciousness

The facial movement data include squeezing eyes and eyebrows, turning head, biting and compressing lips, smiling and etc…. These unconscious actions while watching the video can somehow reflect emotional changes when going through information-facing screens and thus indicate the fact that we are actually manipulated by machine’s consciousness.

machine con.png

Eye

Eyebrow

Jaw

Mouth

System Design

System design

The whole system is divided into 2 parts: At the entrance, visitors are greeted by screens playing the premade video. As visitors watched, they would unconsciously respond to certain pieces with facial expressions. At the same time, the camera Zig Sim Pro (an App that send sensor parameters in smartphone to PC by OSC) would capture the facial movements and export the real-time data in OSC format. The 2nd part is separated from the entrance place, that is, TouchDesigner processes the data to produce an abstract dynamic visual effect and eventually projects the output on the screen that can be watched by another group of visitors.


Overall, constantly transforming, the piece takes as input the facial movement of current visitors who are unaware of being monitored and analyzed.

pictures.png
Screen Shot 2022-12-07 at 5.48.15 AM.png
Reaction video

Facial capture test

Reaction Video

When we are actually facing screens, there are countless occasions that may make us do different actions. Our premade video is aimed to reproduce the information explosion in real life. In order to get as many facial responses as possible, we added various dynamic elements, such as fast-moving objects, delicious food photos, flashing images, etc., that may make visitors respond obviously so that the facial data can be easily collected by Zig Sim Pro.

Working Process

working1.png
5.png
7.png
22.png
4.png
Visual output

Visual Output

By adjusting the parameters of the node (Noise), basic shapes would form some random dynamics. With different rates, intensities, and random ranges, we have various visual outputs.

output6.gif
output5.gif
output2.gif
output1.gif
output3.gif
output4.gif

OSC Connections

18.png
23.jpg
3.png
2.png
1.png

© 2022 By WENYI GU.

bottom of page