An interactive motion-capture installation
About this project
This interactive motion-capture installation creates dynamic visual output by capturing every visitor’s facial movement data secretly when watching and reacting to a premade video on different screens, visualizing the complicated relationship between humans and machines.
Keywords:
Motion-capture Touchdesigner
Interactive exhibition
Group project
10/2022 - 12/2022
​My role
Conceptual design, filmmaking, and installation design.
Key features
As the digital world tempts us to obtain an ever-growing amount of information online, digital media is becoming mainstream all around the world and the information carrier is changed to all kinds of devices, including phones, tablets, computers, and LEDs. No one could imagine what life would be like if there were no screens to show us the world.
As a generally acknowledged fact, humans have dominance over machines: we manipulate devices and decide what we want to see. But what if the fact is totally the opposite? What if the machine has its own consciousness to manipulate and decide what to show us? Every time when we browse the latest news it keeps pushing lots of tragedies which could aggravate vicarious trauma in our minds, or it just keeps selling anxiety by showing others’ the effortless life you are thirsty for. It is as if the machines make the decision to present information thus manipulating our bodies and emotions through our daily lives.
Somehow in the future, humans and machines may exist as equal individuals without distinction, the relationship could be far more complicated than just served and manipulated.
Prewords
In my other project "Be Framed", I focused on the mechanical motion of people. For example, workers in the assembly line, master copy fakers in oil painting village, and art examination students in China. They are poeple in machienry mode. I would like to further extend the concept in this project. The role of people and machines are reversed. Machines seem to have the ability of deep thinking, instead of people.
Somehow, people and machines are all in a formless frame. People are restricted in the frame of life and pressure. Machines are following some kind of rule all the time. Therefore, I create a fenced-look installation in the project "Be Framed". In this project, I will explore machine consciousness in a visual way through screens.
Installation of project "Be framed"
Installation
The format and shape of the final visual output convey the original intention that we humans are actually controlled by screens unconsciously. To achieve this goal, we decided to abstract the dynamic facial data as intertwined wires to visualize the complicated relationship between humans and machines.
Vernissage day
Research
Artistic Research
INSIGHTS: Translate human movement information into machine’s language; real-time data transfer; human’s actions with consciousness; human control the machine.
Data
The data in this project is a medium connecting the input of human reactions to the screen in the digital world and the final visual output. By capturing, processing, and transforming real-time data, the influences of digital information on humans in real life can be clearly explained and visualized. However, every visitor’s reactions to the video are unpredictable, so data is random to generate various outputs.
System
Armed with digital media, we get information as diverse as possible which has brought us unprecedented satisfaction. In the Internet era, the constraints of space and time have been broken through thus humans are actually dependent and small in this huge system. The machine is connected by a network to show us a vast volume of information, at the same time, humans are connected with each other by various mediums of information presented by the machine system.
Machine’s Consciousness
The facial movement data include squeezing eyes and eyebrows, turning head, biting and compressing lips, smiling and etc…. These unconscious actions while watching the video can somehow reflect emotional changes when going through information-facing screens and thus indicate the fact that we are actually manipulated by machine’s consciousness.
Eye
Eyebrow
Jaw
Mouth
System Design
The whole system is divided into 2 parts: At the entrance, visitors are greeted by screens playing the premade video. As visitors watched, they would unconsciously respond to certain pieces with facial expressions. At the same time, the camera Zig Sim Pro (an App that send sensor parameters in smartphone to PC by OSC) would capture the facial movements and export the real-time data in OSC format. The 2nd part is separated from the entrance place, that is, TouchDesigner processes the data to produce an abstract dynamic visual effect and eventually projects the output on the screen that can be watched by another group of visitors.
Overall, constantly transforming, the piece takes as input the facial movement of current visitors who are unaware of being monitored and analyzed.
Facial capture test
Reaction Video
When we are actually facing screens, there are countless occasions that may make us do different actions. Our premade video is aimed to reproduce the information explosion in real life. In order to get as many facial responses as possible, we added various dynamic elements, such as fast-moving objects, delicious food photos, flashing images, etc., that may make visitors respond obviously so that the facial data can be easily collected by Zig Sim Pro.
Working Process
Visual Output
By adjusting the parameters of the node (Noise), basic shapes would form some random dynamics. With different rates, intensities, and random ranges, we have various visual outputs.
OSC Connections
