Lacunae

At the height of COVID, we wanted to make an experience where people could be together, but "from a distance", which is how this idea and art piece came about.

Lacunae was a free, multi-venue exhibition that took place during MONA FOMA 2021 in Launceston, Tasmania. Three locations were used in our piece: a church, a florist's and a local theatre.

Each site was set up with an Intel RealSense D435 unit and a rear-projection setup. Participants who moved around and interacted within the space would see themselves as well as those in the other two spaces at the same time.

I was tasked with creating the underlying framework for this exhibition in Unity. Intel RealSense D435 motion capture units were used to track participants at each of the three sites. I implemented the underlying motion capture, positional audio system and the blending effects for each "layer" of the screen.

The screen consisted of four looping videos being played simultaneously: one background layer, and a video for each site's silhouette. I wrote a shader which blended the four textures together based on a mask.

The primary camera was set up to output to a rendertexture, which was used as the mask for the blending shader before it was blit to the screen.

To facilitate this multiplayer experience a small relay server was also set up in order to pass movement information between the three sites. The server was really dumb, and simply sent all information it received from one of the sites back to the other two sites.

Each site's aesthetics and audio were unique, and had some thematic connection to the site they were in. The florist and theatre exhibitions made use of a dynamic audio system where moving close to invisible parts of the space would increase the volume of a given audio channel.

The church's soundscape was a little different, and was something that the person playing at the church would "conduct". The system calculated the angle between a person's shoulder and their hand, which allowed different audio tracks to play based on hand position. It lead to a lot of swaying and arm flapping!

The video assets were created by Darryl Rogers of Soma Lumia, and the audio assets were created by Ian Chia.