Colin Wren's Blog

As a digital artist, most of the work I produce ends up on a Computer Monitor or the output through another medium that the user can interact with. The one thing I want to break away from is the flat 2D environment that the Computer Monitor forces on the viewer.

In my Double Project I worked with Augmented Reality, I used the ARToolKit library to create a interactive soundscape that responded to the positions of the markers (ARToolKit tracks these by comparing the pixels in a frame to a .patt file) so when the marker was at the bottom left of the screen the audio would be clear and when the marker was at the top right the audio had effects applied heavily. The setup for this was a reversal from traditional ARToolKit uses which required expensive head mounted devices that allowed the audience to see the objects in 3D by having a stationary marker. I instead used a stationary camera but the markers can be rotated.
Thanks to the leaps in mobile devices (the iPhone for instance) ARToolKit will soon make the leap to these portable devices, though this makes me think of how the user of such a device will be able to use custom markers and get the custom 3D objects that the marker is going to be using (one of the basic ARToolKit functions is allowing a marker to show a custom 3D model, though these files are normally held in a directory relative to the application). A web database could be used that stores the .patt file used for reading the marker and the appropriate 3D image to use, though this would require a way of telling the user what .patt file to download. This could be handled by the device by taking a picture of the marker, and working out the marker’s basic properties (such as how big the shapes are, does it have straight lines, curved lines or letters) and then asking the user to choose from a list of similar markers, then downloading the appropriate files. Another approach would be to use the GPRS on the iPhone to pinpoint the user’s location then send the lat and long coords to a database and as before download the file. The markers creator would upload the files and details to a website (use google maps to set where the marker is). The strength of this system is that it allows anyone to upload markers and also allows the user to download files in advance. So if they are planning a trip to the Tate Modern and want to see if there’s a marker nearby they could look up the Tate Modern on the website and see if someones put one up and download the files to their phone.

I was thinking about how these devices would enable the user total 3D immersion with they surroundings if those surroundings were altered using augmented reality. The user could be walking through a underpass and spot a marker on the wall, and if they had the right .patt file (or downloaded it) they would then be able to see the 3D object (and audio?) that the marker’s owner wanted them to see (let’s say the marker makes shark break through the wall). As the user then walks around the marker the device would rotate the 3D object in the right direction making the model seem almost real. If for instance the user is using an Touch Screen phone then they would also be able to interact with the object, for instance touch the shark’s nose and it jolts back.

This type of interaction is ideal for digital art, many digital artists want their work to be interactive and immersive. Due to technical limitations they can’t have full immersion because the medium they are presenting their work in is (in real space) 2D. There are ways that the artist tries to get around this limitation, for instance a while back I went to the Walker Gallery in Liverpool when visiting a friend, at that time Airside had their Insyde installation up. The installation had you walking into a giant shipping crate and then jumping onto images projected onto the floor by projectors, doing so would then make things onscreen happen. Though this blew my mind at the time and was a fun experience I was still aware that I was looking at 2 separate things. The projected images, though acting as a form of interacting with what was happening on screen were still part of real life and for me to utilize them I had to break away from the screen to see where I had to step.

This temporary need to go back to the real world spoils the immersive environment that the artist is trying to put across. I would like to break away from this need to go back to real life so I can proceed with the artist’s immersive environment. If the interaction with the art is done within the screen (like the shark interaction example I gave 2 paragraphs ago) then the audience can become more immersed. In the Augmented Reality Gallery the works of art would be held within the device (let’s say an iPhone application (which could be updated when a new exhibition is put on, this would help to draw people back)) and the markers on the floors, walls and ceilings would just be placeholders for the art work. The audience would walk around the gallery with their headphones in (the gallery would use 3D sound which would determine how close the audience is to a piece of art and then adjust the mix accordingly), looking through their device’s screen (or through head mounted displays connected to the devices). As they move around the gallery they can see all the works (like in a normal gallery) within sight and as they move around a certain work the device would translate the position and this helps cement their position in the environment. This would mean that the Artist wouldn’t need to create an environment (though they could create an environment within the Gallery’s environment) and the gallery environment(both real and digital) could be used as a platform. In the next 2 paragraphs I’m going to talk about 2 different works I feel sum up the possibilities an Augmented Reality Gallery could open up for digital artists.

A pile of bricks
If you’ve seen Equivalent VIII by Carl Andre and you’ve paid attention I think you know where I’m going. Imagine going to an Augmented Reality Gallery, being given an iPhone and told to use the iPhone as your ‘eyes’ in the gallery. You wander around the Gallery looking through the iPhone(like when your taking a picture) and suddenly notice there’s a piece of wood on the floor as you get nearer to it a pile of bricks appear where the wood had been. Boring? Or the best minimalist work ever? For those of you that said boring here’s the twist. You notice that the sign that said ‘look through your iPhone’ now says ‘Press me to rearrange the work’, You press the sign and suddenly the bricks start to hover and then fall into a massive unordered heap. You can now pick up the bricks and make them into whatever you want. For those who said they loved the work in it’s unaltered state there’s an information icon floating above the work which when pressed will start an audio clip on the work.
This type of work allows the audience become immersed in work without the need for any special equipment (other than the iPhone) and each user will have a different experience, though the artist could enable the audience to upload their creation. This work will appeal to art lovers and art haters because they are given 2 paths to follow.

Capcom did it!
This example comes from Resident Evil 2 but I feel would be a great work. For those of you have played this game I’ll be doing a real life version of the Licker intro. Imagine the scene. Similar to the Pile of Bricks in the last paragraph, your walking around looking through your iPhone appreciating the works when you suddenly spot a pile of blood on the floor. You can’t interact with it like the other works so you decide to see if it’s actually part of the Gallery or just another art work. You look at the floor and sure enough it’s a pile of blood, real blood. You look up and notice that on the ceiling there’s a pattern. You put your iPhone in front of your eyes again and now there’s a skinless creature with giant claws and it’s brain exposed. You weren’t expecting this and so you jump back with shock. The iPhone accelerometer notices the movement and suddenly the thing on the ceiling starts to lash out at you. You decide to move around the work thinking that because everything else in the gallery has been fixed in place you will be able move and see the back of the creature. As you do though the creature rotates itself so it’s always facing you. Your really disturbed now.
This experience is one that I’d love to have, to be so immersed in the Gallery’s environment that I’m disturbed by something that I know is just some code and a 3D model. I know that there’s nothing on the ceiling because I saw the pattern but it still caught me off guard.
Works like this would make way for more immersive experiences with work, and maybe even spawn a new type of spacial game. Outdoors exhibitions could take place or graffiti artists could use the markers to create their own 3D graffiti.

The 2 works I just talked about present the audience with 2 different experiences. The pile of bricks give them the ability to re-imagine an artist’s work, making new art in the process. The Licker example gives them a truly immersive experience that so far digital art cannot offer. These are just 2 examples I can think of that an Augmented Reality Gallery would produce. There could be combinations of the 2 examples, it’s up the artist’s to determine what happens. The big question though is. How would an Augmented Reality Gallery work? Would the works reset once the user turns away from the work or would it keep the state in the devices memory (keeping the state the user left the work in would help the immersion of the work). Would the audience use their own devices or ones the gallery supplies? How much would you charge to get in? Would there be cheaper rates for people who are returning? Though these are business questions they do raise the question of the price of such a project. If the iPhone and Android were to be the devices that the gallery used the price would be relatively low, though a lot of time would have to spent in the development of the viewing application. If the gallery used it’s own special headmounted display the price would be high but it would result in less time being spent developing the application because there wouldn’t need to be any work arounds to get the device to function the way it’s needed to. I’d choose the first option due to there being a ARToolKit app for the iPhone already in development.
A bigger question though is how would one curate an Augmented Reality Gallery, how do you decide which works to show and which not to show? How would you plan out the gallery space and how would the works all sit well together? Would 2D works be allowed to be displayed or would it be 3D only? These are all up to the curator but I can’t see there being too much of an issue if 2D work was displayed. If the Licker can rotate to always be head on for the viewer then so can 2D work.

This has been a short introduction to my idea of an Augmented Reality Gallery and I hope to develop it over the next few months and submit either the whole gallery idea or just a proposal for an artwork to the Arts Council at some point.