Nick Gellatly
Light on things that are not screens
Thursday morning (22 October) was the practical follow-up to last week’s session on projection mapping. Last week we talked about it. This week we got our hands on a projector, a copy of Resolume Arena and a table of small white models, and found out how much harder it is than it looks on YouTube.
We were joined by students from the Interior Design course, who made the models. They had built the surfaces, and we had the software to light them.
Picking up from Week 4
In Week 4 (Part 2) I wrote about projection mapping as one half of the “phygital”: joining a physical object to a digital layer so that the object itself becomes the screen. The academic name for it, from Bimber and Raskar (2005), is spatial augmented reality. Instead of handing people a headset or a phone, you put the digital content straight onto the real surfaces around them, so the room does the work and everyone sees the same thing at once.
The example that stuck with me was Led by Donkeys projecting onto the White Cliffs of Dover. It works because the surface already means something before a single pixel lands on it. The cliffs bring the meaning; the projection only gives them a voice.
The kit and the software
The set-up was modest: two laptops on a table, a small projector, a white cyclorama wall and Resolume Arena 7. Resolume is ‘VJ’ software, built for nightclubs and festivals. But it has exactly the tools projection mapping needs, called Advanced Output.
The work happens in two places:
- Input Selection is where you choose which part of your image or video you want to use.
- Output Transformation is where you decide where that part ends up in the real world.
The building blocks are called slices. A slice starts life as a rectangle, but you can turn it into a polygon, drag its corners with Edit Points and rotate it until it matches the face of the object in front of you. Each face of a model gets its own slice. Then you press Save & Close and go and look at the actual object, because the laptop screen will lie to you.
Cutting a triangle and mapping the stairs
My target was a small white staircase model with a slanted top: a set of steps under a long diagonal. A rectangular image thrown at it just spills over the edges and onto the wall behind, which instantly gives the game away.
So before going anywhere near Resolume, I edited the uploaded images into a triangle, so the picture itself already had the same shape as the side of the stairs. Then, in Advanced Output, I made a polygon slice, dragged its three points onto the corners of the stair diagonal and nudged them until the edge of the image ran along the edge of the model instead of past it.
A few things I learned along the way:
- Prepare the image first. Shaping the picture before it goes in saves a lot of fiddling with masks later. The software can crop, but it is easier to start with the right shape.
- Small moves matter. A corner a few pixels out on the laptop is a centimetre out on the model, and the eye notices straight away.
- Look at the object, not the screen. I spent the first part of the session staring at the laptop. The second part, the useful part, was spent looking at the stairs and adjusting until it looked right in the room.
- Shadows are part of it. Anything that sticks out casts a shadow onto whatever is behind it, and you can either fight that or use it.
What it means for my project
In the Beginning Was the Word is an audio play: the voices are finished and the visitor walks the Sedgwick Museum with earbuds in. So I am not about to start projecting dinosaurs onto the display cases. But Thursday did make two things clearer.
First, the Crystal Formation entrance I floated in Week 4 is more realistic than I thought. A single projector, a few polygon slices and a white faceted shape would be enough to make a crystal “grow” or shimmer at the door, in step with the shimmer in the audio. That is a small, achievable piece of phygital, not a full building job.
Second, the Led by Donkeys lesson applies here too. The best surfaces already mean something. In the Sedgwick that might be a slab of rock, a fossil or the edge of a cabinet, lit just enough to point the listener’s eye at the thing the Catalogue is talking about. The light should serve the object, not compete with it.
If I take this forward, the Interior Design students are exactly the people to ask about building the surface.
Choose your own adventure
The afternoon session turned from where a story is told to who gets to steer it. Having spent the week building an audio play that walks people through the Sedgwick in a fixed order, it was useful to spend an afternoon on stories that hand the direction to the audience.
Stories that make you work
The theory came from a Prezi – written by course lecturer Tina Burton, on the ergodic, cybertext, aporia and epiphany. The terms come from Espen Aarseth’s Cybertext (1997), and once they are untangled they are surprisingly practical.
- Ergodic literature is writing that needs “non-trivial effort” to get through. Turning pages does not count. Choosing a path, solving a puzzle or physically moving through a space does.
- Cybertext is Aarseth’s word for texts that work like machines, producing different versions of themselves depending on what the reader does.
- Aporia is the moment you are stuck: the locked door, the dead end, the choice you cannot see past.
- Epiphany is the moment it clicks and the way forward opens.
The rhythm between the last two, stuck then unstuck, is a big part of what makes interactive work feel rewarding rather than merely busy.
Three examples
Oat the Goat. A short animated story from New Zealand about kindness and bullying, in which the viewer chooses what Oat does at key moments. It is a good reminder that branching does not have to mean complicated: a handful of well-placed choices, each with a clear consequence, is enough to make a young audience feel responsible for how the story turns out.
J. R. Carpenter. Emily introduced us to the work of this artist and writer, who makes pieces that move between print, performance and the web. The Gathering Cloud takes the idea of “the cloud” literally, setting the weather against the very physical, energy-hungry machinery behind the internet. What struck me was how the form makes the argument: you read it on the thing it is criticising.
Karen, by Blast Theory. An app in which a life coach called Karen asks you questions, and gradually gets more personal than you expected. Behind the friendly chat she is building a profile of you, and at the end you can buy a report of what she found. It is unsettling in the best way: the interactivity is the point, because it makes you feel what handing over your data actually feels like.
Branching, agency and a Twine workshop
The core idea was the branching narrative: letting the audience make choices that change what happens, as in a video with several endings or scenes that only some viewers ever see. It came with a set of good, awkward questions:
- How do I tell the story, and how much agency do I give the audience?
- Does that make me a new kind of storyteller, more designer than narrator?
- And how would I actually measure whether a story is truly immersive?
Then we tried it. Twine is a free, open-source tool for writing interactive stories. It runs in a web browser, and you write in “passages”, little cards of text joined by links in double square brackets, which Twine draws as a map of arrows so you can see every path at once. The Twine Cookbook has ready-made recipes for the trickier things, such as keeping score or remembering a reader’s earlier choices.
My first attempt is The Flat White, a nine-passage story which was based on TikTok star Rory Sutherland’s frustration at people choosing complicated coffees while he’s in the queue with a tight deadline.
Even at nine passages, the map view made one thing obvious: every choice you offer doubles the writing you owe. Branching is generous to the reader and quite demanding of the writer.
What it means for my project
In the Beginning Was the Word is not a branching story. Everyone hears the same play in the same order. But the session made me realise it is more ‘ergodic’ than I had given it credit for.
- The effort is physical. The listener has to walk the gallery, find each coloured star and stand in the right place. Aarseth’s “non-trivial effort” is, in my case, the whole length of the museum.
- There is aporia and epiphany built in. Looking for the next star is a small moment of being lost, and finding it is a small reward. The bigger epiphany is designed: after a whole tour of the left ear (the label) and the right ear (the life) disagreeing, they finally say the same words at Jonathan Cutbill.
- The agency is real, if quiet. Visitors can pause, skip the two content-noted sections, choose a queer dress code, or simply wander off. That is closer to Karen’s gentle, personal kind of interaction than to Oat the Goat’s forks in the road.
It also gave me a way to answer the question of how to measure immersion: when my testers walk it, I want to know where they got lost, where they lingered and whether they looked up from their phones. Those three things will tell me more than a questionnaire.
And I can already see a post prototype Twine version for the zine or the website: a short “choose your code” story that sends readers to the green carnation, the single earring or the carabiner, each with its own slice of queer history.
Leave a Reply