The Voices of VR Podcast
Kent Bye:
Alright. So my name is Kent Bye, and I do the Voices of VR podcast. And here back again doing another episode in celebration of the 15th anniversary of the IFA Doc Lab. I’m here today with Francesca Panetta. Francesca, why don’t you go ahead and introduce yourself and tell me a bit about what you do in the realm of immersive media?
Francesca Panetta:
Sure! Yeah, I am a freelance immersive artist, director, journalist. I spent a long time at The Guardian experimenting in new forms of storytelling for them, and then more recently worked at MIT as a creative director there, again experimenting in new forms of storytelling. There, I was using AI. So, done kind of all kinds of different types of techniques over the last 10 years or so. Now independent.
Kent Bye:
Okay, great! So maybe you could give a bit more context as to your background and your journey into this type of interactive storytelling.
Francesca Panetta:
Yeah, well, my background is in sound. So I trained as a classical musician and then radio producer, originally at the BBC in radio. And then I moved over to The Guardian in 2006 when podcasting was really just starting, and it was a fantastic place for experimenting in new forms of documentary and journalism. Quite early on, I saw that even though podcasting now doesn’t feel like particularly experimental media, back 15 years ago, suddenly without the restrictions of broadcast time limits or specific formats, it felt like a very free space.
I began to get a bit of a taste of what working with high-level journalists and kind of stories could be when you’re unbound from very traditional formats. So we really stripped apart what radio could sound like, what podcasting could be, and began to experiment. Then I moved from there to working with developers, designers, data journalists, and filmmakers to begin to make some of these web documentaries that we all remember—everything from the kind of scrolly newspaper formats to the more glossy NFB kind of formats.
I partnered with a number of different organizations during that period to really see what different types of feature-making could look like. So I partnered with the Imperial War Museum to do a big interactive on the First World War, with the National Film Board of Canada to do a piece about digital morality. We did a piece about climate change and forest fires in Tasmania. So the topics have always been on a really wide variety of subjects. And then out of that, I set up a virtual reality department in partnership with Google, which lasted for 3 years, where we made 13 pieces of VR and 3 60 content, really trying to experiment in different aesthetic forms and different types of interactivity, with the kind of luxury of a decent budget and fantastic infrastructure from The Guardian. What could different VR look like for nonfiction?
Kent Bye:
And so with the IFA, when was the first time that you came across the IFA Doc Lab community?
Francesca Panetta:
I can’t remember which year now. It feels like a long time ago. I mean, it looks like you had the 7 Deadly Sins at Doc Lab in 2014. I don’t know if that was the… So it would have been way before that, you know, a few years before that. I’d been coming for a while. I mean, it’s a really, really special community. It’s an exciting group of people, both in terms of the DocLab people who put it on, but also the audience members as well as the makers. It feels very welcoming but also has a kind of spark and excitement being here.
I mean, it’s interesting thinking about what makes it unique as a community because I definitely feel that. And part of the reason to go to DocLab is of course for the programming, but I’d say for me, it’s about the community. It’s a place to be inspired and connect with people, to grow ideas. And, I mean, Kent, this is your series. I wonder if you’ve managed to think what it is that has made that possible from talking to all the people that you have done. Like, are you finding threads about what it is that gives it that kind of magic?
Kent Bye:
Well, I think part of the thing that I’ve been meditating on a lot lately, at least, is these concepts of potential and how non-interactive media has no potential, meaning that it’s already completely authored and there’s no participation. There’s no requirement for interaction. And just the ways that as people watching these pieces are making choices, then they’re able to see something that may be right at the right moment, but they need to see it. Or it’s something where their participation is somehow changing the outcome of what is made, and it’s actually in collaboration with the audience in a way.
So I feel like the audience of folks at the doc lab are grounded in documentary. It seems to be that there’s a reflection of these deeper patterns of character or story that are grounded in something that has a very established context. And I feel like with fiction, it tends to be a little bit more… But you have to build the entire world to be able to tell that story. Whereas with nonfiction, the sort of larger contextual aspects are there, and you’re able to then start from there and then start to experiment maybe a little bit more with these other aspects of potential and interaction. So I’d love to hear some of your reflections on that since you’ve been kind of at this frontier of immersive and interactive media for a number of years now.
Francesca Panetta:
Yeah, so you think it’s because of the documentary aspect of it that marks it out from the other film festivals? And maybe that’s right, that that focus and who that attracts is different. I also think that the team here does a lot of work in community building, you know, the way they program the events deliberately brings people together. It feels effortless and that it’s kind of fun, but it’s also, I think, really carefully choreographed to build community as well.
So I think it is both the setup of being documentary and interactive, but also some clever people behind the scenes who are bringing those people together and making sure that those conversations between people happen and that there are those spaces for them as well.
Kent Bye:
Well, maybe it’s good to chat about some of the different specific pieces that you’ve had at the doc lab over the years, and we can maybe unpack some of those different themes from there. So was the 7 Deadly Sins, do you recall if that was the first project that you had at Doc Lab in 2014?
Francesca Panetta:
Yeah, I can’t remember, but it sounds about right. That was a project that The Guardian did in collaboration with the National Film Board of Canada. And it was a long project for us to make. And probably my first kind of experience of the difference between newspaper goals and deadlines and production processes and the National Film Board of Canada’s. So the kind of desires for the project were quite similar, but it was a real challenge to figure out what that workflow was because I had been working on… I was special projects editor at the time, and I had been given a lot of freedom to have these long, long leads for projects. But even still, it was nothing like the National Film Board of Canada’s processes. They invited me, Lockdown invited me out to Vancouver to do a kind of labbing session for a week, and that was just incredible. Suddenly I found that the kind of process of ideation and development was very, very different from working in traditional media and really, really fun.
So that was the beginning of that project. And then we kind of went back and forth with different story ideas and different interactions. We were sure that at the heart of it, we wanted to be asking audiences, has their sense of ethics changed by coming through the Internet? Like when you’re on the internet, do you have a different sense of morality and ethics? And we ended up—with the core of the piece being a kind of questionnaire, like is it alright if someone else does this? Is this something that you do? And then you saw the statistics. And then we wrote a number of Guardian stories for it, and we had a wonderful illustrator, and we did a number of videos as well. So in terms of interactivity, it was a relatively simple piece. And in terms of what an interactive documentary can take in form, it was not one of the most complicated, but actually was incredibly successful in terms of the numbers of people who visited it and in terms of how people liked it. Like, I think that it is something that people have been thinking about a lot.
And the stories were very strong. And it also shows that, at that time, there were extremely complicated interactive web documentaries, you know, these nonlinear ones which would go in all kinds of places. I think it also shows that having a kind of clear concept in a clear form can be extremely helpful for audiences as well. That marrying something can in a huge number of branching tree kind of lines, I mean, it can be great, but it’s just not very sometimes. So it was a kind of very clean-cut effective piece.
Kent Bye:
Yeah, it’s still online. I was watching some of it this morning while I was waiting. And yeah, just to be able to go into one of the 7 Deadly Sins. And so there’s different themes that are there, and then you have a video and then there are a number of stories and then you have these different questionnaires. Like, generally, like are you illegally downloading stuff online? Do you think that’s okay? And then you get the survey results. So then you see, okay, this is like a little bit of a sampling to see what’s the larger normative standards here for what are people doing and whether or not they think that’s okay, whether they are condemning themselves or condemning other people.
So yeah, there’s that element of choice that you’re able to pick what theme is maybe the most interesting to you and then watch or read everything about that or kind of jump around if you get the gist of one of the topics or not. So yeah, I think as I think about the principles of that participation, as I’m getting reflections of the larger community from that, based upon people who do it, then it becomes a little bit more interesting in terms of the statistics. But also just to be able to make those choices of like what order, because you could sit down and watch that as a linear piece and then read all of it, but then you kind of can control your own journey and then jump around if you want to. So you don’t have to be a completionist, and you can kind of dip in and dip out when you want to. So I thought it was a really nice structure in that sense.
Francesca Panetta:
But as you reflect on the principles of interaction, just curious, as you’ve gone through your different pieces, maybe some big takeaways in terms of interactivity when it comes to the type of work that you’ve done over the years.
Francesca Panetta:
I mean, I think that each new technology has got different affordances. So like, trying to compare what type of interactivity works well in VR compared to an interactive web feature or an Alexa is completely different. I feel like the story and developing the interactive design need to be done in parallel, and they kind of talk to each other. So as you’re building the pieces, you’re currently experimenting and seeing what works and the language of that technology, the language of the interactivity.
So I use this as an example because I remember this being the first time this was extremely clear to me. So I was building a number of location-based audio pieces. So these are pieces where you were in a specific place, listening to stories in a nonlinear way. You can explore a part of a city or a park and hear stories. And you know, I’d come from radio production, and I thought, well, this is fine, I know how to do this. And you put the content on location and found that none of that content worked at all well because in radio you describe everything, and here you are on-site. In fact, you’re more like a tour guide, and it’s a very, very different language. At the same time, you’re inviting the audiences to be exploring that space, looking around, going and finding things in that space.
And so, over the course of maybe 6 months of making the first piece, we developed a whole new language of scripting, but also how what worked in terms of building the rules like the code of interaction, you know, how pieces stopped and started and faded out, how many pieces you could put on one location, but also what the language was, you know, lots of second-person storytelling, very short clips, very direct language, as if you were a tour guide. And it made me realize that even though this supposedly was the same medium, audio, it had a completely different language which played to that particular platform. And every time I start with a new platform—yes, of course, you bring your storytelling skills from before, but we need to learn the affordances for that particular technology and what is gonna work and how you can develop, you know, the practices that you have been using over the years to fit and to work with it, you know, it’s like it’s got its own character, and you need to figure out what that is and what’s gonna work within it.
It was the same with VR, you know, trying to pack too much into it, trying to have, like, dense bits of audio over the top when you are in this place trying to explore didn’t work at all. So that was about trying to figure out what does embodied storytelling look like, what are the interactions that are meaningful, and you’re not just kind of putting in because you’ve been told at some conference that interactivity is key in VR, and you’re kind of putting in things for the sake of it, which I’ve seen also, you know, done and seen a lot of. So what is the language that works both in terms of like the content but also in terms of the interaction? And I guess when you say what have I learnt over the years, it’s probably that it just takes a while to explore that and get to know it as new technologies develop.
Kent Bye:
I remember talking to you at Sundance 2016, where you had 6 by 9, and you mentioned that same thing around the second-person language to be able to, rather than having other people talk about their experiences about solitary confinement, you were the one that was immersed in the prison and fell, and you were being directed to pay attention to specific… And so that same type of guided tour aspect, but to really draw out specific details of that room to get people more immersed, rather than to listen to someone else’s direct experience that may be the result of having people feel less immersed in their own first-person perspective of what they’re experiencing.
Francesca Panetta:
So yeah, that seemed to be a big inclusion as well as you’re talking about that. And the next piece that I see from Doc Lab 2016 was Underworld, a virtual experience of the London sewers. Maybe you could talk a little bit about that project?
Francesca Panetta:
Yeah, this is a piece we made for Daydream when Daydream were launching, and we just started this collaboration with Google. We thought it would be interesting to do a piece about urban exploration. So there are these urban explorers who do these kind of incredible physical feats like climbing to the tops of skyscrapers but also exploring places that are forbidden. And there is, you know, both the excitement of urban exploration but also a different relationship with how you connect with cities in particular and how that can relate to the kind of layers of history as well.
So this was a piece about a really important part of the history of London, which was in the late 19th century when there was a huge amount of development both above and under the ground in London. And so as you are being the urban explorer in the sewers, you’re also learning about this period of great development. So it’s kind of part game, trying to escape out of the sewers but also learning about the world of urban exploration of the history of London at the same time.
Kent Bye:
Nice! And are those Daydream experiences still available? Or if people wanted to see some of those different VR experiences from The Guardian, is it dependent on the Daydream, or are they available for folks to be able to check out?
Francesca Panetta:
Some of them! So if you go to The Guardian site, you can see all of the pieces there. Some of them we ported over to different platforms. Daydream’s no longer available, so some of them are now 360 videos. Some of them we ported over to Oculus, so it slightly depends on the pieces.
Kent Bye:
Okay, great! Yeah, I know you had done 6 by 9, but I haven’t seen some of the other pieces, so I’ll have to go check those out. So moving into Doc Lab 2017, you had a piece called Limbo. Maybe you could tell us a little bit about Limbo?
Francesca Panetta:
Yeah, Limbo was another VR piece that we made at The Guardian. This was a 360 video, and it was about asylum seekers in the UK. So the title that we gave it, Limbo, came from the interviews that we did. Almost always when I’m starting pieces, we do a lot of research at the beginning. So like quite traditional journalistic research, talking to experts, talking to grassroots people. In this case, we traveled right across the UK talking to asylum seekers about their experience and this is why we’re trying to figure out what the storyline is, what the voices are gonna be in the piece, you know, what the angle is gonna be.
What came up over and over again was that there’s a sense of limbo for asylum seekers. And the reason for this is that they’re not allowed to work. They’re given a very small amount of money to exist on while they’re waiting for an interview. So what happens in the UK if you wait to be given an interview to see whether you’ll be given the status to exist, to stay in the UK? So you’re given a very small amount of money, and you basically are just waiting for that interview to happen. It’s very difficult to interact with the rest of the members of your community because there’s so little money even for traveling or to do anything.
So a lot of people described how they would sit in the rooms that they had been given while they were waiting for the interview, and it felt to them like limbo. And they talked to us about the PTSD that they could be experiencing from having fled from countries like Syria. So the kind of nightmares they’d have, the difficulty it might be to talk to their family back home. I talked to an Iranian doctor who it was too dangerous to talk to his daughter and his wife back home. So he was literally just waiting there, waiting for someone to give him a date for an interview where he would then maybe be let into the country or not.
So this piece, we decided we’d call Limbo, but also we’d give it this aesthetic as well. They said they didn’t feel like they belonged there, didn’t feel like home, and yet this now was their new home. So we worked with ScanLab, a great production company in London, who live-scan the environment. We shot this in Manchester, and we tried to give it this very kind of luminous feeling using point cloud aesthetic. And you’re toured through Manchester and into the house that someone is living in, and through this, you hear the voices of the people that we interviewed and a script that we wrote through it trying to portray this kind of experience.
Kent Bye:
Okay, was that featured as part of the Museum of Other Realities retrospective piece?
Francesca Panetta:
Okay, I think so!
Kent Bye:
Yeah! I think I saw it there. It was part of one of the immersive story retrospectives there, so I was able to see that. Very cool!
Francesca Panetta:
And we actually had a chance to chat in 2019 at TalkLab about In the Event of the Moon Disaster, where you were at MIT doing deepfake explorations of recreating a speech that never happened of Nixon. In that, I actually saw that quite a bit over the next couple of years. It’s still percolating out. The first time I had seen that. But you spent a number of years at MIT. Maybe you can just kind of reflect on that project, but also some of the other things that you were doing at MIT and artificial intelligence and the intersections between AI and storytelling. What type of specific things you were looking at that intersection?
Francesca Panetta:
Yeah, so this project came out of a number of brainstorms that I was having while I was at Harvard first and then at MIT, and a kind of concern around misinformation and deepfakes. So I’d say a lot of the story ideas that I get are probably from the topic matter first rather than specifically thinking about AI. It was more a concern around misinformation. So the project developed out of one of these kind of regular brainstorms that we were having at MIT and at Harvard, where it was at the time, and it was the 50th anniversary of the moon landing, and we knew of this kind of beautiful speech. And so the speech was if the moon landing had not been successful and Nixon, who was the president at the time, would have delivered this on TV.
So yeah, we talked to Casper, the curator of DocLab, about this project, and we did a bit of fundraising. And then I moved to MIT and very quickly needed to put this together for DocLab. I think I’d been at MIT for like 4 months, and we’d got a bit of funding from Mozilla. And it was a tight turnaround. I am not a machine learning expert, so we were really in the hands of the two companies that we collaborated with: Sokani AI and Respecher. I would say that, you know, from an artist’s point of view, my co-director and I, Halsey Bagand, and I were not sure how realistic this was gonna turn out. You know, there is some media where I’m pretty sure what the results are gonna be. I would say this was not the case here.
But, like, is Nixon’s voice gonna sound anything like… You know, they’re gonna be like weird lines around the edge where they’ve put the dialogue replacement part in? But they did an amazing job, and it came together in time for Doc Lab. And then Casper and his team built this beautiful set because we were, yeah, Halsey and I were in Boston. So really all the hard work of sourcing the furniture and finding a nice wallpaper and all of the kind of space books they found. I mean, we had regular calls with them looking at different pictures of couches and rugs and things. But they really did a wonderful job of scurrying around Amsterdam, finding all these bits and pieces, and it looked beautiful when we turned up.
Yeah, and the TV too! It made a big difference because it was of the era as well. So you’re watching this piece on a TV from the sixties. So that, I think, made a big difference as well. It really does.
Kent Bye:
Yeah! Well, I’m curious what you think some of the ultimate potential of all these immersive technologies and immersive storytelling might be and what they might be able to enable.
Francesca Panetta:
I think what it does is find new ways to intrigue and engage with audiences. So, you know, we know that audiences have got like quite low attention spans at the moment. So I think that things that feel novel and intrigue and draw you in is appealing for audiences. I also think that this space has quite a lot of latitude for storytelling. So, you know, the potential for making a kind of hybrid piece like we did for In the Event of the Moon Disaster, that it gives a space for that kind of potentially more creative and artistic space for storytelling as well as the technology and the types of interaction. It’s got a kind of generosity in terms of what you then do with the content.
I think that that means that you can make pieces that can be very emotive, very provocative, very intriguing in a way that maybe more traditional spaces can’t do. So I think that, you know, my aim is always to try and engage audiences in really visceral, strong ways that makes them care, that moves them, that makes them care about the subject matters that I really care about and think are really important to society at the moment. So it’s about trying to use whatever techniques we can to do that, and I think that this space has a lot of freedom in it.
And it means that you get some kind of wacky pieces and wacky experiments, and I’ve certainly made lots of those myself, which might not really beautifully engage people, but that’s part of the process. It’s only through really the kind of trying of new things that you’re going to get to new forms of storytelling and audience engagement. So, you know, that’s why I love this kind of forum is that it provides a platform for that kind of work and experimentation; it gives it a validity; it gives it an audience, which, you know, sometimes when you’re poking around on your own and trying things that you don’t really know who that audience is, and you’d have a community to support you in doing that. So these kind of forums are really useful for that.
Kent Bye:
Is there anything else that’s left unsaid that you’d like to say to the broader immersive community or the DocLab community?
Francesca Panetta:
Well, I would thank DocLab for supporting projects like this over the years and providing this space and encourage more of the community, either the audiences to be coming to these festivals and particularly DocLab and making stuff. I think just persevere. This is a tough time for audiences and for makers when we haven’t had these physical spaces where we don’t know necessarily what audiences want and what those platforms are gonna be. And my feeling is at the moment, this is a time of kind of supporting each other and persevering. And maybe I’m speaking for myself, but I feel like a lot of the community—these have been hard years and it continues to be so. So that’s what I would like to encourage.
Kent Bye:
Well, Fran, thank you so much for joining us today. I enjoyed your work over the years and also just the deep insights you have from doing this deep dive of storytelling and really focusing on the stories and seeing what the affordances of all these new media might be through all these different pieces that you’ve been doing over the years. So yeah, thanks for taking the time to be able to talk about your own journey and some of your own work and also your experiences at Doc Lab over the years. So thanks for joining us today.
Francesca Panetta:
Pleasure! Thanks for having me.
Kent Bye:
So that was Francesca Panetta, a freelance immersive artist, director, and journalist. She spent a long time at The Guardian experimenting with new forms of storytelling, then as a creative director at MIT, experimenting with new forms of storytelling using AI, and now as an independent artist. This conversation was recorded on Tuesday, December 2021, as a part of the collaboration with IFA’s Doc Lab in order to celebrate their 15th year anniversary. If you’d like to support the Voices of VR podcast, then please do consider becoming a member at patreon.com/VoiceofVR. Thanks for listening!