Telling factual stories in virtual reality, 360 degree video and augmented reality

Slides from a lecture on the MA in Data Journalism and the MA in Media Production at Birmingham City University, explaining what types of stories and projects suit immersive technologies such as VR and AR, considerations when using them, and techniques employed in the field.

1. Immersive media
MED7334 Narrative

2. What I’ll cover
●Definitions and differences
●Editorial choices to consider in VR, AR and 360
●Key concepts: immersion, presence, and remediation

3. What are we talking about?
●Virtual reality (VR): ‘Virtual’ as in not real? Or not ‘here’?
●360-degree video: real but not here/now
●Augmented reality (AR): Here, but with a (typically
data-driven) layer ‘augmenting’ what we see

5. Typical reasons for using 360/VR
●Setting as character
●Setting “unfilmable”/doesn’t exist/conceptual
●Multiple things happening (e.g. events)
●Spectacle: ‘experience’ (e.g. Blick)
●Desire to create ‘empathy’ (hard to measure)

11. Will the UK Brexit? http://news-lab-brexit.appspot.com/vr/

14. “Positioning” the user?
●Participant (spoken to, a ‘character’)
●Observer (invisible)
●Control pace/outcome — or no control?
●“Avatar anthropomorphism”
Bosch, Gensch and Rath-Wiggins (in press); Lugrin, Latt & Latoschik 2015

15. Key concepts: immersion...
●Isolation in the physical environment
●Perception of being included in the virtual
environment
●‘Natural’ interactions and control perception
●Perception of moving within the environment

16. 2 forms? High and low-tech “VR”
●Cardboard and mobile phone-based VR
●Oculus Rift, PC-/PS4-based VR
●Implications for immersion, proprioception

17. RAIR: Response-As-If-Real
“If you are there (PI [Place Illusion]) and what
appears to be happening is really happening (Psi
[Plausibility Illusion]) then this is happening to
you! Hence you are likely to respond as if it were
real. We call this ‘response-as-if-real’ RAIR.”
Slater, M (2009)

18. ...and
presence

20. ●Remediation: see Bolter & Grusin (1999)
●See behind the scenes of one VR documentary
●Why do we hide the camera operator/kit?
A remediation of documentary?

21. A remediation of theatre?
●Set ‘design’/setting as character
●An ‘experience’, not just a story
●Directing attention to a part of the stage (but
also considering all parts as potential focus)

22. “Barker’s innovation was rather
in the viewer’s interface with
the image. The panorama was an
apparatus that would isolate and
control what it was possible to
see. His specifications included
an isolated central viewing
platform exactly halfway up the
height of the canvas…”
Robert Barker’s panoramas

23. “Barker wasn’t taking the picture
out of the frame, he was making
the frame big enough to include
the spectator as well.”

24. ●Narration (voice over)
○Descriptive (what am I seeing)
○Complementary (e.g. interview),
○Directing attention
●Direct address (by ‘characters’)
Editorial choices: audio (verbal)

26. The role of non-verbal audio
●Music
●Wild track
●“Internal” audio, e.g. heartbeats, breathing
●Audio cues (directing attention)
●Haptic feedback

28. Spatial/binaural audio
Capturing sound, not just images, in an immersive
way, e.g. via a 360 or ambisonic microphone
Or via 2 mics, facing opposite directions, 7” apart
See: NPR guide to recording spatial audio for 360-degree video and BBC Academy: What is spatial audio?

29. Binaural sound

30. Ambisonic or 360 sound

31. Ambisonic/360 sound

32. No ‘frame’ but editorial choices
●Mounted camera, starting position
●Moving? (Motion sickness)
●Point of view angle?
●Cuts?
●Captions?

33. “When we compose a shot in VR
there is some sort of primary
action that is happening within a
few feet of the camera but there
should also be things that are
happening around that, maybe in
the medium distance or the far
distance and one of the things
that we struggled with is how
much information can we deliver
without overwhelming people?


https://video.arstechnica.com/watch/culture-and-entertainment-b
ehind-the-scenes-making-a-virtual-reality-documentary

34. “Shelmerdine’s ‘Catatonic’
(2015) cleverly situates the
viewer in a wheelchair. By
doing this, the piece creates
movement that [is] integrated
in the story [and] solves the
proprioception problem by
giving the user arms and legs
within the shot – and because
it’s on a wheelchair, it makes
sense, at least from the
narrative point of view, that
they are not under your
control”
- Joan Soler-Adillon and Carles Sora (in press)

35. Some VR/360 tools
●Google Street View, Fyuse
●Discovery VR, Within, Jaunt VR, NYT VR
●Ricoh Theta, Samsung Gear 360, GoPro, Kodak
PIXPRO
●Dual-lens (+stitching) vs single-lens

37. https://twitter.com/dougalshawBBC/status/1274368
870946668545

38. AR
[Augmented Reality]

39. AR: orienting the augmentation
●Phone sensors (GPS + gyroscope): where am I
and which way is the camera facing?
●QR code: where is the surface?
●Object recognition, e.g. Snapchat filters

43. Key points
●Have a clear reason for using VR: what is it about your story
that suits this medium?
●Consider editorial decisions such as narration, positioning,
edits, and the use of sound
●Consider immersion and presence and related concepts like
proprioception

44. Reading
●BBC Storytelling and Audience Insights into VR
●OR: any research from
https://pinboard.in/u:paulbradshaw/t:vr+research or
https://pinboard.in/u:paulbradshaw/t:ar+research

45. Workshop
●Come up with an idea for a piece of immersive storytelling
●What technology and why?
●Perspective and why?
●How will it use audio?
●Will it have cuts? How/where will it be positioned/framed?