Spatial Audio Lessons Learned from Unscramble the Oracle BOSE AR

3 Lessons from our BOSE AR Audio-First game. While photorealistic graphics grab the lion share of attention, deeply immersive experiences must address all of the senses. This talk covers a post mortem of an audio-only Unscramble the Oracle (BOSE AR) game using fully spatialized interactive audio scapes that require no vision. Come learn about the challenges we faced creating this first of a kind interactive walkable audiobook. Learn how storytelling, navigation, character interaction change when the player can only can hear. Together we look at how we succeeded and stumbled in creating an audio world to explore with native head, body, and interactions How we leveraged characters, specialize sound, and audio UX. What audio attributes increased spatial positioning, what reduced it, and how create audio highlight states. Additionally, it offers lessons from designing to the other senses hearing and touch including several techniques to cross talk and fool the senses.

Pick up any smart phone and gestures to operate it are nearly universal. Put on a set of smart glasses or play a mobile AR game and they are hobbled by out dated UX/UI standards that interfere with the new opportunities on the emerging AR platforms for spacial computing. In contrast to the UX/UI guidelines for WIMP interfaces; precious little has been written about best practices for AR. With screenless information, spatial computing, floating menus, mid air gestures, restyled 3DOF and 6DOF controllers, new graphics, audio, and haptics AR offers novel interaction patterns that demand new interface language and standards. Come join us for a dive into the emerging UX / UI standards for AR and spatial computing with inspiring examples from Leap Motion, North, mixed reality, Magic Leap, as well as some from XEO's own laboratory and latest experiments.��Takeaway
Attendees will walk away with concrete examples and techniques for spatial audio design for AR as well as traditional games. Position is the power of immersive audio. Attendees will learn how precise location separates one audio source from another and the motion of these audio sources unlocks new story telling possibilities. Additionally how to create flexible layouts for the Real World in AR. The lessons in immersive audio also offers lessons for highly dynamic flat screen game worlds that are more than a backdrop for gameplay. Additionally, attendees will learn how to enhance button and object interaction in an audio only world.
Intended Audience
Audio engineers, game designers, and developers of all game platforms will walk away with insights in how leverage psychological properties of sound and how to use use spacialized audio to tell stories, crate interactive audio objects and menus, and interact with a game world that they can hear but not see.��These lessons are also suitable for developers of flat screen games wishing to create more immersive experiences

1. Nicole Lazzaro
Hear Me Feel Me Let Me Play: Lessons from Unscramble
the Oracle a BOSE AR Audio-First AR Game
GDC Summer SF/Online
August 4, 2020
@NicoleLazzaro

2. Hear Me Feel Me Let Me Play:
Lessons from
Unscramble the Oracle
a BOSE AR Audio-First AR Game

3. Spatial Audio Down sampled to Stereo
Best with
Headphones

4. Stanford Psychology
4 Keys to Fun
\o/
Fiero!
Tilt World
the first iPhone game
TiltWorld.com
Tilt World
Plant 16K trees in
Madagascar
Unscramble the Oracle
Follow the White Rabbit
@NicoleLazzaroFree Download
4K2F.com

5. Thank You! Unscramble the Oracle TEAM!
•David Fox writing & Sfx
Electric Eggplant.com
•Patrick O’ShaughnesseyNarr.Engine
Patched Reality.com
•WonkubusWonk Walk, VO. & Art
(Michael Crabbs)
MichaelCrabbs.com
•Nicole Lazzaro Game Design & UX
XEODesign.com

6. Audio First Choose Your Own Adventure You Play Walking Around the Block
How We Designed an Audio First GameImage: Wonkubus

7. Lessons from Beatles, Saturday Night Fever, And Stonehenge
How We Designed an Audio First GameImage: Wonkubus

8. BLUETOOTH SPEAKERS & IMU
BOSE AR
Smart Phone

9. Audio First AR Head Pose Robust
BOSE AR Game Jam
PlayCrafting
July 26, 2019
Image: Wonkubus

10. Design is
Constraint

11. Push Limits Learn Audio Spatialization For All games
•Audio First
•1st person
•“Phone in Pocket”
•Fully Playable
•Walk & Look Around
•Explorable Audio World
•Interactive Audio Gameplay
•Only playable in spatial audio
•Core Loop Required Audio Spatialization
Audio Goals
Game
Technology Did Well
1.Head pose
2.Tap headset to select
3.Position audio (Resonance Plugin)
Challenge
✓Responsive
✓Accurate
X 6DoF (only 3DoF)

12. 3 Challenges for Audio-First Games
StandWalkChooseSpatial Audio >
Image: Wonkubus

13. Stand1

14. Maximize Audio Position Data Around Player
•Immersive Alive Audio World
•Sound Outside Head
•No Headlock
•Positional Clarity
Audio Goals
Stand

15. Why Does Doing This Not Invert the Sound?

16. Beatles Abbey Road 1stto Record in Stereo and Synthesizer
More: https://www.pbs.org/newshour/arts/the-beatles-abbey-road-revolutionized-how-a-track-could-be-recorded-listen-closely-here
Here Comes the Sun ©1969 EMI

17. Want Spatial Audio to Be Out There in the World Around the Player
More: https://www.pbs.org/newshour/arts/the-beatles-abbey-road-revolutionized-how-a-track-could-be-recorded-listen-closely-here

18. Surround Sound ≠Spatial Audio
ØStereo
ØAmbisonic
ØBinaural
ØASMR

19. Head Motion While Standing -Spatial Audio Field

20. Unscramble the Oracle

21. Unscramble the Oracle 6DoF
(Degrees of Freedom for Movement)
“Find the Frog”
Walk Around Rotate Look Up & DownStereo Inside the Head
Zero Degrees of Freedom

22. Head Pose = Degrees of Freedom (DoF)
No DoF
Headphones
No Motion
Head Lock
6 DoF
Walk forward/backward
Sideways left/right
Jump up/down
3 DoF
Look up/down
Left/right
Tilt Left/Right
Spatialization Requires Motion

23. Head Tilt Shows Curiosity
Also Also Increased Spatialized Sampling for Elevation
Amplifies the distortion Between Two Ears
for all Other direction of Head Motion
containing elevation information

24. Auxel
AUXEL
“Audio Pixel”
Audio Source With Positional Information

25. How Big is an Auxel?

26. Rotation
Elevation
Distance
Audio Position -Auxel = RED
Auxel = R5 E3 D3
“Audio Pixel”
5
How many discrete audio positions can players hear?
3
3?
45 Positions 1-3 at a time

27. While Standing in Place Distance Is Less Important
5
Unless Auxel is Moving Relative to Player
Auxel = R5 E3 D3
“Audio Pixel”
Rotation
Elevation
Distance
3?

28. 3
AuxelsNeed Spacing
Position Time Memory
5 3
The Sound of Silence

29. Easier to Localize Auxelswhen Hear Spatial Distortion
Sharp AttackRepeating Pattern
TIME TO: Move Head LATENCY HEADLOCK & OVERSHOOT

30. Spatialization Tips
•Respond to Head Movement Esp. Rotation
•TIME TO DISCOVER
•Length or Long Enough Loops
•Identifiable & Separation
•Mono Sources in Game World No Reflections
•Sharp Attack
•Pitch Above and Below
Audio Goals
Stand

31. Walk2

32. Walk Through the Soundscape

34. Maximize Sense of Forward Motion Through Scene
•Audio Only Storytelling
•Audio Depth Perception
(how Far Out there)
•Walk Anywhere and Stay within Bounds
•Respond to Player’s Walk Speed
Audio GOALS
Walk
3

35. How Did FCG Cross the Road?
Image: Wonkubus

36. Walking Focuses On Distance Cues
Auxel = R5 E3 D3
“Audio-Pixel”
Rotation
Elevation
Distance
53
3?
How many discrete audio positions can players hear?

37. Walk Audio First Lampposts
3.5 DoF

38. Detect Walking from Head Pose
3.5 DoF
At Least 1 Auxel Apart

39. Fast Motion AuxelsOverlap
Fast Player Motion
A1A0A2A0

40. Alternate Audio Lampposts for More Spatialization AR+AD
3.5 DoF
“on rails”
Top Down View
AD -Distance
AR -Rotation

41. Spatial Audio Walk –Believable Real World Overlay
3.5 DoF
As DesignedAs Used
Keep Players In Audio World
Seat BounceWalk Around
the Block
WalkWander

42. Walk -Unscramble the Oracle

43. Spatialization Tips
•Narrative Audio Beats
•Group Audio Sources Side to Side & 3 Depths
•Auxel Space for Slow & Fast Motion
•Show Progress Pass By Head w/in 1 Meter
•Footsteps are Respective and Same Position
•Actions Like Pull Slice Pizza Off Door
Audio GOALS
Walk
3

44. CHOICE3
Putting the Choosein Choose Your Own Adventure
Image: Wonkubus

45. Choice
Peck 1 for Adventure
Image: Nicole Lazzaro

46. Mysterious

47. Maximize Choice From Audio Menus in World
•4 State Buttons
•Choose Between Options
•Options Integrated into World
Audio GOALS
Choose

48. Psychology Constraints of the Ear
Audio menus required more resolution than walk paths
compared to visual menus Auxel Menus
1.Low resolution
2.Brain’s short term memory buffers
3.Hard to refresh audio information
4.Experience 1 audio source at a time
5.Mentally map audio source with an object, event, or
interaction opportunity

49. Spatial Audio Dead Zone
“Find the Frog”

50. Simple Interaction & Spatial
Image: Nicole Lazzaro

52. 4 State Audio Buttons -No Visuals
Plan B
oTap the Phone
oTap Audio Prompt
oYes/No Nods
4 State Audio Buttons
I’m a Button
Highlight
Activated
Un HighlightedGaze “beak cursor” to interact
Heard what you looked at
Double Tap Headset To Choose
Constant Sounds Annoying!
Added Icons for Who Was Speaking

53. Menus Focuses On Rotation & Elevation
Auxel = R5 E3 D3
“Audio-Pixel”
Rotation
Elevation
Distance
5
3?

54. Spatial Audio Cues to Find Menu Option (Find The Frog)
High Pitch Wing Flap Loop
Fairy Chicken Godmother
Button
Low Pitch Frog Croak Loop
[ empty Audio Position]
On/Off Highlight Audio
Double Tap Headset
To Choose

55. Stonehenge Audio Menus -Chicken Flap Audio Cue Moves
Tells story of Chicken flying to a new interaction

56. 3DoF Audio-Menus
I am The Pizza DoorRibbit Ribbit
Do You Want to
Wade Across?
Menus 3DoF
No Distatnce

57. Spatialization Tips
•4-5 AuxelsHorizontally Spaced
•Beware Dead Zone Front & Behind
•Latency: Bluetooth, WiFi, HRTF
•Frogs > Turnips
•Character & Story Beats = Easier to Remember
•Begin Audio Memorably
•Reinforce with Icons
Audio GOALS
Choose

58. Opportunity
Knocks4
Hand Held & Haptics

59. Phone In Hand = Detailed Tracking & Haptics
Challenge Phone and Face Look Different Places = Less Immersion

60. Maximize Rotation Motion Handheld
•Respond to Auxelswith Phone
•Make Hand Motion As Natural As Head Motion
•Haptics to enhance Audio Menu UX
Audio GOALS
Hold In
Hand

61. Feel Buttons As Well As Hear them
Haptics Offers Surface Textures For Audio

62. Spatial Haptics Amplify Audio Interaction & Position
We Need a PBR for Haptics

63. Spatialization Tips
•Audio Tracks Player’s Phone Hand
•Minimize looking left Head and
Moving Phone Right
•Button States with Haptic Signatures
•Haptic Surface and medium Move Phone
Through Like Water, Forest Branches, Sand
Audio GOALS
Hold In
Hand

64. Wrap Up5

65. 3+Challenges for Audio-First Games
StandWalkChooseHold & HapticsSpatial
Audio >

66. Rotation
Elevation
Distance
Audio Position -Auxel = RED
Auxel = R5 E3 D3
“Audio Pixel”
5
How many discrete audio positions can players hear?
3
3?
45 Positions 1-3 at a time

67. 3
AuxelsNeed Spacing
Position Time Memory
5 3
The Sound of Silence

68. Audio GOALS
Stand
Walk
Choose
HOLD
Spatialization Tips
1.Respond to head Pose and Movement
2.Space at Least 1 Auxel apart
3.Contrast Audio Pattern To Spatialize Well
4.Narrative Cues Expand Short Term Memory

69. Thank You! Unscramble the Oracle TEAM!
•David Fox writing & Sfx
Electric Eggplant.com
•Patrick O’ShaughnesseyNarr.Engine
Patched Reality.com
•WonkubusWonk Walk, VO. & Art
(Michael Crabbs)
MichaelCrabbs.com
•Nicole Lazzaro Game Design & UX
XEODesign.com

70. Sign Up for Updates Unscramble the Oracle
surveymonkey.com/r/UTOSignUp

71. Contact Me For More
•[email protected]•@NicoleLazzaro•XEODesign.com
Free Download
4K2F.com
Follow the
White Rabbit
TiltWorld.comUnscramble
the Oracle