IVE 2024 Short Course - Lecture 2 - Fundamentals of Perception

Lecture 2 from the IVE 2024 Short Course on the Psychology of XR. This lecture covers some of the Fundamentals of Percetion and Psychology that relate to XR.

The lecture was given by Mark Billinghurst on July 15th 2024 at the University of South Australia.

1. Lecture 2: Fundamentals
The Psychology of XR
July 15
th – 19
th 2024

2. Virtual Reality (VR)
●Users immersed in Computer Generated environment
○HMD, gloves, 3D graphics, body tracking

3. The First VR Experience …
https://www.youtube.com/watch?v=pAC5SeNH8jw

4. Virtual Reality Definition
●Defining Characteristics
○Sense of Immersion
■User feels immersed in computer generated space
○Interactive in real-time
■The virtual content can be interacted with
○Independence
■User can have independent view and reaction to
environment

5. David Zeltzer’s AIP Cube
Autonomy – User can to react to events
and stimuli.
Interaction – User can interact with
objects and environment.
Presence – User feels immersed through
sensory input and output channels
Interaction
Autonomy
Presence
VR
Zeltzer, D. (1992). Autonomy, interaction, and presence. Presence: Teleoperators & Virtual Environments, 1(1),127-132.

6. Augmented Reality (AR)
•Virtual Images blended with the real world
•See-through HMD, handheld display, viewpoint tracking, etc..

7. Augmented Reality Definition
●Defining Characteristics [Azuma 97]
○Combines Real and Virtual Images
■Both can be seen at the same time
○Interactive in real-time
■The virtual content can be interacted with
○Registered in 3D
■Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

9. Milgram’s Mixed Reality (MR) Continuum
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, (1994) A Taxonomy of Mixed Reality Visual Displays
Internet of Things

10. Apple Vision Pro (2024)
●Transitioning from AR to VR
●Spatial Computing – interface seamlessly blending with real world

11. https://www.youtube.com/watch?v=oP6CrLcMKO4

12. Extended Reality (XR)
Augmented Reality Virtual Reality
Real World Virtual World
Mixed Reality
Extended Reality
Internet of Things

13. Goal of Virtual Reality
“.. to make it feel like you’re actually
in a place that you are not.”
Palmer Luckey
Co-founder, Oculus

14. Creating a Good VR Experience
●Creating a good experience requires multisensory input
○Integrating multiple perceptual cues

15. Example: Shard VR Slide
●Ride down the Shard at 100 mph - Multi-sensory VR
https://www.youtube.com/watch?v=HNXYoEdBtoU

16. Creating Illusions
●Virtual Reality
○You’re immersed in a place
●Augmented Reality
○Virtual content is in your place
●Mixed Reality
○Seamless moving from RW in VR

17. Perception

18. What is Reality?

19. How do We Perceive Reality?
●We understand the world through our senses:
○Sight, Hearing, Touch, Taste, Smell (and others..)
●Two basic processes:
○Sensation – Gathering information
○Perception – Interpreting information

20. Motivation
●Understand: In order to create a strong sense of Presence
we need to understand the Human Perception system
●Stimulate: We need to be able to use technology to provide
real world sensory inputs, and create the VR illusion
VR Hardware Human Senses

21. Senses
●How an organism obtains information for perception:
○Sensation part of Somatic Division of Peripheral Nervous System
○Integration and perception requires the Central Nervous System
●Five major senses (but there are more..):
○Sight (Opthalamoception)
○Hearing (Audioception)
○Taste (Gustaoception)
○Smell (Olfacaoception)
○Touch (Tactioception)

22. Relative Importance of Each Sense
●Percentage of neurons in
brain devoted to each
sense
○Sight – 30%
○Touch – 8%
○Hearing – 2%
○Smell - < 1%
●Over 60% of brain
involved with vision in
some way

23. Other Lessor Known Senses..
●Proprioception = sense of body position
○what is your body doing right now
●Equilibrium = balance
●Acceleration
●Nociception = sense of pain
●Temperature
●Satiety = state of being fed or gratified to or beyond capacity
●Thirst
●Micturition = amount of CO2 and Na in blood

24. Sight

25. The Human Visual System
●Purpose is to convert visual input to signals in the brain

26. The Human Eye
●Light passes through cornea and lens onto retina
●Photoreceptors in retina convert light into electrochemical signals

27. Photoreceptors – Rods and Cones
●Retina photoreceptors come in two types, Rods and Cones
○Rods – 125 million, periphery of retina, no colour detection, night vision
○Cones – 4-6 million, center of retina, colour vision, day vision

28. Human Horizontal and Vertical FOV
●Humans can see ~135
o
vertical (60
o
above, 75
o
below)
●See up to ~ 210
o
horizontal FOV, ~ 115
o
stereo overlap
●Colour/stereo in centre, black and white/mono in periphery

29. Vergence + Accommodation
●saas

30. https://www.youtube.com/watch?v=p_xLO7yxgOk

31. Visual Acuity
Visual Acuity Test Targets
●Ability to resolve details
●Several types of visual acuity
○detection, separation, etc
●Normal eyesight can see a 50 cent coin at 80m
○Corresponds to 1 arc min (1/60
th of a degree)
○Max acuity = 0.4 arc min

32. Stereo Perception/Stereopsis
●Eyes separated by IPD
○Inter pupillary distance
○5 – 7.5cm (avge. 6.5cm)
●Each eye sees diff. image
○Separated by image parallax
●Images fused to create 3D stereo view

34. Depth Perception
●The visual system uses a range of different
Stereoscopic and Monocular cues for depth
perception
Stereoscopic Monocular
eye convergence angle
disparity between left
and right images
diplopia
eye accommodation
perspective
atmospheric artifacts (fog)
relative sizes
image blur
occlusion
motion parallax
shadows
texture
Parallax can be more important for depth perception!
Stereoscopy is important for size and distance evaluation

35. Common Depth Cues

36. Depth Perception Distances
●i.e. convergence/accommodation used for depth perception < 10m

37. Properties of the Human Visual System
●Visual acuity: 20/20 is ~1 arc min
●Field of view: ~200° monocular, ~120° binocular, ~135° vertical
●Resolution of eye: ~576 megapixels
●Temporal resolution: ~60 Hz (depends on contrast, luminance)
●Dynamic range: instantaneous 6.5 f-stops, adapt to 46.5 f-stops
●Colour: everything in CIE xy diagram
●Depth cues in 3D displays: vergence, focus, (dis)comfort
●Accommodation range: ~8cm to ∞, degrades with age

38. Creating the Perfect Illusion
Cuervo, E., Chintalapudi, K., & Kotaru, M. (2018, February). Creating
the perfect illusion: What will it take to create life-like virtual reality
headsets?. InProceedings of the 19th International Workshop on
Mobile Computing Systems & Applications(pp. 7-12).
●Technology to create life-like VR HMDs
●Compared to current HMDs
○2 − 10× higher pixel density
○20 − 30× higher frame rate

39. Comparison between Eyes and HMD

40. When Will We Achieve Life-like VR Displays?
●Could achieve visual fidelity by 2025
○BUT:
■GPUs not fast enough for high framerate (140 Tflops by 2025, need 10x for 1800Hz)
■Wireless life like VR requires 2.7Tbps, c.f. wireless HD standard of 25Gbps
Display FR and Pixels/degree

41. Hearing

42. Anatomy of the Ear

43. Auditory Thresholds
●Humans hear frequencies from 20 – 22,000 Hz
●Most everyday sounds from 80 – 90 dB

44. Sound Localization
●Humans have two ears
○localize sound in space
●Sound can be localized
using 3 coordinates
○Azimuth, elevation, distance

45. Sound Localization
https://www.youtube.com/watch?v=FIU1bNSlbxk

46. Sound Localization (Azimuth Cues)
Interaural Time Difference

47. HRTF (Elevation Cue)
●Pinna and head shape affect frequency intensities
●Sound intensities measured with microphones in ear
and compared to intensities at sound source
○Difference is HRTF, gives clue as to sound source location

48. Accuracy of Sound Localization
●People can locate sound
○Most accurately in front of them
■2-3° error in front of head
○Least accurately to sides and behind head
■Up to 20° error to side of head
■Largest errors occur above/below elevations and behind head
●Front/back confusion is an issue
○Up to 10% of sounds presented in the front are perceived
coming from behind and vice versa (more in headphones)
BUTEAN, A., Bălan, O., NEGOI, I., Moldoveanu, F., & Moldoveanu, A. (2015). COMPARATIVE RESEARCH ON SOUND
LOCALIZATION ACCURACY IN THE FREE-FIELD AND VIRTUAL AUDITORY DISPLAYS. InConferenceproceedings of»
eLearning and Software for Education «(eLSE)(No. 01, pp. 540-548). UniversitateaNationalade AparareCarol I.

49. Touch

50. Haptic Sensation
●Somatosensory System
○complex system of nerve cells that responds to
changes to the surface or internal state of the body
●Skin is the largest organ
○1.3-1.7 square m in adults
●Tactile: Surface properties
○Receptors not evenly spread
○Most densely populated area is the tongue
●Kinesthetic: Muscles, Tendons, etc.
○Also known as proprioception

51. Cutaneous System
●Skin – heaviest organ in the body
○Epidermis outer layer, dead skin cells
○Dermis inner layer, with four kinds of mechanoreceptors

52. Mechanoreceptors
●Cells that respond to pressure, stretching, and vibration
○Slow Acting (SA), Rapidly Acting (RA)
○Type I at surface – light discriminate touch
○Type II deep in dermis – heavy and continuous touch
Receptor TypeRate of
Acting
Stimulus
Frequency
Receptive FieldDetection Function
Merkel DiscsSA-I 0 – 10 Hz Small, well definedEdges, intensity
Ruffini
corpuscles
SA-II 0 – 10 Hz Large, indistinctStatic force,
skin stretch
Meissner
corpuscles
RA-I 20 – 50 Hz Small, well definedVelocity, edges
Pacinian
corpuscles
RA-II 100 – 300 HzLarge, indistinctAcceleration,
vibration

53. Spatial Resolution
●Sensitivity varies greatly
○Two-point discrimination
Body
Site
Threshold
Distance
Finger 2-3mm
Cheek 6mm
Nose 7mm
Palm 10mm
Forehead 15mm
Foot 20mm
Belly 30mm
Forearm 35mm
Upper Arm 39mm
Back 39mm
Shoulder 41mm
Thigh 42mm
Calf 45mm
http://faculty.washington.edu/chudler/chsense.html

54. Proprioception/Kinaesthesia
●Proprioception (joint position sense)
○Awareness of movement and positions of body parts
■Due to nerve endings and Pacinian and Ruffini corpuscles at joints
○Enables us to touch nose with eyes closed
○Joints closer to body more accurately sensed
○Users know hand position accurate to 8cm without looking at them
●Kinaesthesia (joint movement sense)
○Sensing muscle contraction or stretching
■Cutaneous mechanoreceptors measuring skin stretching
○Helps with force sensation

55. Information Processing

56. Simple Perception Action Model
Wickens, C. D., & Carswell, C. M. (2021). Information processing.Handbook of human factors and ergonomics, 114-158.
Open Loop
Closed Loop

57. Simple Sensing/Perception Model

58. Human Information Processing Model
Wickens, C. D. (1992), Engineering Psychology and HumanPerformance, 2nd ed., HarperCollins, New York.

59. Creating the Illusion of Reality
●Fooling human perception by using
technology to generate artificial sensations
○Computer generated sights, sounds, smell, etc

60. Reality vs. Virtual Reality
●In a VR system there are input and output devices
between human perception and action

61. Using Technology to Stimulate Senses
●Simulate output
○E.g. simulate real scene
●Map output to devices
○Graphics to HMD
●Use devices to
stimulate the senses
○HMD stimulates eyes
Visual
Simulation
3D GraphicsHMD Vision
System
Brain
Example: Visual Simulation
Human-Machine Interface

62. Example Birdly - http://www.somniacs.co/
●Create illusion of flying like a bird
●Multisensory VR experience
○Visual, audio, wind, haptic

63. https://www.youtube.com/watch?v=gHE6H62GHoM

65. HMD Basic Principles
●Use display with optics to create illusion of virtual screen

66. Simple Magnifier HMD Design
p
q
Eyepiece
(one or more lenses)Display
(Image Source)
Eye f
Virtual
Image
1/p + 1/q = 1/f where
p = object distance (distance from image source to eyepiece)
q = image distance (distance of image from the lens)
f = focal length of the lens

67. Vergence-Accommodation Conflict
●Looking at real objects, vergence and focal distance match
●In VR, vergence and accommodation can miss-match
○Focusing on HMD screen, but accommodating for virtual object behind screen

68. AR Vergence and Accommodation
●Fixed focal distance for OST displays
●Accommodation conflict between real and virtual object

69. AR – Focal Rivalry
●Optical see-through AR displays with fixed focal length
○E.g. Hololens focal length ~2m
●When real objects < focal length, can’t keep virtual object in focus
○Either real or virtual become blurry

70. Example
●People made errors twice as large on connect the dots task in AR vs. real world
○Connect virtual numbers either – without AR 0.9 mm average error, using AR 2.3 mm error
Focus on Ruler Focus on Virtual Image
Condino, S., Carbone, M., Piazza, R., Ferrari, M., & Ferrari, V. (2019). Perceptual limits of optical see-through visors for
augmented reality guidance of manual tasks.IEEE Transactions on Biomedical Engineering,67(2), 411-419.

71. Using Multiple Image Planes

72. MagicLeap Display
●Optical see through AR display
○Overlay graphics directly on real world
○40
o x 30
o FOV, 1280 x 960 pixels/eye
●Waveguide based display
○Holographic optical element
○Very thin physical display
●Two sets of waveguides
○Different focal planes
■Overcomes vergence/accommodation problem
○Eye tracking for selecting focal plane
●Separate CPU/GPU unit

74. Distortion in Lens Optics
A rectangle Maps to this

75. HTC Vive Optics

76. To Correct for Distortion
●Must pre-distort image
●This is a pixel-based
distortion
●Use shader
programming

77. VR Distorted Image

78. Interpupillary Distance (IPD)
nHorizontal distance between
a user's eyes
nDistance between the two
optical axes in a HMD
nTypical IPD ~ 63mm

79. Field of View
Monocular FOV is the angular
subtense of the displayed image as
measured from the pupil of one eye.
Total FOV is the total angular size of the
displayed image visible to both eyes.
Binocular(or stereoscopic) FOV refers to the
part of the displayed image visible to both eyes.
FOV may be measured horizontally,
vertically or diagonally.

80. Typical VR HMD FOV

81. Foveated Displays
●Combine high resolution center
with low resolution periphery

82. Varjo Display
Varjo resolution
Non-Varjo resolution
Focus area (27°x 27°)
70 PPD, 1920 x 1920px
115°FOV
30 PPD
2880 x 2720px
●1 LCD (wide FOV)
●1 uOLEDpanel (centre)

83. Varjo XR-3 Demo – Threading a Needle
https://www.youtube.com/watch?v=5iEwlOEUQjI

84. Perception Based Graphics
●Eye Physiology
○Rods in eye centre = colour vision, cones in periphery = motion, B+W
●Foveated Rendering
○Use eye tracking to draw highest resolution where user looking
○Reduces graphics throughput

85. Foveated Rendering
●https://www.youtube.com/watch?v=lNX0wCdD2LA

86. Typical VR Simulation Loop
●User moves head, scene updates, displayed graphics change

87. ●Need to synchronize system to reduce delays
System Delays

88. Typical System Delays
●Total Delay = 50 + 2 + 33 + 17 = 102 ms
○1 ms delay = 1/3 mm error for object drawn at arms length
○So total of 33mm error from when user begins moving to when object drawn
TrackingCalculate
Viewpoint
Simulation
Render
Scene
Draw to
Display
x,y,z
r,p,y
Application Loop
20 Hz = 50ms500 Hz = 2ms30 Hz = 3ms60 Hz = 17ms

89. Effects of System Latency
●Degraded Visual Acuity
○Scene still moving when head stops = motion blur
●Degraded Performance
○As latency increases it’s difficult to select objects etc.
○If latency > 120 ms, training doesn’t improve performance
●Breaks-in-Presence
○If system delay high user doesn’t believe they are in VR
●Negative Training Effects
○User train to operative in world with delay
●Simulator Sickness
○Latency is greatest cause of simulator sickness

90. Simulator Sickness
●Visual input conflicting with vestibular system

91. What Happens When Senses Don’t Match?
●20-30% VR users experience motion sickness
●Sensory Conflict Theory
○Visual cues don’t match vestibular cues
■Eyes – “I’m moving!”, Vestibular – “No, you’re not!”

92. Avoiding Motion Sickness
●Better VR experience design
○More natural movements
●Improved VR system performance
○Less tracking latency, better graphics frame rate
●Provide a fixed frame of reference
○Ground plane, vehicle window
●Add a virtual nose
○Provide peripheral cue
●Eat ginger
○Reduces upset stomach

93. Many Causes of Simulator Sickness
●25-40% of VR users get Simulator Sickness, due to:
●Latency
○Major cause of simulator sickness
●Tracking accuracy/precision
○Seeing world from incorrect position, viewpoint drift
●Field of View
○Wide field of view creates more periphery vection = sickness
●Refresh Rate/Flicker
○Flicker/low refresh rate creates eye fatigue
●Vergence/Accommodation Conflict
○Creates eye strain over time
●Eye separation
○If IPD not matching to inter-image distance then discomfort

94. Motion Sickness
●https://www.youtube.com/watch?v=BznbIlW8iqE

95. Applying HIP to XR Design

96. Human Information Processing Model
●High level staged model from Wickens and Carswell (1997)
○Relates perception, cognition, and physical ergonomics
Perception Cognition Ergonomics

97. Design for Perception
●Need to understand perception to design AR/VR
●Visual perception
○Many types of visual cues (stereo, oculomotor, etc.)
●Auditory system
○Binaural cues, vestibular cues
●Somatosensory
○Haptic, tactile, kinesthetic, proprioceptive cues
●Chemical Sensing System
○Taste and smell

98. Depth Perception Problems
●Without proper depth cues AR interfaces look unreal

99. Which of these POI are near or
far?

100. Types of Depth Cues

101. Improving Depth Perception
Cutaways
Occlusion
Shadows

102. Cutaway Example
●Providing depth perception cues for AR
https://www.youtube.com/watch?v=2mXRO48w_E4

103. Design for Cognition
●Design for Working and Long-term memory
○Working memory
■Short term storage, Limited storage (~5-9 items)
○Long term memory
■Memory recall trigger by associative cues
●Situational Awareness
○Model of current state of user’s environment
■Used for wayfinding, object interaction, spatial awareness, etc..
○P
■Landmarks, procedural cues, map knowledge
■Support both ego-centric and exo-centric views

104. Micro-Interactions
▪Using mobile phones people split their attention
between the display and the real world

105. Time Looking at Screen
Oulasvirta, A. (2005). The fragmentation of attention in mobile
interaction, and what to do with it.interactions,12(6), 16-18.

106. Dividing Attention to World
●Number of times looking away from mobile screen

107. Design for Micro Interactions
▪Design interaction for less than a few seconds
○Tiny bursts of interaction
○One task per interaction
○One input per interaction
▪Benefits
○Use limited input
○Minimize interruptions
○Reduce attention fragmentation

108. NHTSA Guidelines - www.nhtsa.gov
For technology in cars:
•Any task by a driver should be interruptible at any time.
•The driver should control the pace of task interactions.
•Tasks should be completed with glances away from road <2 seconds
•Cumulative time glancing away from the road <=12 secs.

109. Make it Glanceable
●Seek to rigorously reduce information density. Successful designs afford for
recognition, not reading.
Bad Good

110. Reduce Information Chunks
You are designing for recognition, not reading. Reducing the total # of information chunks will greatly
increase the glanceability of your design.
1
2
3
1
2
3
4
5
(6)
Eye movements
For 1: 1-2 460ms
For 2: 1 230ms
For 3: 1 230ms
~920ms
Eye movements
For 1: 1230ms
For 2: 1230ms
For 3: 1230ms
For 4: 3690ms
For 5: 2460ms
~1,840ms

111. Ego-centric and Exo-centric views
●Combining ego-centric and exo-centric cue for better situational awareness

112. Cognitive Issues in Mobile AR
●Information Presentation
○Amount, Representation, Placement, View combination
●Physical Interaction
○Navigation, Direct manipulation, Content creation
●Shared Experience
○Social context, Bodily Configuration, Artifact manipulation, Display space
Li, N., & Duh, H. B. L. (2013). Cognitive issues in mobile augmented reality: an embodied perspective.
InHuman factors in augmented reality environments(pp. 109-135). Springer, New York, NY.

113. Information Presentation
•Consider
•The amount of information
•Clutter, complexity
•The representation of information
•Navigation cues, POI representation
•The placement of information
•Head, body, world stabilized
•Using view combinations
•Multiple views

114. Example: Twitter 360
●iPhone application
●See geo-located tweets in real world
●Twitter.com supports geo tagging

115. But: Information Clutter from Many Tweets
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah
Blah

116. Solution: Information Filtering

117. Information Filtering
Before After

118. Outdoor AR: Limited FOV

119. ●Show POI outside FOV
●Zooms between map and panorama views
Zooming Views

120. https://www.youtube.com/watch?v=JLxLH9Cya20

121. Design for Physical Ergonomics
●Design for the human motion range
○Consider human comfort and natural posture
●Design for hand input
○Coarse and fine scale motions, gripping and grasping
○Avoid “Gorilla arm syndrome” from holding arm pose

122. Gorilla Arm in AR
●Design interface to reduce mid-air gestures

123. XRgonomics
●Uses physiological model to calculate ergonomic interaction cost
○Difficulty of reaching points around the user
○Customizable for different users
○Programmable API, Hololens demonstrator
●GitHub Repository
○https://github.com/joaobelo92/xrgonomics
Evangelista Belo, J. M., Feit, A. M., Feuchtner, T., & Grønbæk, K. (2021, May). XRgonomics: Facilitating the Creation of
Ergonomic 3D Interfaces. InProceedings of the 2021 CHI Conference on Human Factors in Computing Systems(pp. 1-11).

124. XRgonomics
https://www.youtube.com/watch?v=cQW9jfVXf4g

125. System Design Guidelines - I
●Hardware
○Choose HMDs with fast pixel response time, no flicker
○Choose trackers with high update rates, accurate, no drift
○Choose HMDs that are lightweight, comfortable to wear
○Use hand controllers with no line-of-sight requirements
●System Calibration
○Have virtual FOV match actual FOV of HMD
○Measure and set users IPD
●Latency Reduction
○Minimize overall end to end system delay
○Use displays with fast response time and low persistence
○Use latency compensation to reduce perceived latency
Jason Jerald, The VR Book, 2016

126. System Design Guidelines - II
●General Design
○Design for short user experiences
○Minimize visual stimuli closer to eye (vergence/accommodation)
○For binocular displays, do not use 2D overlays/HUDs
○Design for sitting, or provide physical barriers
○Show virtual warning when user reaches end of tracking area
●Motion Design
○Move virtual viewpoint with actual motion of the user
○If latency high, no tasks requiring fast head motion
●Interface Design
○Design input/interaction for user’s hands at their sides
○Design interactions to be non-repetitive to reduce strain injuries
Jason Jerald, The VR Book, 2016

127. Questions?
[email protected]