Understanding the Impact of the Reality-Virtuality Continuum on Visual Search using Physiological Measures

Presentation at Mobile HCI of the paper "Understanding the Impact of the Reality-Virtuality Continuum on Visual Search using Physiological Measures" .

Chiossi, Francesco and Gruenefeld, Uwe and Hou, Baosheng James and Newn, Joshua and Ou, Changkun and Liao, Rulu and Welsch, Robin and Mayer, Sven

https://doi.org/10.1145/3676528

1. Francesco Chiossi , Uwe Gruenefeld, Baosheng James Hou, Joshua Newn, Changkun Ou, Rulu Liao, Robin Welsch and Sven Mayer Mobile HCI ’24, Melbourne Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking Features

2. 2 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Mixed Reality (MR) systems are increasingly integrated into daily activities, enhancing fields such as productivity, entertainment, and healthcare. The success of MR interactions depends on how well users can perceive, process, and understand visual information in these complex environments. Mixed Reality

3. 3 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Although MR systems offer significant potential, the cognitive load induced by these environments varies. We need to understand how different MR manifestations (AR, AV, VR) affect our visual and perceptual processing. Understanding Information Processing in Mixed Reality (I) Van den Oever , 2022

4. 4 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Kim, 2022 Understanding Information Processing in Mixed Reality (II) Although MR systems offer significant potential, the cognitive load induced by these environments varies. We need to understand how different MR manifestations (AR, AV, VR) affect our visual and perceptual processing.

5. 5 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search David-John, 2021 Understanding Information Processing in Mixed Reality (III) Although MR systems offer significant potential, the cognitive load induced by these environments varies. We need to understand how different MR manifestations (AR, AV, VR) affect our visual and perceptual processing.

6. 6 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search We identify a lack of research that systematically compares different actualities, such as AR, AV, and VR, when searching for relevant information

7. Do different MR manifestations impact performance and perceived workload differently? 7 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search RQ1 How does cognitive resource allocation in a visual search task vary across the continuum, as indexed by FRP ? RQ2 RQ3 Does the MR continuum impact eye tracking correlates of visual search efficiency and workload?

8. Do different MR manifestations impact performance and perceived workload differently? 8 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search RQ1 How does cognitive resource allocation in a visual search task vary across the continuum, as indexed by FRP ? RQ2 RQ3 Does the MR continuum impact eye tracking correlates of visual search efficiency and workload?

9. Do different MR manifestations impact performance and perceived workload differently? 9 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search RQ1 How does cognitive resource allocation in a visual search task vary across the continuum, as indexed by FRP ? RQ2 RQ3 Does the MR continuum impact eye tracking correlates of visual search efficiency and workload?

10. Do different MR manifestations impact performance and perceived workload differently? 10 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search RQ1 How does cognitive resource allocation in a visual search task vary across the continuum, as indexed by FRP ? RQ2 RQ3 Does the MR continuum impact eye tracking correlates of visual search efficiency ?

11. Do different MR manifestations impact performance and perceived workload differently? 11 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search RQ1 How does cognitive resource allocation in a visual search task vary across the continuum, as indexed by FRP ? RQ2 RQ3 Does the MR continuum impact eye tracking correlates of visual search efficiency and workload?

12. Experiment Design Within – Subjects Study Design ( N = 24 ) Manifestation (AR vs. AV. vs. VR) X Perceptual Load (Low vs. High) Task Visual Search Measures EEG (Fixation Related P3 amplitude), Eye Tracking Features, Accuracy, NASA-TLX 12 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search

13. Trial Structure 13 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search

14. 14 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Augmented Reality Low Perceptual Load

15. 15 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Augmented Reality High Perceptual Load

16. 16 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Augmented Virtuality Low Perceptual Load

17. 17 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Augmented Virtuality High Perceptual Load

18. 18 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Virtual Reality Low Perceptual Load

19. 19 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Virtual Reality High Perceptual Load

20. Experimental Conditions Participants performed a visual search task across the MR continuum ( AR , AV , and VR ) with two possible levels of Perceptual Load ( Low and High ) 20 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search

21. Behavioral Data Eye Tracking Features Fixation Related Potentials Subjective Scores 21 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Results

22. 22 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Behavioral Results Accuracy & Reaction Times Accuracy was higher under low perceptual load conditions across all MR environments, with no significant differences between AR, AV, and VR. Reaction times were significantly faster in AV and VR compared to AR , showing overall improvement under low perceptual load conditions.

23. 23 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Eye Tracking Results Fixation & Saccades & Pupil Size Fixation Duration was lower in AV & VR. Saccade frequency decreased under low perceptual load , indicating fewer saccades in less demanding tasks, while average pupil size was largest in AR and decreased in VR, with the most significant decrease observed under high perceptual load conditions

24. 24 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Fixation Related Potential FRP - P 3 Amplitude Fixation Onset Fixation Related- P3 amplitude is significantly higher in AV and VR environments than in AR , indicating that these settings require fewer processing resources. No effect were found for the Perceptual Load conditions.

25. 25 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Subjective Results (I) Likert Scales & NASA TLX Subjective reports reveal that AR was perceived as significantly more difficult, distracting, and overwhelming compared to AV and VR , particularly under high perceptual load . NASA TLX scores replicate results from Likert Scales.

26. Summary of Results (I) Impact of MR Environments (AR, AV, VR) AR : highest cognitive demand, with increased P3 amplitudes and larger pupil sizes, resulting in slower reaction times and less efficient visual search. AV & VR : Both environments required less cognitive demand than AR, with lower P3 amplitudes. They showed faster reaction times and more efficient visual search behaviors, particularly in VR. Participants perceived AV and VR as less demanding and distracting, with VR being rated the least demanding and most efficient. 26 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search

27. Summary of Results (II) Perceptual Load Perceptual load significantly affects task performance and cognitive load in MR environments (AR, AV, VR). Low perceptual associated with higher accuracy , faster reaction times, and reduced cognitive load, with tasks seen as less demanding. 27 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search

28. Discussion RQ1 : Impact of Manifestation on User Performance 28 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Accuracy : No significant differences between AR, AV, and VR, likely due to task duration (5 seconds) causing a Ceiling effect. Reaction Times : Participants were significantly faster in AV and VR compared to AR. Perceived Workload : AR was rated as more distracting and overwhelming, with a higher workload than AV and VR. Key Findings Implications for AR AR environments may introduce challenges in time-sensitive tasks, where delayed responses could have serious consequences. Designers should carefully balance virtual content to enhance performance without overwhelming users' cognitive resources. Context-Specific Recommendations In critical scenarios, prioritize user safety by minimizing distractions in AR. In less critical or remote tasks, blending more virtual elements to create an AV environment may improve performance.

29. 29 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Discussion RQ2 : Cognitive Resource Allocation per Manifestation P3 Amplitude Increased P3 amplitude in AV and VR compared to AR, indicating more efficient attentional processing in these environments. The richer textures and visual noise in AR increased cognitive demands, leading to delayed and diminished ERPs compared to AV and VR. Key Findings Cognitive resource allocation is significantly affected by the visual complexity of the environment. To optimize attentional resources, designers should reduce complexity in AR or introduce features like directional cues or contextual filtering to aid in information processing. Implications for MR Design

30. 30 Chiossi et al. | Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Discussion RQ3 : Eye Tracking Correlates of Visual Search Efficiency Fixation Durations : VR led to shorter fixation durations than AV and AR, i.e. more efficient visual processing in VR. Transparency Effect : AR with more transparent or see-through objects, increased attentional load due to decreased visual salience. Key Findings Perceptual Load did not significantly impact pupil size. Low perceptual load condition consistently led to faster visual processing, as evidenced by shorter fixation durations and reduced fixation counts and saccade frequency. Visual Search Efficiency