Research Directions in Heads-Up Computing

Keynote speech given by Mark Billinghurst at the workshop on Heads Up Computing at the UbiComp 2024 conference. Given on October 5th 2024. The talk discusses some research directions in Heads-Up Computing.

1. RESEARCH DIRECTIONS IN
HEADS-UP COMPUTING
Mark Billinghurst
[email protected]
Workshop on Heads-up Computing
Ubicomp 2024
October 5
th 2024

2. 1967 – IBM 1401 – half of the computers in the world, $10,000/month to run

3. 2013 Google Glass

4. The Incredible Disappearing Computer
1960-70’s
Room
1970-80’s
Desk
1980-90’s
Lap
1990-2000’s
Hand
2010 -
Head

5. Heads-Up Computing
A new interaction paradigm that offers
seamless computing support for
humans' daily activities.
Zhao, S., Tan, F., & Fennedy, K. (2023). Heads-Up Computing
Moving Beyond the Device-CenteredParadigm.Communications
of the ACM,66(9), 56-63.

6. Moving from device centric interactions to human centric

8. Devices Becoming Available
•Socially acceptable form factor, all day use, enhancing everyday activities
Meta Ray-ban
Vuzix z100
Xreal Air 2

9. Meta Orion

10. Many Possible Research Directions
•Collaborative System
•Novel Interfaces
•Cross Reality Systems
•Hybrid Interfaces
•Prototyping Tools
•Social issues

11. COLLABORATIVE SYSTEMS

12. A wide variety of communication cues used.
Speech
Paralinguistic
Para-verbals
Prosodics
Intonation
Audio
Gaze
Gesture
Face Expression
Body Position
Visual
Object Manipulation
Writing/Drawing
Spatial Relationship
Object Presence
Environmental
Face to Face Communication

13. Smart Glass for Remote Collaboration
•Camera + Processing + AR Display + Connectivity
•First person Ego-Vision Collaboration

14. AR ViewRemote Expert View

15. Empathy Glasses
•Combine together eye-tracking, display, face expression
•Implicit cues – eye gaze, face expression
++
Pupil Labs Epson BT-200 AffectiveWear
Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. InProceedings of the
34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.

16. Remote Collaboration
•Eye gaze pointer and remote pointing
•Face expression display
•Implicit cues for remote collaboration

18. Shared Sphere – 360 Video Sharing
Shared
Live 360 Video
Host User Guest User
Lee, G. A., Teo, T., Kim, S., & Billinghurst, M. (2017). Mixed reality collaboration through sharing a
live panorama. InSIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications(pp. 1-4).

20. 3D Live Scene Capture
•Use cluster of RGBD sensors
•Fuse together 3D point cloud

21. Scene Capture and Sharing
Scene ReconstructionRemote ExpertLocal Worker

22. AR ViewRemote Expert View

23. View Sharing Evolution
•Increased immersion
•Improved scene understanding
•Better collaboration
2D 360 3D

24. NOVEL INTERFACES

25. Interface Challenge
•Need to support
•Natural user interaction
•Rich input expression
•In a way that
•Is socially acceptable
•Ergonomic, reliable
•Can be used all day
•Uses few devices

26. Avoiding Gorilla Arm
•Design interface to reduce mid-air gestures

27. Multi-Scale Gesture
•Combine different gesture types
•In-air gestures – natural but imprecise
•Micro-gesture – fine scale gestures
•Gross motion + fine tuning interaction
Ens, B., Quigley, A., Yeo, H. S., Irani, P., Piumsomboon, T., & Billinghurst, M. (2018). Counterpoint: Exploring Mixed-Scale Gesture Interaction
for AR Applications. InExtended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems(p. LBW120). ACM.

28. https://www.youtube.com/watch?v=TRfqNtt1VxY&t=23s

29. Gaze and Pinch Interaction
Pfeuffer, K., Mayer, B., Mardanbegi, D., & Gellersen, H. (2017, October). Gaze+ pinch interaction
in virtual reality. In Proceedings of the 5th symposium on spatial user interaction (pp. 99-108).

30. https://www.youtube.com/watch?v=NzLrZSF8aDM

31. Physiological Sensor Input
•Using physiological sensors for implicit input
•Systems that recognize user intent/activity
•EEG
•Measuring brain activity
•EMG
•Measuring muscle activity

32. https://www.youtube.com/watch?v=K34p7RwjWt0

33. Meta EMG Demo
•https://www.youtube.com/watch?v=WmxLiXAo9ko

34. Sensor Enhanced VR HMDs
Eye tracking, heart rate,
pupillometry, and face camera
HP Omnicept Project Galea
EEG, EEG, EMG, EDA, PPG,
EOG, eye gaze, etc.

35. Multiple Physiological Sensors into HMD
•Incorporate range of sensors on HMD faceplate and over head
•EMG – muscle movement
•EOG – Eye movement
•EEG – Brain activity
•EDA, PPG – Heart rate

37. •Measure physiological cues
•Brain activity
•Heart rate
•Eye gaze
•Show user state
•Cognitive load
•Attention
Showing Cognitive Load in Collaboration

38. Demo

39. CROSS REALITY SYSTEMS

40. Transitioning Along the RV Continuum
Reality VirtualityAugmented RealityAugmented Virtuality
Mixed Reality
Using the same device to experience Reality, AR and VR

41. Why Transition Along the RV Continuum ?
•Combining the best of AR/VR
•Augmented Reality is preferred for:
•augmenting real objects
•co-located collaboration
•Virtual Reality is preferred for:
•experiencing world immersively
•sharing views, remote collaboration

42. Collaboration Between Points on the RV Continuum
Reality VirtualityAugmented RealityAugmented Virtuality
Collaboration between people in reality, AR and VR
Mixed Reality

43. Why Collaboration Across the RV Continuum ?
•Enabling everyone to connect
•Shared experience, shared understanding
•Using the ideal interface for the task
•E.g. AR for interacting in real world, VR for providing remote help

44. The MagicBook (2001)
Reality VirtualityAugmented
Reality (AR)
Augmented
Virtuality (AV)
Billinghurst, M., Kato, H., & Poupyrev, I. (2001). The MagicBook: a transitional AR interface.Computers & Graphics,25(5), 745-753.

45. MagicBook Demo

46. Features
•Seamless transition from Reality to Virtuality
•Reliance on real decreases as virtual increases
•Supports egocentric and exocentric views
•User can pick appropriate view
•Independent Views
•Privacy, role division, scalability
•Collaboration on multiple levels:
•Physical Object, AR Object, Immersive Virtual Space
•Egocentric + exocentric collaboration
•multiple multi-scale users

47. Apple Vision Pro (2024)
•Transitioning from AR to VR
•Spatial Computing – interface seamlessly blending with real world

48. Cross Reality (CR)
An emerging technology focusing on the
concurrent usage of, or the transition between
multiple systemsat different points on the
reality-virtuality continuum (RVC).
Simeone, Adalberto L., Mohamed Khamis, Augusto Esteves, Florian Daiber, MatjažKljun, KlenČopičPucihar,
Poika Isokoski, and Jan Gugenheimer. "International workshop on cross-reality (xr) interaction." InCompanion
Proceedings of the 2020 Conference on Interactive Surfaces and Spaces, pp. 111-114. 2020.

49. HYBRID SYSTEMS

50. The Incredible Disappearing Computer
1960-70’s
Room
1970-80’s
Desk
1980-90’s
Lap
1990-2000’s
Hand
2010 -
Head

51. Computers Don’t Disappear
Room Desk Lap Hand Head
1960’s
1980’s
2010’s

52. Current Generation of Smart Glasses
•Trend towards lightweight thin displays
•Offload processing onto second device
•High bandwidth connectivity to cloud services

54. Hybrid Interfaces
AR HMDs tethered to handheld devices/phones
○New opportunity for interaction/display
○Wide field of view of HMD and Precise input of HHD

55. Motivation
Complementary nature of
HMDs/HHDs

56. Previous Work
Our previous work
•Use touch input on phone to interact with AR HMD content
•Use tablet to provide 2D view of 3D AR conferencing space
Bleeker 2013Budhiraja2013

58. Example: Secondsight
A prototyping cross-device interface
•Enables an AR HMD to "extend" the screen of a smartphone
Key Features
•Can simulate a range of HMD Field of View
•Enables World-fixed or Device-fixed content placement
•Supports touch screen input, free-hand gestures, head-pose selection
Reichherzer, C., Fraser, J., Rompapas, D. C., & Billinghurst, M. (2021, May). Secondsight: A framework for cross-device augmented
reality interfaces. InExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems(pp. 1-6).

59. Content Placement
© 2021 SIGGRAPH. All Rights Reserved.

60. Input

61. Map application
© 2021 SIGGRAPH. All Rights Reserved.

62. PROTOTYPING TOOLS

63. Challenges with Prototyping HUC Systems
•Cross platform support
•Need for programming skills
•Building collaborative systems
•Need to build multiple different interfaces
•Connecting multiple devices/components
•Difficult to prototype hardware/display systems

64. Second Sight Implementation
Hardware
•Meta2 AR Glasses (82
o FOV)
•Samsung Galaxy S8 phone
•OptiTrack motion capture
system
Software
•Unity game engine
•Mirror networking library

65. Google -The Cross-device Toolkit -XDTK
•Open-source toolkit to enable
communication between Android
devices and Unity
•Handles
•Device discovery/communication
•Sensor data streaming
•ARCorepose information
•https://github.com/google/xdtk
Gonzalez, E. J., Patel, K., Ahuja, K., & Gonzalez-Franco, M. (2024, March). XDTK: A Cross-Device Toolkit for Input & Interaction
in XR. In2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)(pp. 467-470). IEEE.

66. VRception Toolkit
•First multi-user and multi-environment rapid-prototyping toolkit for non-experts
•Designed for rapid prototyping of CR systems
•Supports prototyping in VR or in Unity3D
Gruenefeld, U., et al. (2022, April). Vrception: Rapid prototyping of cross-reality systems in virtual reality. InProceedings of the
2022 CHI Conference on Human Factors in Computing Systems(pp. 1-15).

67. https://www.youtube.com/watch?v=EWzP9_FAtL8

68. SOCIAL ISSUES

69. Social Acceptance
•People don’t want to look silly
•Only 12% of 4,600 adults would be willing to wear AR glasses
•20% of mobile AR browser users experience social issues
•Acceptance more due to Social than Technical issues
•Needs further study (ethnographic, field tests, longitudinal)

70. Ethical Issues
•Persuasive Technology
•Affecting emotions
•Behavior modification
•Privacy Concerns
•Facial recognition
•Space capture
•Personal data
•Safety Concerns
•Sim sickness, Distraction
•Long term effects
Pase, S. (2012). Ethical considerations in augmented reality applications. InProceedings of the International Conference on
e-Learning, e-Business, Enterprise Information Systems, and e-Government (EEE)(p. 1). The Steering Committee of The
World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp).

71. https://www.youtube.com/watch?v=tpOAiAAHDoA
Example: I-Xray Face Recognition

72. Diminished Reality

73. CONCLUSIONS

74. Summary
•Heads-Up Computing – new interaction paradigm
•Moving from device focused to human focused
•Important research areas
•Remote communication, Cross Reality, Novel Interface
•Prototyping Tools, Privacy and Security
•New research opportunities emerging
•XR + AI + Sensing + ??

76. www.empathiccomputing.org
@marknb00
[email protected]