The Metaverse: Are We There Yet?

Keynote talk by Mark Billinghurst at the 9th XR-Metaverse conference in Busan, South Korea. The talk was given on May 20th, 2024. It talks about progress on achieving the Metaverse vision laid out in Neil Stephenson's book, Snowcrash.

1. THE METAVERSE
Mark Billinghurst
[email protected]
May 20
th 2024
Are we there yet?

2. VR in 1994
$150,000+, SGI Onyx
Magnetic tracking
Virtual Research VR4
743x230 pixel/eye
60 degree FOV
$6,000
2 million polygons/sec
30 Hz

3. Greenspace (1994)
•First trans-Pacific shared VR
•4 people – 2 in Japan, 2 in USA
•$50K USD phone bill

4. Snowcrash (1992)
•Neal Stephenson’s coined Metaverse
•mentioned 117 times
“.. a computer-generated universe thathis
computer is drawing onto his goggles and
pumping into his earphones.. this imaginary
place is known as the Metaverse.”

5. Hiro Protagonist
“He is wearing shiny goggles that
wrap halfway around his head ..
He turns off his view of the Metaverse
entirely, making the goggles totally
transparent.”

6. Laser Scanned Retinal Display
•Laptop computer with R, G, B laser scanner
•72 fps, 2K pixel resolution on a side
“.. a narrow beam of any colour
can be shot out of the innards of
thecomputer, up through that
fisheye lens..
..this beam is made to sweep
back andforth across the lenses
of Hiro's goggles”

7. Virtual Retinal Scanner (1991 - )
•Laser scanning display
•Horizontal + vertical MEMs scanners
•Commercialized through Microvision (1993)
1993
2002

8. Snowcrash Metaverse
“The people are pieces of software
called avatars. They arethe audio-
visual bodies that people use to
communicate with each other..”
“.. Hiro goes into the Metaverse
and looks down the Street and
sees buildingsand electric signs
stretching off into the darkness..”

9. Todays Technology
See-through to VR transition3,660 x 3,142 pixel/eye
Hand Tracking
Social
VR
Realistic
Avatars

10. •asf
Are were there yet?

11. Seven Layers of the Metaverse (Jon Radoff)

12. Are We There Yet?
Mostly
There
Work
to Do

13. Would you wear this HMD?

14. Social Acceptance
•People don’t want to look silly
•Only 12% of 4,600 adults surveyed would be willing to wear AR glasses
•20% of mobile AR browser users experience social issues

15. Snowcrash Gargoyles
“It's a gargoyle..
Instead of using laptops, they wear their computers
on theirbodies, broken up into separate modules
that hang on the waist, .. onthe headset.
They serve as human surveillance devices,
recording everythingthat happens around them..
Gargoyles represent the embarrassing side of the
Central Intelligence Corporation.
Nothing looks stupider…”
Google Glass - 2013

16. Google Glass (2013)
•First consumer wearable computer
•Camera + processor + display on head
•Novel gesture/touch interaction and UI

17. Roblox (2006 - )
•Large scale Social 3D/VR platform
•70 million DAU, 300 million MAU
•~ 2 million people playing right now
•User Generated Content
•3 million developers
•Over 5.5 million environments
•Multi-platform
•Mobile, Tablet, PC/Mac, VR

18. Glass vs. Roblox
•Glass
•Technically amazing, but poor social experience
•Socially isolating - “What is he seeing on his screen?”
•Creates privacy concerns – “Is she filming me?”
•Expensive, little utility compared to phone/smart watch
•Roblox
•Content/social focus first
•Creates community, Free + Social
•User generated content/developer rewards
•Multi-platform, extends familiar user experience

19. Designing for the Whole User Experience

20. Avoiding Glass 2.0
•Design for the whole user
experience
•Creating new types of user
experiences
•Improving connection with
the real world
•Improving connection
between people

21. Creating a Better Experience
•Metaverse Roadmap (2007)
•Connecting Metaverse back to real world
•The Metaverse is the convergence of:
•1) virtually enhanced physical reality
•2) physically persistent virtual space

22. Metaverse Taxonomy
•Key Components
•Virtual Worlds
•Augmented Reality
•Mirror Worlds
•Lifelogging
Two Dimensions
Intimate <-> External
Augmentation <-> Simulation

23. Sensing
Immersing
Augmenting
Capturing

24. Expanded Research Opportunities

25. Example: Connecting between Spaces
•Augmented Reality
•Bringing remote people into your real space
•Virtual Reality
•Bringing elements of the real world into VR
•Mirror World
•Capturing real world in real time

26. 3D Live Scene Capture
•Use cluster of RGBD sensors
•Fuse together 3D point cloud

27. Scene Capture and Sharing
Scene ReconstructionRemote ExpertLocal Worker

28. AR ViewRemote Expert View

29. Empathic Computing
•Covering the entire Metaverse
•AR, VR, Lifelogging, Mirror Worlds
•Transforming collaboration
•Observer to participant
•Feeling of doing things together
•Supporting Implicit collaboration

30. NeuralDrum
•Using brain synchronicity to increase connection
•Collaborative VR drumming experience
•Measure brain activity using 3 EEG electrodes
•Use PLV to calculate synchronization
•More synchronization increases graphics effects/immersion
Pai, Y. S., Hajika, R., Gupta, K., Sasikumar, P., & Billinghurst, M. (2020). NeuralDrum: Perceiving Brain
Synchronicity in XR Drumming. InSIGGRAPH Asia 2020 Technical Communications(pp. 1-4).

31. Set Up
•HTC Vive HMD
•OpenBCI
•3 EEG electrodes

32. https://www.youtube.com/watch?v=aG261GfiR90

33. Results
"It’s quite interesting, I actually felt like my
body was exchanged with my partner."
Poor Player Good Player

34. Conclusion
•Many of the Metaverse technologies are here today
•Research need in some areas
•Acceptance more about social issues than technical
•Needs further study (ethnographic, privacy, ethics, etc)
•Need to expand definition of Metaverse
•Connect user experience back to real world
•Explore the entire Metaverse for opportunities
•Opportunities for research across the boundaries

35. Delivering the Entire Metaverse

37. www.empathiccomputing.org
@marknb00
[email protected]