Introduction to Augmented Reality

AR101 Lecture - Introduction to Augmented Reality. Lecture providing an introduction to AR, the history of AR and some example applications. Presented by Mark Billinghurst at the AR101 summer school at the ISMAR 2016 conference, September 18th 2016.

1. INTRODUCTION TO
AUGMENTED REALITY:
HISTORICAL PERSPECTIVES
AND APPLICATIONS
Mark Billinghurst
First Augmented Reality School AR101
September 18
th
2016
Merida, Mexico

2. • adsf
We enjoy connecting to
the world around us..

3. • jg
And to each
other ..

4. The Digital Divide
• Screens are windows into digital spaces
• Separation between digital and physical world
• Separation between people

5. Making Interfaces Invisible
Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented
interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on
User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.

6. Graphical User Interfaces
• Separation between real and digital worlds
• WIMP (Windows, Icons, Menus, Pointer) metaphor

7. Ubiquitous Computing
• Computing and sensing embedded in real world
• Particle devices, RFID, motes, arduino, etc

8. Virtual Reality
• 1985…

9. Virtual Reality
• Immersive VR
• Head mounted display, gloves
• Separation from the real world

10. Augmented Reality
• Star Wars - 1977

11. Augmented Reality Definition
• Defining Characteristics [Azuma 97]
• Combines Real and Virtual Images
• Both can be seen at the same time
• Interactive in real-time
• The virtual content can be interacted with
• Registered in 3D
• Virtual objects appear fixed in space
Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.

12. 2008 - CNN
https://www.youtube.com/watch?v=thOxW19vsTg

13. • Put AR pictures here
Augmented Reality Examples

14. AR vs VR

15. Where Can You Use AR/VR?

16. Making Interfaces Invisible
Rekimoto, J. and Nagao, K. 1995. The world through the computer: computer augmented
interaction with real world environments. In Proceedings of the 8th Annual ACM Symposium on
User interface and Software Technology. UIST '95. ACM, New York, NY, 29-36.

17. From Reality to Virtual Reality
Ubiquitous Computing Augmented Reality Virtual Reality

18. Milgram’s Reality-Virtuality continuum
Mixed Reality
Reality - Virtuality (RV) Continuum
Real
Environment
Augmented
Reality (AR)
Augmented
Virtuality (AV)
Virtual
Environment
"...anywhere between the extrema of the virtuality continuum."
P. Milgram and A. F. Kishino, Taxonomy of Mixed Reality Visual Displays
IEICE Transactions on Information and Systems, E77-D(12), pp. 1321-1329, 1994.

19. Augmented Virtuality
• VR with windows into the real world

20. Ubiquitous Computing Continuum
• Weiser Continuum

21. Milgram – Weiser Continuum
• Newman, J., Bornik, A., Pustka, D., Echtler, F., Huber, M., Schmalstieg, D., & Klinker, G. (2007,
March). Tracking for distributed mixed reality environments. In Workshop on Trends and Issues
in Tracking for Virtual Environments at the IEEE Virtual Reality Conference (VR’07).

22. Metaverse
• Neal Stephenson’s “SnowCrash”
• The Metaverse is the convergence of:
• 1) virtually enhanced physical reality
• 2) physically persistent virtual space
• Metaverse Roadmap
• http://metaverseroadmap.org/

23. Metaverse Dimensions
• Augmentation technologies that layer information
onto our perception of the physical environment.
• Simulation technologies that model reality
• Intimate technologies are focused inwardly, on the
identity and actions of the individual or object;
• External technologies are focused outwardly,
towards the world at large;

24. Metaverse Components
• Four Key Components
• Virtual Worlds
• Augmented Reality
• Mirror Worlds
• Lifelogging

26. Mirror Worlds
• Mirror worlds are informationally-enhanced virtual
models of the physical world.
• Google Earth, MS Street View, Google Maps

27. My House – May 2013

28. LifeLogging
• Technologies record and report the intimate
states and life histories of objects and users
• FitBit, Microsoft Band, Shine, etc

29. Gordon Bell: LifeLogging
1 TB to store 65 years of data

30. Narrative Clip
• Wearable camera
• Automatic picture capture - 2 pics/minute
• http://getnarrative.com

31. Summary
• AR has three key features
• Combines Real and Virtual Images
• Interactive in real-time
• Registered in 3D
• AR can be classified with other technologies
• Invisible Interfaces
• Milgram’s Mixed Reality continuum
• Milgram-Weiser continuum
• MetaVerse

32. HISTORY OF AR

34. Pepper’s Ghost (1862)
• Dates back to Giambattista della Porta (1584)

35. The Master Key (1901) – AR Glass
• "It consists of this pair of spectacles.
While you wear them every one you
meet will be marked upon the
forehead with a letter indicating his or
her character. The good will bear the
letter 'G,' the evil the letter 'E.' …
Thus you may determine by a single
look the true natures of all those you
encounter.”
L. Frank Baum

36. Early HUD (1958)
F16 – Head Up Display

37. Early HMD Patents

38. First HMD
• Philco Headsight (1961) – Remote Camera Viewing

39. Sutherland HMD
• 1968: Sutherland / Sproull’s
first HMD system
• see-through

40. Demo – Sutherland HMD (1968)
• https://www.youtube.com/watch?v=NtwZXGprxag

41. US Airforce HMDs
1960 - 70’s: US Air Force HMDs (T. Furness III)

42. SuperCockpit Program
1970 - 80’s: US Air Force Super Cockpit (T. Furness III)

43. Modern Airforce HMDs
• Honeywell Integrated Helmet and Display Sighting System
(IHADSS) on AH-64 Apache attack helicopter in 1985

44. F35- HMD Demo (2014)
• https://www.youtube.com/watch?v=Ay6g66FbkmQ

45. First Industrial Use
• Early 1990’s: Boeing coined the term “AR.” Wire harness
assembly application begun (T. Caudell, D. Mizell).

46. Academic Research Beginning
• 1994: Motion stabilized display [Azuma]
• 1995: Fiducial tracking in video [Bajura / Neumann]
• 1996: UNC hybrid magnetic-vision tracker

47. Development of the Field
• 1996: MIT Wearable Computing efforts
• 1998: Dedicated conferences begin (ISMAR)
• Late 90’s: Collaboration, outdoor, interaction
• Late 90’s: Augmented sports broadcasts

48. Development of Tools
• 1996 CyberCode (Rekimoto)
• First matrix code tracking
• 1999 ARToolKit (Kato & Billinghurst)
• Open source tracking library

49. Tracking Demos
https://www.youtube.com/watch?v=TqGAqAFlGg0
ARToolKit
Cybercode

50. Mobile/Wearable Systems
• 1995 Navicam (Rekimoto)
• Handheld AR
• 1997 Touring Machine (Feiner)
• Backpack AR
• GPS, see-through display

51. History Summary
• 1960’s – 80’s: Early Experimentation
• 1980’s – 90’s: Basic Research
• Tracking, displays
• 1995 – 2005: Tools/Applications
• Interaction, usability, theory
• 2005 - : Commercial Applications
• Games, Medical, Industry

52. 2007 - AR Reaches Mainstream
• MIT Technology Review
• March 2007
• list of the 10 most exciting
technologies
• Economist
• Dec 6
th
2007
• Reality, only better

53. 2009 - AR in Magazines
• Esquire Magazine
• Dec 2009 issue
• 12 pages AR content
• Many Others
• Wired
• Colors
• Red Bull
• Etc

54. Esquire Demo
https://www.youtube.com/watch?v=LGwHQwgBzSI

55. Google Searches for AR
• Two factors drive rise in AR popularity in 2009
• Browser based AR, Mobile phone AR

56. 2008 - Browser Based AR
• Flash + camera + 3D graphics
• ARToolKit ported to Flash
• High impact
• High marketing value
• Large potential install base
• 1.6 Billion web users
• Ease of development
• Lots of developers, mature tools
• Low cost of entry
• Browser, web camera

57. Demo: GE SmartGrid Example
• https://www.youtube.com/watch?v=vJO_AZkCL9U

58. 2005 - Mobile Phone AR
• Mobile Phones
• camera
• processor
• display
• AR on Mobile Phones
• Simple graphics
• Optimized computer vision
• Collaborative Interaction

59. AR Advertising (HIT Lab NZ 2007)
• Txt message to download AR application (200K)
• See virtual content popping out of real paper advert
• Tested May 2007 by Saatchi and Saatchi

60. 2008: Location Aware Phones
Nokia Navigator
Motorola Droid

61. 2009 - Outdoor Information Overlay
• Mobile phone based
• Tag real world locations
• GPS + Compass input
• Overlay graphics on live video
• Applications
• Travel guide, Advertising, etc
• Wikitude, Layar, etc..
• iOS/Android, Public API released

62. Layar Demo (2009)
• https://www.youtube.com/watch?v=b64_16K2e08

63. Google Glass (2011 - )

64. Epson Moverio BT-200
▪ Stereo see-through display ($700)
▪ 960 x 540 pixels, 23 degree FOV, 60Hz, 88g
▪ Android Powered, separate controller
▪ VGA camera, GPS, gyro, accelerometer

65. Hololens (2016)
• Integrated system – Windows
• Stereo see-through display
• Depth sensing tracking
• Voice and gesture interaction

66. View Through Hololens
• https://www.youtube.com/watch?v=RddvMLwT__g

67. • Weak AR
• Imprecise tracking
• No knowledge of environment
• Limited interactivity
• Handheld AR
• Strong AR
• Very accurate tracking
• Seamless integration into real world
• Natural interaction
• Head mounted AR
Strong vs. Weak AR

68. Augmented Reality Today
• Key Technologies Available
• Robust tracking (Computer Vision, GPS/sensors)
• Display (Handheld, HMDs)
• Input Devices (Kinect, etc)
• Developer tools (Vuforia, ARToolKit)
• Commercial Business Growing
• Gaming, GPS/Mobile, Online Advertisement
• >$2 Billion USD in 2016

69. AR Business Today
• Marketing
• Web-based, mobile
• Mobile AR
• Geo-located information and service
• Driving demand for high end phones
• Gaming
• Mobile, Physical input (Kinect, PS Move)
• Upcoming areas
• Manufacturing, Medical, Military

70. AR Business Today
• Around $600 Million USD in 2014 (>$2B 2016)
• 70-80+% Games and Marketing

71. AR Commercial Landscape

72. Crossing the Chasm - 5-10 years
http://www.gartner.com/newsroom/id/3114217

73. Market Projections
cf. 2014 computer game market = $84 Billion USD

74. Forecast Number of Users

75. Pokemon GO
Killer Combo: brand + social + mobile + geo-location + AR

76. Pokemon GO Effect
• Fastest App to reach $500 million in Revenue
• Only 63 days after launch, > $1 Billion in 6 months
• Over 500 million downloads, > 25 million DAU
• Nintendo stock price up by 50% (gain of $9 Billion USD)

77. Summary
• Augmented Reality has a long history going
back to the 1960’s
• Interest in AR has exploded over the last few
years and is being commercialized quickly
• AR is growing in a number of areas
• Mobile AR
• Web based AR
• Marketing experiences

78. SAMPLE AR APPLICATIONS

79. • Web based AR
• Flash, HTML 5 based AR
• Marketing, education
• Outdoor Mobile AR
• GPS, compass tracking
• Viewing Points of Interest in real world
• Eg: Junaio, Layar, Wikitude
• Handheld AR
• Vision based tracking
• Marketing, gaming
• Location Based Experiences
• HMD, fixed screens
• Museums, point of sale, advertising

Typical AR Experiences

80. AR Books – Markerless Tracking

81. HIT Lab NZ ACMI Book Demo
https://www.youtube.com/watch?v=DfzTFPsq224

82. Example: Haunted Book/AR Book
• Camera hidden in lamp object
• AR content seamlessly integrated into real book
• Natural page turning/manipulation interaction
Scherrer, C., Pilet, J., Fua, P., & Lepetit, V. (2008, September). The haunted book. In Proceedings of the 7th
IEEE/ACM international Symposium on Mixed and Augmented Reality (pp. 163-164). IEEE Computer Society.

83. Gaming: Rock-em Sock-em
• Shared AR Demo
• Markerless tracking

84. Rock-em Sock-em Demo
https://www.youtube.com/watch?v=hXtq1qBMLIw

85. Sales and Marketing
• Connect with brands and branded objects
• Location Based Experiences
• Lynx Angels
• Web based
• Rayban glasses
• Mobile
• Ford Ka campaign
• Print based
• Red Bull Magazine

86. Pepsi AR Experience (2014)
• Video see-through AR in bus shelter
• Bus shelter appears under attack

87. Pepsi Demo
https://www.youtube.com/watch?v=Go9rf9GmYpM

88. Assembly and Maintenance
© 1993 S. Feiner, B. MacIntyre, &
D. Seligmann, Columbia University
© 1996 S. Feiner, B. MacIntyre, &
A. Webster, Columbia University

89. Maintenance Systems
• Ngrain
• http://www.ngrain.com/
• Training authoring tool
• Model based AR tracking
• ScopeAR
• http://www.scopear.com/
• Remote assistance
• Image based tracking

90. Ngrain Example
https://www.youtube.com/watch?v=MISyP9C1WB8

91. Medical AR Trials
! Sauer et al. 2000 at
Siemens Corporate
Research, NJ
! Stereo video see through
F. Sauer, Ali Khamene, S. Vogt: An Augmented Reality Navigation System with a
Single-Camera Tracker: System Design and Needle Biopsy Phantom Trial,
MICCAI 2002

92. TUM Medical Demo
https://www.youtube.com/watch?v=29OSzQbxpcQ

93. Interactive Museum Experiences
" BlackMagic
" Virtual America’s Cup
" 410,000 people in six months
" MagicPlanet
" TeManawa science museum
" Virtual Astronomy
" Collaborative AR experience
" AR Volcano
" Interactive AR kiosk
" Scienceworks museum, Melbourne

94. Digital Binocular Station
http://www.DigitalBinocularStation.com/

95. Demo: Digital Binocular Station
https://www.youtube.com/watch?v=mqLAKBX0om4

96. CityViewAR Application
• Visualize Christchurch before the earthquakes

97. CityViewAR
https://www.youtube.com/watch?v=fdgrXxJx4SE

98. Example: Holoportation
• Augmented Reality + 3D capture + high bandwidth
• http://research.microsoft.com/en-us/projects/holoportation/

99. Holoportation Video
https://www.youtube.com/watch?v=7d59O6cfaM0

100. Summary
• AR technology can be used to develop a
wide range of applications
• Promising application areas include
• Games
• Education
• Engineering
• Medicine
• Museums
• Etc..

101. www.empathiccomputing.org
@marknb00
[email protected]