COMP 4010 - Lecture11 - AR Applications
6 likes1,874 views
The document covers a lecture on augmented reality (AR) applications by Bruce Thomas and Mark Billinghurst, highlighting the growth of the AR business and potential future applications across various sectors including education, marketing, and cultural heritage. It emphasizes the importance of user acceptance, the need for immersive collaboration through AR tools, and presents case studies like the visualization of Christchurch post-earthquake. The discussion also touches on AR's role in enhancing communication and empathy through technology, pointing out both successes and challenges in implementation.
COMP 4010 - Lecture11 - AR Applications
- 1. LECTURE 11: EXAMPLE AR APPLICATIONS COMP 4010 – Virtual Reality Semester 5 – 2017 Bruce Thomas, Mark Billinghurst University of South Australia October 26th 2017
- 2. AR Business Today • Around $600 Million USD in 2014 (>$2B 2016) • 70-80+% Games and Marketing
- 3. • Web based AR • Marketing, education • Outdoor Mobile AR • Viewing Points of Interest • Handheld AR • Marketing, gaming, education • Location Based Experiences • Museums, point of sale, advertising Typical AR Experiences
- 6. What types of AR Applications in Future? • Applications suitable for AR • Involve 3D spatial data • Require physical interaction • Involve connection between real and digital worlds • Require collaboration, connecting between people • Focus on Intelligence Augmentation (IA not AI) • Enhancing people performing real world tasks
- 7. Examples • Good for AR • Sharing virtual masks • Medical visualization on real body • CAD model on real printed manual • AR assembly instructions shown on real model • Bad for AR • ATM locations overlaid on camera view (use map instead) • AR tower defense game (better than non-AR version?) • Language learning with AR flip cards (why AR?) • AR names on faces (social acceptance, not better than 2D)
- 8. Example:SocialAcceptance • People don’t want to look silly • Only 12% of 4,600 adults would be willing to wear AR glasses • 20% of mobile AR browser users experience social issues • Acceptance more due to Social than Technical issues • Needs further study (ethnographic, field tests, longitudinal)
- 13. Other Important AR Application Domains • Education • Learning on real world tasks • Visualization • Enhancing real world view • Conferencing • Face to face and remote collaboration • Empathic Computing • Sharing Understanding
- 14. VISUALISATION
- 16. Christchurch Earthquakes (2011 onwards)
- 17. Christchurch Before and After
- 19. • Professional solutions available • Autodesk REVIST, ESRI ArcGIS, Grass,etc GIS Desktop Visualization Tools
- 20. Interactive AR Maps • Markerless tracking • 3D model overlay • Gesture input
- 21. Enhanced City Plans • CERA – CCDU Plan • Using tablet to track off printed maps • Overlay 3D city models onto real maps
- 22. Earthquake AR Project • Goal: • To allow people to see Christchurch as it was • To provide a tool for visualizing the city as it could be • Technology • Mobile AR platform • Smart phone hardware • 3D content
- 23. HIT Lab NZ Outdoor AR Platform • Cross platform • Android, iPhone • 3D onsite visualization • Intuitive user interface • Positions content in space • Camera, GPS, compass • Client/Server software architecture • Targeting museum guide/outdoor site applications
- 24. Earthquake Reconstruction • See past, present and future building designs • Earthquake survivor stories shown on map view • Collect user comments • Android platform
- 25. Demo https://www.youtube.com/watch?v=vFFUKJzxTH4
- 26. Architecture Android application Database server Postgres Web Interface Add models Web application java and php server
- 27. CityViewAR • Using AR to visualize Christchurch city buildings • 3D models of buildings, 2D images, text, panoramas • AR View, Map view, List view • Available on Android market
- 29. User Experience • While walking in the real world people can see text, 2D images and 3D content on their own phones
- 30. Interface Design (1/2) Browsing Interface ContentFront face Title Screen AR View Map View List View Detail View Image Gallery PanoramaInstruction & Information
- 31. Examples for Situated Visualization Textual labels for architectural sights Image: Raphael Grasset Pollution levels on the street Image: Sean White and Steve Feiner
- 32. Hydrological Data Visualization Hydrosys displays locations of stations in a global sensor network as well as interpolated temperature plotted as geodesic contours Image: Eduardo Veas and Ernst Kruijff
- 33. Raw data Filtered data Visual structures View Data transformation Visual mapping View transformation Visualization Pipeline The visualization pipeline includes three stages: data transformation, visual mapping, and view transformation
- 34. 34 Making the Invisible Visible • Hidden structures & information • Supermans X-Ray Vision • Spatial arrangement problem
- 36. AR Navigation • Many commercial AR browsers • Information in place • How to navigate to POI
- 37. 2D vs. AR Navigation? VS
- 38. AR Navigation Study • Users navigate between Points of Interest • Three conditions • AR: Using only an AR view • 2D-map: Using only a top down 2D map view • AR+2D-map: Using both an AR and 2D map view • Experiment Measures • Quantitative • Time taken, Distance travelled • Qualitative • Experimenter observations, Navigation behavior, Interviews • User surveys, workload (NASA TLX)
- 39. HIT Lab NZ Test Platform – AR View
- 40. HIT Lab NZ Platform – Map View
- 41. Distance and Time No significant differences
- 42. Paths Travelled • Red – AR • Blue – AR + Map • Yellow - Map
- 43. Navigation Behaviour • Depends on interface • Map doesn’t show short cuts
- 44. Survey Responses
- 45. User Comments • AR • “you don't know exactly where you are all of the time.” • “using AR I found it difficult to see where I was going” • Map • “you were able to get a sense of where you were” • “you are actually able to see the physical objects around you” • AR+MAP • “I used the map at the beginning to understand where the buildings were and the AR between each point” • “You can choose a direction with AR and find the shortest way using the map.”
- 46. Lessons Learned • User adapt navigation behaviour to guide type • AR interface shows shortcuts • Map interface good for planning • Include map view in AR interface • 2D exocentric, and 3D egocentric • Allow people to easily change between views • May use Map far away, AR close • Difficult to accurately show depth Dünser, A., Billinghurst, M., Wen, J., Lehtinen, V., & Nurminen, A. (2012). Exploring the use of handheld AR for outdoor navigation. Computers & Graphics, 36(8), 1084-1095.
- 47. CONFERENCING
- 50. A wide variety of communication cues used. Speech Paralinguistic Para-verbals Prosodics Intonation Audio Gaze Gesture Face Expression Body Position Visual Object Manipulation Writing/Drawing Spatial Relationship Object Presence Environmental Face to Face Communication
- 51. Evolution of Communication Tools
- 53. Communication Seams • Technology creates artificial seams in communication • Separation between real and virtual space Task Space Communication Space
- 55. Augmented Reality Conferencing • Augmented Reality • Combines real and virtual world • Interactive in real time • Registered in 3D • AR Conferencing • Seamless blending of task/communication space • Natural spatial presentation/interaction • Support for realistic communication cues • Independent views of shared content
- 56. Augmented Reality Conferencing • Tele-Presence • Bringing a remote person into your space • Tele-Existence • Feeling like you are in a remote space
- 57. TELE-PRESENCE
- 58. AR Video Conferencing (2001) • Bringing conferencing into real world • Using AR video textures of remote people • Attaching AR video to real objects Billinghurst, M., & Kato, H. (2002). Collaborative augmented reality. Communications of the ACM, 45(7), 64-70.
- 59. Multi-View AR Conferencing Billinghurst, M., Cheok, A., Prince, S., & Kato, H. (2002). Real world teleconferencing. Computer Graphics and Applications, IEEE, 22(6), 11-13.
- 60. Holoportation (2016) • Augmented Reality + 3D capture + high bandwidth • http://research.microsoft.com/en-us/projects/holoportation/
- 62. Social VR • Facebook Spaces, AltspaceVR • Bringing Avatars into VR space • Natural social interaction
- 63. Demo: Facebook Spaces VR https://www.youtube.com/watch?v=PVf3m7e7OKU
- 64. Mini-Me – Miniature Avatar • Using a miniature Avatar to show communication cues
- 65. TELE-EXISTENCE
- 66. Example: Google Glass • Camera + Processing + Display + Connectivity • Ego-Vision Collaboration (But with FixedView)
- 67. First Person Sharing Through Glass https://www.youtube.com/watch?v=LyKaUT1bArw
- 68. Current Collaboration onWearables • First person remote conferencing/hangouts • Limitations • Single POV, no spatial cues, no annotations, etc
- 69. Social Panoramas (ISMAR 2014) • Capture and share social spaces in real time • Supports independent views into Panorama
- 70. Implementation • Google Glass • Capture live image panorama (compass + camera) • Remote device (tablet) • Immersive viewing, live annotation
- 71. User Interfaces Glass View Tablet View
- 73. Lessons Learned • Good • Communication easy and natural • Users enjoy have view independence • Very natural capturing panorama on Glass • Sharing panorama enhances the shared experience • Bad • Difficult to support equal input • Need to provide awareness cues
- 74. JackIn – Live Immersive Video Streaming • Jun Rekimoto – University of Tokyo/Sony CSL Kasahara, S., & Rekimoto, J. (2014, March). JackIn: integrating first-person view with out-of-body vision generation for human-human augmentation. In Proceedings of the 5th Augmented Human International Conference (p. 46). ACM.
- 75. JackIn Hardware • Wide angle cameras – 360 degree video capture • Live video stitching
- 77. JospehTame – Tokyo Marathon • Live streaming from Tokyo marathon • http://josephta.me/en/tokyo-marathon/
- 78. Shared Sphere – 360 Video Sharing Shared Live 360 Video Host User Guest User Theta S 360 Camera Hi-res Camera Epson BT-200 See-through HMD Oculus Rift HMD Leap Motion
- 79. Mixed Space Collaboration (2017) • Make 3D copy of real space • AR user in real world, VR user in 3D copy of real space • Hololens for AR user, HTC Vive for VR user • Share virtual body cues (head, hands, gaze information) Real World Virtual World
- 80. AR User/VR User Displays • HTC Vive (VR User), HoloLens (AR User) • Room scale tracking • Gesture input (Leap Motion)
- 81. Virtual Body Cues • Virtual head, hands • View frustums
- 84. Empathic Computing 1. Understanding: Systems that can understand your feelings and emotions 2. Experiencing: Systems that help you better experience the world of others 3. Sharing: Systems that help you better share the experience of others
- 85. 1. Understanding: Affective Computing • Ros Picard – MIT Media Lab • Systems that recognize emotion
- 86. Appliances That Make You Happy • Jun Rekimoto – University of Tokyo/Sony CSL • Smile detection + smart appliances
- 87. Happiness Counter Demo https://vimeo.com/29169237
- 88. 2. Experiencing: Virtual Reality “Virtual Reality is the ultimate Empathy Machine” Chris Milk • Within • http://with.in/
- 89. Virtual Reality as Empathy Medium "Virtual reality offers a whole different medium to tell stories that really connect people and create an empathic connection." Nonny de la Peña http://www.emblematicgroup.com/
- 90. Using VR for Empathy • USC Project Syria (2014) • Experience of Terrorism • Project Homeless (2015) • Experience of Homelessness
- 91. Hunger • Experience of homeless waiting in food line https://www.youtube.com/watch?v=wvXPP_0Ofzc
- 92. 3. Sharing Can we develop systems that allow us to share what we are seeing, hearing and feeling with others?
- 93. The Machine to Be Another (2014) • Art project exploring issues of human identity • One user has impression they’re in the body of another • VR HMDs, stereo cameras, video mixing • http://www.themachinetobeanother.org/
- 94. Machine to Be Another • https://www.youtube.com/watch?v=_Wk489deqAQ
- 95. CHILDHOOD • Kenji Suzuki, University of Tsukuba • What does it feel like to be a child? • VR display + moved cameras + hand restrictors
- 97. Technical Requirements •Basic Requirements •Make the technology transparent •Wearable, unobtrusive •Technology for transmitting •Sights, Sounds, Feelings of another •Audio, video, physiological sensors
- 98. Gaze and Video Conferencing • Gaze tracking • Implicit communication cue • Shows intent • Task space collaboration • HMD + camera + gaze tracker • Expected Results • Gaze cues reduce need for communication • Allow remote collaborator to respond faster Gupta, K., Lee, G. A., & Billinghurst, M. (2016). Do You See What I See? The Effect of Gaze Tracking on Task Space Remote Collaboration. IEEE Transactions on Visualization and Computer Graphics, 22(11), 2413-2422.
- 99. Experiment Set Up • Lego assembly • Two assembly areas • Remote expert
- 100. Experiment Design • 4 conditions varying eye-tracking/pointer support • 13 pairs subjects • Measures • Performance time • Likert scale results, Ranking results, User preference
- 101. Task Performance • Performance Time (seconds)
- 102. Ranking • Average Ranking Values
- 103. Key Results • Both the pointer and eye tracking visual cues helped participants to perform significantly faster • The pointer cue significantly improved perceived quality of collaboration and co-presence • Eye-tracking improved the collaboration quality, and sense of being focused for the local users, and enjoyment for the remote users • The Both condition was ranked as the best in user experience, while the None condition was worst.
- 104. Empathy Glasses (CHI 2016) • Combine together eye-tracking, display, face expression • Impicit cues – eye gaze, face expression ++ Pupil Labs Epson BT-200 AffectiveWear Masai, K., Sugimoto, M., Kunze, K., & Billinghurst, M. (2016, May). Empathy Glasses. In Proceedings of the 34th Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM.
- 105. AffectiveWear – Emotion Glasses • Photo sensors to recognize expression • User calibration • Machine learning • Recognizing 8 face expressions
- 106. Integrated System • Local User • Video camera • Eye-tracking • Face expression • Remote Helper • Remote pointing
- 107. System Diagram • Two monitors on Remote User side • Scene + Emotion Display
- 109. Ranking Results "I ranked the (A) condition best, because I could easily point to communicate, and when I needed it I could check the facial expression to make sure I was being understood.” Q2: Communication Q3: Understanding partner HMD – Local User Computer – Remote User
- 110. Lessons Learned • Pointing really helps in remote collaboration • Makes remote user feel more connected • Gaze looks promising • Shows context of what person talking about • Establish shared understanding/awareness • Face expression • Used as an implicit cue to show comprehension • Limitations • Limited implicit cues • Task was a poor emotional trigger • AffectiveWear needs improvement
- 111. CONCLUSIONS
- 112. AR Applications • Current applications mostly gaming and marketing • Application types ideal for AR • Involve 3D spatial data • Require physical interaction • Involve connection between real and digital worlds • Emphasis on Intelligence Amplification • Promising Areas for future applications • Visualization • Conferencing • Empathic Computing