![Investigating integration between auditory and visual location
information presented in an augmented-reality (AR) device
Hiraku Okumura1,2, Sushrut Khiwadkar1, and Sungyoung Kim1
1 Rochester Institute of Technology, Rochester, NY, USA, 2 Yamaha Corporation, Hamamatsu, Japan
Background
Method
Devices Visual Auditory
VR HMD (Oculus Rift, HTC Vive, Sony Playstation VR, etc.) Immersive Immersive
AR
HMD (Microsoft, HoloLens, etc.) Immersive Immersive
Glasses (Sony SmartEyeGlass, Epson Moverio, etc.) Simple
Immersive?
Simple?
The leap of the Virtual Reality (VR) and Augmented
Reality (AR) technologies allows more people to
access head mounted displays (HMDs) and smart
glasses in daily uses. HMDs can provide users with
immersive visual information, while smart glasses
display relatively simple visual information.
We presented a target visual object, the green small square, in the AR
glasses, Sony SmartEyeGlass (Fig. 1), at the 0° (face-forward center), -
5° and -10° in a counter clockwise from the center. Auditory stimuli,
White noise, were presented through the headphone to have target
locations in the horizontal plane from +45° to -45° with a 5-degree
interval.
The subjects were forced to choose between Yes or No as a response to
the question: ”Is the sound coming from the direction of the video cue”.
Discussion & Future work
[1] Kaneko, S., et al., “Ear Shape
Modeling for 3D Audio and Acoustic
Virtual Reality: The Shape-Based
Average HRTF,” in Audio
Engineering Society 61st
International Conference on Audio
for Games, February 2016.
Reference
Result
Panning
HRTF
Video cue angle (Horizontal)
0º -5º -10º
App
Host Android device
(Android tablet)
Laptop PCMAX
Bluetooth
Video Cue
Wireless Network
AKG K550
−45°45°
0° −10°
Sony
SmartEyeGlass
Audio Cue
By integrating proper Auditory information, we can enhance perceived immersion as well as improve clarity (and thus understandability) of visual
information. In audio engineering, binaural techniques based on HRTF (Head Related Transfer Function) are very popular rendering methods for a
pair of headphones. HRTF-based techniques can generate virtual sound sources all around a user and generate more immersive auditory image.
However, they require more computational costs. On the other hand, conventional amplitude-based panning methods can generate virtual sound
sources between loudspeakers, which require less computational costs. How much different are those two approaches in the context of AR?
To answer this question, we have conducted a study that compared two audio rendering techniques–HRTF-based and panning-based one. The
objective of the study was to quantitatively investigate possible benefits of employing panning-based rendering methods for AR devices.
• Audio rendering
• HRTF Dataset: Shape-based average HRTF (30-people) [1]
• Panning Method: Sine-Cosine panning law
• AR Device: Sony SmartEyeGlass
• Screen size (Angle of view) ± 10º (horizontal) x ± 5º
(vertical)
• Display color Monochrome (green)
Eyewear Controller
Fig.2 System Configuration
Fig.3 The user interface for subjectsFig.1 Sony SmartEyeGlass
The results show that two methods–amplitude-based panning and HRTF-based methods–did not produce any
significant differences for the tested AR device. This implicates that we can employ simple panning methods for
AR applications that mainly display simple visual information and save the computational cost.
As the first step, the current study has concentrated on audio/visual interaction along the horizontal plane. We
plan to extend the current work to the integration in the median plane as well as integration of perceived
distance. Furthermore, other types of the AR device such as Microsoft Hololens are being tested for multimodal
integration for more complex visual cues.](https://image.slidesharecdn.com/135d0a78-52f9-4f2c-96d7-8eee020ae999-161206202100/85/Poster02-1-320.jpg)
Poster02
- 1. Investigating integration between auditory and visual location information presented in an augmented-reality (AR) device Hiraku Okumura1,2, Sushrut Khiwadkar1, and Sungyoung Kim1 1 Rochester Institute of Technology, Rochester, NY, USA, 2 Yamaha Corporation, Hamamatsu, Japan Background Method Devices Visual Auditory VR HMD (Oculus Rift, HTC Vive, Sony Playstation VR, etc.) Immersive Immersive AR HMD (Microsoft, HoloLens, etc.) Immersive Immersive Glasses (Sony SmartEyeGlass, Epson Moverio, etc.) Simple Immersive? Simple? The leap of the Virtual Reality (VR) and Augmented Reality (AR) technologies allows more people to access head mounted displays (HMDs) and smart glasses in daily uses. HMDs can provide users with immersive visual information, while smart glasses display relatively simple visual information. We presented a target visual object, the green small square, in the AR glasses, Sony SmartEyeGlass (Fig. 1), at the 0° (face-forward center), - 5° and -10° in a counter clockwise from the center. Auditory stimuli, White noise, were presented through the headphone to have target locations in the horizontal plane from +45° to -45° with a 5-degree interval. The subjects were forced to choose between Yes or No as a response to the question: ”Is the sound coming from the direction of the video cue”. Discussion & Future work [1] Kaneko, S., et al., “Ear Shape Modeling for 3D Audio and Acoustic Virtual Reality: The Shape-Based Average HRTF,” in Audio Engineering Society 61st International Conference on Audio for Games, February 2016. Reference Result Panning HRTF Video cue angle (Horizontal) 0º -5º -10º App Host Android device (Android tablet) Laptop PCMAX Bluetooth Video Cue Wireless Network AKG K550 −45°45° 0° −10° Sony SmartEyeGlass Audio Cue By integrating proper Auditory information, we can enhance perceived immersion as well as improve clarity (and thus understandability) of visual information. In audio engineering, binaural techniques based on HRTF (Head Related Transfer Function) are very popular rendering methods for a pair of headphones. HRTF-based techniques can generate virtual sound sources all around a user and generate more immersive auditory image. However, they require more computational costs. On the other hand, conventional amplitude-based panning methods can generate virtual sound sources between loudspeakers, which require less computational costs. How much different are those two approaches in the context of AR? To answer this question, we have conducted a study that compared two audio rendering techniques–HRTF-based and panning-based one. The objective of the study was to quantitatively investigate possible benefits of employing panning-based rendering methods for AR devices. • Audio rendering • HRTF Dataset: Shape-based average HRTF (30-people) [1] • Panning Method: Sine-Cosine panning law • AR Device: Sony SmartEyeGlass • Screen size (Angle of view) ± 10º (horizontal) x ± 5º (vertical) • Display color Monochrome (green) Eyewear Controller Fig.2 System Configuration Fig.3 The user interface for subjectsFig.1 Sony SmartEyeGlass The results show that two methods–amplitude-based panning and HRTF-based methods–did not produce any significant differences for the tested AR device. This implicates that we can employ simple panning methods for AR applications that mainly display simple visual information and save the computational cost. As the first step, the current study has concentrated on audio/visual interaction along the horizontal plane. We plan to extend the current work to the integration in the median plane as well as integration of perceived distance. Furthermore, other types of the AR device such as Microsoft Hololens are being tested for multimodal integration for more complex visual cues.