22CDH01-UNIT V ER DEVELOPMENT -IMMERSIVE DESIGN THEORY

P.GOWSIKRAJA M.E., (Ph.D.,)
Assistant Professor
Department of Computer Science and Design
UNIT V- Extended Reality(XR) Development
KONGU ENGINEERING COLLEGE (AUTONOMOUS)
DEPARTMENT OF COMPUTER SCIENCE AND DESIGN
22CDH01-HONOR DEGREE-IMMERSIVE DESIGN THEORY

UNIT V- Extended Reality(XR) Development
1. Augmented Typography:
Legibility and readability
Creating visual contrast
Take control
Design with purpose
2. Color for XR:
Color appearance models
Light interactions
Dynamic adaptation
Reflection
3. Sound Design:
Hearing what you see
Spatial sound
Augmented audio
Voice experiences
Power of sound.

Augmented Typography:-
♣ To exploring how to optimize your type in augmented experiences.
♣ Though many of the topics discussed also apply to virtual reality, the
emphasis best practices for AR, because that provides less control of
the environment.
UNDERSTAND LEGIBILITY AND READABILITY
CREATE VISUAL CONTRAST
TAKE CONTROL

Evolution
Back to the basics
Legibility
KEEP IT SIMPLE
THINK BIG
CONSIDER X-HEIGHT
STAY SANS DETAIL
Type made for XR
ARone
Type is meant to be read
Readability
GIVE IT SPACE
ARone Halo
SAY MORE WITH LESS
MAKE A CASE
LIMIT LINE LENGTH
WEIGH IN
KEEP IT FLAT

Evolution
Back to the basics
Legibility
KEEP IT SIMPLE
THINK BIG
CONSIDER X-HEIGHT
STAY SANS DETAIL
Type made for XR
Arone
Type is meant to be read
Readability
GIVE IT SPACE
ARone Halo
SAY MORE WITH LESS
MAKE A CASE
LIMIT LINE LENGTH
WEIGH IN
KEEP IT FLAT

Displays covered in type are around us everywhere—from gas station signage to
airport information boards to mobile applications in the palm of our hands.
Type on displays. Presenting text that will appear on a variety of screens creates
design challenges.
Evolution: Throughout the evolution of screens, typefaces have been designed to
improve the user experience.(CRT)
Screens have grown bigger, brighter & more light weight.

Back to the basics:-
Designing typography for ease of reading within XR involves similar
considerations as designing for a screen.
typographical marks include:
♣ Letters
♣ Numbers
♣ Punctuation
♣ Dingbats/symbols

Legibility is based on learned from designing for screens, simple is better.
Typefaces that are made from simple shapes translate better into lower
resolution displays, such as screens.
Legibility How easily distinguishable one letter is from another within a
typeface.
KEEP IT SIMPLE. Typefaces created from simple shapes work better than
overly styled type.
Geometric Type. Look for letterforms
that are created with basic geometric
shapes, right angles, and horizontal
finishing strokes.
GEMOMETRY

THINK BIG. In print you can have body
copy ranging from 8 to 12 points is too
small for pixel-based type. 14 to 16
pixels or larger is optimal size.
CONSIDER X-HEIGHT. The height of
the lowercase letters is called the x-
height. Not all typefaces have the same
x-height.

STAY SANS DETAIL:- create typefaces specifically for
screen type, so these are good places to start.
Helvetica, Verdana, and Georgia are some classics,
but this list continues to grow thanks to the availability
of the Web Open Font Format and fonts being designed
for both print and web formats
ARone Halo
SERIF

Type is meant to be read:
Selecting a typeface that is legible to users means that they can easily distinguish the
characters from one another.
Readability The spacing and arrangement of characters and words in order to make the
content flow together to aid reading it.
Many of these remain connected to the foundations of typography, but just need some
optimization for XR. Keep these guidelines in mind:
GIVE IT SPACE:-
Increasing your overall tracking, the space between two or more characters, will help
with readability.
With many of the displays you often see a bit of a halo effect around the text, so by
tracking out your type you can avoid the overlay of the halos and the letters themselves.

SAY MORE WITH LESS. Reducing the amount of copy, especially
paragraphs of type, is a better practice with tool tips, explainer type,
closed captioning, and an audio track.
AROne is typeace tool
MAKE A CASE. Select a case that works for the content.
Uppercase is hard to read in large amounts, but can add hierarchy for
headers or shorter phrases you want to stand out.
LIMIT LINE LENGTH To reduce eye strain, keep the length of your lines of
type to 50 to 60 characters per line.

♣ WEIGH IN. Varying your weights of type is a great way to add hierarchy to
your designs and can help guide the user’s eye though the page.
♣ Light and Extra Bold weights are much less legible than regular,
medium, or bold weights.
♣ KEEP IT FLAT. 2D type is easier to read than 3D type. Type that is
extruded and volumetric becomes much harder to read.
♣ It makes sense if you consider we aren’t as used to reading type in 3D;
most of our reading is two dimensional.
♣ Logotypes are an exception as they can work as a 3D element in an
experience.

Viewing Distance- display and text
type
Spatial Zones
UI zone
Focal zone
Environmental zone
Different spatial zones
IMMERSIVE TYPE
UI TYPE
ANCHORED TYPE
RESPONSIVE TYPE

Creating visual contrast:-
Designers are used to considering
reading distance when designing.
A poster or billboard is expected to
be viewed from a further distance
than a brochure or postcard.
There are different design
considerations as a result of the
distance between the user and the
design element.

Viewing Distance. The viewing distance from text to our eyes changes based
on the medium.
Within an XR experience there may be type that is:
● Placed within the 3D space
● Static (such as any type that is part of the UI)
● Anchored within the environment
● Responsive
In print media, when choosing from the wide range of type options, you can start
by selecting from two main text types: display and text type.
Display type Type found in large headings and titles; typically 16+ points.
Text type Type found in paragraphs and meant for longer reading; typically 8 to
12 points and sometimes called body text.

Spatial Zones. Showing the three main
spatial zones in relationship to the user’s
display.
♣ UI zone
♣ Focal zone
♣ Environmental zone
UI ZONE The closest text to the user is
within this space.
This type is anchored to the camera position
on a mobile device or HMD making this
information constant in placement and view.

FOCAL ZONE This is an optimal placement for some of the main part of the experience, including
any essential type.
This is the ideal reading distance for essential type within the experience. This space is within 3 to
16 feet from the user.
ENVIRONMENTAL ZONE It can be used for positioning, landmarks, and to add any additional
environmental context within the experience.
Keep your type in the center zone of what you are designing to avoid the pixels from blurring on the
edge of your peripheral sight. Here are the optimal degrees to remember:
Field of view: 94°
Head turn limit: 154°
Maximum viewing at one time: 204

With a 3D experience, there are important type design considerations for
each kind of type relative to the different spatial zones.
Immersive type
UI type
Anchored type
Responsive type
IMMERSIVE TYPE This type needs to act like a 3D object, but will most
likely be a flat 2D element (for readability).
● This type is integrated into the 3D environment. As such, it should
match the perspective of the planes where it is placed.

♣ This dynamic type will rely on spatial computing to map out the space in
advance of the experience or having the user select a vertical or horizontal plane
where the type will be placed.
♣ UI TYPE This type remains static in the experience.
♣ This should be 2D and remain in one place on the screen, such as the navigation
bar or on the top and bottom of the screen.
♣ This text is critical for the user experience and often provides identifying
information, such as the name of the app or experience.
♣ UI type must be easy to find, easy to see, and easy to use, because it plays an
essential role in the approachability of the experience for a user.

ANCHORED TYPE This type is connected to a
specific plane or object within the environment.
♣ Anchored type stays pinned to one specific
location or object to identify it, like the
business labels in this AR navigation app
prototype.
♣ Example, the tags pinned to the surrounding
businesses and landmarks around help the
user identify them.
♣ These visual tags are anchored to the
physical location.

RESPONSIVE TYPE Just as websites have to create responsive layouts and size ratios for
the desktop displays, tablets, and mobile devices, that same concept applies in XR
environments.
♣ Currently, type in HMDs uses pixel or bitmap type, instead of vector or outline type
which would allow it to be scalable.
♣ CSS is a benefit for a similar system within AR.
♣ Based on user movement and the viewing angle, this approach allows type to
automatically adjust for optimal readability.
♣ Black-on-white text is not as effective across all devices because you cannot reproduce
pure black in a transparent or see-through display, which is used for many AR/MR
experiences.
♣ Without the pure black there may not be enough contrast between the type and the
background for readability.

Viewing Distance- display and text type
Spatial Zones
UI zone
Focal zone
Environmental zone
Different spatial zones
IMMERSIVE TYPE
UI TYPE
ANCHORED TYPE
RESPONSIVE TYPE

Take control
Where type is used? Tags
How type is used
How to view type? perspective distortion.
Customization
Minimize
Design with purpose
DESIGN CHALLENGE
MAKE AN AUGMENTED EYE CHART

Take control
You are probably aware by this point that there are a lot of uncontrollable
components to working within augmented and mixed realities.
Where type is used
To achieve more control is to consistently place your type in the same
location within the experience, from the entry point until the end.
After the user sees type repeatedly show up in the same place multiple
times, they will start to look to that place for the information when they
need it.

This can apply to the UI type that
helps users figure out how to
navigate through an experience,
but it can also relate to the
immersive type that is part of the
3D space.
Example, in tagAR the name tags
always appear above people’s
heads.

Tags. An augmented name tag from the mobile application tagAR.
These tags always appear directly above each person’s head, making it easier
to see their name and make eye contact at the same time.
Example: cars with projected GPS directions appearing in the road in front
of them use this approach to take advantage of constants within the driving
experience.
How type is used
To allow the users to start associating a specific style with a specific
function, give a role to each of the type styles within the experience.

It does take time to initially set up the styling for each of the needed styles,
such as:
Main header (h1)
Secondary header (h2)
Additional headers (max 6)
Body type (p)
How to view type
 The positioning and orientation are relative to the user and their gaze.
This added control ensures that people will view the type without any
perspective distortion.

 When users view text in 3D space from extreme angles, the type can get
bent and misshapen
 Perspective Distortion. As type gets warped to fit into a 3D scene, the
use of extreme perspectives makes the type more distorted, reducing the
readability of the message.
 Perspective distortion A warping of the appearance of an object or image
often caused by viewing it from an extreme angle or how it is placed into a
3D scene.

Customization
♣ Knowing that people will each have a different experience based on their
physical location and environment, you can design for this.
♣ Using your user research to identify the most common places people
interact with the experience, you can create different experiences for each.
♣ When a user first launches the experience, they would have to provide
information about their physical environment.
♣ Reduce the effort for users (and yourself) by providing a list to choose.

Minimize
♣ What information is essential to be included in the copy?
♣ It is important to look through all the wording you are including in an
experience and be as efficient as possible.
♣ reading large amounts of type in XR is not optimal. Using simple icons,
arrows, illustrations, photographs, videos, or even a combination of
these like a data visualization or an infographic communicating the
same information in a visual way.

Design with purpose
 The key takeaway to understand leaving this chapter is efficiency.
 There are many challenges in displaying type in AR—everything from
working with lower resolution screens to choosing the best typeface to
be viewed up close and far away and everywhere in between.
 These challenges reveal the need for efficiency of all the type you include
within an experience.

DESIGN CHALLENGE
MAKE AN AUGMENTED EYE CHART
The goal of this challenge is to help test your typographic design choices at varying
reading distances in augmented reality.
1. Based on some of the suggestions in this chapter select three typefaces and
font weights that you think will be legible in AR.
2. Using Adobe Illustrator or Photoshop, design an eye chart with different
letters in each row. Use these letters in order and add line breaks as shown in
the figure.
EFPTOZLPEDPECFDEDFCZP
3. As you go down each line, reduce the point size as shown.
4. Save this file as a JPG.
5. Launch Adobe Dimension. From the basic shape library select a plane.

6. Using the widget tool on the plane, rotate the
plane on the z-axis (blue) to lift the plane up
vertically, as you would expect to see a traditional
eye chart. Then position the plane on the x-axis
(magenta) to lift it up off the ground.
7. Now you need to add your eye chart to the plane.
To do this, move to the right side of the screen and
select your Plane layer in the Scene panel. Click
the arrow on this layer to view your customization
properties. Find the Properties panel, and double-
click the base color. Toggle from selecting a color
to selecting an image. Here you can upload the JPG
you saved earlier.

8. Adjust the positioning of your plane as needed to make sure you can view
the letters correctly.
9. Now, the fun really begins. You are going to share this to Adobe Aero.
While still in Adobe Dimension, choose File > Export > Selected for Aero.
Choose Export from the pop-up window, and then save the file in your
Creative Cloud Files folder. This should be the default folder that comes up,
but if you don’t see it, you can find it in your user files.
10. Using a mobile device or iPad, launch the Adobe Aero application. When
it prompts you to choose an image, choose your eye chart from your Creative
Cloud files. Place this on a plane so you can start testing.

11. Make sure you are clear to
move around the image. View the
type up close, and step back away
from it. How is the readability
affected? Take notes.
12. Based on your findings, choose
a different typeface and repeat the
process to help identify which
typefaces to try out in your next
AR project.

2. Color for XR: Color appearance models
Color space
Additive
Subtractive
Linear versus gamma color space
Usability
●Legibility and readability
●Contrast
●Vibrancy
●Comfort
●Transparency

2. Color for XR: Color appearance models
♣ The color is a personal and dynamic relationship.
♣ Color creates an emotional impact, as we carry cultural meanings to
the hues surrounding our society.
♣ “Not all reds are the same. Some are more intense,
some more passionate, some more full of life, and
some more cautionary”.
♣ The term color space is used to describe the capabilities of a display or
printer to reproduce color information.

Color space A specific organization of colors that
determines the color profile that is used to support
the reproduction of accurate color information on a
device. RGB and CMYK are two common
examples.
Cyan, Magenta, Yellow, and Key (black)

Additive color/RGB
Subtractive color/CMYK
Red, Green, and Blue each a color value of
between 0 and 255. To create over 16 million
color combinations.
The common color profile for this is called
CMYK (cyan, magenta, yellow and key
black). (mixing base)
8-bit sRGB color format preferred input for
images on many XR devices.
the term key is a direct reference to the key
plate used in the printing process. four-color
printing process
Each color has a specific value in the HSB or
HSL format. provides a numeric value to the
hue, saturation, and brightness (or
lightness) of the color.
These four colors can produce over 16,000
different color combinations.

Linear and Gamma
color space:
The difference between
increasing the shading
incrementally in the
linear color space
versus using the
gamma correction,
which is nonlinear.

Linear Color Space Gamma Color Space
when creating digital images, there is
a need of more accuracy and variety
in the dark tones.
Once an image or graphic has been
gamma corrected, it should, in theory,
be displayed “correctly” for the human
eye.
To accommodate for this sensitivity of
the way the brain perceives shades,
gamma correction, also referred to as
tone mapping, was created.
To replicate that in a way that is
mathematical correct, the
linear color space was created to
match our physical space.

Linear Color Space Gamma Color Space
Linear color space Numeric color
intensity values that are
mathematically proportionate.
Gamma correction A process that
increases the contrast of an image in
a nonlinear way to adjust for the
human eye’s perception and the way
displays function.
This has also become a standard within software focused on immersive
experiences such as Unity and Unreal Engine.

♣ Tint The increased lightness of color by the addition of white.
♣ Shade The increased darkness of a color by the addition of black.
♣ HMDs will support linear only, while others support gamma only.
♣ Some will allow a combination: linear color with some gamma
corrections.
♣ Usability:-
♣ Selecting colors that will make the experience usable.
1. Legibility and readability
2. Contrast
3. Vibrancy
4. Comfort
5. Transparency

♣ Legibility and readability refer not only to the color of type (text), but
also to color of the elements surrounding the text.
♣ White is the most common color for text and icons in XR.
♣ Red text on a black background is hard to read because they are both
dark.
♣ Select colors that have varying shades, so you don’t have a dark color on
a dark color; instead, you want light on dark or dark on light.

Contrast
● When you have two colors that are close in shade or even saturation,
they will start to vibrate off one another.
● To avoid this effect, select colors that have visual contrast.
● It means opposite qualities such as light& dark or saturated &
desaturated.
Color Vibration. Colors that are close in tonal range start to vibrate when
placed in close proximity.
● Contrast is essential for keeping your experience accessible.
● Making sure your color choices have solid contrast will make the
experience usable for a greater number of users.

Vibrancy
♣ A color at its purest form is called chroma. when the color is fully saturated,
without the addition of gray. These pure colors are bright and vibrant.
♣ Vibrancy increases the brightness of the desaturated tones.
♣ The energy of a color caused by increasing or decreasing the saturation of the least
saturated tones.
♣ Vibrancy can also change the energy of the color and, as a result, the overall
experience.
♣ Bright oranges and reds will grab your attention over desaturated greens or grays.

Comfort
To create a positive user experience, you want the user to be comfortable.
If the colors you select are too intense or create too much strain, then this
will cause discomfort.
If a user is met with too much discomfort, they will likely leave the experience
to find a different one that is more comfortable.
Larger areas of color in XR, especially vibrant and fully saturated colors, will
be hard on the eyes. So, use these brighter colors sparingly to attract
attention, but don’t use them in large quantities.

Comfort
♣ To create a positive user experience, you want the user to be comfortable.
♣ If the colors you select are too intense or create too much strain, then this will
cause discomfort.
♣ If a user is met with too much discomfort, they will likely leave the experience to
find a different one that is more comfortable.
♣ Larger areas of color in XR, especially vibrant and fully saturated colors, will be
hard on the eyes.
Comfort:-
The colors will change in appearance between the computer you create them on and
the actual device that plays the XR experience, it is important to test your designs.
View the colors in context, and then make adjustments to improve the ease of use.

Transparency:- Color will be displayed differently based on the kind of display you
use.
An optical see-through (OST) display, such as the Microsoft HoloLens 2, AR
glasses, or smart glasses will show all elements as more transparent, due to the
nature of the technology.
Video see-through (VST) displays, such as mobile AR experiences that use the
camera to view the physical world, have different considerations. Because any
graphics or objects will be applied directly on top of the camera view in a VST-
based experience, they can be displayed fully opaque.

If the amount of transparency in a 3D
model or object,then you can reserve
opaque colors for UI elements so that
they stand out on the display. The
perception of color is directly connected to
the light in the scene.
To ensure that users see the colors that
you select for the design, you need to
design the lighting as well.

Light interactions:
● Type of light
● POINT LIGHT
● SPOT LIGHT
● AREA LIGHT
● DIRECTIONAL OR PARALLEL LIGHT
● AMBIENT LIGHT
● Color of light- Light Temperatures
● Lighting setup
● Soft lighting
● One-point lighting
● Three-point lighting
● Sunlight
● Backlight
● Environmental
● Direction and distance of light:- Falloff, Feathering
● Intensity of light:- 100% (the highest brightness)
● Shadows

Adjusting light in a scene or onto an object does not just mean that you
are simply brightening or darkening;
Believable immersion relies in the use of light and its accompanying
shadow.
With the exception of some stylistic deviations, you will want your
lighting to mimic the real world.
Type of light: Think about lighting design as you would think about
determining the colors of a composition: Identify the key areas that you
would like to have the most attention.
The brightest and most vibrant colors will attract attention first.

POINT LIGHT A point light will emit light in
all directions from a single point.
This light has a specific location and
shines light equally in all directions,
regardless of orientation or rotation.
Examples are lightbulbs and candles.
SPOT LIGHT A spot light works just like a
spotlight used in stage design.
It emits light in a single direction, and you
can move the direction of the light as needed.
Example: stage spot light for a soloist.

AREA LIGHT This light source is confined within
a single object, often in a geometric shape such
as a rectangle or sphere shape. Examples are a
rectangular florescent light and a softbox
light.
DIRECTIONAL OR PARALLEL LIGHT Parallel
rays that mimic the sun; these lights are
infinite, just like sun infinitely lights. This means
that the position of these lights doesn’t matter,
only their direction and brightness. An obvious
example is sunlight.

AMBIENT LIGHT Ambient light applies to the full scene. You cannot choose a
specific location for this light, and it will change the overall brightness of the
scene. Example: natural, indirect light from a window.
Color of light
If you have ever gone lightbulb shopping or bought Christmas lights, then you’ve
seen how many different colors of light there are.
Even if you want just “plain white” light, you are greeted with a magnitude of
options. The reason is that no light is pure white.
Light is made up of three colors: red, green, and blue. Mixing these colors in
different proportions alters the color of the light we see, thanks to the additive
property we discussed earlier.
Light has a color temperature;

it can be warm or cool depending on the proportional mix of colors.
2700K is a warmer, yellower white;
7000K is a cooler, bluer white; daylight is 6400K.
Light Temperatures. The temperatures of various kinds of lights using the
Kelvin scale for measurement.

Lighting setup: It is quite to use more than one light in your scene, just as
you would in the real world. You can have window light and a table lamp in
the same space.
Example: add additional lights to the scene, you need to control the
relationship of the lights.
Soft lighting:Soft lighting is the best choice if you need to add evenly
distributed lighting to your scene.
The name actually refers to the soft quality of shadows in the scene, making
the overall contrast feel balanced and calm.
This kind of lighting is frequently used for
portrait photography.

Soft Light. One soft light provides equal
light across the 3D sphere.
One-point lighting: The one-point
lighting technique uses a single light and,
as a result, will create a dynamic mood.
It also creates harsher shadows where
the light is not illuminating the object.
One-point light hits the 3D sphere
making the light and shadows more
dramatic.

Three-point lighting
The three-point lighting technique
uses three lights—key, rim, and fill
—each of which has a specific role in
the overall lighting setup.
Three lights are set up around the 3D
sphere to demonstrate the positions
of the rim light (backlight), key
light, and fill light.

Three-Point Light:
Three lights are set up around the 3D sphere to demonstrate the positions of
the rim light (backlight), key light, and fill light.
● Key light illuminates the focal point of the scene or object and is the
primary light in the scene.
● Rim light illuminates the back your subject, separating it from the
background and adding depth.
● Fill light fills in more light in the scene to reduce or eliminate harsh
shadows and even out the overall lighting.

Sunlight
In the sunlight approach there is a single light
source: the sun.
If you are looking to replicate an outdoor scene,
then you should use direct sunlight as your
lighting.
Unlike in the real world, however, you easily can
move the direction of the sun in a 3D scene to
mimic the type of sunlight you prefer: sunrise,
high noon, sunset, or something in between.

Backlight
A primary light source behind your
object is a backlight.
This technique is not as commonly
used, but it can create some
mystery and drama to the scene as
needed.

Environmental
The environmental lighting approach pulls lighting from an image that is
imported into the program.
This works best when using high-dynamic-range imagery (HDRI) for which
the luminosity data of the image, specifically the darkest and lightest
tones, are captured at a larger range.
These images can be used to replicate the lighting in the image in the 3D
scene.
Environmental. The light was created to mimic the lighting from the
background image and replicated on the 3D sphere.

Direction and distance of light
The relationship between the light and the shadow provides a lot of
information, to control the look and feel of that transition.
the light weakens so too will the shadow. This weakening of a light along
its outer edge is called falloff.
The falloff has a radius and a distance, and you can control it.
Lights with a smooth falloff have a high radius and a large distance that
will show a gradient blur that slowly goes from light to dark.

Falloff :
● The visual relationship of shadow and light as illumination decreases
while becoming more distant from the light source.
● The edge of the light can be controlled through edge or cone feathering
to soften the line between the light and the shadow.
● This is how you can edit and control the edge itself.
● This option is often available for any lighting that is a cone shape, such
as a spot light.
Feathering :-
The smoothing, softening, or blurring of an edge in computer
graphics.

Intensity of light
Once you have the kinds of lights, their position, their roles in the scene,
and their color properties identified, the next step is to determine how
bright the light should be. This is the intensity.
The default is 100% (the highest brightness), but this amount can be edited to
make the light dimmer. The strength of the light can also be called energy.
Shadows
Wherever there is a light, there must be an accompanying shadow, where there
is light falloff or light is blocked by another object.

Shadows also play a big part in our ability to perceive where an object is
in space.
A shadow that connects to the bottom of the object tells us that the
object is sitting directly on the plane.
Example, natural sunlight casts stronger shadows than artificial light.
♣ The terms soft light and hard light actually reference the
characteristic of the shadows the types of light create.
♣ Soft lighting provides a more even light across all of the subject and,
in turn, creates soft shadows with a fuzzy edge.
♣ Hard lighting provides more dramatic lighting on an object, creating
sharp edges on shadows.

Shadows. 3D rendering highlighting where
the main light source is (the setting sun)
and how the light falls off into increasing
shadow inside the cave.
The farther away from the sunlight, the
darker the shadows become.

Dynamic adaptation
Lighting estimation
● Brightness
● Light color
● Color correction values
● Main light direction
● Ambient intensity
● Ambient occlusion
Environmental reflections
● Diffusion
● Roughness
● Metalness

Dynamic adaptation:
The idea of the copycat. It allows you to learn and adapt to new
interactions by imitating what someone else is doing—learning as you go
along. This simple concept can be applied to a larger scale, as we look at
imitation in AR.
With dynamic backgrounds and environments, the light and the
properties of the light will constantly change. such as Google’s ARCore and
Apple’s ARKit framework, evaluate environmental light and repeat it as digital
light. The basic method used is called lighting estimation.

Using sensors, cameras, and algorithms, the computer creates a
picture of the lighting found within a user’s physical space and then
generates similar lighting and shadows for digital objects added to the
space.
This is a key attribute in the ARCore and ARKit frameworks.
Lighting estimation A process that uses sensors, cameras, machine
learning, and mathematics to provide data dynamically on lighting
properties within a scene.

Lighting estimation
When using this lighting estimation method, the computer and AR
development framework work together to analyze the:
● Brightness
● Light color
● Color correction values
● Main light direction
● Ambient intensity
● Ambient occlusion

Brightness:
For each pixel on the display, the average lighting intensity can be
calculated and then applied to all digital objects is called pixel
intensity, and it adjusts the overall brightness based on calculating the
average overall available light in the environment.
Light color and color correction
The white balance can be detected and checked dynamically to allow
for color correction of any digital objects within the scene to react to
the color of the light.
It will enhance the color balance allows changes to occur smoothly and
more naturally instead of abrupt adjustments(the illusion or realism).

Main light direction: By
identifying the main directional light, the
software ensures that digital objects added to
the scene will have shadows cast in the same
direction as other objects around them.
It also enables specular highlights and
reflections to be correctly positioned on the
object to match the environment.
If you want to make sure that all the
shadows and highlights are following
consistently from the singular directional
light.

Having this consistent direction of light may seem minor, but it is
something that the brain sees and perceives without us even realizing.
The intensity of the light and also the falloff of those shadows.
Light Direction:
3D rendering showing a prominent main light source that can be seen
as it enters through the window opening.
Ambient intensity: how multiple lights can work together to create a
full lighting setup, As an important part of the light estimation scan,
ARCore can re-create what Google calls “ambient probes” that add
an ambient light to the full scene coming from a broad direction to
create a softer overall tone.

It works with the directional light to help
the digital objects blend more seamlessly
into the scene.
Again, it is about replicating or
imitating the real-world scene.

Ambient occlusion
Every time you add a computer-generated light,
it will produce a generated shadow. Those
shadows need to fall into the physical space
to make to make them believable.
● When you add an ambient light, it should
both cast a shadow on the object and
have the shadows occlude all around it.
● When the light hits the object itself, such
as on a piece of fabric, each wrinkle
should show a shadow.

Something like a brick wall should have shadows created inside of every
groove. Ambient light will hit multiple surfaces, and each one will create
their own shadow. This shadow casting is called ambient occlusion.
Ambient occlusion Simulation of shadows both on an object itself and
also on the other objects around it created by the addition of an ambient
light source.

Environmental reflections
Take a look at the reflections to see environmental reflections, or places
where pieces of the space are reflected.
Depending on the material of the objects, the relative reflectiveness
will change.
When you add a digital object to a scene, especially an object that has a
metallic or glass surface, it should respond to the light around it in the
form of a reflection.
For these virtual objects, the reflections have to happen in real time and
adjust according to the space to lend realism and believability to the
objects.

Reflection. A metallic sphere reflects images from the environment
surrounding it.When creating your 3D objects, you can adjust several
properties to affect how reflective an object is.
● Diffusion
● Roughness
● Metalness
Diffusion Even distribution of light across an object’s surface.
Each material you apply to your 3D object has a base color or texture.
Adjusting an object’s diffusion property affects the amount and color of
light that is reflected at each point of an object.

Diffusion: The diffusion stays consistent as you look around the object.
It is a property that is applied equally along the material’s surface.
Because this is an even distribution of light, it will result in a
nonreflective surface.
In 3D software, the default diffusion color is white, unless you change it
otherwise.

Roughness:
♣ If the surface is smooth and shiny like a car’s chrome bumper, it will be
highly reflective. But if the surface has tiny bumps and cracks along the
surface like the surface of a rock or brick, then it will be less reflective.
♣ This roughness property can change how matte or shiny an object can
become. Increasing the roughness and using brighter colors will diffuse
the light across the surface more, making it appear matte or rough.
♣ Reducing the amount of roughness, in addition to using darker colors, will
cause the material to appear smooth and shiny.
♣ Materials that are shiny will also create specular highlights.

Metalness:- For the physical surface of an object, you can set multiple properties
to determine how metallic or nonmetallic it is.
♣ Light that cannot travel through an object will reflect back, and more
metallic surfaces will produce sharper reflections.
♣ The grazing angle makes the surface appear more or less mirror-like.
♣ If the surface reflects the light sharply and has a mirror-like quality, it will
appear more metallic.
♣ These properties can be adjusted to lower or increase the metalness to
change the appearance of an object’s surface.
♣ If the surface is made more metallic and mirror-like, this will increase the need
for environmental reflections on the object’s surface.
♣ Reflective surfaces also pick up colors and reflect images.

Reflection: Light and color work together to create a sense of depth and
realism.
♣ To create and design digital objects, they should be reflective of the
environment around them.
♣ This process starts with selecting the appropriate color appearance mode
for your experience, works through adding and adjusting any custom
lighting options, and should come to life by adapting to the physical
spaces that the object augments.

LIGHTING DESIGN
To creating some lighting setups.
To get started, you need to add a sphere to your scene. Do not apply any
materials to the sphere so you can see the way the lights change the
surface. Using the lighting setups create the following: One-point light
(soft and hard), Three-point light (add a key, fill, and rim light)
● Sunlight, Backlight and Your own custom setup
For each lighting setup you create, go to your Render options, and save a
PNG file for each. You can name each lighting setup accordingly. Save
these images in a folder, and use them for reference as you work on more
complex 3D models. This will create a lighting reference library for you.

Sound Design: Hearing what you see
Exploring how sound plays an essential role in creating an immersive experience. From
how sound is created to how we can re-create it in a digital space, there are a lot of
exciting things happening within 3D sound experiences.
HEARING WHAT YOU SEE It is important first to understand how we hear so that we
can then look at the best ways to re-create that sound to create realism in a soundscape.
SPATIAL SOUND Just as in physical spaces, sound has direction and distance. There
is different technology that will help create a sense of 3D sound.
AUGMENTED AUDIO Just as we can add a layer of visuals into a user’s view, we can
also add a layer of ambient audio to what they hear.
VOICE EXPERIENCES With XR devices becoming more hands free, voice is becoming an
intriguing way to interact with a computer.

HEARING WHAT YOU SEE
Listening
Sound localization
How do we hear sound?
Loudness, pitch
Raw audio to be captured and edited
Music and voice audio
Transferable and sharable audio
How is sound useful?
How do we use sound in XR?
● Ambient sound
● Feedback sound
● Spatial sound

HEARING WHAT YOU SEE
Find a place that you can sit comfortably for about
five minutes, and bring a notebook and something to
write with.
It can be inside or outside—it really can be anywhere.
1. Close your eyes, and be still. Bring your awareness to listening. Try to
avoid moving your head as you do this. Don’t turn your neck toward a
sound; try to keep your neck at a similar orientation.
2. Listen for what you hear. See if you can identify what sounds you are
hearing.

3. Then go one step further and try to identify where those sounds are
coming from.
Are they close? Far?
Which direction are they coming from?
Keeping yourself as the central axis, do you hear them in front of you?
Behind you?
To the left or right of you?
Up high or down low?
4. When five minutes are up, draw out what you heard by placing a circle in
the middle of the page to represent you, and then map out all the sounds
that you heard around you in the locations you heard them from.

♣ Sound localization: start to pay attention to where you place the sound in context
to yourself. Also consider how you determine the source of the sound. This is called
sound localization. The ability of a listener to identify the origin of a sound based
on distance and direction.
♣ How do we hear sound?
♣ Sound is created through the vibration of an object. This causes particles to
constantly bump into one another, sending vibrations as sound waves to our ears
and, more specifically, to our eardrums.
♣ When a sound wave reaches the eardrum, it too will vibrate at the same rate.
♣ To do this, the sound has to travel from the ear to the brain along the
auditory nerve.
♣ Sound requires an element or medium to travel through, such as air,
water, or even metal.

Loudness The intensity of a sound, measured in relation to the space that
the sound travels.
we can detect a wide range of sound, we need a way to measure the intensity
of the sound. This is called loudness, which uses the unit of decibels (dB) to
measure how loud or soft a sound is.
To help you add some perspective to dB measurements:
● A whisper is between 20 and 30 dB.
● Normal speech is around 50 dB.
● A vacuum cleaner is about 70 dB.
● A lawn mower is about 90 dB.
● A car horn is about 110 dB.

Pitch:-
Sound changes depending on how fast the object is vibrating. The faster the vibration, the higher
the sound. This pitch is measured using frequency, or how many times the object vibrates per
second.
Pitch The perceived highness or lowness of a sound based on the frequency of vibration.
Frequency is measured in hertz (Hz).
The human hearing ranges from 20 to 20,000 Hz. However, our hearing is most sensitive to
sounds ranging in frequency between 2000 and 5000 Hz.
Raw audio to be captured and edited: Uncompressed formats allow you to work with the
highest quality file, and then you can compress the files to be smaller afterward.
● Music and voice audio: Lossless audio compression files maintain the audio quality but
also the larger file sizes.
● Transferable and sharable audio: Lossy audio compression formats produce smaller files
sizes, which facilitates sharing.

How is sound useful?
● Sound is much like a ripple in water; it starts in a central spot, and
then it slowly extends out gradually getting smaller and smaller (or
quieter and quieter) as it moves away from the center.
● Even if you hear a sound from far away, you can still detect where the
sound is coming from or at least an approximate direction.
● You can tell the difference between footsteps walking behind you or
down another hallway.
● You can tell the difference between a crowded restaurant and an empty
one from the lobby, all because of the sound cues of chatter.

● how light can be used to understand space and depth, and sound can
also be used to calculate distance and depth.
● Through the use of SONAR (sound navigation and ranging), you can
measure the time it takes for a sound to reflect back its echo.
How do we use sound in XR?
There are many ways that sounds play a role in our understanding of space.
Within XR there are three main ways sound is used.
● Ambient sound
● Feedback sound
● Spatial sound

Ambience for reality
In order to really create a sense of “being there,” sound adds another layer
of realness.
When you see a train approaching, that comes with the expectation of
hearing the wheels on the tracks, the chugging sound of the engine, steam
blowing, and the whistle, horn, or bell.
Listen for sounds that you can mimic to re-create the scene. Sounds that are
noise intensive and have consistent looping, such as fans, wind, or waves,
do not work as well in this medium, however, so you want to avoid them.
Sounds that have a start and stop to them will be more effective and less
intrusive.

When designing for AR and MR, you can rely more on the natural ambient noise
that will be in the user’s physical space.
Providing feedback
● For the user experience, sound can be a great way to provide feedback
about how the user is interacting within space.
● Hearing a sound when you select an interactive element will reinforce that
you have successfully activated it.
● These sounds can be very quiet, such as a click, or louder, such as a chime.
● Just be sure to use these sounds in a consistent way, so that the user will
start to associate the sounds with their actions.
● Sound cues can guide interactions.

● You can also use sound to direct the user to look or move to another
location, to make sure they see an object that may not be in their
gaze.
● It can also be used in VR to alert the user when they are close to the
edge of their space boundaries.
Creating depth
Because our understanding of sound is 3D, it makes sense that you would
also re-create sound to reflect depth.
It also provides information to the user, such as how close or far away an
object is.
This topic is such an essential part of XR sound design, we are going to
dive into how make your sound have depth next.

Spatial sound
Single-point audio capture:
Binaural
Ambisonic
Paradise case study
Virtual Design Environment
Behind the Scenes

To re-create sound in a spatial environment, look at two components.
● How the sound is recorded
● How the sound is played back through speakers or headphones
The traditional types of audio recordings are mono and stereo.
Mono sound is recorded from a single microphone.
stereo is recorded with two microphones spaced apart.
Stereo is an attempt to create a sense of depth by having different sounds
heard on the right and left sides of a recording.
It is intended to create a sense of 3D audio.

♣ The concept of 360-degree sound has been experimented with for years, looking
at how surround sound can allow sound to come from different speakers all
around the room creating a full 3D audio experience.
♣ Single-point audio capture:
♣ stereo recordings sound even more natural, one option is a binaural audio
recording format.
♣ To record binaurally, you record from two opposite sides and place each
microphone inside a cavity to replicate the position and chamber of an ear.
♣ Headphones are needed to accurately listen to binaural sound.
Binaural A method of recording two-channel sound that mimics the human ears
by placing two microphones within a replicated ear chamber positioned in
opposite locations to create a 3D sound.

Ambisonic audio uses four channels (W, X, Y, and Z) of sound
versus the standard two channels.
An ambisonic microphone is almost like four microphones in
one. You can think of this as 2D (stereo) versus 4D (ambisonic)
sound.
Ambisonic microphones have four pickups, each pointed and
oriented in a different direction making a tetrahedral
arrangement.
Ambisonic Microphone. Able to capture audio from four
directions at once, this Sennheiser ambisonic microphone is
creating a spatial audio recording from nature.

Ambisonic A method of recording
four-channel sound that captures a
sphere of sound from a single point to
reproduce 360° sound.
It was developed by the British
National Research Development
Council in the 1970s, more
specifically, by engineer Michael
Gerzon.

Paradise case study
Many XR experiences rely on an individual user experience, where each
person will have their own set of headphones on or be inside their own
virtual space.
This is a feature for which there is more and more of a demand.
In a social situation, the sound may not track with the user.
Paradise is an interactive sound installation and gestural instrument for
16 to more than 24 loudspeakers.
For this collaborative project, Douglas Quin and Lorne Covington joined
their backgrounds in interaction design and sound design to create a fully
immersive sound experience that they optimized for four to eight people.

Paradise case study
The installation allows users to “compose a collage of virtual acoustic
spaces drawn from the ‘natural’ world.” As users move through the space
and change their arm positioning,
sensors activate different soundscapes from wilderness and nature to create
a musical improvisation.
Motions can change the density of sounds, the volume of them, the motion or
placement of the sound in the space, and the overall mix of the sounds
together.

Paradise Experience:- Visitors
react to the interactive
soundscape environment of
Paradise. Venice International
Performance Art Week, 2016.
Photograph used by permission
of Douglas Quin

This experience was reimagined for
both interior and exterior spaces.
Changing the location “creates a
different spatial image,” Quin
explained when I spoke with him
and Covington about the challenges
of the project.

The interior
exhibit required
more planning
based on how
sound would be
reflected and
reverbed by the
architecture of the
space.

Behind the Scenes. This behind-the-scenes screen capture shows the
installation environment for Paradise.
Numbers indicate loudspeakers. The green rectangular blocks are visitors.
The large red circles are unseen zones of sounds that slowly rotate.
Sounds are activated when a visitor breaks the edge of a circle.
The triangles with colored balls are sound sources for any given sound (with
volume indicated by the size of each ball).

Augmented audio
AR Sound
How does it work?
Speaker Closeup
More than a speaker
How can you get your own?
Imagine going for a bike ride while listening to your favorite playlist,
receiving voice-driven instructions, and still being able to hear car engines,
sirens, and horns honking all around you.
This is the power of augmented audio.

Augmented audio The layering of digital sound on top of, but not blocking
out, the ambient sounds of an environment.
Augmented audio, also referred to as open-ear audio, allows you to hear all
the ambient sounds around you while adding another layer of audio on top of
it.
Smartglasses, also known as audio glasses, come in different shapes from a
number of different manufacturers.
Many of these come in a sunglasses option, as they are most likely to be
used outside. However, many come with customizable lens options to
personalize your experience.

Bose was the first to the market and started off demonstrating the technology
using 3D printed prototypes at South by Southwest (SXSW) Conference and
Festival in 2018.
I remember walking by the Bose house in Austin, Texas, where they had
taken over part of a local restaurant to showcase their AR glasses.
I was intrigued. I wanted to know how Bose, known for their high-quality
speakers and headphones, was entering the world of AR.
The experience started by connecting the sunglasses to my phone through
Bluetooth.
Advantage of the processing power of a smartphone, Bose could keep them
lightweight and cool.

One person in the group spotted a famous
actor stepping out of their vehicle for a film
premiere at the festival and was able to tell
everyone else as we continued listening to our
guided tour.
AR Sound. 3D printed prototypes of the
original Bose AR glasses at SXSW 2018.
To be clear, these glasses and similar pairs
from other developers don’t show the user any
visuals. They are just to provide audio.

It allow for voice interactions without needing to take out a phone.
They allow the user to interact hands-free and ears-free.
They are essentially replacements for headphones or earbuds that allow the
user to still hear everything around them at the same time.
How does it work?
Using what is called open-ear technology, a speaker is built into each arm
of the audio glasses.
What helps make them augmented, while also staying private, is the position
and direction of the speakers.
One speaker is placed on each arm of the glasses near the temple so that the
sound is close, but still allows other sounds to enter the ear cavity.

The speakers point backward from
the face, so they are angled right
toward the ears.
This angle reduces how much of
the sound can be heard by others
around the wearer.
Even in a 3D printed prototype
there was not much sound
escaping from the glasses, and very
little could be heard even by those
standing on either side.

Speaker Closeup. The speakers on the Bose
AR glass prototypes are near the ear.
In addition to the speakers themselves, there is
also a head-motion sensor built in that can
send information from the multi-axis points to
your smartphone.
This allows the app to know both the wearer’s
location as well as what direction they are
looking.

More than a speaker
Listening is only half of the conversation. To allow for user feedback,
these glasses also include a microphone.
This allows the glasses to connect with the user’s voice assistant (more on
this in the next section).
Once again, this function helps maintain the hands-free functionality.
It also allows the glasses to be used for phone calls and voice memos for
those who want to communicate on the go.
Many models have the option to turn the microphone feature on and off for
privacy when needed. This is an important consideration.

To further customize the experience, one arm of the glasses is equipped with
a multi-function button that you can tap, touch, or swipe.
This allows you change your volume, change tracks, make a selection, and
navigate within an experience—without having to access your phone
directly.

Although Bose has recently announced they would stop manufacturing their audio
sunglasses line, they are still currently available for purchase as of this writing.
They were the first to the market, but decided to not continue manufacturing the
glasses as they didn’t make as much profit as the company had hoped.
When interviewed about, this a Bose spokesperson said, Bose AR didn’t become
what we envisioned. It’s not the first time our technology couldn’t be commercialized
the way we planned, but components of it will be used to help Bose owners in a
different way.
We’re good with that. Because our research is for them, not us.
Roettgers, J. (2020, June 16). Another company is giving up on AR. This time, it’s
Bose. Protocol. www.protocol.com/bose-gives-up-on-augmented-reality.

Since Bose’s first launch, others have stepped up production of their own
version of Bluetooth shades.
Leading the way is Amazon with their Echo Frames, which bring their well-
known Alexa assistant into a pair of sunglasses.
Everything many have learned to love about having a voice-powered home
assistant is now available on the go.
Other options to check out include audio glasses from GELETE, Lucyd,
Scishion, AOHOGOD, Inventiv, and OhO.

If you are looking to use your glasses for more than just audio
communication, some of shades on the market also include cameras allowing
for some action-packed capture. Leading the way in this market is Snapchat
with their Spectacles Bluetooth Video Sunglasses.
Audio glasses might remain as stand-alone audio devices. Augmented audio
may also be incorporated into full visual and auditory glasses. But in either
case, the focus on exceptional sound quality will pave the way.

Augmented audio
AR Sound
How does it work?
Speaker Closeup
More than a speaker

Voice experiences & Power of sound
Voice experiences
● VUI for voice user interface
● NLP for natural language processing
●Not a replacement- Virtual Keyboard
●Context
●Scripts
Power of sound

Voice experiences
Voice is now an interface. Voice interfaces are found in cars, mobile devices,
smartwatches, and speakers.
They have become popular because of how they can be customized to the
user’s environment, the time of day, and the uniqueness of each situation.
Alexa, Siri, and Cortana have become household names, thanks to their
help as virtual assistants.
We are accustomed to using our voice to communicate with other people—
not computers.
it makes sense that companies like Amazon, Apple, and Microsoft try to
humanize their voice devices by giving them names.

It is important to make these interfaces feel conversational to match the
expectations that humans have for any kind of voice interaction.
As stated in Amazon’s developer resources for Alexa, “Talk with them,
not at them.”
This concept has also been supported by Stanford researchers Clifford
Nass and Scott Brave, authors of the book Wired for Speech.
Their work affirms how users relate to voice interfaces in the same way that
they relate to other people.
This makes sense, because that is the most prominent way we engage in
conversation, up until this point.

Voice user interface The use of human speech recognition in order to
communicate with a computer interface.
Alexa is one example of a voice interface that allows a user to interact conversationally.
The challenge of this, of course, is that when we speak to a person, we rely on context
to help them make sense of what we are saying.
Natural language processing, or understanding the context of speech, is the task
that an NLP software engine performs for virtual-assistant devices.
The process starts with a script provided by a VUI designer.
Just as you’d begin learning a foreign language by understanding important key words,
a device like Alexa must do something similar.
This script allows the user to train an assistant to an experience, or skill, as it is called
in VUI design.

Natural language processing
The use of artificial intelligence to translate human language to be
understood by a computer.
With many XR experiences linking to a smartphone or even promoting
hands-free as an added benefit, that opens up the potential for other ways for
users to interact within an experience.
If you are relying on tapping into smartphone technology, then you first
need to understand how to design for it.
VUIs are not reliant on visuals, unlike graphic user interfaces (GUIs).
The first thing to understand is that voice interactions should not be
viewed as a replacement for a visual interface.

Not a replacement
It is important not to get into a mindset that a voice interaction can serve as a
replacement for a visual interaction.
Example, adding a voice component is not an exact replacement for providing
a keyboard.
You also need to be aware that the design approach must be different.
If you show a keyboard to a user, they will likely understand what action to
complete thanks to their past experiences with keyboards.
A keyboard, in and of itself, will communicate to the user that they need to
enter each letter, number, or symbol.
If they are able to do this with both hands, like on a computer, it may be an
easy enough task.

But if they have to enter a long search term or password using an
interface where they have to move a cursor to each letter individually, this
task may be greeted with intense resentment.
One way to overcome this daunting task is to provide a voice input option
instead.
It is often much easier to say a word than type it all out.
However, the process of inputting data this way is much different than
with a traditional QWERTY keyboard or even an alphabet button
selection, and it not as familiar.

When a user sees the letters
of the alphabet or a standard
QWERTY keyboard, they
connect their past experiences
with it, so they can easily start
to navigate to the letter they
want to choose.
But when you are relying on
voice, the user will connect to
their past communication with
other people as how to interact.

Virtual Keyboard. Concept communicating
with friend via screen hologram with full
QWERTY keyboard.
There needs to be something in the UI that
communicates to the user that they can use
their voice to interact.
This is often shown through the use of a
microphone icon.
However, it can also come in the form of speech.
One way to let someone know that they can
speak is by starting the conversation with a
question such as “How can I help you?”

Depending on the device, and where it will be used, design this experience to match what
will work best to communicate to the user that they can use their voice and to start the
conversation.
What do you do first in a conversation?
Before you speak, you might make sure the person you are speaking to is listening. But
if you are speaking to a computer, you don’t have body cues or eye contact to rely on. So,
this active listening state needs to be designed.
Though most of the conversation experience will not use visuals, this is one area where a
visual provides great benefit.
Using a visual to let the user know that the device is listening can help substitute for the
eye contact they are used to when talking with another person.
This can be a visual change, such as a light turning on or a colorful animation, so they
know that what they are saying is being heard.

Tip
Provide visual cues to provide feedback to the user. A great place for this is to
communicate that a device is ready and listening.
Companies like Apple have created their own custom circular animations that
they use across all their devices;
when a user sees Apple’s colorful circle of purples, blues, and white, they connect
it with a voice interaction.
Seeing this animation communicates that the device is ready and listening for a
voice command.
All of this happens instead of a keyboard appearing. So, it isn’t a replacement, but
rather a totally different way of communicating, and therefore in need of a totally
different interface and design.

Context
When people communicate, we use our knowledge of the context to create a
shared understanding. Once the user knows the device is listening, they may
know to start talking, but how does the user know what to say or even what is
okay to say without any prompts?
With a voice interface there is no visual to show what the options are.
It is best practice to have some options voiced, such as “you can ask...” followed
by a few options.
You may be familiar with the options provided by a teleprompt: “Press 1 to talk to
HR, press 2 to talk to the front desk.”
Those are often very frustrating, because they are very one sided and not
conversational. The goal here is to start by understanding the user’s goal.

This is where everything we have been talking about is coming together.
Once you have identified the why in your project, planned out roughly how it
might look, done user research, and created a user flow, you can start to
predict some options that a user may be looking for.
You can start off by letting the user know what their options are based on
this research.
This can be laid out in a set of questions or by asking an open-ended
question.
When you record an interview with someone, it is the best practice to ask
open-ended questions or compound questions.
The reason is that you want the person to answer with context.

If you ask two questions in one, a compound question, it is a natural
tendency for them to clarify the answer as they respond.
Perhaps you ask “What is your favorite way to brew coffee, and what do you
put in it?” Instead of answering “French press and with cream,” it is likely
that they will specify which of the questions they are answering within the
answer itself.

We’re discussing question and answer methods here because such exchanges
point out an important way that humans communicate.
We like to make sure that our answers have the correct context to them. This
is especially true when there is more than one question asked.
So, a likely response would be, “My favorite way to brew coffee is using a
French press, and I like just a little cream in it.”
Traditional media interviews don’t include the questions—so it’s important to
get context in the answer.

The need for context is important to understand as it relates to voice
interfaces:
Humans may not provide needed context in their voice commands.
Having the computer ask questions that are open-ended or have multiple
options will trigger the user to provide more context in their answer, which
will help the device more successfully understand what the user is asking.
Using the power of machine learning and natural language processing,
the device creates a system that recognizes specific voice commands.
These commands must be written out as scripts.

Scripts
Think about all the different ways within just the English language someone can
say no:
nope, nah, no way, not now, no thanks, not this time...
And this is just to name a few. You also need to consider what questions the user
may ask and the answers they may give.
This requires anticipating what the user will say and then linking that response to
activate the next step.
With a traditional computer screen, the user has a limited number of options based
on the buttons and links you provide.
With the input of a click or a tap, the computer knows to load the connected screen
based on that action. With voice, the interaction is reliant only on spoken language.

As part of the voice interface design, an
important step of the process is to create a
script.
This script should embrace the dynamic qualities
of conversation.
A successful script should go through multiple
levels of user testing to identify questions that
users answer—and also all the different ways
they answer.
When the user isn’t given a set number of
options to choose from, the script helps translate
the human response into something actionable
by the computer.

Script Sample. Sample answers collected to show possible answers to the
question “How have you been?”
All likely answers need to be collected to help a voice assistant understand how to
respond based on each possible answer.
While it is easy to tell a computer what to do if someone says “yes” or “no,” it is
less likely that the user will stick to these words.
Computers may easily be able to understand yes and no commands, but what
happens when a user, who is speaking as they always do to other people, says “I’m
going to call it a day.” Or “I’m going to hit the sack.”
These idioms are not going to be understood by a computer, unless they are
taught them. Without understanding the cultural context, it could be understood
that you are going to physically hit a bag, instead of go to sleep.

How do you anticipate responses of users to build into a script? You ask them,
and you listen to them.
How do you anticipate responses of users to build into a script? You ask them,
and you listen to them.
Use a script and thorough user testing to collect anticipated responses to help
keep the experience complete.
Multiple rounds of quality assurance testing must be completed throughout the
whole process.
Think of how color, image choice, and copy set the tone for a design in print or on
the web. In the same way, sound quality, mood, and content of the responses of
the voice will set the tone for the voice experience.
As you can imagine, this is no small task.

To do this well requires a team of people dedicated to designing these VUI skills.
However, scripts that are created have components that can be used across the different
skills.
The conversational experiences we have will start to build a relationship with the device
and will also help establish a level of trust.
These experiences have a way of making an experience personal. Think about how nice it
is when someone knows and uses your name.
What if they could also learn and know your favorite settings and words? What if they
could mimic these preferences as they guide you, provide clear instructions, and help
reduce your anxiety by saying the right words?
This is what voice experiences can do. They are rooted in our experiences of
conversations with friends and colleagues, so it is no surprise that we start to trust them
like one too

Power of sound
Sound design should not be an afterthought; it makes or breaks the
experience.
Once you start to notice how sound plays a role in your physical world, you
can start to design ways for sound to create more immersion in your XR
experiences.
Using audio can help an experience feel more real, enhance your physical
space, or even help you interact with a computer interface, hands-free.

SOUND LOCALIZATION DESIGN
In the beginning of this chapter, you played the role of the listener. You
directed your awareness to the sounds that happened around you.
This time, you can take what you learned from that experience, and what you
have learned in this chapter, to design your own soundscape.
To do this, draw a chart similar to the sound localization diagram you
created from your listening experience.
However, this time you are going to design what sounds will be happening,
and where.

● Think about where the experience will be happening, and if it is for VR
or AR, as that will determine how much ambient sound you will need
to plan for.
● Think about the distance and intensity of the sound from the user’s
perspective.
If you want the extra challenge, you can then record the sounds and bring
them into a sound editor of your choice, such as Apple Logic Pro or Adobe
Audition, to start editing them.

To create a full immersive experience, you
will need to bring the edited sounds into a
program, such as Unity Pro or Unreal
Engine, that will allow you to spatial map
out the location of the sounds.

22CDH01-UNIT V ER DEVELOPMENT -IMMERSIVE DESIGN THEORY

More Related Content

Similar to 22CDH01-UNIT V ER DEVELOPMENT -IMMERSIVE DESIGN THEORY (20)

More from GOWSIKRAJA PALANISAMY (20)

Recently uploaded (20)

22CDH01-UNIT V ER DEVELOPMENT -IMMERSIVE DESIGN THEORY