ShopTalk: Toward Independent Shopping by People with Visual Impairments
0 likes231 views
Shoptalk is a system designed to assist individuals with visual impairments in finding shelved products in grocery stores using verbal directions and layout descriptions. It employs a topological map and a barcode connectivity matrix to navigate the store without needing additional hardware, while experiments showed a 100% success rate for participants in retrieving products. The study demonstrated significant improvements in distance, time, and barcode scans needed over repeated shopping tasks.
![ShopTalk: Toward Independent Shopping by People with
Visual Impairments
Vladimir Kulyukin John Nicholson Daniel Coster
Dept. of Computer Science Dept. of Computer Science Dept. of Mathematics and
Utah State University Utah State University Statistics
Logan, Utah, U.S.A. Logan, Utah, U.S.A. Utah State University
vladimir.kulyukin@ john.nicholson@ Logan, Utah, U.S.A.
usu.edu aggiemail.usu.edu daniel.coster@usu.edu
ABSTRACT shifts to the search space where a small amount of locomo-
ShopTalk, a proof-of-concept system designed to assist indi- tion may be required to place the shopper directly within
viduals with visual impairments with finding shelved prod- reach of the target product. When the target product is
ucts in grocery stores, is built on the assumption that sim- within reach, the product is considered to be in the shop-
ple verbal route directions and layout descriptions can be per’s haptic space which requires no locomotion, because the
used to leverage the O&M skills of independent visually im- shopper can now physically grasp the target product.
paired travelers to enable them to navigate the store and
retrieve shelved products. This paper introduces ShopTalk 2. SOFTWARE AND HARDWARE
and summarizes experiments performed in a real-world su- ShopTalk guides VI shoppers through a store to specific
permarket. products, requiring no additional hardware instrumentation
beyond what is already installed in the store. ShopTalk uses
Categories and Subject Descriptors the assumption that simple verbal route directions and store
K.4 [Computers and Society]: Social Issues—Assistive layout descriptions can be used to leverage the orientation
technologies for persons with disabilities and mobility (O&M) skills of independent VI travelers en-
abling them to navigate the store and retrieve products.
Two data structures represent the store environment. A
General Terms topological map models the locomotor space using nodes to
Experimentation, Human Factors, Design represent decision points and edges to store movement in-
structions. The topological map is the only software instru-
1. INTRODUCTION mentation requirement for ShopTalk and is built by walking
through the store, noting decision points of interest, and
Shopping complexes top the list of the most functionally
then representing them in a graph. The other structure,
challenging environments for visually impaired (VI) individ-
barcode connectivity matrix (BCM), takes advantage of the
uals [7]. A typical modern supermarket stocks an average
inventory systems used by many grocery stores. These sys-
of 45,000 products, and has a median store size of 48,750
tems place barcodes labels on the shelves immediately be-
square feet [1]. One can distinguish two types of grocery
neath every product. The BCM associates each shelf bar-
shopping. In small-scale shopping, the shopper buys only a
code with five types of location information: 1) the aisle, 2)
few items that can be carried by hand or in a hand basket;
the side of the aisle, 3) the shelf section, 4) the shelf, and 5)
large-scale shopping necessitates the use a shopping cart [4].
the relative position of the product on the shelf.
The scope of this paper is limited to finding the desired gro-
ShopTalk’s hardware (see Fig. 1) consists of a computa-
cery items in a supermarket when small-scale shopping for
tional unit, a small headphone, a numeric keypad, a wireless
products stocked on the aisle shelves.
barcode scanner and its base station, and a USB hub con-
We view the process of shopping as a shopper moving
necting all components. The user wears a small backpack to
through space, initially defined as a spatial dichotomy [4],
help carry the components. The barcode scanner, carried in
but since extended to a trichotomy [3]. In the locomotor
a shopping basket and retrieved when needed, has stabiliz-
space, the shopper travels from her current location to the
ers attached to take advantage of the fact that many stores
general area, somewhere in an aisle, of the target product, i.e.
have shelves that curl down with a small lip at the bottom.
the next product on the shopping list. Next, the shopper
The stabilizers rest on the shelf lips when a shopper is scan-
ning barcodes, making it easier for the shopper to align the
scanner with shelf barcodes.
ShopTalk issues route instructions in two modes: loca-
tion unaware mode (LUM) and location aware mode (LAM).
LUM instructions, used in the locomotor space, are gener-
ated using the topological map. LUM is location-unaware
because there are no sensors to detect the shopper’s current
Copyright is held by the author/owner(s).
ASSETS’08, October 13–15, 2008, Halifax, Nova Scotia, Canada. location. Instead, the shopper detects environmental cues to
ACM 978-1-59593-976-0/08/10. make sense of the verbal instructions. There is some research](https://image.slidesharecdn.com/shoptalktowardindependentblindshoppingbypeoplewithvisualimpairmentsassets08-111221142327-phpapp01/85/ShopTalk-Toward-Independent-Shopping-by-People-with-Visual-Impairments-1-320.jpg)
![had each participant shop for the three products five times.
The 117 meter route began at the store entrance, went to
each of the three products, and ended at a cashier lane.
The experiment was designed to test five hypotheses. Hy-
pothesis 1 (H1 ) was that using only LUM route directions,
VI shoppers can successfully navigate the locomotor space
in a store. Hypotheses 2 (H2 ) was that verbal LAM instruc-
tions based on the BCM are sufficient to guide VI shoppers
to target products in the search and haptic spaces. The re-
maining three hypotheses were that as participants repeat-
edly perform a shopping task, the total distance (H3 ) and
the total time (H4 ) needed to find the products approaches
the those of a VI shopper guided by a sighted person, and
the number of barcode scans needed to find the products
decreases (H5 ).
The product retrieval success rate was 100%. All ten par-
ticipants were able to find all three products in every run
providing evidence in support of H1 and H2 . ANOVA anal-
ysis for the collected data found statistical significance in
decreases in distance, time, and number of barcode scans
Figure 1: The ShopTalk hardware. supporting H3 , H4 , and H5 . Figure 2 shows how the aver-
age total time fell over repeated runs; similar results were
1200 seen for distance and barcode scans. Fuller data analysis
all participants
participants w/ low-level vision
participants w/ complete vision loss
can be found in [5].
time with sighted guide
1000
4. ACKNOWLEDGMENTS
The first author would like to acknowledge that this re-
average run time (seconds)
800
search has been supported, in part, through NSF grant (IIS-
600 0346880), the National Eye Institute of the National Insti-
tutes of Health under Grant (1 R41 EY017516-01A1).
400
5. REFERENCES
200
[1] Food Marketing Institute Research. The Food Retailing
Industry Speaks 2006. Food Marketing Institute, 2006.
0
0 1 2 3 4 [2] F. Gaunet and X. Briffault. Exploring the functional
run id
specifications of a localized wayfinding verbal aid for
blind pedestrians: Simple and structured urban areas.
Figure 2: The average total run time for each run. Human-Computer Interaction, 20(3):267–314, 2005.
[3] V. Kulyukin, C. Gharpure, and D. Coster.
Robot-assisted shopping for the blind: Proof-of-concept
evidence [2, 6] that people with visual impairments share
design and feasibility evaluation. Assistive Technology.
route descriptions and guide each other over cell phones.
To appear.
Following LUM instructions is conceptually no different from
[4] V. Kulyukin, C. Gharpure, and C. Pentico. Robots as
being guided on a cell phone.
interfaces to haptic and locomotor spaces. In HRI ’07:
A barcode scan switches the mode of instruction from
Proceeding of the ACM/IEEE international conference
LUM to LAM which is reserved for the search and haptic
on Human-robot interaction, pages 325–331, New York,
spaces. When a barcode is scanned, the exact location of
NY, USA, 2007. ACM.
the shopper is known, and ShopTalk issues location aware
instructions on how to proceed to the target product. In [5] V. Kulyukin, J. Nicholson, and D. Coster. Shoptalk:
the search space, barcode scans use the BCM to guide the Toward independent shopping by people with visual
shopper through the search space to the haptic space. In impairments. Technical Report USU-CSATL-1-04-08,
the haptic space, the process of scanning guides the shopper Department of Computer Science, Utah State
to the target product’s exact location on the shelf. University, April 2008.
[6] V. Kulyukin, J. Nicholson, D. Ross, J. Marston, and
F. Gaunet. The blind leading the blind: Toward
3. EXPERIMENTS AND RESULTS collaborative online route information management by
Ten participants were recruited from the local area. Three individuals with visual impairments. In Proceedings
reported total vision loss, the remaining seven reported par- from the AAAI Spring Symposium, Palo Alto, CA,
tial vision. With the cooperation of a local supermarket, the March 2008. AAAI.
locations of shelf barcodes were recorded for three aisles (9, [7] R. Passini and G. Proulx. Wayfinding without vision:
10, and 11), a total of 4,297 products. One product was ran- An experiment with congenitally totally blind people.
domly chosen from each aisle for a total of three products. Environment and Behavior, 20(2):227–252, 1988.
The experiment, performed during regular business hours,](https://image.slidesharecdn.com/shoptalktowardindependentblindshoppingbypeoplewithvisualimpairmentsassets08-111221142327-phpapp01/85/ShopTalk-Toward-Independent-Shopping-by-People-with-Visual-Impairments-2-320.jpg)
More Related Content
Similar to ShopTalk: Toward Independent Shopping by People with Visual Impairments (20)
ShopTalk: Toward Independent Shopping by People with Visual Impairments
- 1. ShopTalk: Toward Independent Shopping by People with Visual Impairments Vladimir Kulyukin John Nicholson Daniel Coster Dept. of Computer Science Dept. of Computer Science Dept. of Mathematics and Utah State University Utah State University Statistics Logan, Utah, U.S.A. Logan, Utah, U.S.A. Utah State University vladimir.kulyukin@ john.nicholson@ Logan, Utah, U.S.A. usu.edu aggiemail.usu.edu [email protected] ABSTRACT shifts to the search space where a small amount of locomo- ShopTalk, a proof-of-concept system designed to assist indi- tion may be required to place the shopper directly within viduals with visual impairments with finding shelved prod- reach of the target product. When the target product is ucts in grocery stores, is built on the assumption that sim- within reach, the product is considered to be in the shop- ple verbal route directions and layout descriptions can be per’s haptic space which requires no locomotion, because the used to leverage the O&M skills of independent visually im- shopper can now physically grasp the target product. paired travelers to enable them to navigate the store and retrieve shelved products. This paper introduces ShopTalk 2. SOFTWARE AND HARDWARE and summarizes experiments performed in a real-world su- ShopTalk guides VI shoppers through a store to specific permarket. products, requiring no additional hardware instrumentation beyond what is already installed in the store. ShopTalk uses Categories and Subject Descriptors the assumption that simple verbal route directions and store K.4 [Computers and Society]: Social Issues—Assistive layout descriptions can be used to leverage the orientation technologies for persons with disabilities and mobility (O&M) skills of independent VI travelers en- abling them to navigate the store and retrieve products. Two data structures represent the store environment. A General Terms topological map models the locomotor space using nodes to Experimentation, Human Factors, Design represent decision points and edges to store movement in- structions. The topological map is the only software instru- 1. INTRODUCTION mentation requirement for ShopTalk and is built by walking through the store, noting decision points of interest, and Shopping complexes top the list of the most functionally then representing them in a graph. The other structure, challenging environments for visually impaired (VI) individ- barcode connectivity matrix (BCM), takes advantage of the uals [7]. A typical modern supermarket stocks an average inventory systems used by many grocery stores. These sys- of 45,000 products, and has a median store size of 48,750 tems place barcodes labels on the shelves immediately be- square feet [1]. One can distinguish two types of grocery neath every product. The BCM associates each shelf bar- shopping. In small-scale shopping, the shopper buys only a code with five types of location information: 1) the aisle, 2) few items that can be carried by hand or in a hand basket; the side of the aisle, 3) the shelf section, 4) the shelf, and 5) large-scale shopping necessitates the use a shopping cart [4]. the relative position of the product on the shelf. The scope of this paper is limited to finding the desired gro- ShopTalk’s hardware (see Fig. 1) consists of a computa- cery items in a supermarket when small-scale shopping for tional unit, a small headphone, a numeric keypad, a wireless products stocked on the aisle shelves. barcode scanner and its base station, and a USB hub con- We view the process of shopping as a shopper moving necting all components. The user wears a small backpack to through space, initially defined as a spatial dichotomy [4], help carry the components. The barcode scanner, carried in but since extended to a trichotomy [3]. In the locomotor a shopping basket and retrieved when needed, has stabiliz- space, the shopper travels from her current location to the ers attached to take advantage of the fact that many stores general area, somewhere in an aisle, of the target product, i.e. have shelves that curl down with a small lip at the bottom. the next product on the shopping list. Next, the shopper The stabilizers rest on the shelf lips when a shopper is scan- ning barcodes, making it easier for the shopper to align the scanner with shelf barcodes. ShopTalk issues route instructions in two modes: loca- tion unaware mode (LUM) and location aware mode (LAM). LUM instructions, used in the locomotor space, are gener- ated using the topological map. LUM is location-unaware because there are no sensors to detect the shopper’s current Copyright is held by the author/owner(s). ASSETS’08, October 13–15, 2008, Halifax, Nova Scotia, Canada. location. Instead, the shopper detects environmental cues to ACM 978-1-59593-976-0/08/10. make sense of the verbal instructions. There is some research
- 2. had each participant shop for the three products five times. The 117 meter route began at the store entrance, went to each of the three products, and ended at a cashier lane. The experiment was designed to test five hypotheses. Hy- pothesis 1 (H1 ) was that using only LUM route directions, VI shoppers can successfully navigate the locomotor space in a store. Hypotheses 2 (H2 ) was that verbal LAM instruc- tions based on the BCM are sufficient to guide VI shoppers to target products in the search and haptic spaces. The re- maining three hypotheses were that as participants repeat- edly perform a shopping task, the total distance (H3 ) and the total time (H4 ) needed to find the products approaches the those of a VI shopper guided by a sighted person, and the number of barcode scans needed to find the products decreases (H5 ). The product retrieval success rate was 100%. All ten par- ticipants were able to find all three products in every run providing evidence in support of H1 and H2 . ANOVA anal- ysis for the collected data found statistical significance in decreases in distance, time, and number of barcode scans Figure 1: The ShopTalk hardware. supporting H3 , H4 , and H5 . Figure 2 shows how the aver- age total time fell over repeated runs; similar results were 1200 seen for distance and barcode scans. Fuller data analysis all participants participants w/ low-level vision participants w/ complete vision loss can be found in [5]. time with sighted guide 1000 4. ACKNOWLEDGMENTS The first author would like to acknowledge that this re- average run time (seconds) 800 search has been supported, in part, through NSF grant (IIS- 600 0346880), the National Eye Institute of the National Insti- tutes of Health under Grant (1 R41 EY017516-01A1). 400 5. REFERENCES 200 [1] Food Marketing Institute Research. The Food Retailing Industry Speaks 2006. Food Marketing Institute, 2006. 0 0 1 2 3 4 [2] F. Gaunet and X. Briffault. Exploring the functional run id specifications of a localized wayfinding verbal aid for blind pedestrians: Simple and structured urban areas. Figure 2: The average total run time for each run. Human-Computer Interaction, 20(3):267–314, 2005. [3] V. Kulyukin, C. Gharpure, and D. Coster. Robot-assisted shopping for the blind: Proof-of-concept evidence [2, 6] that people with visual impairments share design and feasibility evaluation. Assistive Technology. route descriptions and guide each other over cell phones. To appear. Following LUM instructions is conceptually no different from [4] V. Kulyukin, C. Gharpure, and C. Pentico. Robots as being guided on a cell phone. interfaces to haptic and locomotor spaces. In HRI ’07: A barcode scan switches the mode of instruction from Proceeding of the ACM/IEEE international conference LUM to LAM which is reserved for the search and haptic on Human-robot interaction, pages 325–331, New York, spaces. When a barcode is scanned, the exact location of NY, USA, 2007. ACM. the shopper is known, and ShopTalk issues location aware instructions on how to proceed to the target product. In [5] V. Kulyukin, J. Nicholson, and D. Coster. Shoptalk: the search space, barcode scans use the BCM to guide the Toward independent shopping by people with visual shopper through the search space to the haptic space. In impairments. Technical Report USU-CSATL-1-04-08, the haptic space, the process of scanning guides the shopper Department of Computer Science, Utah State to the target product’s exact location on the shelf. University, April 2008. [6] V. Kulyukin, J. Nicholson, D. Ross, J. Marston, and F. Gaunet. The blind leading the blind: Toward 3. EXPERIMENTS AND RESULTS collaborative online route information management by Ten participants were recruited from the local area. Three individuals with visual impairments. In Proceedings reported total vision loss, the remaining seven reported par- from the AAAI Spring Symposium, Palo Alto, CA, tial vision. With the cooperation of a local supermarket, the March 2008. AAAI. locations of shelf barcodes were recorded for three aisles (9, [7] R. Passini and G. Proulx. Wayfinding without vision: 10, and 11), a total of 4,297 products. One product was ran- An experiment with congenitally totally blind people. domly chosen from each aisle for a total of three products. Environment and Behavior, 20(2):227–252, 1988. The experiment, performed during regular business hours,