Recognition of unistroke gesture sequences
1 like708 views
This document presents an approach for recognizing multiple gestures in a single stroke sequence using dynamic time warping. It extends the $1 recognizer to segment an input gesture sequence into its constituent gestures by finding the best match between templates and the input. The approach employs dynamic time warping to calculate distances between templates and input points without restrictions on sequence length or composition. It achieves good accuracy even with a small number of training samples by splicing off matched portions and repeating on remaining input.
![References
[1] Lyddane, Donald. United states attorneys’ bulletin. Technical report, United States Department
of Justice Executive Office for United States Attorneys, May 2006.
[2] J. Yang, Y. Xu, and C. S. Chen. Gesture interface: Modeling and learning. In Robotics and
Automation, 1994. Proceedings., 1994 IEEE International Conference on, pages 1747–1752, 1994.
[3] D. O. Tanguay Jr. Hidden markov models for gesture recognition. Master’s thesis, Massachusetts
Institute of Technology, 1995.
[4] J.O. Wobbrock, A.D. Wilson, and Y. Li. Gestures without libraries, toolkits or training: a $1
recognizer for user interface prototypes. In Proceedings of the 20th annual ACM symposium on
User interface software and technology, pages 159–168. ACM, 2007.
[5] Zygmunt Pizlo. Perception viewed as an inverse problem. Vision Research, 41(24):3145–3161,
November 2001.](https://image.slidesharecdn.com/recognitionofunistrokegesturesequences-121207230456-phpapp02/85/Recognition-of-unistroke-gesture-sequences-10-320.jpg)
Recognition of unistroke gesture sequences
- 1. Recognition of Unistroke Gesture Sequences Presented by Justin Permar & Arvind Krishnaa Jagannathan
- 2. Problem Statement To develop a gesture recognition system, which will segment the input sequence consisting of multiple gestures, drawn in one stroke, into the constituent. Our primary objective in solving this problem is to have a minimal set of training data in order to quickly build a prototype system.
- 3. Related Work • Recognition of individual gestures in unistroke and multistroke sequences of digits – Yang et al: Training HMMs for continuous sequences • Individual gesture recognition : Extracting several features from an input and constructing Hidden Markov Models for each gesture – Tanguay • Recognizing an individual unistroke gesture sequence, using a distance measure from existing templates - $1 Recognizer - Wobbrock et al
- 4. Our Approach – Salient Features • Extend $1 recognizer to identify multiple gestures in a gesture sequence. • Hardest problem is that of segmentation – Classified in literature as “Inverse Perception Problem”. • Based on a “Visual Affinity” approach – a type of geometric match. • Employs Dynamic Time Warping to decide the distance from a given template. • No restriction on gestures in the input sequence – • Arbitrarily long • Can contain any pre-defined gesture. New gestures can also be added to the system.
- 5. Approach – Training mode • In training mode, the user draws a template of either an existing or new gesture. • Only individual gestures are drawn – no need to train the system with gesture sequences. • If misclassification occurs during training, user can indicate the intended gesture. • Score next to the shape (“star”) is based upon a distance measure between the input and the indicated gesture.
- 6. Approach – Match using Dynamic Time Warping • Euclidean distance used as the distance metric between the neighboring points of input sequence and each template. d = (𝑥 𝑘 −𝑥 𝑖 )2 + (𝑦 𝑘 −𝑦 𝑖 )2 where (𝑥 𝑘 , 𝑦 𝑘 ) correspond to the points on the template under consideration and (𝑥 𝑖 , 𝑦 𝑖 ) correspond to the points on the input sequence. • For a given set of input points, if there is a template which gives a score lower than a fixed threshold, then that gesture is recorded as being part of the input. • The identified portion of the input is “spliced off ” and the matching procedure is repeated for the remaining input.
- 7. Approach – Segmentation and Recognition Candidate input points Slide the input set of points Remaining input after splicing No Match Match with “v” (First dot) off the matched template Input Gesture Sequence Blue Dot – Start of the Sequence Red Dots – Points at which sequence is segmented Next set of candidate points Sliding the set of points from remaining input – No Match Match with “Arrow” (Final dot)
- 8. Results and Interpretation Sequence Length Segmentation Sequence Length Accuracy Rate (%) Sequence Length Relaxed Accuracy (%) Accuracy(%) 1 57.5 1 71.5 2 56.19 2 32.16 2 71.5 3 56.37 3 25.72 3 52.27 Table:3 Relaxed Accuracy Table:1 Overall Accuracy – Exact Match Table:2 Segmentation Accuracy 𝑅𝑒𝑝𝑜𝑟𝑡𝑒𝑑 𝐺𝑒𝑠𝑡𝑢𝑟𝑒𝑠 𝑖𝑛 𝑖𝑛𝑝𝑢𝑡 𝑅𝑒𝑙𝑎𝑥𝑒𝑑 𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = 𝑇𝑜𝑡𝑎𝑙 𝐺𝑒𝑠𝑡𝑢𝑟𝑒𝑠 𝑖𝑛 𝑖𝑛𝑝𝑢𝑡 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒 Training Size Vs. Accuracy Rate graph, indicates the following: 1. Even for relatively low training samples (𝑛 = 3) fairly good accuracy rates are reported. 2. An optimum recognition rate is achieved at (𝑛 = 5) 3. Too many templates, cause incorrect recognition due to “confusion” caused by many variations of an individual gesture.
- 9. Discussion • Training only on individual gestures lose features indicating a transition from one gesture to another (such as a pause) • Construct HMMs for each individual gesture in the “gesture library” and then chain HMMs to construct arbitrarily long sequences of gestures. • Compromise training time and computation for improved accuracy.
- 10. References [1] Lyddane, Donald. United states attorneys’ bulletin. Technical report, United States Department of Justice Executive Office for United States Attorneys, May 2006. [2] J. Yang, Y. Xu, and C. S. Chen. Gesture interface: Modeling and learning. In Robotics and Automation, 1994. Proceedings., 1994 IEEE International Conference on, pages 1747–1752, 1994. [3] D. O. Tanguay Jr. Hidden markov models for gesture recognition. Master’s thesis, Massachusetts Institute of Technology, 1995. [4] J.O. Wobbrock, A.D. Wilson, and Y. Li. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th annual ACM symposium on User interface software and technology, pages 159–168. ACM, 2007. [5] Zygmunt Pizlo. Perception viewed as an inverse problem. Vision Research, 41(24):3145–3161, November 2001.