Sign Language Recognition with Gesture Analysis

ISSN 2350-1022
International Journal of Recent Research in Mathematics Computer Science and Information Technology
Vol. 3, Issue 1, pp: (6-11), Month: April 2016 – September 2016, Available at: www.paperpublications.org
Page | 6
Paper Publications
Sign Language Recognition with Gesture
Analysis
1
Suhas Dhavale, 2
Rutwik Gunjal, 3
Prashant Umbare, 4
Sushil Daswadikar,
5
Prof. Hema Kumbhar
1,2,3,4,5
Computer Engineering, PVPIT Bavdhan, Maharashtra, India
Abstract: The main communication methods used by deaf people are sign language, but opposed to common
thought, there is no specific universal sign language: every country or even regional group uses its own set of signs.
The use of sign language in digital systems can enhance communication in both directions: animated avatars can
synthesize signals based on voice or text recognition; and sign language can be translated into various text or sound
forms based on different images, videos and sensors input. The ultimate goal of this research, but it is not a simple
spelling of spoken language, so that recognizing different signs or letters of the alphabet (which has been a common
approach) is not sufficient for its transcription and automatic interpretation. Here proposes an algorithm and
method for an application this would help us in recognising the various user defined signs. The palm images of
right and left hand are loaded at runtime. Firstly these images will be seized and stored in directory. Then
technique called Template matching is used for finding areas of an image that match (are similar) to a template
image (patch). Our goal is to detect the highest matching area. We need two primary components- A) Source image
(I): In the template image in which we try to find a match. B) Template image (T): The patch image which will be
compared to the template image. In proposed system user defined patterns will be having 60% accuracy while
default patterns will be provided with 80% accuracy.
Keywords: Alphabetical Gesture, Audio Output, Colour Detection, Feature Extraction , Gesture Recognition,
Pattern Matching, Specific user defined gestures.
I. INTRODUCTION
It has been difficult for Deaf people to acquire writing system or oral language or sign language. But, the Deaf people
have the right to an identity, language and culture. They have the right to access the available human symbolic
communication, interaction, learning, etc. Sign Language, is the natural language of the Deaf, it is efficient and capable
way of providing complex linguistic functionalities. Deaf culture is basically the term that defines social movement that
regards deafness as a difference in human experience – rather than a deficiency.
The Deaf people face difficulties in acquiring a writing system, be it of the oral language, or of the Sign Language. Sign
Languages share a commonality with other oral languages from minority groups that have their own cultural and
traditional means of maintaining folk language art forms. The remainder of this paper further describes the plight of the
Deaf, and their need for understanding Sign Language; and briefly discusses Gesture or Sign Recognition.
II. LITERATURE SURVEY
Sign Languages are tools for deaf people which can be used use for their communication, information gathering and
access, education needs, among others. Information Systems, whose role should be to facilitate those processes, still do not
present a natural interaction for the deaf. The challenge is to devise a framework with which to work towards tackling
those drawbacks. The present study presents such a framework to support sign language recognition and interaction to
serve as standard that should be used by Computer Vision in order to get back the field's genuine task of developing
effective as well as efficient technologically derived services that take the deaf's needs into consideration towards social
inclusion. [D. R. Antunes, C. Guimaraes, L. S. Garcia, L. E. S. Oliveira, and S. Fernandes, “A framework to support
development of sign language human-computer interaction: Building tools for effective information access and inclusion
of the deaf,”[2]].

ISSN 2350-1022
Page | 7
Paper Publications
The tracking 3D objects articulated is a theoretically interesting and challenging problem. For example, the 3D tracking of
human hands has various numbers of diverse applications including but not limited to human activity recognition,
interaction between Human and Computer, understanding human grasping power, robot learning by demonstration, etc. To
develop an effective and efficient solution, one has to struggle with a number of factors such as the high dimensionality of
the problem, in chromatically manner the uniform appearance of a hand and the severe self-occlusions that occur while a
hand is in action. [I.Oikonomidis, N. Kyriazis, and A. Argyros, “Efficient model-based 3d tracking of hand articulations
using kinect,”[5]].
Image oriented real-time fitting algorithm for the hand, using an object recognition by parts approach, and we use this
hand-modeler in an American Sign Language (ASL) digit recognition application. So, we created a realistic 3D hand
model that represents the hand with 21 different parts. The classification results are stored into a local mode searching
algorithm to estimate the joint locations for the hand skeleton. As an application of the system, we also describe a support
vector machine (SVM) that is based on recognition module for the 10 digits of ASL based on our method, which attains a
recognition rate of 99.9% on live depth images in real-time. [C. Keskin, F. Kira, Y. E. Kara, and L. Akarun, “Real time
hand pose estimation using depth sensors.”[4]]
Here, we propose a model-based approach to the problem. It is an optimization problem that minimizes the discrepancy
between the 3D structure and appearance of hypothesized 3D hand model instances, and actual visual observations. [I.
Oikonomidis, N. Kyriazis, and A. Argyros, “Efficient model-based 3d tracking of hand articulations using kinect”[5]]
A. Architecture:
Use cameras for capturing different angles of the scene and reconstruct a standard view of the actor. We need to train the
system first about the object information. The use of vocabularies with single words for the recognition of sign languages
is therefore inappropriate due to the large amount of data required for training the system. A video database (LIBRAS-HC-
RGBDS) containing all 61 hand configurations of LIBRAS was acquired with the Kinect sensor, using specially developed
acquisition software.
Fig.1: System Architecture
The first step of Image Acquisition as the name suggests is of acquiring the image during runtime. We acquire this image
through integrated webcam. The images will be stored in the directory as soon as they are captured and the image captured
recently will be acquired and will be compared with the images stored for specific letter in the database. The application
picks up all the images specified in database and find the keypoints one by one of each image and finds the number of
matched keypoints, the comparisons with highest matched keypoints in an image will take the advance and will produce
as an output.
In orientation detection we will take the input of hand movement in any form or any gesture orientation will be detected
through the described section of feature extraction. At last when the whole process is complete we will then convert the
gesture into its recognized character or alphabet through application. The single dimensional array of twenty six character
corresponding to alphabets has been passed where the image number stored in database is provided in the array.

ISSN 2350-1022
Page | 8
Paper Publications
Below in Fig N we can see simple implementation flow of each module sequentially.
Fig.N: Implementation Flow
B. Proposed Methodology:
The proposed algorithm consisted of following major steps which are namely Image Acquisition, Background detection
and removal, Colour Detection, Pattern Matching, Vector Calculation, Gesture Database and Gesture Recognition which is
also shown in below given Fig 2.
All the major steps mentioned above are explained in detail in later part of the paper with its proper module working and
what output is expected by specific module.
Fig.2: Data flow diagram.

ISSN 2350-1022
Page | 9
Paper Publications
1. Image Acquisition:
The initial step of Image Acquisition is of acquiring the image during runtime through integrated webcam or external
webcam. As soon as images are captured, they are stored in directory. And then the images stored in directory and recently
captured image are compared and this comparison will give the specific gesture, translated text for the following gesture
and its audio output. Here images will be captured through basic code in .NET of opening a webcam. Capturing of the
image frames per second which will be stored in directory where all input images are stored and compared with recently
captured image.
We have provided application interface with START & Recognize button. Clicking on button, open up the webcam and
when user gives user defined gesture he/she can give specific hand position name in Hand Position Name column.
2. Background Detection and Removal:
2.1. Blob Analysis:
Blob analysis method is aimed at detecting regions captured digitally. It detects different factors like colour, brightness in
surrounding region. Methods used here is Local Extrema based on finding maxima and minima of function.
The basic scenario of the Blob Analysis solution consists of the following steps:
a) Extraction:-
Techniques are applied to obtain a region corresponding to the objects (or single object) being inspected.
In any object there are many edges or points, that can be extracted to provide a description of object. Here we are using
Contour Extraction. This technique is applied to digital images to extract their boundaries. Contour tracing used to extract
information about the general shape of digital image. Its different characteristics will be examined to classify patterns.
Hence this will produce more accurate features.
b) Refinement:-
The extracted region is often flawed by external noise of various kind. In refinement stage the region is enhanced using
region transformation techniques.
c) Analysis:-
In step, the refined region is subject to measurements and the final results are computed. If the region represents multiple
objects, it is split into individual blob each of which is inspected separately.
2.2. HSV Model:-
HSV stands for „Hue Saturation Value‟. It is often called HSB i.e. „Hue Saturation Brightness‟. HSV basically used for
colour detection. In hand gesture recognition system, hand tracking & segmentation are most important & challenging
steps towards gesture recognition. Skin colour detection & complex background are major challenges in hand gesture
recognition. Here we will see what does exactly HSV does in short.
1. Hue:-
Hue represent the dominant colour like red, green, blue, purple etc.
2. Saturation:-
Saturation measures colourfulness of area in proportion to its brightness.
3. Value:-
It is related to colour luminance.
2.3. Template Matching:-
Template matching is a technique for finding areas of an image that match to a template image (patch). Our goal is to
detect the highest matching area. We need two primary components:
I. Source Image (I):-
Source image is the image in which we expect to find a match to the template image.

ISSN 2350-1022
Page | 10
Paper Publications
II. Template Image (T):-
In this patch image which will be compared to the template image.
3. Results:
i. GUI:
ii. Output:

ISSN 2350-1022
Page | 11
Paper Publications
III. CONCLUSION
A system for recognizing sign language hand configurations as described, which will additionally provide the facility to
each individual to define and upload his own sign language into the system i.e. User defined gestures, since every country
or even regional group uses its own set of signs. Additionally we have also provided audio output for our desired output.
However some disadvantages are quality may vary according to light & brightness in surrounding. Camera quality may
also cause problem. Speed of gesture matching with text may vary a bit.
Although research in sign language recognition began several decades ago, it is still in its infancy, as no such systems have
been deployed on large scale to date. We can look forward in developing a more accurate & efficient system for deaf as
well as for blind people.
ACKNOWLEDGMENTS
It gives us great pleasure in presenting the preliminary project report on „Sign Language recognition with Gesture
Analysis‟.
We would like to take this opportunity to thank our internal guide Prof. H.V.Kumbhar for giving us all the help and
guidance we needed. We are really grateful to them for their kind support. Their valuable suggestions were very helpful.
We are also grateful to Prof. N.D.Kale, Head of Computer Engineering Department, JSPM PVPIT, Bavdhan, Pune for his
indispensable support, suggestions.
REFERENCES
[1] M. Mohandes, M. Deriche, and J. Liu, “Image based and sensor based approaches to Arabic Sign language
recognition”, IEEE transactions on Human-Machine Systems, VOL. 44, No. 4, August 2014.
[2] D. R. Antunes, C. Guimaraes, L. S. Garcia, L. E. S. Oliveira, and S. Fernandes, “A framework to support
development of sign language human-computer interaction: Building tools for effective information access and
inclusion of the deaf,” in Proc. Fifth Int Research Challenges in Information Science (RCIS) Conf, 2011, pp. 1–12.
[3] S. C. W. Ong and S. Ranganath, “Automatic sign language analysis: A survey and the future beyond lexical
meaning,” IEEE Trans. Pattern Anal.Mach. Intell., vol. 27, no. 6, pp. 873–891, Jun. 2005.
[4] C. Keskin, F. Kira, Y. E. Kara, and L. Akarun, “Real time hand pose estimation using depth sensors.” in ICCV
Workshops. IEEE, 2011, pp.1228–1234.
[5] I.Oikonomidis, N. Kyriazis, and A. Argyros, “Efficient model-based 3d tracking of hand articulations using kinect,”
Oikonomidis ET AL.: TRACKING HAND ARTICULATIONS USING KINECT
[6] M. M. Zaki and S. I. Shaheen, “Sign language recognition using a combination of new vision based features,”
Pattern Recog. Lett., vol. 32, no. 4,pp. 572–577, 2011.

Sign Language Recognition with Gesture Analysis

More Related Content

What's hot (19)

Viewers also liked (10)

Similar to Sign Language Recognition with Gesture Analysis (20)

Recently uploaded (20)

Sign Language Recognition with Gesture Analysis