Georges Antoniadis
580 California St., Suite 400
San Francisco, CA, 94104
Automatic Identification of Arabic Dialects
2010, Language Resources and Evaluation
https://doi.org/10.13016/M2BK16T26visibility
…
description
5 pages
link
1 file
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
In this work, automatic recognition of Arabic diale cts is proposed. An acoustic survey of the proporti on of vocalic intervals and the standard deviation of consonantal intervals in nine dialects (Tunisia, Morocco, Algeria, Egypt, Syria, Lebanon, Yemen, Golf's Countries and Iraq) is performed using the platform Alize and Gaussian Mixture Models (GMM). The results show the complexity of the
Related papers
Arabic dialects identification: North African dialects case study
2020
Arabic is the fourth most used language on the Internet and the official language of more than 20 countries around the world. It has three main varieties, Modern Standard Arabic, which is used in books, news and education, local Dialects that vary from region to another, and Classical Arabic, the written language of the Quran. Maghrebi dialect is the Arabic dialect language used in North African countries, where internet users from these countries feel more comfortable using local slangs than native Arabic. In this study, we present a large dataset of regional dialects of three countries, namely Algeria, Tunisia, and Morocco, then we investigate the identification of each dialect using a machine learning classifiers with TF-IDF features. The approach shows promising results, where we achieved accuracy up to 96%.
Using prosody and phonotactics in Arabic dialect identification
Interspeech 2009
While Modern Standard Arabic is the formal spoken and written language of the Arab world, dialects are the major communication mode for everyday life; identifying a speaker's dialect is thus critical to speech processing tasks such as automatic speech recognition, as well as speaker identification. We examine the role of prosodic features (intonation and rhythm) across four Arabic dialects: Gulf, Iraqi, Levantine, and Egyptian, for the purpose of automatic dialect identification. We show that prosodic features can significantly improve identification, over a purely phonotactic-based approach, with an identification accuracy of 86.33% for 2m utterances.
Spoken Arabic Dialect Identification Using Motif Discovery
The Egyptian Journal of Language Engineering, 2018
In traditional Dialect Identification (DID) approaches, regardless of the level and type of features used for identification, they use either predefined references such as phones, phonemes, or even acoustic sounds that characterize a language/dialect, or involve some sort of transcription of the input data. The transcription may be manual or automatic using tools such as ASRs, Tokenizers, or Phone Recognizers. In this paper, we introduce a new approach based on analyzing the speech signal directly and extracting the features that characterize the dialect without any predefined references and without any sort of transcription. The main idea is that we find the repeated sequences (motifs) of the dialect by treating the speech signal as a times series, so we can apply motif discovery techniques to extract the repeated sequences directly from the speech signal. For motif extraction, we adopted an extremely fast parameter-free Self-Join motif discovery algorithm called Scalable Time series Ordered-search Matrix Profile (STOMP). We implemented the new approach in two stages; in the first we built a base line system in which we extracted 12 Mel Frequency Cepstral Coefficients (MFCC) from each motif, in the second stage we built an improved system using 39 coefficients by adding 13 Delta coefficients, 13 Delta-Delta coefficients, and 1 Log Energy coefficient. In both systems, we used Gaussian Mixture Model-Universal Background Model (GMM-UBM) as a classifier. We applied our new approach on three different motif lengths 500ms, 1000ms, and 1500ms using 1gmm component up to 2048gmm components. We downloaded the data set from Qatar-Computing-Research-Institute domain. We carried out our experiments on different Arabic dialects: the Egyptian (EGY), Gulf (GLF), Levantine (LEV), and North African (NOR).The base line results were very competitive with the traditional, more sophisticated approaches, while the improved system showed very good result. The improvement was so significant that we can consider the new approach as competitive, simple, and dialect-independent approach.
Effective Arabic dialect classification using diverse phonotactic models
2011
We study the effectiveness of recently developed language recognition techniques based on speech recognition models for the discrimination of Arabic dialects. Specifically, we investigate dialect-specific and cross-dialectal phonotactic models, using both language models and support vector machines (SVMs). Techniques are evaluated both alone and in combination with a cepstral system with joint factor analysis (JFA), using a fourdialect data set employing 30-second telephone speech samples. We find good complementarity from different features and modeling paradigms, and achieve 2% average equal error rate for pairwise classification.
Text and Speech-based Tunisian Arabic Sub-Dialects Identification
2020
Dialect IDentification (DID) is a challenging task, and it becomes more complicated when it is about the identification of dialects that belong to the same country. Indeed, dialects of the same country are closely related and exhibit a significant overlapping at the phonetic and lexical levels. In this paper, we present our first results on a dialect classification task covering four sub-dialects spoken in Tunisia. We use the term ’sub-dialect’ to refer to the dialects belonging to the same country. We conducted our experiments aiming to discriminate between Tunisian sub-dialects belonging to four different cities: namely Tunis, Sfax, Sousse and Tataouine. A spoken corpus of 1673 utterances is collected, transcribed and freely distributed. We used this corpus to build several speech- and text-based DID systems. Our results confirm that, at this level of granularity, dialects are much better distinguishable using the speech modality. Indeed, we were able to reach an F-1 score of 93.7...
Automatic Dialect Detection in Arabic Broadcast Speech
Interspeech 2016, 2016
In this paper, we investigate different approaches for dialect identification in Arabic broadcast speech. These methods are based on phonetic and lexical features obtained from a speech recognition system, and bottleneck features using the i-vector framework. We studied both generative and discriminative classifiers, and we combined these features using a multi-class Support Vector Machine (SVM). We validated our results on an Arabic/English language identification task, with an accuracy of 100%. We also evaluated these features in a binary classifier to discriminate between Modern Standard Arabic (MSA) and Dialectal Arabic, with an accuracy of 100%. We further reported results using the proposed methods to discriminate between the five most widely used dialects of Arabic: namely Egyptian, Gulf, Levantine, North African, and MSA, with an accuracy of 59.2%. We discuss dialect identification errors in the context of dialect code-switching between Dialectal Arabic and MSA, and compare the error pattern between manually labeled data, and the output from our classifier. All the data used on our experiments have been released to the public as a language identification corpus.
Statistical Analysis of Arabic Phonemes for Continuous Arabic SpeechRecognition
2012
Although Arabic is the world's second most spoken language in terms of the number of speakers, Arabic automatic speech recognition (AASR) did not receive the desired attention from the research community. In this paper, we introduce thorough statistical analysis of the Arabic phonemes from a widely used Arabic corpus that was developed by King Fahd
Algerian Modern Colloquial Arabic Speech Corpus (AMCASC): regional accents recognition within complex socio-linguistic environments
Language Resources and Evaluation, 2016
The Algerian linguistic situation is very intricate due to the ethnic, geographical and colonial occupation influences which have lead to a complex sociolinguistic environment. As a result of the contact between different languages and accents, the Algerian speech community has acquired a distinctive sociolinguistic situation. In addition to the intra-and inter-lingual variations describing day-today linguistic behavior of the Algerian speakers, their speech is characterized by the presence of many linguistic phenomena such as bilingualism and code switching. The study of automatic regional accent recognition in such a type of environment is a new idea in the field of automatic languages, dialect and accent recognition especially that previous studies were conducted using monolingual evaluation data. The assessment of the effectiveness of GMM-UBM and i-vectors frameworks for accent recognition approaches through the use of the Algerian Modern Colloquial Arabic Speech Corpus (AMCASC), which is a linguistic resource collected for this purpose, shows that not only the recording conditions mismatch, channels mismatch, recordings length mismatch and the amplitude clipping which have a non-desirable effect on the effectiveness of these acoustic approaches but also language contact phenomena are other perturbation sources which should be taken into consideration especially in real life applications.
Helpful Statistics in Recognizing Basic Arabic Phonemes
—The recognition of continuous speech is one of the main challenges in the building of automatic speech recognition (ASR) systems, especially when it comes to phonetically complex languages such as Arabic. An ASR system seems to be actually in a blocked alley. Nearly all solutions follow the same general model. The previous research focused on enhancing its performance by incorporating supplementary features. This paper is part of ongoing research efforts aimed at developing a high-performance Arabic speech recognition system for learning and teaching purposes. It investigates a statistical analysis of certain distinctive features of the basic Arabic phonemes which seems helpful in enhancing the performance of a baseline HMM-based ASR system. The statistics are collected using a particular Arabic speech database, which involves ten different male speakers and more than eight hours of speech which covers all Arabic phonemes. In HMM modeling framework, the statistics provided are helpful in establishing the appropriate number of HMM states for each phoneme and they can also be utilized as an initial condition for the EM estimation procedure, which generally, accelerates the estimation process and, thus, improves the performance of the system. The obtained findings are presented and possible applications of automatic speech recognition and speaker identification systems are also suggested.
Arabic Dialects System using Hidden Markov Models (HMMs
WSEAS TRANSACTIONS on COMPUTERS, 2022
The Arabic language has many different dialects and it must be recognized before using the automatic speech recognition (ASR). On the other hand, it is observed in all Arab countries that the standard Arabic language is widely written and used in an official speech, newspapers, public administration, and schools but it is not used in daily conversations instead the dialect is widely spoken in daily life and rarely written. In this paper, we examine the difficult task of properly identifying various Arabic dialects and propose a system developed to identify a set of four regional and modern standard Arabic speeches, based on speech recognition using Hidden Markov Models (HMMs) algorithms. HMMs have become a very popular way to build a speech recognition system. It is set as hidden states and possibilities of transition from one state to another. Due to the similarities and differences between the Arabic dialects, speeches collected from the ADI5 datasets were retrieved from the MGB-3 challenge source. We proposed an Arabic Dialect Identification System called "Building a System for Arabic Dialects Identification based on Speech Recognition using Hidden Markov Models (HMMs)" that takes Input as speech utterances and produces output as dialect being spoken. During the training phase, speech utterances from one or more dialects were analyzed to capture the important properties of audio signals in terms of time and frequency. During the testing phase, previously unseen test utterances were utilized to the system, and the system outputs the dialect associated with the model of dialect that most closely matches the test utterance. The proposed model of the system shows promising results of the model for each dialect match.
Automatic Identification of Arabic Dialects
Mohamed BELGACEM, Georges ANTONIADIS, Laurent BESACIER
Université de Grenoble, France Laboratoire LIDILEM & LIG : GETALP
Address
E-mail: [email protected], [email protected], [email protected]
Abstract
In this work, automatic recognition of Arabic dialects is proposed. An acoustic survey of the proportion of vocalic intervals and the
standard deviation of consonantal intervals in nine dialects (Tunisia, Morocco, Algeria, Egypt, Syria, Lebanon, Yemen, Golf’s
Countries and Iraq) is performed using the platform Alize and Gaussian Mixture Models (GMM). The results show the complexity of
the automatic identification of Arabic dialects since. No clear border can be found between the dialects, but a gradual transition
between them. They can even vary slightly from one city to another. The existence of this gradual change is easy to understand: it
corresponds to a human and social reality, to the contact, friendships forged and affinity in the environment more or less immediate of
the individual. This document also raises questions about the classes or macro classes of Arabic dialects noticed from the confusion
matrix and the design of the hierarchical tree obtained.
for the construction of our corpus of Arabic speech with
1. Introduction and problem
its different dialects, an interesting approach consists in
The existence of language’s dialects is a challenge to using broadcast news data.
Natural Language Processing (NLP) in general, because it In the data we have chosen, there is a variety of speakers
adds another set of dimensions variation from a known (Tunisian, Algerian, Moroccan, Egyptian, Iraqi,
standard. The problem is particularly interesting in Arabic Armenian, Gulf...) and themes (newspapers, TV shows,
and its dialects (J. Meyer et al., 2003). This work aims to political debates, sports, education, social ...). Several
highlight the different dialectical phenomena of the people can speak on a given subject successively or
Arabic speech and try to build an automatic identification simultaneously. The acoustic quality of the recording can
system of Arabic dialects using GMM models. To address change significantly over time.
this issue we need as first part a vocal corpus. However,
At this moment, we are particularly interested in news
the limited work about this topic and the lack of Arabic
(diary, flash, press reviews, including weather and stock
speech corpora lead us to build one, as a first work.
markets, economy, politics, social events ...) in the audio
2. Construction of the corpus document. Any other form of registration (ads, games,
drama ....) will not be transcribed. Following this
approach, we recorded the equivalent of 10 hours of
Arabic speech with good quality of various dialects from
(10 Arabic TV channels and radio: Al-Jazeera, Tunisia,
Morocco, Algeria, Egypt, Syria, Lebanon, Yemen…).
Figure 1: Construction of our dialectal Arabic corpus
2.1 Recording of Arabic Speech
Regarding the collection of vocal data in large quantities
3437
transcription, and verification. These steps can be
conducted in parallel or otherwise applied sequentially
over long portions of the signal, depending on the choice of
the transcriber.
At this moment, we managed to transcribe only 37% of our
corpus (3 hours and a half). Transcribed broadcasts are
mixture of Arabic dialects as they are political debates.
3. Automatic identification of dialects
3.1 Introduction
The system designed for this particular task is based on
state of the art Gaussian Mixture Models (GMMs) used for
Automatic Speaker Verification (SV). We used tools
available in "free version" (LIA_SpkDet and ALIZE)
(Bonastre, Wils, Meignier. 2005) and developed at LIA. In
continuation of this work, we propose here a first study on
* Gulf: This group contains several countries (Iraq,
the extraction of information useful for the characterization
Kuwait, Saudi Arabia, Bahrain, Qatar, Yemen, Oman ...
of Arabic dialects. We will focus particularly on three
sources of informations: (1) complete filtered signal
* Non-native Arabic: English, French, Iranian, Israeli …
coming from our corpus of different dialects, (2)
informations about each speaker (male, female, origin ...),
Figure 2: Statistics of our dialectal Arabic corpus
(3) grouping speakers for each dialect. The corpus used in
this study and the system of automatic segmentation by
2.2 Automatic Segmentation by Speakers
speaker are already described in Part One.
To go faster in the transcript and make clear all information
from our speakers, we used an automatic speaker 3.2 Experiments
segmentation system (from GETALP team of LIG
laboratory). After this automatic segmentation, we did a The experiments were carried out after "adaptation" of the
manual verification to improve performance and assigned SV system of LIA. This system (called LIA_SpkDet) is
to each speaker the necessary informations (name, sex, based entirely on ALIZE open platform (Bonastre, Wils,
origin, dialect, studio or telephonic recording ...). Meignier. 2005) designed and constructed under the
Mistral project. LIA_SpkDet is also distributed as free
2.3 Transcripts of our Wav File using software.
Transcriber
We describe in this section a set of conventions to structure,
3.3 Protocol
annotate and transcribe recordings of radio or broadcast
news. The speech of each speaker must be transcribed We used in this work, the corpus describes in the first part
orthographically. This represents the transcript itself. So far, and to cut out it in two parts:
it is the most important part and therefore maximum The corpus of training is composed of 30 one minute
attention should be focused on it. The different steps of sequences of each dialect (Tunisia, Morocco, Algeria,
transcription are: the sound segmentation, the Egypt, Syria, Lebanon, Yemen, Golf' S Countries and Iraq).
identification of speech parts and the speakers, the The sequences of word result from televisual contents in
identification of thematic sections, the orthographic various environments of sound recording (studio, report,
3438
sport but not of simultaneous speakers…).
The corpus of test is composed of 9 one minute sequences
of each dialect so that our speakers of the corpus of test are
not included in our corpus of training. The sequences result
from same televisual contents of the corpus of training.
Set Dialect Duration Male Female
(mn)
Train 9 270 235 235
Test 9 90 45 45
Total 9 360 280 280
Table 1: Division of our corpus: training and test.
3.4 Results
Figure 4: Classification of Arabic dialects using the
method of ascending hierarchical clustering
The previous figure shows the concept of classes and
subclasses obtained. We can divide Arabic dialects in 2
classes:
Figure 3: Confusion matrix obtained with GMMs on our
• Middle East Dialect: Currency in 2 subclasses
Arabic dialects corpus
Golf‘s Countries dialect and Yemin, Iraq dialect.
3.5 Classes of useful informations • The second class Dialect is divided into three
subclasses, Morocco and Algeria represent the
Our results show both that our system is able to
far left, Egypt and Tunisia the right end and Syria
distinguish the dialects of Arabic-speaking listeners
Maghreb and / or the Middle East. It is also interesting to and Lebanon are an intermediate zone, but also
notice that, using a method of ascending hierarchical shows that Syria and Lebanon are closer to the
clustering (see figure 4), we can draw automatically some dialects of the Maghreb as the Middle East.
large families, theoretically and logically related to large
geographic areas (these “families” being widely accepted 4. Discussion
by the linguistic community (Abu-Haidar.F, 1991)).
We can distinguish especially the dialects of the east and In this paper, we propose our system of automatic
west, ie the Gulf’s Countries and Maghreb (western and identification of Arabic dialects. This study was
eastern Arab world). The first region corresponds to the conducted using our vocal corpus of different dialects and
Gulf countries, Iraq, Yemen and the second is the region the SV system of LIA called LIA_SpkDet and based
of the Maghreb, Egypt, Lebanon and Syria. entirely on ALIZE free platform.
3439
From an experimental point of view, the results and the Dialects ». Dans Proceedings of Eurospeech’99, 395- 398,
concept of classes obtained by our system seem most Budapest (Hongrie).
interesting in this study, in that they can confirm almost
Reynolds D. A, Quatieri T.F, Dunn R.B. (2000)
automatically the theoretical results of specialists from
« Speaker verification using adapted Gaussian mixture
Arabic that show the 2 classes in Arabic dialects with the
models, digital signal processing (dsp) ». In a review
Middle East representing the eastern dialects (Gulf
countries) and as second-class dialects of North Africa, journal - Special issue on NIST 1999 speaker recognition
Egypt, Lebanon and Syria. Although this division is an workshop 10 (1-3), pages 19–41.
oversimplification of Arabic dialectology (Barkat.M, J. Meyer, F. Pellegrino, M. Barkat-Defradas &
1999), it is widely accepted by the linguistic community F.Meunier. (2003) « The Notion of Perceptual Distance:
and can be supported by linguistic and geographic data.
The Case of Afro-Asiatic Languages». Dans Proceedings
(Rjaibi-Sabhi.N, 1993).
of the 15th International Congress of Phonetic Sciences,
5. Conclusion Barcelone, 3-9 September 2003.
Bonastre J.-F., Wils.F, Meignier.S. (2005) « Alize, a free
This work presented a system that automatically identifies
toolkit for speaker recognition ». In ICASSP-05,
the Arabic dialects. The difficult task of properly
Philadelphia, USA, volume 39, pages 430–451
identifying various Arabic dialects was examined. Since
the Arabic language has many different dialects, they must
be identified before Automatic Speech Recognition can
take place. Due to the limited availability of Arabic speech
databases, it was necessary to create new data for dialects.
A new model has been presented in this work based upon
the features of Arabic dialects; namely, a model that
recognizes the similarities and differences between each
dialect. The model utilized in this work was the Gaussian
Mixture Models (GMM) (Reynolds, Quatieri, Dunn.
2000).
Therefore this new initialization process is used and yields
better system performance of 73.33%.
6. References
Abu-Haidar.F. (1991) « Variabilité et invariance du
système vocalique de l’arabe standard ». Thèse de Doctorat
en Sciences du Langage, Université de Besançon.
Rjaibi-Sabhi.N. (1993) «Approches Historique,
Phonologi-que et Acoustique de la Variabilité Dialectale
Arabe : Caractérisation de l’origine Géographique en
Arabe Standard», Thèse de Doctorat, Université de
Besançon.
Barkat.M, Pellegrino.F & Ohala.J.J. (1999) « Prosody
as a Distinctive Feature for the Recognition of Arabic
3440
3441