Downloaded 42 times
Uploaded byJitendra Kumar Yadav
NLP - updated (Natural Language Processing))
The document discusses natural language processing (NLP), a subfield of AI focused on enabling computers to understand human languages, with applications including automatic summarization, sentiment analysis, text classification, and virtual assistants. It outlines the development of an NLP project related to cognitive behavioral therapy (CBT), emphasizing the stages of problem scoping, data acquisition, processing, and modeling. The document also explains challenges in NLP and techniques like tokenization, stemming, and the Bag of Words model, alongside the TF-IDF algorithm for identifying word importance.
More Related Content
Similar to NLP - updated (Natural Language Processing))
![SHS_Core_CAE_Q3_LE1 FOR THIRD [FINAL].pdf](https://cdn.slidesharecdn.com/ss_thumbnails/shscorecaeq3le1final-251116055110-e3081055-thumbnail.jpg?width=640&height=640&fit=bounds)
NLP - updated (Natural Language Processing))
- 1. NATURAL LANGUAGE PROCESSING -JitendraKumar Yadav DAV Public School, Gumla
- 2. NLP It isthe sub-field of AI that is focused on enabling computers to understand and process human languages. It is a subfield of Linguistics, Computer Science, Information Engineering, and Artificial Intelligence. It is concerned with the interactions between computers and human (natural) languages, in particular how to program computers to process and analyse large amounts of natural language data.
- 3. APPLICATIONS OF NATURAL LANGUAGEPROCESSING Automatic Summarization Sentiment Analysis Text classification Virtual Assistants
- 4.
- 5. AUTOMATIC SUMMARIZATION… Itis the process of shortening a set of data computationally, to create a summary that represents the most relevant information within the original content. It comes out as the solution to information overload. It is about understanding emotional meanings within the information.
- 6.
- 7. SENTIMENT ANALYSIS… Itis about identifying sentiment among several posts or even in the same post where emotion is not always explicitly expressed. Companies use NLP applications, such as sentiment analysis, to identify opinions and sentiment online to help them understand what customers think about their products and services. Ex- “I love the new iPhone” and, a few lines later “But sometimes it doesn’t work well” where the person is still talking about the iPhone and overall indicators of their reputation.
- 8.
- 9. TEXT CLASSIFICATION… Textclassification makes it possible to assign predefined categories to a document and organize it to help finding the information needed. For example, an application of text categorization is spam filtering in email.
- 10.
- 11. VIRTUAL ASSISTANTS… Anapplication program that understands natural language voice commands and completes tasks for the user. Benefits of AI Assistants: Improved customer support Ease of key data collection Personalized user experience Examples: Chatbots, Voice Assistants, AI Avatars, Domain Specific Virtual Assistants, etc.
- 12. LET’S TALK ABOUTA SCENARIO
- 13. THE WORLD ISCOMPETITIVE NOWADAYS
- 14. THE WORLD ISCOMPETITIVE NOWADAYS… Everybody wishes to give their best even in a tiniest task. And, when people are unable to meet these expectations, they get stressed/depression. People often get depressed due to reasons like peer pressure, studies, family issues, relationships, etc.
- 16. IS THERE ANYTHERAPY FOR THIS?
- 17.
- 18. CBT Cognitive BehaviouralTherapy (CBT) is considered to be one of the best methods to address stress as it is easy to implement on people and also gives good results. It includes understanding the behaviour and mindset of a person in their normal life and help people overcome their stress and live a happy life.
- 19. How an NLPproject on “CBT” will be developed? To understand this lets go through AI Project Cycle.
- 20. PROBLEM SCOPING Mostof the therapists cure patients out of depression using CBT technique. But, People do not wish to seek the help of a psychiatrist willingly. They try to avoid such interactions as much as possible. Thus, there is a need to bridge the gap between a person who needs help and the psychiatrist.
- 21. PROBLEM SCOPING WhoCanvas – Who has the problem? People suffering from stress/depression. What Canvas – What is the nature of the problem? People who need help are reluctant to consult a psychiatrist and hence live miserably. Where Canvas – Where does the problem arise? When they are going through a stressful period of time. Why Canvas – Why do you think it is a problem worth solving? People get a platform where they can talk and vent out their feelings anonymously. (4Ws CANVAS)
- 22.
- 23. DATA ACQUISITION Tounderstand the sentiments of people, we need to collect their conversational data so the machine can interpret the words that they use and understand their meaning. Such data can be collected from various means: 1. Surveys 2. Observing the therapist’s sessions 3. Databases available on the internet 4. Interviews, etc.
- 24. DATA EXPLORATION Thetextual data collected needs to be processed and cleaned so that an easier version can be sent to the machine. The text is normalised through various steps and is lowered to minimum vocabulary since the machine does not require grammatically correct statements but the essence of it.
- 25. MODELLING Once thetext has been normalised, it is then fed to an NLP based AI model. In NLP, modelling requires data pre- processing only after which the data is fed to the machine. Depending upon the type of chatbot to be made, an appropriate AI model is used to develop the foundation of the project.
- 26. EVALUATION The reliabilityof AI model is observed on the basis of outputs by feeding the test dataset into the model and comparing it with actual answers.
- 27. EVALUATION… If the model’s outputdoes not match the true function at all, the model is said to be underfitting and its accuracy is lower. Case-I
- 28. EVALUATION… If the model’s performancematches well with the true function, then the model has optimum accuracy and it is called a perfect fit. Case-II
- 29. EVALUATION… If the Modelperformance is trying to cover all the data samples even if they are out of alignment to the true function, then this is said to be overfitting and this too has a lower accuracy. Case-III
- 30. CHATBOTS One ofthe most common applications of Natural Language Processing is a chatbot. An Al software that can simulate a real human conversation with real-time responses to users based on reinforced learning. AI Chatbots either use text messages, voice commands, or both.
- 31. CHATBOTS… Ex- • MitsukuBot https://www.pandorabots.com/mitsuku/ • CleverBot https://www.cleverbot.com/ • Jabberwacky http://www.jabberwacky.com/ • Haptik https://haptik.ai/contact-us • Rose http://ec2-54-215-197-164.us-west-1.compute.amazonaws.com/speech.php • Ochatbot https://www.ometrics.com/blog/list-of-fun-chatbots/
- 32. CHATBOTS… There are2 types of chatbots: Ex- bots deployed in the customer care section of various companies
- 33. CHATBOTS… Ex- Google Assistant,Alexa, Cortana, Siri, etc.
- 34. HUMAN LANGUAGE VS COMPUTERLANGUAGE Human brain continuously processes everything what it gets around, makes sense and stores it in some place. When someone whispers, the focus of our brain automatically shifts(giving more priority) to that speech and starts processing automatically. While, the computer understands the language of numbers. Everything that is sent to the machine has to be converted to numbers.
- 35. DIFFICULTIES DURING PROCESSING NATURALLANGUAGE BY A MACHINE There are structures/characteristics in the human language that might be easy for a human to understand but extremely difficult for a computer to understand. Different syntax, same semantics: 2+3 = 3+2 Different semantics, same syntax: 2/3 (Python 2.7) ≠ 2/3 (Python 3) Arrangement of the words and meaning
- 36. DIFFICULTIES DURING PROCESSING NATURALLANGUAGE BY A MACHINE… => His face turned red after he found out that he took the wrong bag. => His face turns red after consuming the medicine. Both the sentences might have multiple meanings. Multiple Meanings of a word
- 37. DIFFICULTIES DURING PROCESSING NATURALLANGUAGE BY A MACHINE… => Chickens feed extravagantly while the moon drinks tea. Both the sentences might have multiple meanings. Perfect Syntax, but no Meaning
- 38. We may facethese challenges if we try to teach computers how to understand and interact in human language. So, lets see how does NLP do this magic?
- 39. DATA PROCESSING (TEXT NORMALISATION) It involves preparing and cleaning text data for machines to be able to analyze it. This process puts data in workable form and highlights features in the text that an algorithm can work with. There are several ways this can be done, including:
- 40. DATA PROCESSING… SentenceSegmentation: In this process the whole corpus is divided into sentences. Each sentence is taken as a different data so now the whole corpus gets reduced to sentences.
- 41. DATA PROCESSING… SentenceSegmentation:
- 42. DATA PROCESSING… Tokenisation: Itis the process of breaking down the sentences into smaller units(tokens) to work with.
- 43.
- 44. DATA PROCESSING… RemovingStopwords, Special Characters and Numbers: It is the process of removing common words, special characters, etc(which do not add any essence to the information) are removed from text so, unique words that offer the most information about the text remain. Some examples of stopwords are: a, an, are, for, etc.
- 45. DATA PROCESSING… Convertingtext to a common case: In this process the whole text is converted into a similar case(lower case). This ensures that the machine is case- insensitive.
- 46. DATA PROCESSING… Convertingtext to a common case:
- 47. DATA PROCESSING… Stemming: Here,the remaining words are reduced to their root words. It is the process in which the affixes of words are removed and the words are converted to their base form.
- 48.
- 49. DATA PROCESSING… Lemmatization: Theprocess in which a word is converted to its meaningful root form. Stemming and lemmatization both are alternative processes to each other as the role of both the processes is same – removal of affixes. But the difference between both of them is that in lemmatization, the word we get after affix removal (also known as lemma) is a meaningful one.
- 50.
- 51. BAG OF WORDSALGORITHM A Natural Language Processing model which helps in extracting features out of the text which is very helpful in machine learning algorithms. The occurrences of each word is counted and the vocabulary for the corpus is constructed.
- 52.
- 53. BAG OF WORDS… Thestep-by-step approach to implement bag of words algorithm: 1. Text Normalisation: Collect data and pre- process it. 2. Create Dictionary: Make a list of all the unique words occurring in the corpus. (Vocabulary). 3. Create document vectors: For each document in the corpus, find out how many times the word from the unique list of words has occurred. 4. Create document vectors for all the documents.
- 54.
- 55. BAG OF WORDS… Hereare three documents having one sentence each. After text normalisation, the text becomes: Note that no tokens have been removed in the stopwords removal step. It is because we have very little data and since the frequency of all the words is almost the same, no word can be said to have lesser value than the other.
- 56. BAG OF WORDS… Listdown all the words which occur in all three documents: therapy stress
- 57. BAG OF WORDS… Inthis step, •The vocabulary is written in the top row. •Now, for each word in the document, if it matches with the vocabulary, put a 1 under it. •If the same word appears again, increment the previous value by 1. •And if the word does not occur in that document, put a 0 under it.
- 58. BAG OF WORDS… Sincein the first document, we have words: aman, and, anil, are, stressed. So, all these words get a value of 1 and rest of the words get a 0 value. therapy stress
- 59. BAG OF WORDS… Thisgives us the document vector table for our corpus. But the tokens have still not converted to numbers. This leads us to the final steps of our algorithm: TFIDF therapy stress
- 60. BAG OF WORDS… Aplot of occurrence of words versus their value
- 61. TFIDF TFIDF stands forTerm Frequency and Inverse Document Frequency. It helps in identifying the value for each word. Let us understand each term one by one. Term Frequency: Term frequency is the frequency of a word in one document. It can easily be found from the document vector table.
- 62.
- 63. TFIDF… Inverse Document Frequency: Itthe total number of documents divided by the document frequency. IDF = Total no. of documents The document frequency therapy stress therapy stress
- 64. TFIDF… TFIDF(W) = TF(W)* log( IDF(W) ) therapy
- 65. TFIDF… After calculating allthe values: Conclusion: The value of a word is inversely proportional to the IDF value of that word. therapy
- 66. TFIDF… Ex- Total Number ofdocuments: 10 Number of documents in which ‘and’ occurs: 10 Therefore, IDF(and) = 10/10 = 1 Which means: log(1) = 0. Hence, the value of ‘and’ becomes 0. On the other hand, Number of documents in which ‘pollution’ occurs: 3 IDF(pollution) = 10/3 = 3.3333… Which means: log(3.3333) = 0.522; Which shows that the word ‘pollution’ has considerable value in the corpus.
- 67. APPLICATIONS OF TFIDF:- DocumentClassification Topic Modelling Information Retrieval System Stop word filtering Thank You!!!
Editor's Notes
- #4 Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very commonly used by the majority of people in their daily lives.
- #5 Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very commonly used by the majority of people in their daily lives.
- #6 Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very commonly used by the majority of people in their daily lives.
- #9 Artificial Intelligence nowadays is becoming an integral part of our lives, its applications are very commonly used by the majority of people in their daily lives.