analyzing public sentiments using twitter feeds
- 1. By Muhammad Tahir Orakzai NATIONAL INSTITUTE OF MANAGEMENT, PESHAWAR Analyzing Public Sentiments Using Twitter Feeds 23rd SMC Current Issue Presentation Faculty Advisor Akbar Ali Khan 2nd May 2018
- 2. • Introduction • Acronyms • What is Analysis? • What is Sentiment Analysis? • Social Media & Twitter Feeds • Importance of Public Sentiment / opinion • Sentiment Analysis • Evolution of WWW • Data Mining: KDD • Sentiment Analysis Process • Case Studies: Pros & Cons • Conclusion • Recommendations 2 Sequence of Presentation
- 3. Glossary of Terms 3 @ : at the rate of # : Hashtag Preceding a Word AI : Artificial Intelligence Bot: Web Robot Com : Commercial DM : Direct Message EDU : Education HTTP : Hyper Text Transfer Protocol ICT : Information Communication Technology IM : Instant Message KDD : Knowledge Discovery of Data NET : Network NLP : Natural Language Processing OS : Operating System PC : Personal Computer PEW : Family Name not and acronym PPT : Power Point Presentation Pros & Cons : Pro et Contra (Latin Word, for and Against) RAND : Research & Development SA : Sentiment Analysis SMS : Sort Message Send SWOT : Strengths, Weaknesses, Opportunities, Threats UGC : User Generated Content URL : Uniform Resource Locator Web 2.0 : Social Web WWW : World Wide Web
- 4. Heard of these terms? • Analysis • Public Sentiment Analysis • Twitter Feeds: Social Media • User Generated Content (UGC) • WWW: Web 2.0 • Data Mining: KDD • Metadata • Artificial Intelligence (AI) 4 Introduction
- 5. “The beginning of wisdom is the definition of terms.” Socrates (470–399 B.C.)1 1 https://knowledgism.com/glossary-2/the-importance-of-the-definition/?v=47e5dceea252 5 Introduction
- 6. • A systematic examination and evaluation of data or information, by breaking it into its component parts to uncover their interrelationships. • Opposite of synthesis.2 2 http://www.businessdictionary.com/definition/analysis.html 6 Analysis
- 7. • The process of computationally identifying and categorizing opinions expressed in a piece of text, especially in order to determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or neutral.3 3 https://en.oxforddictionaries.com/definition/sentiment_analysis 7 Sentiment Analysis
- 8. Sentiment analysis has many other names 4 • Opinion mining • Opinion extraction • Sentiment mining • Subjectivity analysis 4 https://web.stanford.edu/class/cs124/lec/sentiment.pptx 8 Sentiment Analysis: Synonyms
- 9. • the collective opinion of many people on some issue, problem, etc., especially as a guide to action, decision, or the like 5 5 http://www.dictionary.com/browse/public-opinion 9 Public Opinion
- 11. • Online social networking service • 140 (280) character - “tweets” • Registered users can read & post tweets (UGC) • Communicate and stay connected • Tweets - words, photos, videos and links 6 6 https://en.wikipedia.org/wiki/Twitter 11 Twitter
- 12. • Created & Launched in 2006 • 330+ million active users • 500 million tweets are generated everyday • Microblogs: SMS of the Internet • Audience common man - celebrities • Discussion on current affairs and share personal views on various subjects 7 https://en.wikipedia.org/wiki/Twitter 12 Twitter7
- 13. • Retweet: a tweet that you forward to your followers that always retains original attribution • @: the @ sign is used to call out usernames in tweets • #: any word or phrase immediately preceded by the # symbol. By clicking a hashtag you can see other tweets containing the same keyword or topic • Follower: another person who receives your tweets on their Home stream • Direct Messages: private messages sent from one Twitter user to another. • Home: timeline displaying a stream of tweets from accounts you have chosen to follow 8 https://blog.hubspot.com/marketing/34-twitter-terms-defined-list 13 Twitter: Terminology 8
- 14. • Democratic governments rest on the consent of the governed • Hence, major shifts in public opinion should trigger a shift in public policy • This attention to public opinion has created an industry in public opinion polling and survey research. 14 Public Opinion Importance
- 15. • Opinions are the influencers of our behaviors • Before making a decision, we usually seek opinion of others: • Buy a product, rent a car, reserve a hotel room, looking for a good restaurant … • Individuals • Organizations • Governments 15 Public Opinion Importance
- 16. • General: is this review positive or negative? • Product: what do people think about the new product? • Public sentiment: how is consumer confidence? is despair? or increasing? • Politics: what do people think about this candidate or issue? • Prediction: predict the outcomes or assess trends from sentiments 16 Importance of Twitter SA
- 17. 17 Buzz Tweet
- 18. 9 https://medium.com/@vivekmadurai/web-evolution-from-1-0-to-3-0-e84f2c06739 18 Evolution of WWW : Rise of Social Media 9
- 20. 11 https://twitter.com/wef/status/930131155248209920 20 Scale of Internet Today 11
- 21. • Twitter “feed” is any list of tweets that constantly updates when new tweets that fit the specified criteria pop up. • An ongoing stream of Twitter messages (tweets) 12 https://blog.hubspot.com/marketing/34-twitter-terms-defined-list 21 Twitter Feeds 12
- 22. • Opinion Mining, Text Mining • Sentiment and Subjectivity Analysis • Artificial Intelligence • Natural Language Processing • Computational Linguistics • Etc. 22 Knowledge Discovery from Data
- 23. • Sociologists • Statisticians • Researchers • Linguists • Computer Experts 23 Back End Professionals
- 24. 13 http://www.academicstuff.co.in/Web-Sentiment-Analysis-for-Scoring-Positive.html Opinion of People Feature Extraction Sentence Extraction Sentiment Classification Sentence Analysis Data Mining Techniques (Supervised + Unsupervised) Results Overall Review Score Calculation 24 Process 13
- 25. • Simplest task: • Is the attitude of this text positive or negative? • More complex: • Rank the attitude of this text from 1 to 5 • Advanced: • Detect the target, source, or complex attitude types 14 https://web.stanford.edu/~jurafsky/slp3/slides/7_Sent.pdf 25 SA : Tasks
- 26. • Easy: search the Web and find a Sentiment Analysis tool and URLs • http://www.twitalyzer.com/index.asp • http://twendz.waggeneredstrom.com/ • http://www.sentiment140.com/ • http://www.blogmeter.it • http://twitrratr.com/ • http://www.socialmention.com • http://www.lovewillconquer.co.uk/ • Many more…. • And professional sites for companies • www.radian6.com • www.sysomos.com 26 Automated Analysis
- 27. • People express opinions in complex ways • In opinion texts, lexical content alone can be misleading • Intra-textual and sub-sentential reversals, negation, topic change common • Rhetorical devices/modes such as sarcasm, irony, implication, etc. • Unstructured and also non-grammatical • Lexical Variation • Out of Vocabulary Words • Extensive usage of acronyms like asap, lol, afaik 15 https://web.stanford.edu/~jurafsky/slp3/slides/7_Sent.pdf 27 Text (Data) Mining Challenges
- 28. 28 At present there are two different schools of thoughts on using social media for analyzing public opinion: • The Pro School– RAND Corporation • The contra-group – PEW Research Center Pros & Cons of SA
- 29. Potential policy usage of computer based analysis of social media: • Make informed assessments of public opinion • Forecast events, such as large-scale protests • Assess the impact of political events or actions on public opinion • Inform outreach efforts to foreign populations • Pinpoint intelligence gaps 16 https://www.rand.org/pubs/research_briefs/RB9685.html 29 Case Study – 1: A Case Study of Iranian Public Opinion After the 2009 Elections 16
- 30. • The reaction on Twitter often differs a great deal from public opinion as measured by surveys • Twitter May Be Different at Times • Often it is the overall negativity that stands out • Reliability? 17 http://www.pewresearch.org/2013/03/04/twitter-reaction-to-events-often-at-odds-with- overall-public-opinion/ 30 Case Study – 2: Twitter Reaction to Events Often at Odds with Overall Public Opinion 17
- 31. Advantages • Huge data is available for analysis • Low cost than traditional surveys & polls • Faster way of getting insight from user generated data • New area for researchers: evolving in Pakistan • Can be used in: • Disaster Management • Public health • Assessing efficiency of services and infrastructure • Measuring degree of citizens satisfaction 31 Conclusion
- 32. • Useful tool for SWOT Analysis of organizations and Governments • 80% of all data consists of words: the Sentiment Engine is an essential tool for making sense of it all • More accurate and insightful perceptions and feedback 32 Conclusion Advantages
- 33. • Irony/sarcasm • Slang • Languages • Geographical variations • Text Analysis Problem • Inference Problem • Supervised Data Mining 33 Conclusion Disadvantages
- 34. • How many people do have access to computer • How many of them are using twitter • Are we targeting a segment of society • Is the sample representative of the society • Often public opinion is contradictory • Lower Taxes, More Government Action • Also, the public may be uninformed or misinformed on many topics • Osama being involved in the 9/11 attacks • Weapons of mass destruction in Iraq 34 Conclusion Reliability
- 35. • Misinterpretation – If tweets are misunderstood and then spread virally, this can create long-lasting negative effects • Hacks – Twitter feed can be hacked and used to spread misinformation • Use of BoTs - a social networking account powered by artificial intelligence can change the results • Opinion vary by demographics, social groups, sentiment, topics, time 35 Conclusion Reliability
- 36. • Develop a culture in the government to capitalize on ICT systems • Knowledge updation of public sector organizations • Public policy formulation: Feedback • Effective use for intelligence gathering • can be a great vehicle to get closer to the citizens: Arab Spring phenomenon • Sentiment analysis as an alternative research technique for collecting and analyzing textual data on the internet especially opinion polls. 36 Recomendations
- 37. 1. P. Anderson, What is Web 2.0? Ideas, technologies and implications for education. JISC Technology and Standards Watch (2007) 2. A. Bifet, E. Frank, Sentiment discovery in Twitter streaming data (2010) 3. A. Bifet, G. Holmes, R. Kirkby, B. Pfahringer, MOA: massive online analysis. J. Mach. Learn. Res. 11, 1601– 1604 (2010). http://moa.cs.waikato.ac.nz 4. Z. Bodnar, Generating small business customers with social media marketing: small business case studies (2014), Hubspot http://www.HubSpot.com. 5. C. Burke, Twitter search tools (2011), http://charleneburke.com/2011/02/12-twitter-search-tools/ 6. M. Cha, H. Haddadi, F. Benevenuto, K.P. Gummadi, Measuring user influence in Twitter: the million follower fallacy, in Proceedings of the 4th International AAAI Conference on Weblogs and Social Media, 2010. 7. D. Dasgupta, R. Dasgupta, Social networks using Web 2.0. IBM Developer Works (2009), http://www.ibm.com/deveoperWorks/ 8. https://www.rand.org/pubs/research_briefs/RB9685.html 9. http://www.pewresearch.org/2013/03/04/twitter-reaction-to-events-often-at-odds-with-overall-public- opinion 10. https://www.researchgate.net/publication/286841798_Sentiment_Analysis_for_Government_An_Opti mized_Approach 11. N. Agarwal and H. Liu. Modeling and Data Mining in Blogosphere, vol- ume 1 of Synthesis Lectures on Data Mining and Knowledge Discovery. Morgan and Claypool, 2009. 12. A. Anagnostopoulos, R. Kumar, and M. Mahdian. InÀuence and corre- lation in social networks. In Proceeding of the 14th ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 7–15, New York, NY, USA, 2008. ACM. 13. A. Go and L. Huang. Twitter sentiment analysis using supervision, in Proceedings of the Workshop on Languages in Social Media 2011; 3-38. 14. https://www.sciencedirect.com/science/article/pii/S1877050916320130 15. Meena Rambocas, João Gama Marketing Research: The Role of Sentiment Analysis References