Twitter sentiment analysis project report
Download as DOCX, PDF5 likes8,606 views
The document outlines a project focused on sentiment analysis of Mr. Narendra Modi's brand image using Twitter data, utilizing Apache Flume and Hortonworks Hadoop. It describes the process of fetching unstructured data from Twitter, transforming it into a tabular format in Hive, and visualizing sentiment analysis results using Tableau. The project goals include analyzing tweet metrics and sentiments globally while providing graphical tools for ad-hoc analysis.
Twitter sentiment analysis project report
- 1. 11/18/2015 Analyze Twitter Data with Hortonworks Hadoop Intermediate Project Report Bharat Khanna UNIVERSITY AT BUFFALO
- 2. 1 Sentiment Analysis of Mr. Narendra Modi’s Brand Image using Twitter Data Summary: - I am doing sentiment analysis of Mr. Narendra Modi’s Brand Image across different nations using data from twitter. For fetching the twitter data, I am using Apache Flume that is open source and by default comes installed in Hortonworks sandbox platform 1.3. After fetching the data from twitter, it would be loaded directly to HDFS (Hadoop Distributed File System). This way I am reducing the extra overhead of transferring the data from local system to HDFS. Data loaded in HDFS is still in unstructured format and not good for Ad-hoc analysis. So I will be converting the JSON data to tabular format and store it in HIVE. Also I would be providing a graphical user interface to end users to run their own ad-hoc analysis. Next step deals with using the dictionary file to score the sentiment of each tweet by the number of positive words compared to number of negative words, and then assigned a positive, negative or neutral sentiment value to eachtweet. I have downloaded the dictionary file from below link. Click here for Dictionary Last part of project is to show results of sentiments analysis in form of visualizations. Here I will be using Tableau for it. I will be connecting Tableau to Hive using Hortonworks ODBC Driver that I downloaded from Hortonworks website (link mentioned in references section). I will show the results of analysis in the form graphs and maps using Tableau’s inbuilt VIZQL server. Data sets and Software: Sentiment Data: - Sentiment Data is unstructured data that represents opinions, emotions, attitudes contained in sources such as social media posts, online blogs, and product reviews etc. Whyuse sentiment Data:- Organizations use sentiment data to know what people feel about their product and what they can do to effectively market their product. How did I fetched Twitter Data: - Created twitter app, configured flume.conf with app credentials and ran flume. All the steps for fetching data from twitter using Apache Flume I have mentioned in a YouTube video and a ppt, the link of which is below. I have alsouploaded video at ublearns discussion forum of DC. YouTube: - https://youtu.be/E1w5SkE7Cco Slide share: - http://www.slideshare.net/bharat3khanna/extracting-twitter-data-using- apache-flume Source code for Flume-Snapshot.jar:- Idownloadedsource code of Flume-snapshot.jarfromgithub and builtthe jarusingmavenpackage inHadoop cluster.
- 3. 2 Click here for Flume Source Code Size of Data: - Though there is no limitation of amount of data I can get from twitter but for this project, I am going to do my analysis on approximately 100 mb of data. AlgorithmsUsed:- IamnotusingMap-Reduce Algorithmhere,sinceIwanttodoanalysis oncomplete data and I don’twant to use aggregatedmeasures.If I wouldhave usedMap Reduce,thenmy lot of data wouldhave beenaggregatedbyreducer.My source data isin JSON format and I am usingHive- serde.jar (serde stands serializer and deserializer) that helps in parsing the JSON data effectively to hive tables. Source code forHive-serde.jar:-Idownloaded source code of Hive-serde.jarfromgithubandbuiltthe jar using maven package in Hadoop cluster. Clickhere forHive-serde.jarsource code Analysis to be done on Twitter data: - I am going to do following analysis using Hive and Tableau:- a) Maximum tweets count per user. b) Count of retweets. c) Geographically mapping people’s sentiments towards Mr. Modi. References: - http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop https://github.com/cloudera/cdh-twitter-example https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html#lexicon http://hortonworks.com/products/releases/hdp-1-3/#add_ons