AI Class Topic 4: Text Analytics, Sentiment Analysis and Apache Spark

Editor's Notes

• #3: As you know, the customer experience is your brand. And every touch point customers have influences of that brand. Those touch points used to be very limited – a TV advertisement, the product packaging, the product itself, maybe a brochure or flyer. If customers wanted information it came from the company, industry publications or a select group of peers. With the rise of social and mobile, those touch points are multiplying. And multiplying exponentially. Today’s customers have huge networks to tap for information and are no longer buying in the ways they used to. While this shift creates new challenges, it also creates a huge opportunity for businesses to reach customers in ways never before possible. As we have just mentioned, the world has gone social. And we all experience it every day in our personal lives. We are sharing on Twitter, Facebook, and blogs. When we want to buy a product, we go online for product reviews, or ask questions on forums. We engage our peers for opinions and Tweet our likes and loves. We post photos and videos for our friends and the world. And this isn’t a trend or a fad – this is a fundamental shift in the way people communicate. But this isn’t just a change in the way consumers behave – these social conversations and behaviors are increasingly moving to the B2B space as well. There is a conversation going on out there and if you’re not listening to it, you’ll be in the dark. You can see the facts on this slide which are meant to illustrate what’s happening in the market – and it’s amazing to quantity some of the numbers for how social the world is becoming. But to highlight a few other facts of what this rise in social means. First: 75% of B2B are likely to use social media to influence their purchase decisions. What that means is that customers aren’t just using social – they’re using it to buy. So if you’re not engaged on social, you’re missing the majority of your target market. These customers are out there talking about your products, brands and competitors and if you aren’t hearing the conversations and using them to inform your strategy, you’ll be at a severe disadvantage. Source: http://www.insideview.com/social-selling?utm_source=infographic&utm_medium=howsocialisb2b&utm_campaign=social-selling Second: 2. Customers are 70% of the way through the purchase process before they contact sales. When you think about this – and the way customers buy, that means they’ve already done their research and are close to a decision before they even contact you. Given this, if you want to be able to influence purchasing decisions, you’ll need to engage long before they ever contact your sales department. Source: SiriusDecisions “When you couple that with the fact that over 500,000,000 photos are uploaded and shared each day on Facebook, 100 hours of video are uploaded to YouTube per minute, there are 190,000,000 tweets per day, and the total piece of content shared on Facebook each month exceeds 70,000,000,000….we can see that our customers are part of this social reality and are using social media as a mechanism to get the information they are looking for as they are looking to purchase products from us. In fact, (as we just stated), they are well down the path of the buying cycle because of it before they even reach out to us to engage.”
• #4: Instructor Walk through the slide animations and use this as points of discussion in your class.
• #5: “Let’s talk a little bit about customer experience.” Walk through the animation of the slide to guide your class discussion.
• #6: “As you can see, leveraging social media as well as monitoring what is happening relative to your social footprint is a vital aspect of many businesses’ day to day marketing, sales, and even customer care efforts. Before we spend too much more time on specifics, let’s review some of the common terms and tools used online that drives this customer behavior.”
• #7: “ Hashtag The hashtag is a word or phrase preceded by the “#” sign. #Hashtags are a simple way to mark the topic (or topics) of social media messages and make them discoverable to people with shared interests. On most social networks, clicking a hashtag will reveal all the public and recently published messages that also contain that hashtag. Hashtags first emerged on Twitter as a user-created phenomenon and are now used on almost every other social media platform, including Facebook, Google+, Instagram, Vine and Pinterest. Bitly/ URL shortening Converting a long URL name into a short one. Also called "URL redirecting," there are free URL shortening services on the Web that take a long URL and convert it to a short one for publication on a Web page or other venue. When the short URL is clicked, the URL shortening service receives the request and redirects it to the long URL.Short URLs are widely used to slim down long URLs printed in magazines and newspapers to make it easier to type into a browser. They are also used in e-mail, because URLs in the message often wind up as two lines of text at the recipient's end that cannot be clicked as one link. Although the recipient can copy the two lines back together in the browser's address bar, the short URL eliminates the potential problem. Permalink The URL address of an individual piece of content. Permalinks are useful because they allow you to reference a specific Tweet, update, or blog post instead of the feed or timeline in which you found it. You can quickly find an item’s permalink by clicking on its timestamp SEO Search engine optimization is the practice of increasing the “organic” visibility of a web page in a search engine, such as Google. Although businesses can pay to promote their websites on search engine results pages (Search Engine Marketing, or SEM), SEO refers to “free” tactics that enhance the search ranking of a page. Sentiment analysis An attempt to understand how an audience feels about a brand, company, or product based on data collected from social media. It typically involves the use of natural language processing or another computational method to identify the attitude contained in a social media message. Different analytics platforms classify sentiment in a variety of ways; for example, some use “polar” classification (positive or negative sentiment), while others sort messages by emotion or tone (Contentment/Gratitude, Fear/Uneasiness, etc.).  Meme An idea, fashion, or behavior that is transmitted from person to person through media, speech, gestures, and other forms of communication. The term was conceived by evolutionary biologist Richard Dawkins in the 1970s, but it has exploded into greater relevance in the past decade with the rise of online culture. In Dawkins’ theory, memes are ideas (or fragments of ideas) that are copied and combined as they move from person to person, much like genes are passed down from generation to generation. Dawkins surmised that we could use the concept of evolution by natural selection to understand how ideas spread and change over time. Some memes spread far and wide, some die out, and others mutate. Social media has made it possible to visualize and measure this phenomenon like never before. For example, we can see hashtags rise and fall in popularity and track how quickly they spread throughout a network. The word meme is itself a meme. The theory isn’t perfect, and it has its share of critics, but it’s an alluringly simple way to think about the spread of ideas. Therefore, people use the word and pass it on. Its meaning has also evolved over time as it has become increasingly used to describe viral social media content. Social Capital The central theme of Social Capital is that social networks have value. It refers to the collective value of all social networks (who people know) and the tendencies or benefits that come from these networks to do things for each other. The interactions and flow of information provides benefit/ value for the people who are connected. Social capital works through multiple channels such as information flows, norms of reciprocity (mutual aid), collective action, and social identity. User-generated content (UGC) Media that has been created and published online by the users of a social or collaboration platform, typically for non-commercial purposes. User-generated content is one of the defining characteristics of social media. It is often produced collaboratively and in real-time by multiple users. Many companies have enthusiastically embraced and encouraged user-generated content as a means of increasing brand awareness and customer loyalty. Instagram contests, Vine video contests, and other UGC-based social campaigns allow businesses to tap into the creative energies of their customers and use their contributions to fuel marketing strategies. Brandjacking The hijacking of a brand to promote an agenda or damage a reputation. Brandjackers don’t hack the social media accounts of target individuals and organizations. Instead, they assume a target’s online identity through indirect means such as fake accounts, promoted hashtags, and satirical marketing campaigns. Clickthrough rate (CTR) This is a common metric for reporting on the number of people who viewed a message or piece of content and then actually performed the action required such as clicking on the ad or link in an email marketing campaign. The actual metric is calculated by comparing the number of clicks to impressions. For example, if 100 people saw your ad in Google and one person clicked on the ad, you would have a click-through-rate of 1.0%. Clickthrough rate (CTR) is most commonly used for search engine marketing and other performance-driven channels as the general philosophy is that the higher your CTR, the more effective your marketing is. Creative Commons Creative Commons is a public copyright license that gives you the ability to use and share otherwise copyrighted material. For social media users, Creative Commons often comes into play when we are looking for images and photos to accompany a social media message or blog post. In both of these cases, unless you are using your own images or have express permission, you can only share Creative Commons images. Sites like Google Image Search and Flickr have filters so you can easily search for Creative Commons photos. Just be careful, as there are different level of Creative Commons which could restrict whether an image could be used commercially, whether it can be modified, and what kind of attribution is required. Crowdsourcing Crowdsourcing refers to the process of leveraging your online community to assist in services, content and ideas for your business. Business examples include getting your audience to volunteer in helping translate your product or by asking your community to contribute content for your blog. Geotargetting A feature on many social media platforms that allows users to share their content with geographically defined audiences. Instead of sending a generic message for the whole world to see, you can refine the messaging and language of your content to better connect with people in specific cities, countries, and regions. You can also filter your audience by language. Astroturfing Astroturfing is a fake grassroots campaign that seeks to create the impression of legitimate buzz or interest in a product, service or idea. Often this movement is motivated by a payment or gift to the writer of a post or comment or may be written under a pseudonym. Return on relationship (ROR) A measurement of the value gained by a person or business from developing a relationship. Measuring ROR isn’t easy; it involves not only analyzing connection growth, but also understanding the impact your customers’ voices have on your brand and reputation. This includes sentiment analysis, as well as engagement metrics for your content, like organic sharing rates. ROR is an alternative (or complementary) metric to social media ROI. Troll In Internet slang, a troll is someone who posts controversial, inflammatory, irrelevant or off-topic messages in an online community, such as an online discussion forum or chat room, with the primary intent of provoking other users into an emotional response or to generally disrupt normal on-topic discussion. SMO Social Media Optimization (SMO) is a set of practices for generating publicity through social media, online communities and social networks. The focus is on driving traffic from sources other than search engines, though improved search ranking is also a benefit of successful SMO. Reach Reach is a data metric that determines the potential size of audience any given message could reach. It does not mean that that entire audience will see your social media post, but rather tells you the maximum amount of people your post could potentially reach. Reach is determined by a fairly complex calculation, that includes # of followers, shares and impressions as well as net follower increase over time. Reach should not be confused with Impressions or Engagement. Share of Voice Share of voice is a metric for understanding how many social media mentions a particular brand is receiving in relation to its competition. Usually measured as a percentage of total mentions within an industry or among a defined group of competitors. Word cloud Word clouds, also known as tag clouds or weighted lists, are a visual representation of text, where the frequency of a word determines its size in the word cloud. This is a great tool for identifying words that are repeated or most common.
• #14: A) Data Moore law: number of transistors in a dense integrated circuit doubles approximately every two years A) To make things even more challenging, you need Distr Processing distributed of etherogenus data sets. Need more hardware to store / process modern data. Distributed Clusters are Pervasive (AWS, Vm, Azure, Rack Space). B) ML Mature Methods for Common Problems
 e.g., classification, regression, collaborative filtering, clustering Traditional tools (Matlab, R, Excel, etc.) run on a single machine Who is going to manage all this? IT Architects? MIT: When there chaos there are opportunities.
• #16: Libraries -SparkSQL (DataFrame manipulation ) processing structured and semi-structured datasets, Hive Support, SQL Windowing (ANSI 2003) Languages: 3 levels a) Command line R b) Shell are available for Python, Scala and R c) IntellJ Idea (Java-Scala) -Spark Streaming (continuous DStream computation )for Scalable, High-t Throughput, FaultTolerant Stream Processing of Live Data Stream (Kafka, Flume, Twitter, etc) -Spark Graphs for Graphs and Graph-Parallel Computation. Graph algorithms: PageRank, Connected Components, Triangle Counting
• #18: General purpose engine, It provides in-memory computing capabilities to deliver speed, a generalized execution model to support a wide variety of applications, and Java, Scala, and Python APIs for ease of development. Spark is built using Scala around the concept of Resilient Distributed Datasets (RDD) and provides actions / transformations on top of RDD.
• #20: RDDs are a 'immutable resilient  distributed collection of records' which can be stored in the volatile memory or in a persistent storage  (HDFS, HBase etc) and can be converted into another RDD through some of the transformations. An action like count can also be applied on an RDD. http://www.thecloudavenue.com/2014/01/resilient-distributed-datasets-rdd.html Partitions are logical buckets for data. Partitions correspond to Hadoop's splits (if the data lives on HDFS) or partitioning schemes in the source storage RDD (and hence the data inside) is partitioned. Spark manages data using partitions that helps parallelize distributed data processing with minimal network traffic for sending data between executors. Data in partitions can be skewed, i.e. unevenly distributed across partitions. e is Spark’s mechanism for re-distributing data so that it’s grouped differently across partitions. Data is often distributed unevenly across partitions. repartition and coalesce operators can repartition a dataset.
• #21: provide distributed machine learning algorithms on top of Spark’s RDD abstraction.
• #23: Orientation
• #24: Model saved as set of files including an object containing the model exportable to PMML Output jar file that can be executed Pipeline is a wrapper for this green part of workflow The goal of the Pipeline API (aka Spark ML or spark.ml given the package the API lives in) is to let users quickly and easily assemble and configure practical distributed machine learning pipelines (aka workflows) by standardizing the APIs for different Machine Learning concepts. + The ML Pipeline API is a new DataFrame-based API developed under the spark.ml package
• #32: Campaign Monitoring Measure overall tone and sentiment of external social conversations related to a campaign Brand Monitoring Measure overall tone and sentiment of external social conversations related to the business Identify Influencers Who should I care about? What’s the impact of their influence? How do I harness them?
• #33: Extending Service Reach Identify specific issues and engage social customers to resolve their issues Early Warning System - Identify trends in subjects and sentiment to anticipate and respond to customer issues Knowledge Management Harness the conversations across the business as a living, breathing knowledge base Knowledge Harvesting Extract, package and re-use best practices and solutions from customers helping customers
• #34: Lead Harvesting Listen for, identify and react to potential sales triggers
• #37: Sentiment analysis is one of the Natural Language Processing fields, dedicated to the exploration of subjective opinions or feelings collected from various sources about a particular subject.
• #38: Ronaldo is the highest paid footballer in the world Every year Cristiano Ronaldo earns a base salary of $45 million.
• #40: Orientation