Download to read offline
![Example: “What are our customers saying?”
sql# SELECT cid, sum(tfxidf)/count(*) AS centroid
FROM (
SELECT id, tfxidf, cid,
row_number() OVER (
PARTITION BY id
ORDER BY distance, cid) rank
FROM blog_distance
) blog_rank
WHERE rank = 1
GROUP BY cid;
-MAP:
NAME t t tNAME: extract_terms
PARAMETERS: [id integer, body text]
RETURNS: [id int, title text, doc _text]
FUNCTION: |
if 'parser' not in SD:
import ...
class MyHTMLParser(HTMLParser):
...
SD['parser'] = MyHTMLParser()
parser = SD['parser']
parser.reset()
parser.feed(body)
id | tfxidf
-----+-------------------------------------------------------------------
2482 | {3,1,37,1,18,1,29,1,45,1,...}:{0,8.25206814635817,0,0.34311110...}
1 | {41,1,34,1,22,1,125,1,387,...}:{0,0.771999985977529,0,1.999427...}
10 | {3,1,4,1,30,1,18,1,13,1,4,...}:{0,2.95439664949608,0,3.2006935...}
...
31© Copyright 2011 EMC Corporation. All rights reserved.](https://image.slidesharecdn.com/36dc0665-4e85-472d-8cac-31cda14087d7-160203132602/75/CTAM-Making-Analytics-Actionable-RJA-FINAL-31-2048.jpg)
More Related Content
Similar to CTAM Making Analytics Actionable RJA FINAL
CTAM Making Analytics Actionable RJA FINAL
- 1. Making Analytics ActionableActionable CTAM Conference MarriottMarquis New YorkMarriott Marquis, New York Robert J. Abate, CBIP, CDMP October 6th, 2011 Solutions Principal, EMC Consulting 1© Copyright 2011 EMC Corporation. All rights reserved.
- 2. Making Analytics ActionableMakingAnalytics Actionable •Topic IntroductionTopic Introduction •Data Sources & Rationalization •Actionable Analytics •Open Exchange Of•Open Exchange Of Ideas 2© Copyright 2011 EMC Corporation. All rights reserved.
- 3. Topic Introduction:p Analytics &Big Data… You Are In The “Data” Business… 3© Copyright 2011 EMC Corporation. All rights reserved.
- 4. The Analytics BusinessOpportunity “Through 2015, organizations integrating high value, diverse new 4© Copyright 2011 EMC Corporation. All rights reserved. Through 2015, organizations integrating high value, diverse new information sources and types into a coherent information management infrastructure will outperform industry peers financially by more than 20%” Gartner July 2011 "The New Value Integrator, Insights from the CFO Study”
- 5. The Information IssueIs… •33% of business leaders make critical decisions without the information they need •50% of business leaders don’t have access to the information across their50% of business leaders don t have access to the information across their organization needed to do their jobs •75% of business leaders say more predictive information would drive better decisions IBM Institute for Business Value, March 2009 5© Copyright 2011 EMC Corporation. All rights reserved.
- 6. Information Trust &Alignment Is Missing…Information Trust & Alignment Is Missing… • Harris Interactive recently polled 23,000 U.S. employees in key industries and functional areas about EIM & EII:in key industries and functional areas about EIM & EII: – Only 37% said they have a clear understanding of what their organization is trying to achieve and why – Only one in five was enthusiastic about their team and the organization’s / corporation’s goals – Only one in five said they have a clear “line of sight”– Only one in five said they have a clear line of sight between their tasks and their team and organization’s goals Only 15% felt that their organization fully enables them– Only 15% felt that their organization fully enables them to execute key goals – Only 20% fully trusted the organization they work for 6© Copyright 2011 EMC Corporation. All rights reserved.
- 7. View Another Way… Ifa football team had these players on the field:these players on the field: • Only 4 of the 11 players on the field would know which goal is theirs • Only 6 of the 11 would care • Only 3 of the 11 would know what position they play and what they are supposed to do • 9 players out of 11 would in some way• 9 players out of 11 would, in some way, be competing against their own team rather than the opponent 7© Copyright 2011 EMC Corporation. All rights reserved.
- 8. We Are AwashIn A “Sea Of Data”1We Are Awash In A Sea Of Data • In the information age, every organization is in the “data” businessbusiness • Data is growing exponentially, so are the challenges • Complexity is causing insight to be lost• Complexity is causing insight to be lost Source: IDC Digital Universe White Paper, Sponsored by EMC, May 2009 8© Copyright 2011 EMC Corporation. All rights reserved. 9 1 “Big Opportunities In Big Data”, Forrester , Brian Hopkins, 5/18/2011
- 9. Pictorial Representation OfInformation 9© Copyright 2011 EMC Corporation. All rights reserved.
- 10. Time Really IsMoney! 10© Copyright 2011 EMC Corporation. All rights reserved.
- 11. Data Is ComingAt Us FasterData Is Coming At Us Faster • In a recent TDWI survey of 450 CIO’s – 17% have a real time data warehouse – 90% plan on having a real time warehousep g – 75% will replace to get to a real-time solution “REAL -TIME IS A RAPIDLY BECOMING A NECESSARY FOUNDATION TO A VALUABLENECESSARY FOUNDATION TO A VALUABLE ANALYTICS SOLUTION!” 11© Copyright 2011 EMC Corporation. All rights reserved.
- 12. Data Is ComingFrom All DirectionsData Is Coming From All Directions • Data is now commonly entering into the enterprise from external sourcesfrom external sources – Marketing Lists – Neilson / AcxiomNeilson / Acxiom – Government – Bloombergg – TransUnion, Experian, Equifax – Radian 6 – Biz360 – Etc. 12© Copyright 2011 EMC Corporation. All rights reserved.
- 13. The Data IntegrationImperative 13© Copyright 2011 EMC Corporation. All rights reserved.
- 14. Summation Of ChallengesSummationOf Challenges • Business mandate to obtain more value out of the data (get answers)of the data (get answers) • Need to adapt and become agile to information and industry-wide changesinformation and industry wide changes • Variety of sources, amount and granularity of data that customers want to integrate isdata that customers want to integrate is growing exponentially • Need to shrink the latency between theeed to s t e late cy bet ee t e business event and the data availability for analysis and decision-making 14© Copyright 2011 EMC Corporation. All rights reserved.
- 15. Data Sources & RationalizationRationalization HowDo We Make Sense Of The Data… 15© Copyright 2011 EMC Corporation. All rights reserved.
- 16. What is Information? •Data = Documents, Pages Files,… • Structured Data / Metadata = Data + Syntax • Semantics = Meaning of Structured Data Elements • Information = Structured Data + SemanticsWhat is Information? Information Structured Data Semantics • Knowledge = Purposeful Combination of Information ABATE Wisdom Triangle Options = “Benefit Driven Usage” Wisdom Business Performance Management Making coherent judgments and inferences from the knowledge gained by evaluating all the possible outcomes Understanding Patterns = “Context & Function” Wisdom Knowledge Business Intelligence & D/W Making sense of these facts so that we can now derive conclusions from related and non-related information Data In Context = “Related” Information Performance Reporting Placing these facts into the context of your business model and operations process Unstructured Data Raw Data Reporting Structured and Unstructured raw elements or images (structured vs. unstructured) 16© Copyright 2011 EMC Corporation. All rights reserved. g ( )
- 17. Where is thedata? • Unstructured Data – Emails and text-based documentation, contract documentation, social media outlets – Web sentiment • Network Performance DataNetwork Performance Data – Gigabytes of event and fault data captured every day from networks – Data from various network layers (service, network and transmission) that are rarely brought together – Stores rich insights into root causes of alarms, network performance i d h kissues and other network events • Network Traffic Data – Captures type of data and traffic passing over network, level of BW usage, type of user – Could enable better traffic management and network investment decisions • Call / IP Detail Records – Typically archived and rarely used for detailed analysis due to volume – Provides insights on call patterns data usage and perceived QoSProvides insights on call patterns, data usage and perceived QoS • Web logs and Call Center Records – Traffic analysis common but rarely look at from specific customer’s journey – Can help move customers from labor-intensive channels to automation 17© Copyright 2011 EMC Corporation. All rights reserved. From article: “Unlocking the value in telecommunications data”
- 18. Analytics Maturity “What IsThe Best Thing That Could Happen?”“What Is The Best Thing That Could Happen?” Business Optimization – Moving Up The Maturity Curve BP Optimization “What If These Trends Continue?”“What If These Trends Continue?” “What Will Happen Next?”“What Will Happen Next?” Forecasting Predictive Analytics “Why Is This Happening?”“Why Is This Happening?” What If These Trends Continue?What If These Trends Continue? dvantage Statistical Analysis Forecasting petitiveAd Drill-Down Queries Exception Alerts Com Standard “Operational” Or “System” Reports Ad-Hoc Reporting 18© Copyright 2011 EMC Corporation. All rights reserved. Business Intelligence / Analytics Maturity Or System Reports
- 19. Why is AnalyticsMaturity Critical? ll d h lImmature organizations allocate a majority of time to data capture, while mature organizations action on data already captured Decision Making & Action Decision Making Analyze Information %TimeSpent Collect Data TimeSpent & Action Analyze Information % Analyze Information Predictive Analytics Collect Data Collect Data Classic Reporting BI & Analytic Tools First Generation EIM 19© Copyright 2011 EMC Corporation. All rights reserved. Less Mature More MatureLess Mature More Mature
- 20. New Technologies HaveChanged Game… Recent Technology Advancements Analytics RamificationsRecent Technology Advancements • MPP data platforms • Advanced in-database analytics • Analytics (Hadoop MapReduce) on Analytics Ramifications • Tackle more complex analytical opportunities to yield new insights • Empower users by delivering actionable• Analytics (Hadoop, MapReduce) on unstructured data • Integrate analytic results with operational systems Empower users by delivering actionable insights directly • Extend reach of BI/Analytics to new devices and next generation user experiences • Data visualization across multiple mediums and devices • Linking structured & unstructured data using semantics & linking • Enable self-service access to data • Increase visibility into data assets facilitating audits and compliance • Elastic architectures that simplifyusing semantics & linking • Self-service BI and data marts • New structured & unstructured “big data” sources • Elastic architectures that simplify management and administration • Collaboration around analytics and data assets associated with specific subject • Agile data warehousing concepts • ELT & data enrichment capabilities • Self-provisioned analytic sandboxes 20© Copyright 2011 EMC Corporation. All rights reserved. • Analytics collaboration
- 21. Big Data, BigPicture Analytics Portal (Collaboration &(Collaboration & Sandboxes New Data WatchesNew Data Watches Continuous Query Correlation &Correlation & NotificationsNotifications Deep Analytic Perspectives• Set-Top Box Analytics h l i i p – Channel Viewing Data • Advertizing Analytics – Interactive Services • Shopping, Home, Auto • Information Services & Web Vid O D d A l ti Deep “Data Pool” Historic, Federated, Multi-Participant, Cleansed … • Video On Demand Analytics – On Demand Programming Sales & Selections • Internet Analytics – IPDR Statistics, Bandwidth Utili ti 21© Copyright 2011 EMC Corporation. All rights reserved. Utilization, …
- 22. Actionable Information: Turning DataInto DecisionsTurning Data Into Decisions Turning Info Into Intelligence 22© Copyright 2011 EMC Corporation. All rights reserved.
- 23. Companies are tellingus that they’re trying to gain key insights about their customers, products, and operations, p , p Who are my most valuable customers? What are my Quality Of Service (QOS) metrics and are they i i ?” Where should I make revenue h i ? aluable custo e s? What are my most important products? What customers are most likely to churn? improving?” growth investments? How do I minimize What are my most successful campaigns? Where can I optimize network effectiveness? How do I minimize portfolio risk? What channels are most effective? What business areas are at greatest risk? Where do I have compliance risk? “77% of respondents are aware of bad decisions that managers have made within their organizations because they did not have access to accurate information” 23© Copyright 2011 EMC Corporation. All rights reserved. “The Fact Gap: The Disconnect Between Data and Decisions” BusinessWeek Research Services
- 24. Evolution to “ContinuousIntelligence” • Industry moving from historical and near- time reporting to real-time analytics that enable real-time actions / decisions • Ability to predict future trends using large volumes of historical / real-time data gaining traction with customers • Challenging economic times are driving service providers to focus on cost-cuttingp g programs (e.g. improved operations, avoiding violations) • Retaining customers is critical as new competition enters market and priceco pet t o e te s a et a p ce becomes second to customer experience • Improving access to real-time, actionable intelligence is a growing trend in the service assurance and sales areas 24© Copyright 2011 EMC Corporation. All rights reserved. se v ce assu a ce a d sales a eas Heavy Reading
- 25. Large-Scale, Real-World Analytics •How do I segment my clients? • Which trial is working better? • How can I predict fraudulent or other • K-means clustering • Mann-Whitney U Test • Logistic regressionHow can I predict fraudulent or other negative events? • Is this user likely to be interested in this ad? Logistic regression • Conjugate Gradient, SVM this ad? • Does this product appeal to some segments more than others? • How do I do hyper targeting of my • Log-likelihood • Cohort analysis• How do I do hyper-targeting of my high-value frequent customers? • Which features of a campaign result in user revisits? • Cohort analysis • Linear regression in user revisits? • Which campaigns have the best return on investment? • Financial models 25© Copyright 2011 EMC Corporation. All rights reserved.
- 26. New industry shiftunderway, and key to success is defining a process where organizations can continuously uncover and publish new insights about the businesspublish new insights about the business 1) Business A na l y t i cs L i fe c ycle 1 2) Bus. & IT Acquires and 1) Business Defines mandate and requirements 17% of1 2 q integrates data 5) Business Consumes insights enterprises feel they have a strategic shortage of 5 Strategic Business Initiative 3) Data Scientists g and measures effectiveness Data Scientists… a role that many did not even 3 4 Scientists Build and refine analytic models 4) IT Publishes new insights know existed 12 months ago1 26© Copyright 2011 EMC Corporation. All rights reserved. 1Source: Enterprise Strategy Group, 2011
- 27. CSP Analytics HotSpotsCSP Analytics Hot Spots Two major focus areas identified by the TM Forum include: • How can we understand the impact of product changes in days rather than months? • How do we value the revenues and costs of each customer? • How do we build a 360o view of the customer’s interaction with us? • How do we get a complete view across multipleHow do we get a complete view across multiple networks (e.g. broadband, wireless, voice)? Customer Analytics 27© Copyright 2011 EMC Corporation. All rights reserved.
- 28. Example: IP TVPerformance Analytics Moving from Reactive…ov g o eact ve… • Network Operations Centers monitor availability and performance from different systems • Correlation is traditionally slow, manual and not integrated with customer care • Care is typically contacted by frustrated customer before they understand customer impacted • Result is slow response and inability to be proactive with customer Report trouble • Sorry, we did not know you were having an issue… • We do not see any issues from this end… • It must be due to something out of our control… NEGATIVE experience Ticketing System experience Customer Data Network Operations Network Monitoring T di i l i i f k 28© Copyright 2011 EMC Corporation. All rights reserved. Traditional monitoring of network availability / health for devices, connections, app servers, etc.
- 29. Example: IP TVPerformance Analytics …to Proactive…to oact ve • Availability, Performance, Customer, Ticket data fed into analytics engine • Data analyzed in real-time and customer experience impacting events identified • Information available to customer care in Service Experience Dashboard Phone call, email, social media… p • Information can be used for credits, auto-notification or to support customer calls • Data can be used for off-line analytics to identify trends / opportunities Confirm Satisfaction • We know you were having issues and resolved it for you… • We appreciate your service and want to give you a credit… • <email sent indicating service issues identified and fixed> POSITIVE experience Ticketing System Customer Data Service Experience Dashboard Data Scientists Other Data Network Operations Network Monitoring Real-time Analytics Engine Analytics Modeling Constantly analyze data to identify trends and h h 29© Copyright 2011 EMC Corporation. All rights reserved. Network Monitoring Traditional monitoring of network availability / health for devices, connections, app servers, etc. improvements. Which regions have the most video issues? What channels do customers use for specific issues?
- 30. Example: Customer RetentionAnalytics • Customer Retention – Identify those at risk ofy abandoning their accounts. – Use logistic regression models, GAM models built in R, or SAS d lscoring models. – Build and execute models in the database, and reduce time to act. Al d di Credit Card Number Probability of Fraud • Also used to predict… – Usage fraud – Customer churn 7125 6289 5972 9510 15% 3955 8125 1327 7120 22% 2190 6379 9218 9290 7% SSN Probability of Churn 611‐43‐2435 15% 812‐35‐1035 22% – Customer profitability – Total customer experience (contract negotiations) 2760 1924 2864 0950 47% 4915 1908 8302 9940 12% 3534 7203 6200 4010 3% 253‐23‐2943 7% 732‐62‐1435 47% 483‐32‐5298 12% 821‐90‐8574 3% 30© Copyright 2011 EMC Corporation. All rights reserved.
- 31. Example: “What areour customers saying?” sql# SELECT cid, sum(tfxidf)/count(*) AS centroid FROM ( SELECT id, tfxidf, cid, row_number() OVER ( PARTITION BY id ORDER BY distance, cid) rank FROM blog_distance ) blog_rank WHERE rank = 1 GROUP BY cid; -MAP: NAME t t tNAME: extract_terms PARAMETERS: [id integer, body text] RETURNS: [id int, title text, doc _text] FUNCTION: | if 'parser' not in SD: import ... class MyHTMLParser(HTMLParser): ... SD['parser'] = MyHTMLParser() parser = SD['parser'] parser.reset() parser.feed(body) id | tfxidf -----+------------------------------------------------------------------- 2482 | {3,1,37,1,18,1,29,1,45,1,...}:{0,8.25206814635817,0,0.34311110...} 1 | {41,1,34,1,22,1,125,1,387,...}:{0,0.771999985977529,0,1.999427...} 10 | {3,1,4,1,30,1,18,1,13,1,4,...}:{0,2.95439664949608,0,3.2006935...} ... 31© Copyright 2011 EMC Corporation. All rights reserved.
- 32. What are thelargest analytics challenges? Data quality and a clear business case are critical to success Analytics programs need to…y p g • Enable an Information Mgmt strategy that will make accurate data a critical business objectivecritical business objective • Focus, focus, focus. Define a clear business case, show small wins, b ldbuild momentum. • Realize this is different than “reporting” and ensure the right platformensure the right platform and support org are put in place day one Challenges – creating data driven relationships from legacy infrastructure TM Forum’s Exploiting Analytics, Sep 2010 32© Copyright 2011 EMC Corporation. All rights reserved. TM Forum s Exploiting Analytics, Sep 2010
- 33. “Think Different”“Think Different”“ThinkDifferent”“Think Different” 33© Copyright 2011 EMC Corporation. All rights reserved. Apple Ad Campaign, 1997Apple Ad Campaign, 1997Apple Ad Campaign, 1997Apple Ad Campaign, 1997
- 34. Q&A 34© Copyright 2011EMC Corporation. All rights reserved.
- 35. THANK YOUTHANK YOU RobertJ. Abate, CBIP, CDMPJ , , Solutions Principal, EMC Consulting [email protected] Blog: infocus.emc.com/author/robert_abate/ (201) 745-7680 35© Copyright 2011 EMC Corporation. All rights reserved.