Etiya White Paper_ABDR
- 2. Contents EXECUTIVE SUMMARY .................................................................................................................................... 0 INTRODUCTION ................................................................................................................................................ 1 ABOUT BIG DATA ......................................................................................................................................................... 2 RETHINKING DATA MANAGEMENT ............................................................................................................... 3 CALL DETAIL RECORDS (CDR) AND TRAFFIC ANALYSIS ....................................................................................................... 3 Traffic Analysis ..................................................................................................................................................... 4 Analytics Big Data Repository ............................................................................................................................................. 4 EXTRACTING BUSINESS VALUE ..................................................................................................................... 5 CAMPAIGN MANAGEMENT ............................................................................................................................. 6 REAL TIME ................................................................................................................................................................... 7 CONCLUSION .................................................................................................................................................... 7
- 3. Executive Summary The new technology advancements, quality issues and cost pressures are among the key drivers forcing communication service providers (CSPs) to shift toward more accurate ways for controlling network planning and enhancing quality of service. With the evolution of networks and access devices, new information has become available, turning extremely rich material into a valuable competitive differentiator for CSPs. New tools could help CSPs differentiate themselves in a very competitive environment. More sophisticated and real-time analytics solutions as well as tools to manage big data have been hailed to decrease churn and to understand how their customers consume their services. By doing so, they are able to increase average revenue per user and offer more customized services, generating new revenues. The authors of this white paper believe that analytics data repository is a tool and a measure that is made available to CSPs, and recommend that the big data repository is implemented as to provide detailed and automated insight into transaction paths and performance. In this white paper, it is aimed to provide a broad understanding of telecom data (structured and unstructured), the challenges of big data analytics and possible solutions to these challenges. In addition, while the term big data is used in the context of the CSPs, same concept applies to Information Technology (IT) organizations.
- 4. 1 Introduction From the beginning of the Information Technology, the demand for storing and processing data has always been ahead of the capability of the technology and tools available. It may be the first time in the history that the data storing and processing technologies have advanced beyond the “simple” needs of people and industry. The past few years have witnessed immense commercial investments in solutions that address the processing and analysis of big data. Big data opens a vast array of applications and opportunities in multiple vertical sectors including, but not limited to, retail, hospitality, energy, media, utilities, healthcare and pharmaceutical, telecom, manufacturing, government and transportation, homeland defense and security, and the emerging technology fields. In the past era, the most of the Communication Service Providers (CSPs) invested heavily on the big data technologies. They have been having challenges on unlocking the data potential accumulated in their environment. In order to create value from the data deluge, big data management and strategies need to be aligned with business strategies and plans. Leveraging big data is sought after. CSPs need to store what is important and extract results to serve their own business objectives. Most of the CSPs are currently seeking ways to utilize the big data, converting hard copies to soft copies and immigrating this information into cloud, turn them into value for their businesses in the form of customer satisfaction, reduced operational costs, increased service quality, and increased profits. The CSPs are extracting the means of knowing more about their customers (their shopping habits, what ticks them off to
- 5. 2 make a purchase) to decrease churn, maintain smarter networks, and to generate new revenues. Big Data refers to a massive volume of both structured and unstructured data that is so large that it is difficult to process using traditional database and software techniques. Telecom structured data sources are many and varied. Some sources are completely static or semi- static while others are very dynamic in nature. The telecom industry was an early adopter of data mining technology and hence many data mining applications exist; such as fraud detection, customer profiling and network fault isolation. The CSPs maintain a great deal of data about their customers. In addition to the general customer data that most businesses collect, the CSPs store the call detail records, (CDR) which precisely describe the calling behavior of each customer. This information can be used to profile the customers and these profiles can then be used for marketing and/or forecasting purposes. Big data offers CSPs a real opportunity to gain a much more holistic view of their operations and their customers, and to further their innovation efforts. About Big Data Big data represents a challenge in the technological capabilities to store and analyze data, in accordance with 3 V’s of big data: Volume, velocity and variety. The telecom industry generates and stores a tremendous amount of data, which is difficult to calculate manually. Data volume that is considered to be big data now will be comparably small data in the near future. Thus, “big” does not describe the real size of data but its relative size to the capabilities of technology of the present day. Policy and rules are just the beginning of new era where telecom networks will adapt themselves in real-time to subscriber’s needs, profile, and quality of experience (QoE) management. Big data comes from a greater variety of sources, both structured and unstructured, in the form of text, audio, video, and human language, as well as semi-structured data including XML and RSS feeds, and multi-dimensional data from a data warehouse to add historic context to big data. The type of these data are called call detail data, which describes the calls that traverse the telecommunication networks, network data, which describes the state of the hardware and software components in the network, and customer data, which describes the telecom customers. Telecom data pose several interesting issues for data mining. The first concerns scale, since databases may contain billions of records and are amongst the largest in the world. A second issue is that the raw data is often not suitable for data mining. Because many data mining applications in the telecom industry involve predicting very rare events, such as phone fraud, rarity is another issue. The fourth data mining issue concerns real-time performance: many data mining applications, such as fraud detection, require that any learned model/rules be applied in real-time.
- 6. 3 Telecom networks are extremely complex configurations of equipment, comprised of thousands of interconnected components. Each network element is capable of generating error and status messages, which leads to a tremendous amount of network data. This data must be stored and analyzed in order to support network management functions. Due to the enormous number of network messages generated, IT technicians cannot possibly handle every message. Rethinking Data Management Customer loyalty is becoming difficult to manage since the Telecom customers are easily attracted to other CSPs by a more attractive offer, plan, or new device. Traditional revenue streams, such as short message service (SMS) and international direct dialing (IDD) calls, are being eroded by over-the-top (OTT) players such as WhatsApp, Skype, Tango, SnapChat, and Viber. Meanwhile the explosive use of mobile devices creates complexity and big demand on network capacity. This section will highlight call detail records and how they can be used to get all the insight into key call quality metrics and variables. Call Detail Records (CDR) and Traffic Analysis Each time a call is placed on a telecom network, descriptive information about the call is saved as a call detail record. The number of CDR generated and stored is huge. Most CSPs store several months of call detail online, leading to the storage of tens of billions of call detail records to be stored at any time. CDRs include adequate information to describe the important characteristics of each call at real-time. At a minimum, each call detail record will include: • the originating and terminating phone numbers, • the date and time of the call, • the duration of the call. CDR is available to be mined and analyzed near real-time. Call detail records are not used directly for data mining, since CDR holds data at the call level. CDRs needs to gathered and summarized into a single record to describe a caller’s behavior. The choice of summary variables is critical in order to obtain a useful description of the associated customer. Some of the variables can be listed as follows: • average call duration • % no-answer calls • % calls to/from a different city code • % of weekday calls • % of daytime calls • average # calls received or originated per day • # unique area codes called during a specific time period These variables can be used to build a customer profile.
- 7. 4 Traffic Analysis In an Etiya study, a month’s one Terabyte network data was analyzed for a leading CSP in Turkey to forecast wireless network traffic and predict anomalies. Etiya performed customer churn analysis on IVR and Call Center data. All tasks were performed in real-time for three scenarios. The first aimed at the single view of the customer and to create a customer storyboard. Etiya enriched the customer data with demographic information and then, calculated Customer Experience Index (CEI) using pre-defined network KPIs. Predictive analytics were followed by specific customer mobility on usage map. The second scenario was to create a proactive customer monitoring profile by predicting Quality of Service (QoS) and Quality of Experience (QoE). An alarm-based solution was created to notify the operation upon a decreased QoS or QoE. Etiya was able to provide and predict the next ten minutes of future events. The third aimed at tasks against a subset of customers; i.e. VIP subscribers. When predefined errors were analyzed, it was found that QoE with number of errors and delays could be forecasted per hour. If human error, call-centers could be conditioned against these possible mistakes. Analytics Big Data Repository At Digital Disruption event of TM Forum in December 2014, laid the groundwork for the creation of a common data repository to eliminate the need for multiple copies of data. At the third phase of the project, Etiya team is a part of developing an innovative Analytics Big Data Repository (ABDR) to help CSPs advance customer experience and reach business growth by avoiding data replication, saving on extract, transform and load (ETL). ABDR provides a unified layer that can support multiple use cases and analytical systems while addressing the challenges such as hardware space and energy costs, etc. In the Etiya use case, the scenario is created as follows: Customers decide on their threshold level for usage types. CRM system receives the selected thresholds. Data is collected per usage type of the customer in the Analytics Big Data Repository: Voice Usage, Internet Usage, and SMS Usage. Usage data is then fed to the Complex Event Processor (CEP). CEP compares the accumulated usage against the selected thresholds and triggers actions to notify customers real-time if they exceed the limits they decided.
- 8. 5 Extracting Business Value Communication service providers conduct analytics programs that enable them to use their internal data to boost the efficiency of their networks, segment customers, and drive profitability with some success. The potential of big data poses a challenge on how to combine large data sets across the entire telecom value chain, from network operations to product development to marketing and customer service and to monetize the data itself as data as service (DaaS). In order to gain leverage, CSPs manage the four key attributes of big data: • Volume - the quantity of data to be captured continues to grow exponentially. • Velocity – the bits and bytes have to be processed at high speed. • Variety - data comes in many formats, from diverse sources. • Value - data needs to be converted into meaningful insights.
- 9. 6 Campaign Management In the advertising business, sensory inputs help the marketers and advertisers correct course of action in order to meet their one specific goal to drive people into a store, to buy a product or service, or register for an event. The internet acts as a collector of these sensory inputs from multichannel, providing a huge variety in data. In an effort to cut through the advertising clutter and ensure that consumers take a note of their ads, some marketers try to increase by buying variety of ad space within the same environment (plane, bus, etc.) or the sensory input by adding empty space in order to accentuate the brand logo or product illustration. In marketing, the challenge comes from activating new campaigns as the capacity of a measuring one campaign may not produce the same reading when measured under the same prescribed conditions. In today’s competitive landscape, the latency is no longer acceptable. Latency is also caused by inefficient approval flow with CSPs. The campaign management needs to stream digital data and run a real-time analysis. Customer Experience Management (CEM) emphasizes the concept of offering targeted and tailored campaigns to customers. Finding the best offer for the customer, requires a detailed analysis on customer behavior. There are technologies and search engines that assemble real time offers to shoppers in 200 milliseconds, using advanced analytics based on location, age, gender, and both historical and immediately preceding online activity, along with the most recent responses of other customers. In today’s campaign management systems, there is a lack of unified customer view for marketing campaigns. Customer behaviors stored data may be stored in different databases.
- 10. 7 Therefore, there is a gap between how often the customers use one service and how much of the data or voice is transferred, using their quota of services and the amount paid for these services. Helping the customers find the optimal price plans mean helping the operator itself. The business implication of happy customers is the increased revenue through increased ARPU, increased acquisition rates and decreased churn rates. Most of the customer related analytics on the quest for finding the best offer relies on the usage behavior of the customer. Usage data is processed and stored in high volumes, therefore analyzing the usage data for each customer is in the scope of the big data technologies. Currently most of the campaign offers are determined through usage analytics by manual processes. Many customer service providers employ ‘Business intelligence’ teams to analyze the information stored in their enterprise Data Warehouses and distribute the campaign proposals to their outbound sales channels periodically. To automate this process, a powerful analytics engine is necessary as well as a good business strategy to implement proper analytic techniques. Real Time Big data technology is capable of processing high volume of data in a very short time. Usage data have an enormous potential to understand the shortcomings and needs of the customers if monitored and understood correctly. Event-based analytics engines are proving very useful in the offering the best campaigns and additional services to the customer at the right time. The power of real-time campaign offers lies in catching the correct timeframe that customer is in need. Customers are much more likely to accept the upsell or cross-selling offers when their quota is about to end, or they tend to buy additional packages when they use their service in a different location or from a different device. The real-time usage event analysis is used to capture these moments to offer the best campaign for the customer. The automation of this process is another benefit for the sake of reducing operational costs. Conclusion Customers decide on their threshold level for usage types. CRM System receives the selected thresholds. Data is collected per usage type of the customer in the Analytics Big Data Repository (ABDR). There will be real-time network optimization due to real-time detection of dropped calls, error perturbation, sensors not reporting data, etc. There will be real-time customer experience management with real-time dropped call resolution, or real time VIP campaign monitoring, etc. With ABDR, last but not least, there will be real-time mobile spam and fraud detection. Big data analytics paves the way for the real-time marketing based on where the customers have been, where the customers are, and a prediction of future behaviors to determine their needs; supplying real-time top-ups and dynamic charging, and eventually predicting what they are purchasing next and knowing the ‘when’.
- 11. 8 About the Authors Bengi Tozeren, MBA Marketing Specialist II, Business Development Tel. +90 (212) 483 – 7101 [email protected] Rukiye Cetiner, MBA Sr.Product Manager, Product Development Tel. +90 (212) 483 – 7101 [email protected] Information on ABDR Catalyst Project: Abdulkerim Mizrak: [email protected] Rukiye Cetiner: [email protected] Request a demo at [email protected] Published by Etiya http://www.etiya.com Yıldız Teknik Davutpasa Kampüsü Teknopark, Cifte Havuzlar Mahallesi, Eski Londra Asfaltı Caddesi 151/1 B No:301 İstanbul, Turkey Information as of 2015 The written statements on this whitepaper concerning Etiya solutions and projects as well as topics around big data are forward-‐looking statements, which involve a number of risks, and uncertainties that could cause actual results to differ materially from those in such forward-‐ looking statements. The risks and uncertainties relating to these statements include, but are not limited to, risks and uncertainties regarding fluctuations in earnings, intense competition in IT services including those factors which may affect our cost advantage, challenges and enablers of written topics, reduced demand for technology in the key focus areas, disruptions in telecommunication networks, political instability, IT security and information governance, legal restrictions on ARPU, OPEX and CAPEX concepts, and unauthorized use of our intellectual property and general economic conditions affecting Telecom and ICT industries.