Service Assurance Constructs for Achieving Network Transformation by Sunku Ranganath

Editor's Notes

• #6: Sunku
• #7: At a very high level, Service Assurance includes 3 key elements – Monitoring, Presentation and Provisioning. Monitoring includes getting the insight into various platform & network counters to correlate against established KPIs The monitoring interfaces need to be able to integrate with both legacy and next gen management/controller systems while providing open standard interfaces Presentation is the ability of reporting the metrics to take relevant action on service level changes. Traditionally this is done by human intervention to address failures/violation but with the amount of data available for processing the goal is to move towards dynamic resolution based on the configured parameters. Provisioning involves idea of reconfiguring platform resources to meet the service level objectives in a service level agreement. Today’s presentation will focus first two elements
• #8: Phase1: Make sure that the relevant telemetry is instrumented where appropriate and that telemetry is exposed through collected to existing management systems. Basically the goal of phase 1 is to interwork with existing management systems. Phase 2: moving to NFV deployment world, we want to make sure that the virtual infrastructure as well as the management and orchestration layer can understand the telemetry and events that it is receiving so that it can take appropriate action based on that telemetry Phase 3: is where we want to integrate with Ml to allow us to make more intelligent placement decisions and adjustments to scheduling policy. More importantly to correlate NFVI failures with VNF performance issues to be able to predict failures to automatically adapt our environment as required based on the current state of the platform and the telemetry of the different sub systems. Phase 1 will continue forever: as in the ingredient teams will be always upstreaming new features to collectd. Phase 2 and Phase 3 is what we are currently implementing
• #10: Platform Telemetry agents: Collectd - https://github.com/collectd/collectd Local Platform Agent: Resource Management Daemon - https://github.com/intel/rmd
• #17: Sunku
• #18: John When we started out we would deploy collectd directly on the platform. This caused issues as every system is different. If we were dealing with customers we would have no control over what they had installed on their system. Within the barometer project we decided to have a SA package. This included Collect influx and graphana containers. This allows the customer to deploy with ease. We then added ansible scripts to make the deployment process a one line command. In short a new customer should be able to take our package, deploy it, and see platform metrics on a web browser within 10 minutes. John When we started out we would deploy collectd directly on the platform. This caused issues as every system is different. If we were dealing with customers we would have no control over what they had installed on their system. At the same time the opnfv community was shifting towards containers. For this reason we decided to containerise collectd. This gave us multiple benefits. We could insure that collectd would be installed on the same environment. Its much quicker to pull the collectd container instead of building and installing it on your system. None of the software within the container would interfere with what is already installed on the system. When we containerised collectd we also containerise other nessary components. Once we get the metrics from collectd we need to save them for which we used influxdb. In an effort to provide end to end When we containerised collectd we also decided to containerise other important components. These included: Influx which we use as our time series database and Grafana which we use to graph the metrics within influx After this when we started talking to customers we found a few issues. Customers were unfamiliar with containers and need help setting them up. Deploying at scale was still slow. For this reason we created ansible scripts to deploy and configure the containers.
• #19: Providing Ansible support has made it easy for us to introduce SA to various customers. Idea is to provide one click install that installs & configures necessary containers like Collectd, Influxdb, Grapahana, Kafka as a message bus VNF Event Stream (VES) containers. We made it easy to install, configure and visualize various NFVi metrics at scale. Various customers are currently using them and playing with it. On the plugins, we have tight collaboration with NFVi BKC that tests the plugins across combinations of various operating systems, platform generations, NICs, etc. This ensures our plugins are up to date with IA platforms. On automation front, we have good integration with opnfv ci functest to ensure barometer deploys with opnfv installers without regression. Barometer currently supports Apex and we will be adding Compass support.
• #20: The downside of Collectd community is that there is no established cadence of releases or merging of pull requests. In order to provide early showcase IA features through Barometer we came up with 3 container approach: Stable container provides latest stable branch of Collectd that has fully tested and validated plugins Master container provides plugins & bug fixes that are on master and not yet in a release Experimental container provides latest and greatest of plugins by cherrypicking the newest pull requests instead of waiting for making onto master. This way we provide early access to telemetry feature set. On downstream, we have strong engineer to engineer partnership with RH on OSP to ensure IA specific plugins make it into OSP releases.
• #23: Sunku The basic goal of ML is to achieve close loop automation using IA metrics NSB is used to generate the data for the ML analysis We have had a few ML teams engaged and they have been able to correlate multiple IA metrics to packet loss on an EPC We are still in the early stages of ML.
• #24: KK Prometheus is a project originally from SoundCloud, then it has been moved to Cloud Native Computing Foundation (CNCF), as second hosted project, right after kubernetes. This is an is an open-source monitoring system, which has few interesting features, like a dimensional data model flexible query language built-in efficient time series database and modern alerting approach The integration with collectd is already in place. We have performed verification how it can be integrated with two ways: One is collectd native plugin, which is serving http endpoint from where Prometheus server can scrap latest metrics. And the second one is Prometheus collectd exporter which acts like a proxy, where collectd is writing data with its network plugin and Prometheus server can get data from there. What is worth to highlight here is that Prometheus is working in pulling model, rather then standard collectd pushing of metrics. So a bit different architecture.   Prometheus is part of many solutions, like NGCO, on which Sunku will tell you more in a moment RH OSP SA framework Or also as a proposal in ONAP to integrate with OOF in the edge area   And there is also container incoming to be part of barometer collection. Bottom line is that all the platform stats can be pulled into Prometheus today without any additional development
• #27: KK Open Network Automation Platform is a project under Linux Foundation Networking governance. By unifying member resources, ONAP is accelerating the development of a vibrant ecosystem around a globally shared architecture and implementation for network automation, with focus on open standards. It „provides a comprehensive platform for real-time, policy-driven orchestration and automation of physical and virtual network functions that will enable software, network, IT, cloud providers and developers to rapidly automate new services and support complete lifecycle management.” Our goal is to enable closed loop automation, currently by engaging on VES and EPA/HPA projects.
• #28: KK So VES or VNF Event Stream is a project which goal is to enable significant reduction in effort required to develop and integrate VNF telemetry-related data into automated VNF management systems, by promoting convergence to a common event stream format and collection system.   In current implementation collectd is sending metrics to kafka bus, from there VES application is picking them up, unifying with given schema and sending to VES collector (which is part of DCAE). It is choosen solution in the core area.
• #31: KK
• #33: KK
• #35: KK What is PNDA? „The scalable, open source big data analytics platform for networks and services” PNDA similar to OPNF or ONAP is a project under Linux Foundation Networking In terms of functionality PNDA aggregates data like logs, metrics and network telemetry Scales up to consume millions of messages per second Efficiently distributes data with publish and subscribe model Processes bulk data in batches, or streaming data in real-time Manages lifecycle of applications that process and analyse data Lets developers to have insight and explore data using interactive notebooks   Entry point for collectd is Kafka bus, there are few ways to ingest data for consumption by analytic apps. In raw json format directly with write_kafka plugin Or with network plugin through logstash with raw json or avro formatting We have performed verification it works properly with RedPNDA and also working with CISCO as a partner for better customer oriented use cases for analytics.   PNDA can integrate with MANO systems supporting closed loop automation on the east/west level. And there is proposal about integrating PNDA in DCAE (part of ONAP) by replacing CDAP with similar functionality.
• #36: KK AR: change color scheme / bigger font Direct ingestion with write_kafka could have better performance But preferred format in PNDA is AVRO, this is data serialization framework with given schema in json. (if remember correctly) check was with generic avro serializer, not pnda one, so additional field mutae may be required, which should not be case with pnda-avro codec. They were same by the time check was made, with different single step of base64 encoding, which was the issue on decoding messages with given example of consumer in redpnda. PNDA AVRO schema { "namespace": "pnda.entity", "type": "record", "name": "event", "fields": [ {"name": "timestamp", "type": "long"}, // time when was generated/ingested by pnda {"name": "src", "type": "string"}, // e.g. collectd {"name": "host_ip", "type": "string"}, // source of host where was generated {"name": "rawdata", "type": "bytes"} // rest of raw data ] }
• #40: Looking to help these with IA features. Feature exposure, provisioning & telemetry. We are looking to enable/fill in these gaps.
• #43: Sunku
• #44: Sunku
• #45: KK
• #46: KK