The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
Download as PPTX, PDF•0 likes•1,046 views
Kentik, founded in 2014 and headquartered in San Francisco, is a network traffic intelligence company that has experienced significant growth, supported by over $38 million in funding and more than 100 customers. The document discusses various network tools and techniques used for traffic analysis, anomaly detection, and performance monitoring, emphasizing the importance of integrating flow data with other analytics for comprehensive insights. It highlights challenges in data fusion and storage, targeting both network engineers and application teams to improve collaboration and understanding of networking issues.
![All contents © Kentik Inc. 6
Network Traffic Instrumentation
• Modern network devices can send traffic summaries =
“NetFlow”
• (Or, often, sFlow, or IPFIX)
• Which are all different protocols but have similar info
• [PROTOCOL, SRC/DST IP, PORT, MAC, VLAN, …]
• These are continuous streams of samples of traffic (*)
• Usually just from the headers - though more advanced
implementations can watch perf and L7 info
6](https://image.slidesharecdn.com/thenetworkknows-v3-avifreedmankentik-170405193317/85/The-Network-Knows-Avi-Freedman-CEO-Co-Founder-of-Kentik-6-320.jpg)
The Network Knows—Avi Freedman, CEO & Co-Founder of Kentik
- 1. The Network Knows Avi Freedman Kentik CEO & Co-founder
- 2. All contents © Kentik Inc. 2 Tools, tools, everywhere… Active Testing (ping/traceroute) APM BI Metric (App/SNMP/Server) BGP Hijack detection NPM Config Management Policy Analysis Event Correlation Routing Analytics Forensics Flow Tools Logging Traffic Engineering Threat Intelligence
- 3. All contents © Kentik Inc. 3 With all those tools, can you: • See when there’s a real problem? • And where the problem is – app, server, network? • Let the network group understand if there are app issues? • Let non-network groups understand the network’s impact (or not)? • Automatically detect traffic anomalies, attacks, and shifts? • Debug CDNs, cloud delivery, and the path to API partners? • And… How often do you hear “is it the network?” 3
- 4. All contents © Kentik Inc. 4 The Network Knows
- 5. All contents © Kentik Inc. 5 The Network Knows • Apps generate traffic • But the network delivers it • And can see authorized/specified • And unauthorized/unspecified traffic • Often including performance and Layer 7 info • And it knows the ‘routing’ – the path traffic will take • And if it’s internal, external, or your or others’ infrastructures 5
- 6. All contents © Kentik Inc. 6 Network Traffic Instrumentation • Modern network devices can send traffic summaries = “NetFlow” • (Or, often, sFlow, or IPFIX) • Which are all different protocols but have similar info • [PROTOCOL, SRC/DST IP, PORT, MAC, VLAN, …] • These are continuous streams of samples of traffic (*) • Usually just from the headers - though more advanced implementations can watch perf and L7 info 6
- 7. All contents © Kentik Inc. 7 + Other Network Telemetry • There’s also SNMP (you can think of NetFlow as a double-click into SNMP data) • As well as logs – interface up/down, fan+cpu+optic failures, re-config, routing up/down, memory or CPU issues • And a lot of work being done on “streaming telemetry” of every detail of a device and its software – will need modern time-series backends • And configs • And topology 7
- 8. All contents © Kentik Inc. 8 Network Nerd Use Cases for Network Knowledge Anomaly Detection Planning and Peering Traffic Engineering DDoS DefensePerformance Analytics Threat Analytics Service Creation Digital Forensics Customer Cost, Prospecting
- 9. All contents © Kentik Inc. 9 But Not Just for Network Nerds! • But systems and app folks should be able to debug also • And network people should be able to know if the blip matters to production traffic • So how do we tie systems together? • Make flow look like metrics and correlate there • Expose via APIs • Last resort – train others in flow usage
- 10. All contents © Kentik Inc. 10 OSS and Vendor Options for Flow • There are open source flow tools: pmacct, NFDUMP/NfSen, SiLK • And vendors (Kentik as SaaS, Arbor as appliance) • And you can DIY: pmacct front-ending Hadoop-ish SQL, or Elastic • NetFlow is UDP so it’s easy to replicate (samplicator) and send to multiple places 10
- 11. All contents © Kentik Inc. 11 OK, What’s so Hard?
- 12. All contents © Kentik Inc. 12 Awesome! What’s so hard? • Often requires fusing (geo, routing, app ID, threat intelligence …) • Flow can be trillions of records/day – think of it as a sampled superset of all of your logs • The OSS flow tools don’t cluster, so can’t store at scale • And don’t integrate with other systems • Metrics systems often choke on the high cardinality of IP addresses and port #s • DIY is hard but possible (usually pmacct+Elastic)
- 13. All contents © Kentik Inc. 13 Network Engineers Distributed Systems Engineers SREs Low level network developers And DIY is hard Resilience / Reliability Geo-distributed ingest Flow friendly data-store BGP Daemon Flow inspection & conversion Network protocols hacking Make all of the above work reliably Train all the other teams on the involved network protocols and their usage Required areas of expertise (because every presentation needs a Venn diagram)
- 14. All contents © Kentik Inc. 14 But don’t give up… • It’s still better to get started! • Even if aggregate-based in a flow tool • I can provide a host agent that will generate metrics along with flow (but be careful if you store IPs/ports in TSDBs)
- 15. All contents © Kentik Inc. 15 How To: Get the data. Fuse the data. Store the data. Use the data. Share the data.
- 16. All contents © Kentik Inc. 16 TCP stats data / app specific data Where to find this data ? Flow data NetFlow, SFlow, IPFIX SNMP, Streaming telemetry Sys/Event logs TACACS & Syslog App Server, Logs, Metrics BGP, IGP Path info NETWORK + + + = Combinatorially useful! + Router Router PCAP agent +User tags, Threat Intel, SDN Control, DNS, ping/trace
- 17. All contents © Kentik Inc. 17 A Broader View of “NetFlow” You can ALSO get performance data from the infrastructure: • Queue Depth • Retransmits per flow • TCP latency • Application Latency From: • Host software (nProbe) • Sensors / Taps • Webserver logs (Nginx) • Cisco AVC supported routers 17
- 18. All contents © Kentik Inc. 18 Fusing data for richer traffic analytics Flow or BGP or SNMP or DNS or logs alone are not enough. This becomes much richer when combined with: • Performance and layer 7 information • BGP attributes • Geography • Tags (rack, department, customer…) • Config changes and software versions • Threat intelligence and known-bad IPs Fusing should be near real-time, performed at ingest and data specific 18
- 19. All contents © Kentik Inc. 19 Summary and Take-Aways
- 20. All contents © Kentik Inc. 20 Quick Demos: Grafana Kentik Host Agent
- 21. All contents © Kentik Inc. 21 Overview Kentik is the network traffic intelligence company. • Founded 2014 • HQ: San Francisco • 100+ Customers • $38M in Funding • 60+ Team Members • 600% Growth in 2016