SQL Server 2008 R2 StreamInsight
0 likes•837 views
Massive amounts of data are being generated from various sources like cell phones, sensors, web logs etc. This ambient data needs to be processed in real-time to enable scenarios like fraud detection, manufacturing process control, network monitoring etc. SQL Server StreamInsight provides a platform to process data streams with low latency queries, enabling near real-time analytics and action. Key capabilities include filtering, correlating, aggregating events over windows using a LINQ-like declarative query language.
SQL Server 2008 R2 StreamInsight
- 1. Petabytes for Peanuts! Making sense of “Ambient Data” SQL Server Stream InsightIng. Eduardo Castro, PhDComunidad [email protected]://ecastrom.blogspot.com
- 2. Key Takeaways…Massive shift in how we process dataIncredible data volumesRemaking how we discoverChanging the Scientific MethodReducing latency & impedanceExtreme Scale Data ProcessingStream Processing (Several Views)From “programs” to “queries”What’s up with this “anti-SQL” stuff anyhow?
- 3. 1997Storage Cost: $~1.00Transfer Time: ½ hour2009Storage Cost: ~0.1₵Transfer Time: 8 sec.1982Storage Cost: $~2000Transfer Time: 1 day“Free” Storage Power
- 4. Ambient Data?Over 84 percent of Americans have cell phones, according to Steve Largent, president and CEO of CTIA. While two trillion minutes were used in 2007, an 18 percent increase over 2006 talk times. More than 48 billion text messages were sent in the month of December 2007, an average 1.6 billion messages per day. The rate of text messaging represented a 157 percent increase over December 2006 texting. http://www.clickz.com/3628985Text Message Traffic in US: 160GB / day 58TB / yearVoice traffic in US (GSM encoding) 200PB / year
- 5. The Old WorldData volumes constrained by human typing speedApp & Data formed closed systemAppAssume 200M people in US typing 8 hr / day @ 10K keystokes / hour: 2TB/hror ~6PB / yearDB
- 6. The Old New WorldAvailable data explodedAvailable DataQuestions toAnswerWhat data shouldwe throw out?Design SchemaDesign ETLWhat if we havea new question?DW Nirvana!
- 7. The New World of Abundant DataSave All Available DataHypothesize Theorize TestNew Question to AnswerAlgorithmicProcessingRun “query”over data…ExploitCorrelation…Correlation isEnough!Analyze reduced dataThe CMS front end of the Large Hadron Collider records 1TB/sec!http://blogs.discovermagazine.com/cosmicvariance/2006/09/27/lhc-factoids/Interesting Read: The Petabyte Age: Because More Isn't Just More — More Is Differenthttp://www.wired.com/science/discoveries/magazine/16-07/pb_intro
- 8. Analyze Model Monitor1Event Stream both stored and processedEvent ProcessingEngine4Produce real time alerts and actionEvent StreamAlerts & Action3Models installed in event processing engineCorrelation Model2Analysis produces event correlation modelsAnalysis
- 9. Extreme Scale Data ProcessingSourceDWTraditional Data WarehouseSourceSourceETLSourceSourceAnalysis / ReportingSourceSourceExtreme ScaleData ProcessingDWNon-traditionalSources12Majority of data filtered or discardedAll data retained and reprocessedAnalysis / ReportingAnalysis
- 10. SQL Server 2008 R2 – StreamInsight TechnologyData volumes are exploding with event data streaming from sources such as RFID, sensors and web logs The size and frequency of the data make it challenging to store for data mining and analysis. The ability to monitor, analyze and take business decisions in near real-time
- 11. SQL Server StreamInsight’sSQL Server StreamInsight’s ability to derive insights from data streams and act in near real time provides significant business benefits. Some of the possible scenarios include: Algorithmic trading and fraud detection for financial services Industrial process control (chemicals, oil and gas) for manufacturing Electric grid monitoring and advanced metering for utilitiesClick stream web analyticsNetwork and data center system monitoring.
- 12. .NETC#LINQStreamInsight Application DevelopmentStreamInsight Application at RuntimeEvent sourcesEvent targetsInputAdaptersOutputAdaptersStreamInsight EngineDevices, SensorsPagers &Monitoring devicesStanding QueriesKPI Dashboards, SharePoint UIWeb serversQuery LogicQuery LogicTrading stationsEvent stores & DatabasesQuery LogicEvent stores & DatabasesStock ticker, news feedsStreamInsight Platform
- 14. EventsRepresent the user payload along with temporal characteristicsStreamsSequence of eventsFlows into (one or more) standing queries in StreamInsightengineQueriesOperate on event streamsApply desired semantics on eventsAdaptersConvert custom data from event sources to / from StreamInsight eventsKey Concepts
- 15. EventComplex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency. requestoutput streaminput streamresponseWhat is CEP?
- 16. LatencyRelational Database ApplicationsCEP Target ScenariosOperational Analytics Applications, Logistics, etc.Data Warehousing ApplicationsWeb Analytics ApplicationsManufacturing Applications Financial Trading ApplicationsMonitoring ApplicationsAggregate Data Rate (Events/sec)Event Processing Scenarios
- 17. Use Case: Customer SegmentationAnalysis of Click Streams on MSN.comWeb Server log streamed into StreamInsightCategorizing user behavior based on URL:Click targetsSearch keywordsSegmentation of user IDs into marketsAdapting navigational structure and ad placement in real timePatterns over time windows: user first clicks PageA, then PageB, then PageC within X secondsHigh performance requirementsMillions of online usersLow latency (seconds)Possible late events
- 19. Use Case: NBC Sunday Night Football1Telemetry Receiver4StreamInsightListener AdapterGeoTag and group by regionSQL AdapterPerfCounter Adapter2Count total eventsCount session startsCount active sessions3
- 20. Use Case: Data CenterPower ConsumptionVisualizeProcess InformationComplex Aggregations/CorrelationsCentraltime seriesarchiveQueryETWInput AdapterQuery21QueryPower MeterInput Adapter3
- 21. ChallengesHow do I …detect interesting patterns?reason about temporal semantics?correlate data?aggregate data?avoid writing custom imperative code?create a runtime environment for continuous and event-driven processing? As a developer, I need a platform!
- 22. Query ExpressivenessSelection of events (filter)Calculations on the payload (project)Correlation of streams (join)Stream partitioning (group and apply)Aggregation (sum, count, …) over event windowsRanking over event windows (topK)
- 23. ProjectionFilterCorrelation (Join)Aggregation over windowsGroup and AggregateQuery Expressivenessvar result = from e ininputStreamgroup e by e.id intoeachGroupfrom win ineachGroup.TumblingWindow(TimeSpan.FromSeconds(10))selectnew { eachGroup.Key,avg = win.Avg(e => e.W) };
- 24. ConclusionCEP Platform & APIEvent-triggered, fast ComputationAPI for Adapters, Queries, ApplicationsDeclarative LINQFlexible Adapter APIExtensibleSupportability
- 25. Q&A
Editor's Notes
- #11: Data volumes are exploding with event data streaming from sources such as RFID, sensors and web logs across industries including manufacturing, financial services and utilities. The size and frequency of the data make it challenging to store for data mining and analysis. The ability to monitor, analyze and act on the data in motion provides significant opportunity to make more informed business decisions in near real-time
- #19: NBC Sunday Night Football: live streaming through SilverlightRich client experience, multiple camera anglesNeeded: track, monitor, analyze user behavior, based on silverlight Media analytics