Boundary Front end tech talk: how it works
2 likes416 views
This document discusses Boundary's real-time data streaming and visualization capabilities. It describes how lightweight collectors intercept and collect meter data via TLS authentication from multiple data sources. The data is stored and streamed in real-time at high resolution with sub-second latency. The streaming UI provides intuitive dashboards to view the continuously updating data. It also outlines the data structure and subscription process, and discusses strategies for optimizing large state dumps and resubscriptions to address data and subscription problems. Potential solutions and next steps are proposed, including stratified queries, top-N limitations, web workers, and using HTML5 local storage.
![JSON
Example port:protocol subscription State Object
{ {
channel:""/query/ac3941b8924a2f73/custom9 "1331234598000:80:6":"{
query92f536bbc93f159418b98f1a973b5dc4e78379 "epochMillis":=1331234598000,
filter979564d6c94c9394f479abae906ff95a3411e9 "portProtocol":="80:6",
aggregation9b5cc18d49098d949c29a08497037b08a30e6" "ingressPackets":"100,
data:"{ "ingressOctets":"8463,
insert:"[[1331234598000,"80:6,"100,"8463,"93," "egressPackets":"93,
9672],"[1331234925000,"8080:6,"242,"39657,"243," "egressOctets":"9672
40865],"…], },
keys:"[epochMillis,"portProtocol], "1331234925000:8080:6":"{
schema:"[epochMillis,"portProtocol,"ingressPackets," "epochMillis":=1331234925000,
ingressOctets,"egressPackets,"egressOctets], "portProtocol":="8080:6",
timestamp:"1331235119001 "ingressPackets":"242,
} "ingressOctets":"39657,
} "egressPackets":"243,
"egressOctets":"40865
}
…
}](https://image.slidesharecdn.com/front-endtechtalk-120824123400-phpapp02/85/Boundary-Front-end-tech-talk-how-it-works-13-320.jpg)
Boundary Front end tech talk: how it works
- 2. Tech Talk - Frontend Stephen Boak ([email protected]) Mark Mahoney ([email protected]) Matt King ([email protected])
- 3. When we say real-time we mean it: JSON being pushed to the browser once a second and visualizations built to constantly move and adapt to streaming data
- 5. Meters ! Lightweight and highly scalable, these sit on the cloud as well as private data centers across virtual or physical servers.
- 6. Intercept Meter data via Transport Collectors Layer Security (TLS) Authentication.
- 7. Collect all of the data at high resolution to gain rich insight into complex Data Store environments & problems before they can impact critical business services.
- 8. Real-time continuous data streaming at Streaming high resolution, low latency intervals.
- 9. Stream data to your dashboard with sub-second latency, Streaming UI providing you with intuitive, powerful dashboard.
- 10. How We Get Data • CometD server (streaker)! • Pulls in data from multiple backend services and streams it out to clients! • Can also aggregate and filter data on demand
- 11. Subscriptions 1 2 3 4 5 Make a Get back a Subscribe to Receive Receive subscription request subscription ID new query state-dump add/remove (query and filters) (Query ID) (unique channel) (inserts & schema) messages !! ! ! ! !
- 12. Data Structure • Schema and key (which fields are concatenated for each record) • State dump with schema for requested time window (up to 100kb state dumps) • Save bandwidth with N-tuples indexed by the schema • Adds/removes update the state with keys (subset of schema)
- 13. JSON Example port:protocol subscription State Object { { channel:""/query/ac3941b8924a2f73/custom9 "1331234598000:80:6":"{ query92f536bbc93f159418b98f1a973b5dc4e78379 "epochMillis":=1331234598000, filter979564d6c94c9394f479abae906ff95a3411e9 "portProtocol":="80:6", aggregation9b5cc18d49098d949c29a08497037b08a30e6" "ingressPackets":"100, data:"{ "ingressOctets":"8463, insert:"[[1331234598000,"80:6,"100,"8463,"93," "egressPackets":"93, 9672],"[1331234925000,"8080:6,"242,"39657,"243," "egressOctets":"9672 40865],"…], }, keys:"[epochMillis,"portProtocol], "1331234925000:8080:6":"{ schema:"[epochMillis,"portProtocol,"ingressPackets," "epochMillis":=1331234925000, ingressOctets,"egressPackets,"egressOctets], "portProtocol":="8080:6", timestamp:"1331235119001 "ingressPackets":"242, } "ingressOctets":"39657, } "egressPackets":"243, "egressOctets":"40865 } … }
- 14. DataSource.js • Negotiates between multiple data sources and subscribers on a page Subscriptions • One data source for N subscribers Data Sources • Abstracts CometD subscription process for JS development Subscribers • Smart enough to resubscribe and notifies subscribers (so that state dumps aren’t doubled up)
- 16. Demo Time
- 17. Data and Subcription Problems • Monolithic, Multi-Purpose Queries • Large State Dumps cause CometD timeout (5-10MBs per second) • No Resubscription • Aggregated all data on front-end
- 18. Data and Subcription Solutions • Stratified, Filterable Queries and server-side aggregation • Top-N Limitations (constrained output) • Resubscription & filtering • Web workers haven’t helped much • Time smoothing (1-second ticks)
- 19. Visualization DOM manipulation is expensive ...so is tweening SVG Canvas
- 20. What’s Next • WebSockets • Historical Data and long term data storage • HTML5 local storage (store data) • Machine-learning for real-time network visualization
- 21. Thank You