KITE Network Instrumentation: Advanced WebRTC Testing
Download as PPTX, PDF1 like1,424 views
This document provides biographies for Dr. "Alex" Gouaillard and Dr. Ludovic Roux, who are experts in WebRTC testing. It discusses their backgrounds, careers, awards, and involvement in WebRTC standardization. It also outlines their company CoSMo Software's vision of contributing to open source to help grow the WebRTC community and ecosystem.
KITE Network Instrumentation: Advanced WebRTC Testing
- 2. Dr. “Alex” Gouaillard （アレックス ） 2017 – CEO – CoSMo Software (SG) 2015 – Principal Architect – Citrix (SG) 2012 – CTO - Temasys Communications (SG) 2009 – Lab Head - A*STaR (SG) 2007 – Faculty - Caltech then Harvard (USA) 2005 – Senior R&D Eng. - Sun Microsystems ERI (PRC) 2005 – PhD (Fr) EE - 3D Images and Video Processing 2005 – PhD (Jp) CS/CG - Discrete Geometry Processing 2001 – M.Eng. (Signal and Image processing) 2001 – M.Sc. (Applied Math) Hobbies: languages (more and more) -- Professional fluency: French, English, Japanese, -- Everyday life: German, Chinese, -- Can understand / is learning: Thai, Tagalog, Catalan “Incognito Mode” (シークレット モード)
- 3. 2018~ Member AOMedia – Real-Time Codec Group 2015~ Co-Chair, WebRTC interoperability A.G., IMTC 2015~ Webrtc in Webkit Initiative (webrtcinwebkit.com) 2012 ~ webRTC WG, ORTC CG, @ W3C 2012 ~ rtcweb, mmusic, payload, … @ IETF 2014 WebRTC Pioneer Award (Individual) 2014 WebRTC Expo - Best Tool Award: Puppet Master @ T. 2013 WebRTC Expo - Best Conferencing Award @ Temasys 2013 WebRTC Expo - Visionary Award @ Temasys 2013~ Corp. Contributor to webrtc.org @ Temasys Dr. “Alex” Gouaillard （アレックス ） “Incognito Mode” (シークレット モード)
- 4. Dr Ludovic Roux Dr. Ludovic Roux 2017 – Chief Scientist Officer - CoSMo SCPL 2015 – Deputy-CTO - Temasys Communications 2014 – Researcher Agency For Science Technology & Research (SG) 2008 – Thales Training & Simulation (France) 2007 – Thales Communications (France) 1999 – JAXA – EORC Earth Observation Research Center (Japan) 1997 – PhD (France) Data fusion, classification of satellite images 1993 – M.Sc. (France) Image and Language Processing 1990 – B.Sc. (France) Computer Science Languages: – Professional fluency: French, English -- Everyday life: JapaneseHobbies: Skydiving
- 5. Our vision and moto We like to give back to grow the community / ecosystem: - We publish most of our results for everybody to use freely - We involve other open source community projects in our advances, share the love - We contribute to other open source projects even if they ”compete”: - First fully functional internet explorer plugin: FREE - H.264 Simulcast implementation: Contributed to all Browsers - KITE testing Engine: FREE - We spend the money to attend all standard committee meetings to make sure the small use cases have a voice at the table - We create and finance projects to push the tech adoption forward: webrtcinwebkit - We organize Hackathon so that browser vendors, SFU vendors, and anybody can finally exchange around WebRTC in an informal and practical manner. - …
- 6. Publications 1. A Lemesle, A Marion, L Roux, A Gouaillard, “NARVAL a no-reference video quality tool for real-time communications”, Proc. of Human Vision and Electronic Imaging, 2019 2. E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi 3. L. Roux and A. Gouaillard, “Real-time communication testing evolution with WebRTC 1.0”, 2017 Principles, Systems and Applications of IP Telecommunications (IPTComm) 4. Grozev, Boris & Ivov, Emil & Budkiewicz, Arnaud & Roux, Ludovic & Gouaillard, Alexandre, “PERC double media encryption for WebRTC 1.0 sender simulcast”. 2017 Principles, Systems and Applications of IP Telecommunications (IPTComm). In collaboration with Jitsi. 5. Amirante, A and Castaldi, T and Gouaillard, A and Miniero, Lorenzo and G. Murillo, S and Romano, Simon Pietro , “Bringing privacy to the Janus WebRTC server: The PERC way“ 2017 Principles, Systems and Applications of IP Telecommunications (IPTComm). In collaboration with Meteecho.
- 7. Early WebRTC Testing – Pre-KITE L. Roux and A. Gouaillard, “Real-time communication testing evolution with WebRTC 1.0”, 2017 Principles, Systems and Applications of IP Telecommunications (IPTComm) WebRTC brings first P2P api to the browsers! Existing tool not applicable. Pioneers: Kurento => ElasTests Very basic problems: 1. it does not connect (ICE, network, …) 2. I cannot differentiate between ICE errors and DTLS errors 3. It seems to connect but media is not flowing
- 8. WebRTC Interop Testing – KITE L. Roux and A. Gouaillard, “Real-time communication testing evolution with WebRTC 1.0”, 2017 Principles, Systems and Applications of IP Telecommunications (IPTComm) Very basic problems: 1. it does not connect (ICE, network, …) 2. I cannot differentiate between ICE errors and DTLS errors 3. It seems to connect but media is not flowing At one point most browsers made it work with themselves but the “divorced parents” syndrome hit us/them. Interoperability
- 9. WebRTC Interop Testing – KITE
- 10. WebRTC Interop Testing – KITE
- 11. WebRTC Interop Testing – KITE
- 12. KITE Architecture Dashboard reporting and output analysis Engine Design Test Conductor Test Dispatcher Test Runner Test Runner Test Runner Test Runner Test Runners Client configuration, and test matrix management. Load Balancing, and managing test retries. Run tests, collect and output RAW results Test Setup and Config WebRTC Interop Testing – KITE
- 13. KITE: Karoshi Interoperability Testing Engine The effortless way to test WebRTC compliance, prevent Karoshi with KITE! • Open-source testing engine for WebRTC – Apache 2.0 license • Available at github.com/webrtc/KITE KITE can be setup on Windows, Mac or Linux. The installation process only takes 10 to 15 minutes. Write automated interoperability test scripts in Java or Javascript and run them on any platforms. KITE supports: • all web browser: Chrome, Firefox, Safari, Edge, Opera... on all OS (Linux, Windows, Mac, iOS and Android) • Mobile Native Apps on Android, iOS • Desktop Native Apps on Windows and MacOS • Electron Apps WebRTC Interop Testing – KITE
- 14. Native Apps Testing • Native App automated testing for WebRTC can prove quite time consuming at first • To help people get quickly started on the right track, we can provide Selenium Grid setup guides and scripts, sample tests and tutorials to test Native Apps on the following platforms: • Windows 10 • MacOS • Android • iOS • We also have an extensive experience writing automated tests for Electron apps WebRTC Interop Testing – KITE
- 15. Sample Tests and Tutorials Additional free WebRTC sample tests are available github.com/CoSMoSoftware/KITE-Sample-Tests The sample tests cover all the major open-source SFUs demos: • Jisti • Janus • Kurento/openvidu • Mediasoup • Medooze (Simulcast demo) Tutorials: Writing a simple KITE Test for Jitsi in Java and/or Javascript Once you've completed the installation and run the sample tests, please try to write your own KITE test following our tutorials. • KITE Test for Jitsi in Java • KITE Test for Jitsi in Javascript WebRTC Interop Testing – KITE
- 16. Test Reports WebRTC Interop Testing – KITE
- 17. Test Reports
- 18. WebRTC Load Testing - KITE
- 19. Load Testing Highly Scalable • Simulate up to 250,000 clients • Run test for unlimited time (days) Integrates with your own cloud, you pay directly to AWS your usage cost Very cost effective by using Spot Instances (<$0.01 to $0.07/h per VM) Server-side System Metrics Monitor your system in real-time during the test for network traffic, CPU & memory utilization, disk I/O Client-side WebRTC Statistics Collect statistics from all the clients during the full duration of the test and analyze foreach audio and video streams, tracks, filtering by Server, Meeting ID, User ID, Clients: in/out bitrates, packet losses, jitter, frame rate, RTT, codec, resolution WebRTC Load Testing - KITE
- 20. WebRTC Load Testing - KITE E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi
- 21. WebRTC Load Testing - KITE E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi
- 22. WebRTC Load Testing - KITE E. André, N. Le Breton, A. Lemesle, L. Roux, A. Gouaillard, “Comparative Study of WebRTC Open Source SFUs for Video Conferencing”, Principles, Systems and Applications of IP 2018. In collaboration with Kurento, Meetecho, MediaSoup, Medooze, Jitsi https://www.3cx.com/blog/news/web-meeting-performance/
- 23. WebRTC Real Time Testing probes
- 24. WebRTC Real Time Testing probes A Lemesle, A Marion, L Roux, A Gouaillard, “NARVAL a no-reference video quality tool for real-time communications”, Proc. of Human Vision and Electronic Imaging, 2019
- 25. WebRTC Real Time Testing probes A Lemesle, A Marion, L Roux, A Gouaillard, “NARVAL a no-reference video quality tool for real-time communications”, Proc. of Human Vision and Electronic Imaging, 2019
- 26. Network Instrumentation Use Case Once basic connection and interoperability problems have been solved, you go from • a binary world (works/doesn’t work) • to 50 shades of grey …. or maybe just 5 shades
- 27. Network Instrumentation Use Case How to test Video Adaptation and how fast it happens in a deterministic and SFU agnostic way as to compare them without bias? How to change and measure the impact of different flavors of the following algorithms families? - BWE, (sender side? receiving side? ….) - CC, (REMB, TIMMBR, Transport-wide-CC, Goog-cc, BBR, ….) - Simulcast, (degradation preference, layer switching, key frames, …) - SVC, (same as above) - jitter buffer (depth, ….) Key to UX: Perceived video quality and network adaptation
- 28. Network Instrumentation Use Case In other words: How do I replicate this experiment, automatically, for any SFU:
- 29. Network Instrumentation • Network conditioning can be done on the clients side, for each client independently • Control bandwidth, packets loss, latency, and jitter. • Simulate Firewall and NATs • on-premises or hosted • native or web apps • Desktop or mobile
- 30. Network Instrumentation • A video is worth 1,000 words.
- 31. How do you make money? Short answer: pretty much like most open-source projects do • Expertise (professional services) • core is fully featured, open source and free, • advanced modules are commercial • Tools and advanced GUIS that save you time, are commercial • Think Red Hat, NGINX, …. No dual-license. IP protection through
- 32. Grid Manager Workflow Writing a test script Setup the grid Uploading test script Running the test and lastly.. viewing the test result AND dashboard
- 33. Setup your grid
- 34. Full control over the grid configuration
- 35. Upload and run your tests
- 36. Reporting and Dashboard Access Allure Reports and Kibana Dashboard from the Grid Manager GUI