FIWARE Global Summit - Real-time Media Stream Processing Using Kurento
1 like•503 views
Kurento is an open source software that simplifies the creation of real-time communication applications involving audio and video streams. It provides a server that abstracts compatibility issues between senders and receivers and allows for manipulation or redistribution of streams. The server includes endpoints for stream input/output and filters for processing or transforming media as it flows through the pipeline. Example applications demonstrated by Kurento include an RTP receiver that redirects streams to a browser and a magic mirror that applies computer vision to detect and overlay images on a face in real-time video.
FIWARE Global Summit - Real-time Media Stream Processing Using Kurento
- 1. Real-Time Media Stream Processing using Kurento Juan Navarro Moreno Kurento Software Developer [email protected] github.com/j1elo www.kurento.org
- 2. Overview – What is Kurento? ● A server that simplifies creation of streaming applications. ● Focused on processing of Audio/Video media streams. ● Supporting WebRTC communications. ● Simple workflow: input → process → output. Kurento is an Open Source project. Hosted on GitHub: https://github.com/Kurento
- 3. Overview – Why Kurento? ● Abstracts compatibility issues between sender and receiver. ● Manipulates or redistributes the streams. ● Extracts information from the streams. Input stream Kurento Media Server Output stream Product-specific data Eg.: FIWARE data-store
- 4. Overview – Why Kurento? Examples: ● Abstraction of incompatible video codecs ● Multi-point distribution of media ● Processing / Storing media
- 5. Overview – Why Kurento?
- 6. Overview – How does Kurento work? A server with two main components: ● Endpoints: where data flows In/Out. ● Filters: composable modules where data is processed or transformed. Media stream Src Sink Media stream Input Endpoint Output Endpoint
- 7. Overview – How does Kurento work? Endpoints and Filters are linked together, mix-and-matched to form a Pipeline. Input Endpoint Src Sink Output Endpoint Src Sink Src Sink Src Sink
- 8. Overview – How does Kurento work? The server is controlled with an RPC API. ● Client applications manipulate Endpoints and Filters through this API. ● Ready-made client SDKs for Java, Node.js and in-browser JavaScript. Components of a Real-World application: ● Kurento Media Server ● Client Application – usually a server too. ● User Interface – common case is a web page.
- 9. Overview – How does Kurento work?
- 11. Component overview Kurento Media Server (aka. “KMS”) ● Core: Basic functionality and RPC API. ● Elements: Endpoints used for input and output of streams. ● Filters: Implementation of all stream- processing modules.
- 12. KMS Elements Protocols and Codecs Media Repository WebRtcEndpoint RtpEndpoint HttpEndpoint PlayerEndpoint RecorderEndpoint
- 13. KMS Elements Different Endpoints for different needs: ● WebRtcEndpoint – Full support of WebRTC standard. • Currently compatible with Chrome and Firefox (Safari and Edge are work-in-progress). • All WebRTC lingo: SDP, (Trickle-)ICE, STUN, TURN, Google REMB. ● RtpEndpoint – For RTP and SRTP streams. • Supports port auto-discovery as an alternative to ICE. ● HttpEndpoint – Accepts GET/POST requests (eg. file uploading to KMS).
- 14. KMS Elements Special Endpoints: ● PlayerEndpoint – Retrieves content from either of the file system, HTTP servers, or RTSP sources. Input-only. ● RecorderEndpoint – Redirects streams to storage. Output-only.
- 15. KMS Filters Process or transform data as it flows through filters. ● Could be a simple transformation. Eg: Convert video to black and white. ● Could be a complex task, involving external libraries. Eg: Apply Computer Vision algorithms and extract features from the video. ● Imagination is the limit.
- 16. KMS Filters Computer Vision Generic filter Zbar FaceDetector PlateDetector GStreamerFilter
- 17. Pipelines ● All elements get created and linked through specific RPC commands from the Client Application. ● As seen before, complex topologies are possible.
- 18. Pipelines SinkSink SRCSRC SinkSink SRCSRCSinkSink SinkSinkSRCSRC WebRtcEndpoint AR Filter RecorderEndpoint HttpGetEndpoint RtpEndpoint MP4 fle stored in media repository Web application using HTML5 <video> tag RTP full duplex client video phone WebRTC full duplex client video application SinkSink
- 19. Client Application ● Implements or includes an RPC client. ● Orchestrates the creation of the Pipeline, with Elements and Filters. ● WebRtcEndpoint follows the standard way of configuration via SDP Negotiation (SDP Offer/Answer Model). • The Client Application is in charge of passing around ICE Candidates. ● RtpEndpoint also uses SDP for configuration, but no ICE. ● Other Endpoints have custom RPC methods for configuration.
- 20. Client Application ● Provides access to external sources such as FIWARE data-stores. ● Proceeds with the rest of the business logic. ● Kurento provides a FIWARE integration package for Java applications. ● Also there are multiple example applications available.
- 21. Client Application Application Server Application Server Media Server Media ServerClientClient Media Negotiation phase 11 Media channel preparation phase (optional) 22 Media exchange phase 33 Specifc app logic Pipeline building Pipeline managed media SDP Ofer (Mangled) SDP Ofer SDP Answer(Mangled) SDP Answer ICE Candidates ICE Candidates
- 22. A closer look to some of the underlying technologies
- 23. A closer look – GStreamer media library Kurento is powered under the hood by the GStreamer project: https://gstreamer.freedesktop.org GStreamer is in charge of all media handling: ● Decoding / encoding of input / output video & audio. ● Establishment of communication streams (eg. RTP Sessions). ● Special interest: Agnostic bin.
- 24. A closer look – GStreamer Agnostic bin Kurento abstracts codec compatibility issues between Endpoints. ● Achieved through an “invisible” intermediate filter – ‘agnosticbin’. ● Applies on-the-fly transcoding iff the codecs are not compatible. WebRtcEndpoint WebRtcEndpoint Agnostic media adapter, “hidden” behind every connection Agnostic media adapter, “hidden” behind every connection H.264VP8 SinkSink SRCSRC SinkSink SRCSRC
- 25. A closer look – SDP Offer/Answer ● The standard signaling format for WebRTC. ● Used to configure codecs & advanced settings in WebRtcEndpoint and RtpEndpoint. ● SDP started simple but has grown to become a mess. See: https://webrtchacks.com/sdp-anatomy/ (The problem is, the IETF RTCWEB working group was/is dominated by players from the SIP world who wanted to re-use their existing code.)
- 26. A closer look – SDP Offer/Answer v=0 o=- 0 0 IN IP4 127.0.0.1 s=- c=IN IP4 127.0.0.1 t=0 0 m=audio 9 RTP/AVPF 96 a=rtpmap:96 opus/48000/2 a=sendonly a=direction:active a=ssrc:445566 cname:[email protected] m=video 9 RTP/AVPF 103 a=rtpmap:103 H264/90000 a=sendonly a=direction:active a=ssrc:112233 cname:[email protected] v=0 o=- 372 372 IN IP4 192.168.1.15 s=Kurento Media Server c=IN IP4 192.168.1.15 t=0 0 m=audio 41654 RTP/AVPF 96 a=rtpmap:96 opus/48000/2 a=recvonly a=direction:passive a=ssrc:3224 cname:user225@host-9b2 m=video 61134 RTP/AVPF 103 a=rtpmap:103 H264/90000 a=recvonly a=direction:passive a=ssrc:2717 cname:user225@host-9b2 SDP Offer to RtpEndpoint: SDP Answer from RtpEndpoint:
- 28. A closer look – Application example: RTP Receiver Kurento RTP Player is a recent addition to the Kurento Tutorials: ● Receives an audio/video stream. ● Redirects it and sends to a browser through WebRTC. ● http://doc-kurento.readthedocs.io/en/latest/user/tutorials.html#rtp-receiver RTP producer
- 29. A closer look – Application example: RTP Player void start(String browserSdpOffer) { // Create and link Endpoints MediaPipeline pipeline = kurento.createMediaPipeline(); RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); rtpEndpoint.connect(webRtcEndpoint); // Configure the RtpEndpoint String fakeSdpOffer = "..."; String kmsSdpAnswer = rtpEndpoint.processOffer(fakeSdpOffer); String browserSdpAnswer = webRtcEndpoint.processOffer(browserSdpOffer); webRtcEndpoint.gatherCandidates(); }
- 30. A closer look – Application example: Magic Mirror ● Receives a video stream from a browser. ● Applies Computer Vision to detect a face and overlay a picture. ● Sends back the modified video to the same Endpoint it came from. ● http://doc-kurento.readthedocs.io/en/latest/user/tutorials.html#webrtc-magic-mirror
- 31. A closer look – Application example: Magic Mirror void start() { // Create and link Endpoints, Filters MediaPipeline pipeline = kurento.createMediaPipeline(); WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build(); FaceOverlayFilter faceOverlayFilter = new FaceOverlayFilter.Builder(pipeline).build(); faceOverlayFilter.setOverlayedImage("mario-hat.png"); webRtcEndpoint.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); webRtcEndpoint.gatherCandidates(); }
- 33. Thank you! http://fiware.org Follow @FIWARE on Twitter Juan Navarro Moreno Kurento Software Developer [email protected] github.com/j1elo www.kurento.org