Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives

Implementing a WebRTC
endpoint in GStreamer:
challenges, problems and
perspectives
8-9 October 2015
Dublin, Ireland
Conference 2015
Miguel París
Jose A. Santos
santoscadenas@gmail.com
mparisdiaz@gmail.com

2
Who we are
José Antonio Santos Cadenas
●
Software Engineer
●
Telematic Systems Master's
●
Developer at Naevatec (Madrid,
Spain)
●
Kurento Media Server (KMS)
manager
●
Integrating with client APIs
●
Miguel París
●
Software Engineer
●
Telematic Systems Master's
●
Researcher at Universidad Rey
Juan Carlos (Madrid, Spain)
●
Kurento real-time manager
●
●
Twitter: @mparisdiaz

What is WebRTC
3
●
WebRTC is a new movement bringing RTC to the WWW
●
It is based on standards
●
Supported by Chrome & Firefox
●
Windows Edge added partial support recently
●
Safari is rumored to be on the way of providing it
●
FOSS implementations for Smartphone platforms (iOS and Android)
●
More tan 1B devices currently support WebRTC
Developing
RTC applications
for the WWW
Before WebRTC After WebRTC
Begin End
• Unified APIs
• Standards
• FOSS
• Multiplatform

WebRTC standards
4
●
Standard APIs: Standardized by the WebRTC WG at W3C
 Media Capture APIs
●
Standardize APIs for media capture
 PeerConnection APIs (WebRTC 1.0 almost finished)
●
Standardize API for P2P media communications
●
Media = Audio + Video + Data
●
DataChannel API: low latency
 WebRTC stats
●
RTP stats for inbound and outbound directions
●
encode/decode stats
●
DataChannel stats

Implementing WebRTC in GStreamer
5
●
Deep interest in the GStreamer community
●
Several implementations
 OpenWebRTC (Ericsson)
●
Design for working as part of client applications
 Kurento Media Server (Kurento.org)
●
Design for working as part of media server infrastructures
 Main differences between them:

OpenWebRTC captures media from camera and mic. KMS doesn't

OpenWebRTC renders to display and speakers. KMS doesn't

OpenWebRTC encodes/decodes audio/video. KMS only if needed.

6
<GstPipeline>
pipeline0
[>]
KmsWebrtcEndpoint
kmswebrtcendpoint1
[>]
GstRTPRtxQueue
rtprtxqueue1
[>]
GstRtpVP8Pay
rtpvp8pay1
[>]
GstRtpOPUSPay
rtpopuspay1
[>]
KmsWebrtcSession
kmswebrtcsession1
[>]
KmsRtcpDemux
kmsrtcpdemux1
[>] GstRtpSsrcDemux
rtpssrcdemux5
[>]
KmsWebrtcTransportSinkNice
kmswebrtctransportsinknice1
[>]
GstNiceSink
nicesink1
[>]
GstDtlsSrtpEnc
dtlssrtpenc1
[>]
GstFunnel
funnel
[>]
GstSrtpEnc
srtp-encoder
[>]
GstDtlsEnc
dtls-encoder
[>]
KmsWebrtcTransportSrcNice
kmswebrtctransportsrcnice1
[>]
GstDtlsSrtpDec
dtlssrtpdec1
[>]
GstSrtpDec
srtp-decoder
[>]
GstDtlsDec
dtls-decoder
[>]
GstDtlsSrtpDemux
dtls-srtp-demux
[>]
GstNiceSrc
nicesrc1
[>]
GstRtpBin
rtpbin1
[>]
GstRtpSsrcDemux
rtpssrcdemux4
[>]
GstRtpSession
rtpsession3
[>]
GstRtpSsrcDemux
rtpssrcdemux3
[>]
GstRtpSession
rtpsession2
[>]
KmsWebrtcEndpoint
kmswebrtcendpoint0
[>]
GstRtpVP8Depay
rtpvp8depay0
[>]
KmsAgnosticBin2
kmsagnosticbin2-1
[>]
GstQueue
queue3
[>]
KmsParseTreeBin
kmsparsetreebin1
[>]
KmsVp8Parse
kmsvp8parse0
[>]
GstFakeSink
fakesink3
[>]
GstTee
tee3
[>]
GstFakeSink
fakesink2
[>]
GstTee
tee2
[>]
GstRTPOpusDepay
rtpopusdepay0
[>]
KmsAgnosticBin2
kmsagnosticbin2-0
[>]
GstQueue
queue1
[>]
KmsParseTreeBin
kmsparsetreebin0
[>]
GstOpusParse
opusparse0
[>]
GstFakeSink
fakesink1
[>]
GstTee
tee1
[>]
GstFakeSink
fakesink0
[>]
GstTee
tee0
[>]
GstRTPRtxQueue
rtprtxqueue0
[>]
GstRtpVP8Pay
rtpvp8pay0
[>]
GstRtpOPUSPay
rtpopuspay0
[>]
KmsWebrtcSession
kmswebrtcsession0
[>]
KmsRtcpDemux
kmsrtcpdemux0
[>]
GstRtpSsrcDemux
rtpssrcdemux2
[>]
KmsWebrtcTransportSinkNice
kmswebrtctransportsinknice0
[>]
GstNiceSink
nicesink0
[>]
GstDtlsSrtpEnc
dtlssrtpenc0
[>]
GstFunnel
funnel
[>]
GstSrtpEnc
srtp-encoder
[>]
GstDtlsEnc
dtls-encoder
[>]
KmsWebrtcTransportSrcNice
kmswebrtctransportsrcnice0
[>]
GstDtlsSrtpDec
dtlssrtpdec0
[>]
GstSrtpDec
srtp-decoder
[>]
GstDtlsDec
dtls-decoder
[>]
GstDtlsSrtpDemux
dtls-srtp-demux
[>]
GstNiceSrc
nicesrc0
[>]
GstRtpBin
rtpbin0
[>]
GstRtpJitterBuffer
rtpjitterbuffer1
[>] GstRtpPtDemux
rtpptdemux1
[>]
GstRtpJitterBuffer
rtpjitterbuffer0
[>] GstRtpPtDemux
rtpptdemux0
[>]
GstRtpSsrcDemux
rtpssrcdemux1
[>]
GstRtpSession
rtpsession1
[>]
GstRtpSsrcDemux
rtpssrcdemux0
[>]
GstRtpSession
rtpsession0
[>]
Legend
Element-States: [~] void-pending, [0] null, [-] ready, [=] paused, [>] playing
Pad-Activation: [-] none, [>] push, [<] pull
Pad-Flags: [b]locked, [f]lushing, [b]locking; upper-case is set
Pad-Task: [T] has started task, [t] has paused task
proxypad40
[>][bfb]
sink
[>][bfb]
sink_audio
[>][bfb]
proxypad42
[>][bfb]
sink
[>][bfb]
sink_video
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
send_rtp_sink_1
[>][bfb]
proxypad33
[>][bfb]
src
[>][bfb]
src
[>][bfb]
send_rtp_sink_0
[>][bfb]
proxypad31
[>][bfb]
send_rtp_src_0
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_src
[>][bfb] rtcp_sink
[>][bfb]
sink
[>][bfb]
src_1
[>][bfb]
recv_rtp_sink_1
[>][bfb]
rtcp_src_1
[>][bfb] recv_rtcp_sink_1
[>][bfb]
src_421259003
[>][bfb] recv_rtp_sink_0
[>][bfb]
rtcp_src_421259003
[>][bfb] recv_rtcp_sink_0
[>][bfb]
proxypad44
[>][bfb]
proxypad45
[>][bfb]
proxypad46
[>][bfb]
proxypad47
[>][bfb]
sink
[>][bfb]
proxypad34
[>][bfb]
rtp_sink_0
[>][bfb]
rtp_sink_0
[>][bfb]
src
[>][bfb]
proxypad36
[>][bfb]
rtcp_sink_0
[>][bfb]
rtcp_sink_0
[>][bfb]
proxypad37
[>][bfb]
rtp_sink_1
[>][bfb]
rtp_sink_1
[>][bfb]
proxypad39
[>][bfb]
rtcp_sink_1
[>][bfb]
rtcp_sink_1
[>][bfb]
proxypad29
[>][bfb]
funnelpad5
[>][bfb]
src
[>][bfb]
funnelpad6
[>][bfb]
funnelpad7
[>][bfb]
funnelpad8
[>][bfb]
funnelpad9
[>][bfb]
rtp_src_0
[>][bfb]
rtcp_src_0
[>][bfb]
rtp_src_1
[>][bfb]
rtcp_src_1
[>][bfb]
src
[>][bfb][T]
proxypad28
[>][bfb]
sink
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
proxypad26
[>][bfb]
proxypad27
[>][bfb]
rtcp_src
[>][bfb]
rtp_sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
dtls_src
[>][bfb]
src
[>][bfb][T]
send_rtp_sink
[>][bfb]
send_rtp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
proxypad30
[>][bfb]
proxypad32
[>][bfb]
send_rtp_src_1
[>][bfb]
proxypad35
[>][bfb]
send_rtcp_src_0
[>][bfb]
proxypad38
[>][bfb]
send_rtcp_src_1
[>][bfb]
sink
[>][bfb]
rtcp_sink
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
sink
[>][bfb]
rtcp_sink
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
proxypad14
[>][bfb]
sink
[>][bfb]
sink_audio
[>][bfb]
audio_src_0
[>][bfb]
proxypad15
[>][bfb]
sink
[>][bfb]
sink_video
[>][bfb]
proxypad24
[>][bfb]
proxypad25
[>][bfb]
video_src_0
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
proxypad23
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
proxypad43
[>][bfb]sink
[>][bfb]
src
[>][bfb][T]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
src_2
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
src_1
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
proxypad19
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
proxypad41
[>][bfb]sink
[>][bfb]
src
[>][bfb][T]
sink
[>][bfb]
src
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
sink
[>][bfb]
src_2
[>][bfb]
sink
[>][bfb]
src_0
[>][bfb]
src_1
[>][bfb]
sink
[>][bfb]
src
[>][bfb]
send_rtp_sink_1
[>][bfb]
proxypad7
[>][bfb]
src
[>][bfb]
src
[>][bfb]
send_rtp_sink_0
[>][bfb]
proxypad5
[>][bfb]
send_rtp_src_0
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_src
[>][bfb]
rtcp_sink
[>][bfb]
sink
[>][bfb]
src_1442068093
[>][bfb]
recv_rtp_sink_0
[>][bfb]
rtcp_src_1442068093
[>][bfb]
recv_rtcp_sink_0
[>][bfb]
src_836061664
[>][bfb]
recv_rtp_sink_1
[>][bfb]
rtcp_src_836061664
[>][bfb]
recv_rtcp_sink_1
[>][bfb]
proxypad16
[>][bfb]
proxypad17
[>][bfb]
proxypad20
[>][bfb]
proxypad21
[>][bfb]
sink
[>][bfb]
proxypad8
[>][bfb]
rtp_sink_0
[>][bfb]
rtp_sink_0
[>][bfb]
src
[>][bfb]
proxypad10
[>][bfb]
rtcp_sink_0
[>][bfb]
rtcp_sink_0
[>][bfb]
proxypad11
[>][bfb]
rtp_sink_1
[>][bfb]
rtp_sink_1
[>][bfb]
proxypad13
[>][bfb]
rtcp_sink_1
[>][bfb]
rtcp_sink_1
[>][bfb]
proxypad3
[>][bfb]
funnelpad0
[>][bfb]
src
[>][bfb]
funnelpad1
[>][bfb]
funnelpad2
[>][bfb]
funnelpad3
[>][bfb]
funnelpad4
[>][bfb]
rtp_src_0
[>][bfb]
rtcp_src_0
[>][bfb]
rtp_src_1
[>][bfb]
rtcp_src_1
[>][bfb]
src
[>][bfb][T]
proxypad2
[>][bfb]
sink
[>][bfb]
sink
[>][bfb] rtp_src
[>][bfb]
proxypad0
[>][bfb]
proxypad1
[>][bfb]
rtcp_src
[>][bfb]
rtp_sink
[>][bfb]
rtp_src
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src
[>][bfb]
sink
[>][bfb]
rtp_src
[>][bfb]
dtls_src
[>][bfb]
src
[>][bfb][T]
send_rtp_sink
[>][bfb]
send_rtp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
recv_rtp_sink
[>][bfb]
recv_rtcp_sink
[>][bfb]
proxypad4
[>][bfb]
proxypad6
[>][bfb]
send_rtp_src_1
[>][bfb]
proxypad9
[>][bfb]
send_rtcp_src_0
[>][bfb]
proxypad12
[>][bfb]
send_rtcp_src_1
[>][bfb]
proxypad18
[>][bfb]
recv_rtp_src_0_1442068093_111
[>][bfb]
proxypad22
[>][bfb]
recv_rtp_src_1_836061664_100
[>][bfb]
sink
[>][bfb] src
[>][bfb][T]
sink_rtcp
[>][bfb]
sink
[>][bfb]
src_100
[>][bfb]
sink
[>][bfb] src
[>][bfb][T]
sink_rtcp
[>][bfb]
sink
[>][bfb]
src_111
[>][bfb]
sink
[>][bfb]
src_836061664
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src_836061664
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
sink
[>][bfb]
src_1442068093
[>][bfb]
rtcp_sink
[>][bfb]
rtcp_src_1442068093
[>][bfb]
send_rtp_src
[>][bfb]
send_rtcp_src
[>][bfb]
recv_rtp_src
[>][bfb]
sync_src
[>][bfb]
Kurento WebRtcEp
The pipeline

Interactive Connectivity Establishment
ICE I
7
●
Using libnice (NiceSrc and NiceSink)

Involved RFCs:
 RFC 5245: Interactive Connectivity Establishment
(ICE): A Protocol for Network Address Translator (NAT)
Traversal for Offer/Answer Protocols
 RFC 5389: Session Traversal Utilities for NAT (STUN)
 RFC 5766: Traversal Using Relays around NAT (TURN):
Relay Extensions to Session Traversal Utilities for NAT
(STUN)

Interactive Connectivity Establishment
ICE II
8
●
Our experience with libnice
 0.1.7 version seem to be stable, but lack many
required features
 0.1.10 version incorporate relevant features but seem
not to provide the appropriate stability yet
●
Leaks of TCP sockets
●
Connection states
●
Connectivity problems (Fails 1/1000)
 Fixed in 0.1.13 ? (we have to check it)

ICE III
9
●
What next needed in ICE for WebRTC?
 IPv6 shall be mandatory in WebRTC.

We need to check IPv6 support in libnice

draft-ietf-rtcweb-transports-09 (mandates IPv6 support)
 HTTP CONNECT protocol for TURN might be important in
corporate networks (connectivity)

draft-hutton-httpbis-connect-protocol-00
 STUN consent freshness may become mandatory for
WebRTC. This may require some adjustments on libnice for
supporting consent negotiation (security)

Draft-ietf-rtcweb-stun-consent-freshness-16
 ICE stated published through events

ICE state should be reported as specified in the WebRTC
1.0 WG draft

DTLS / SRTP I
10
●
Using dtlssrtp(dec|enc) bins
 dtls(dec|enc) manages DTLS handshake
 srtp(dec|enc) decrypts/encrypts RTP packets
●
Involved RFCs:
 RFC 5764: Datagram Transport Security (DTLS)
Extension to Establish Keys for the Secure Real-time
Protocol (SRTP)
 RFC 3711: The Secure Real-time Transport Protocol
(SRTP)

DTLS / SRTP II
11
●
What next for WebRTC?
 New cryptographic framework for group communications

This is under definition at the Privacy Enhanced RTP Conferencing (PERC) IETF
WG

Only early drafts here
 draft-jones-perc-private-media-framework-00
 draft-jones-perc-private-media-reqts-00
 draft-mattsson-perc-srtp-cloud-00
 draft-westerlund-perc-webrtc-use-case-00

PERC shall provide in the next few months (improve efficiency)
 Enabling unicast group communications without requiring infrastructure to
decipher
 New mechanisms enabling unicast group communications without
requiring infrastructure to cipher once per RTC session.

These may require re-factoring of current DTLS stack
 DTLS state published through events.

Full connectivity requires ICE connectivity + successful DTLS handshake.

DTLS is being incorporated by the W3C in the WebRTC 1.0 spec as event
generator dealing with connection health

Congestion control I
12
●
This is an essential feature for providing satisfactory QoE for WebRTC users

How does it work?
 Bandwidth estimator

Based on packet delay (useful RTP ExtHdr: abs-send-time)

Based on packet loss
 REMB (Receiver Estimated Maximum Bitrate)

GStreamer does not provide any kind of features for this off the self

How Kurento estimates bandwidth?
 on-sending-rtcp
 Get packet-loss, bytes-sent, etc. from RtpSession stats
 Advantages: simple
 Drawbacks

Reactive and not predictive

Requires packet loss to detect BW constraints

Getting RtpSession stats quite inefficient

Congestion control II
13
●
For having satisfactory congestion control we should look to the following
drafts
 draft-alvestrand-rmcat-remb-03: REMB mechanism implemented by
Chrome
 draft-ietf-rmcat-cc-requirements-09: Congestion Control Requirements
for Interactive Real-Time Media
 draft-ietf-avtcore-rtp-circuit-breakers-10: Multimedia Congestion Control:
Circuit Breakers for Unicast RTP Sessions
●
In summary, congestion control should:
 be predictive
 be efficient
 be fair
 "break the circuit" (drop the BW) in case of evidence of huge congestion

This is to avoid breaking other types of traffic.

DataChannels
14
●
Kurento has DataChannels support
●
Protocols: DTLS-SCTP
●
Drafts
 draft-ietf-rtcweb-data-channel
 draft-ietf-rtcweb-data-protocol
 draft-ietf-mmusic-data-channel-sdpneg

Impl. based on sctp(dec|enc) provided by Ericsson

What have Kurento done?
 Fixed problems when using Gstreamer 1.4
 Fixed a lot of bugs
 Implementation of the DataChannel protocol

AppSrc, AppSink

Thank you
Suggestions, comments and complains:
http://www.kurento.org
http://www.github.com/kurento
info@kurento.org
Twitter: @kurentoms
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu

Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives (20)

Recently uploaded (20)

Implementing a WebRTC endpoint in GStreamer: challenges, problems and perspectives