Developing rich multimedia applications with FI-WARE.
Download as PPTX, PDF1 like1,502 views
The document discusses the Stream Oriented Generic Enabler (SO GE) which provides multimedia capabilities to the FI-WARE infrastructure through a simple API. It allows sending and receiving media, processing media through elements like computer vision and augmented reality filters, and transforming media formats. Examples are given of media elements like HttpEndpoint, RtpEndpoint, and filters like JackVaderFilter. The SO GE API allows dynamically creating media pipelines by connecting these reusable elements. An overview is also given of developing applications using this API in JavaScript or Java.
Developing rich multimedia applications with FI-WARE.
- 1. Open APIs for Open Minds The Stream Oriented Generic Enabler – SO GE Developing rich multimedia applications with FI-WARE.
- 2. Speakers 1 Luis López • Leader of the Kurento project • As. Prof. at URJC • Coordinator of the Stream Oriented GE at FI-WARE Jose Antonio Santos • Chief Architect of Kurento project • Senior Developer at Naevatec • Researcher at FI-WARE project
- 3. The Stream Oriented Generic Enabler – SO GE 2 • Send and receive media (Multidevice/multiprotocol) • Process media (Computer vision, augmented reality, media indexing, etc.) • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) Provides multimedia capabilities to the FI-WARE infrastructure • No need of protocol/codec low level understanding Exposes those capabilities through a simple to use API • LGPL 2.1 Is distributed through a flexible FOSS licence • Sorry for the instabilities… Is still work in progress
- 4. How media APIs (usually) work 3 Client side - WWW browser - Native API Server side - Media repositories - Media capabilities Hey, I want to send/receive media in this way Here you have the media you requested API API
- 5. The SO-GE API is a WWW inspired API 4 Dynamic WWW page development • Lots of capabilities (ex. Java EE) Multimedia development with the SO-GE • Don’t want to loose any capability Create HTML - DDBB access - Transactions - Security tools - Etc. HTTP request: I want this resource HTTP response: The resource Create Media - Media API - DDBB access - Transactions - Security tools - Etc. HTTP request: I want this media HTTP response: The media is here
- 6. The SO-GE Media API: media elements and pipelines 5 Media Element • Provides a specific media functionality › Building block › Send/receive media › Process media › Transform media • The Media API provides a toolbox of media elements ready to be used. • New media elements can be added Media pipeline • Chain of media elements implementing the desired media logic. • The Media API provides the capability of creating media pipelines by joining media elements of the toolbox Media Element Sink SRC File Source SRC Ogg-demuxer Sink SRC Vorbis-decoder Sink SRC Audio-sink Sink Theora-decoder Sink SRC Video-sink Sink
- 7. The global vision of a SO-GE application 6 SO-GE Media Server Infrastructure Decode Video Augmented Reality Encode Video Computer Vision Video Playing and Recording
- 8. Examples of media elements: HttpEndPoint 7 HttpEndPoint Function • Sends/receives media basing on HTTP transports (HTML5 video tag) • Associates media to a temporary-one- time-use URL Formats • WebM • H.264 support under development Input parameters • None HttpEndPoint Media Source Media Sink HTTP POST requests HTTP answers
- 9. Example of media elements: RtpEndPoint 8 RtpEndPoint Function • Sends/receives media basing on RTP transports (Real Time Communications). • Negotiates media parameterization through SDPs Formats • H.263, MPEG4, H.264, VP8 Input parameters • Input SDP RtpEndPoint Media Source Media Sink RTP input media RTP output media
- 10. Example of media elements: PlayerEndPoint 9 PlayerEndPoint Function • Plays media from file or URI. Formats • WebM • MP4 (H.264) • 3GPP • AVI • Etc. Input parameters • File URI PlayerEndPoint Media Source Media from file or URI
- 11. Example of media elements: RecorderEndPoint 10 RecorderEndPoint Function • Records media into a file. Formats • WebM • H.264 formats under development Input parameters • File path RecorderEndPoint Media Source Media Sink Media to file
- 12. Example of media elements: JackVaderFilter 11 JackVaderFilter Function • Example of filter • Recognizes faces • Adds a Jack Sparrow or Darth Vader hat onto faces Formats • Raw Input parameters • None JackVaderFilter Sink SRC Media Stream Augmented Media stream
- 13. Example of media elements: ZBarFilter 12 ZBarFilter Function • Example of filter • Recognizes Bar and QR codes • Generates events Formats • Raw Input parameters • None ZBarFilter Sink SRC Media Stream Media stream Event
- 14. Developing media application with the SO-GE Runtime environment • Install Kurento Media Server (soon available at FI-LAB) • Install JBoss Java EE container (soon available at FI-LAB) Development environment • Maven (repository: http://repository.kurento.com/archiva/repository/snapshots) • Soon available at maven central › kmf-content-api › Provides handers › kmf-media-api › Provides media elements and pipelines • Eclipse • Javascript libraries (available at the same repository) › kws-content-api 13
- 15. My first application with the SO-GE: play a video 14 HttpEndPoint Media Source Media Sink PlayerEndPoint Media Source Media from file or URI HTTP media streaming Client side Server side
- 16. My first application with the SO-GE: play a video Source code … 15
- 17. My second application: including a simple filter 16 HttpEndPoint Media Source Media Sink PlayerEndPoint Media Source HTTP media streaming JackVaderFilter Sink SRC Client side Server side
- 18. My second application: including a simple filter Source code … 17
- 19. Making it more fun: introducing real time media 18 Media Source Media Sink Real time media HTTP media streaming JackVaderFilter Sink SRC RtpEndPoint Media Source Media Sink Client side Client sideServer Side
- 20. Making it more fun: introducing real time media Source code … 19
- 21. Programming with media events: the ZBarFilter 20 Real time media ZBarFilter Sink SRC RtpEndPoint Media Source Media Sink HttpEndPoint Media Source Media Sink Media events to browser Client side Client sideServer Side
- 22. Programming with media events: the ZBarFilter Source code … 21
- 23. Digging into the media API: implementing your own filters: requirements Requirements • GStreamer › Based on GObject and GLib › C programing skills needed • OpenCV or your preferred image processing software • C++ basic knowledge • Thrift basic knowledge 22
- 24. Digging into the media API: implementing your own filters: GStreamer media element implementation Download media server source code Look for sub-module gst-kurento-plugins Enter into src/filters folder Create your filter as couple of filtername.c filtername.h files • Your filter must inherit from GstVideoFilter • You have to implement virtual method transform_frame_ip › This method gives you each frame to be processed • Be aware of init and finalize functions if you use attributes that need to be freed 23
- 25. Digging into the media API: implementing your own filters: Integration in media server Modify kms-interface/mediaServer.thrift file adding a new value to enum FilterType Create a C++ file to manage your GStreamer filter (by convention we use the same name in camel case) • This class inherits from kurento::Filter class • Its constructor creates a “filterelement” gstreamer element and add your filter name as a parameter (it will be created automatically) • Destructor should free this element Modify kurento::MediaPipeline::createFilter method in order to add your filter type in the switch statement 24
- 26. Facing the FI-WARE challenge using the SO-GE Getting started • Use the FI-WARE Catalogue (ready since 15 minutes ago hopefully!) › http://catalogue.fi-ware.eu/enablers/stream-oriented-kurento/ • Ask the SO-GE team around the FI-WARE stand Augmenting videos in novel ways • Sensor information on live stream videos › Pollution/noise/traffic Red-filtering level • Combine audio-visual & sensor information through augmented reality › Thermometers, charts, gauges on top of a media flow. Make a camera to become an advanced sensor using computer vision • Traffic status › Car counter • Weather › Sky/ground color Conceive novel WWW and Smartphone interaction GUIs • Up to your imagination 25
- 27. http://fi-ppp.eu http://fi-ware.eu Follow @Fiware on Twitter ! Thanks ! 26
- 28. Technical details: Javascript KwsContent API Creating a media connection • conn = new KwsContent(uri, options); • Options › remoteVideoTag: id of the video tag rendering the remote stream › Others for WebRtc (currently not available) • Semantics › Connects to URI and requests media › Upon media reception, renders it on the specified video tag Adding event handlers • conn.onstart( function … • conn.onremotestream (function … • conn.onterminate( function … • conn.onmediaevent( function … 27
- 29. The SO-GE Content API: handlers Handler • Is implemented by the application developer (equivalent to a Servlet) • Defines the sequence of instructions executed when a request for media is received Four types • PlayerHandler › Handles requests for playing media with HTTP transport › Compatible with the HTML5 video tag • RecorderHandler › Handlers request for recording media with HTML transport › Compatible with HTTP file upload mechanism • RtpMediaHandler › Handlers request for establishing RTP media sessions • WebRtcMediaHandler (under development) 28
- 30. Applica on logic Applica on Server (JBosss/Mobicents)Media Server Flash Video Server HTTP ServeletSIP Servlet Media element proxies Media Session (signaling) Server side signaling plane IP Network (remote clients) Media Session (media plane) SIPSignaling Media events RESTfulSignaling RAWHTTPSignaling OtherSignaling Other JEE SOAPSignaling Media applica on (App. logic) Media Repository Input Element Decoder AR Filter CV Filter Encoder Adaptor Output Element Incomingmediastream Outgoingmediastream Media message bus Control commands Technical details: the SO-GE Architecture 29
- 31. PlayerHanlderinternaldetails: sequencediagram 30 Player Handler Servlet Media ServerClient Player Handler (Applica on logic) PlayRequest Applica on Server POST Content URL create() onMediaRequest(playHandler) Addi onal logic play(source) create(h pEndPoint) connect(source, h pEndPoint) h pEndPoint.getURL Assigned URL (JSON object) 200 OK (JSON object) build media pipeline createPipeline create(media-element) create, connect, etc. GET URL Media