Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
8 likes9,227 views
This document discusses Apache Zeppelin and Apache Flink integration. It describes how the Flink interpreter allows users to run Flink jobs within Zeppelin notebooks, accessing features like dynamic forms, angular displays, and progress monitoring. The roadmap includes improving multi-tenancy with authentication and containers, and developing Helium as a platform for packaging and distributing analytics applications on Zeppelin.
Moon soo Lee – Data Science Lifecycle with Apache Flink and Apache Zeppelin
- 1. Data science lifecycle with Apache Flink and Apache Zeppelin (incubating) Flink Forward Moon moon@nflabs.com NFLabs www.nflabs.com
- 2. Content 1. Data science lifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 6. Data Science: people Engineer Data Scientist DevOps Business http://aarondavis.design/
- 7. Content 1. Data science lifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 8. Zeppelin for data scientist
- 9. ProjectTimeline ASF Incubation12.2014 08.2014 Started getting adoption http://zeppelin.incubator.apache.org 12.2012 Commercial Product for data analysis 10.2013 Open sourced a single feature
- 12. Zeppelin 10.2013
- 13. Zeppelin 10.2013
- 14. Zeppelin 08.2014
- 15. Zeppelin 08.2014
- 19. Current Status 1 Release 68 Contributors worldwide 722 Stars on GH 300/900 Emails at users/dev @i.a.o
- 23. Zeppelin & Friends Z-Manager ZeppelinHub …⋯ Collaboration/Sharing Packaging & Deployment Zeppelin + Full stack on a cloud Packages Backend Integration
- 24. OnlineViewer
- 26. Deployment
- 27. As a Service
- 30. Content 1. Data science lifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 31. Flink integration Integrated through Interpreter Data processing system abstraction in Zeppelin
- 33. Writing an Interpreter public abstract void open(); public abstract void close(); public abstract InterpreterResult interpret(String st, InterpreterContext context); public abstract void cancel(InterpreterContext context); public abstract int getProgress(InterpreterContext context); public abstract List<String> completion(String buf, int cursor); public abstract FormType getFormType(); public Scheduler getScheduler(); Must have Good to have Advanced
- 34. Flink Interpreter https://github.com/apache/incubator-zeppelin/blob/master/flink/src/main/java/org/apache/zeppelin/flink/FlinkInterpreter.java Zeppelin Server Thrift Flink Interpreter Interpreter JVM process FlinkILoop ExecutionEnvironment
- 35. Using interpreter Configure Bind use
- 36. Using interpreter Use different interpreters in the same notebook
- 37. Display System Zeppelin Server Flink Interpreter Other Interpreter Zeppelin webapp Websocket, REST Text Html Table Angular
- 38. Display System Select display system through output
- 39. Built in scheduler Built-in scheduler runs your notebook with cron expression.
- 41. DEMO
- 42. Content 1. Data science lifecycle 2. Zeppelin for data science 3. Zeppelin and Flink 4. Project Roadmap
- 43. Flink Integration • ZeppelinContext :Access to Zeppelin provided features • - Dynamic form • - Angular display system • Dependency loading • Auto completion • Cancel • Get progress information
- 44. Thank you Q & A Moon moon@nflabs.com NFLabs www.nflabs.com http://zeppelin.incubator.apache.org/
- 45. Project roadmap
- 46. Multi-tenancy Two approaches 1. Implement authentication,ACL inside of Zeppelin https://github.com/apache/incubator-zeppelin/pull/53 2. Run Zeppelin on top of Docker http://github.com/NFLabs/z-manager
- 48. An Engineer engineer by http://aarondavis.design/
- 50. An Organization engineer by http://aarondavis.design/
- 51. That’s too many! engineer by http://aarondavis.design/
- 52. What is the problem? Too much: Install Configure Cluster resources
- 53. Solution? We have containers + reverse proxy
- 55. Z Manager PoC httpd + mod_php nginx Linux box engineer by http://aarondavis.design/ 2 days, bash + php :(
- 56. Z Manager PoC
- 57. Z Manager http://github.com/NFLabs/z-manager Apache 2.0 Licence Containerized deployment per user Reverse proxy Single binary Simple web application Z Manager SGA to ASF coming *
- 58. Z Manager Auto-update engineer by http://aarondavis.design/ Linux box go + react :) Z Manager process
- 59. Z Manager
- 60. Helium
- 61. People do the similar work with different data New visualization Model & Algorithm Data process pipeline engineer by http://aarondavis.design/
- 62. Package and distribute work New visualization Model & Algorithm Data process pipeline Pkg Repo engineer by http://aarondavis.design/
- 63. Helium https://s.apache.org/helium Platform for on top of Apache Zeppelin Data Analytics Application
- 64. Helium Application = + View Algorithm Zeppelin provided Resources
- 66. -
- 67. Result
- 68. of
- 69. last
- 70. execution
- 71. -
- 72. JDBC
- 73. connection
- 74. (from
- 75. JDBC
- 76. Interpreter)*
- 77. -
- 78. SparkContext
- 79. (from
- 81. -
- 82. Flink
- 83. environment
- 84. (from
- 86. Provided
- 87. by
- 88. user
- 89. created
- 90. Interpreter -
- 91. Provided
- 92. by
- 93. user
- 94. created
- 95. Helium
- 96. application
- 99. ex)
- 100. get
- 101. git
- 102. commit
- 103. log
- 104. data
- 117. -
- 118. ex)
- 119. run
- 120. cpu
- 121. usage
- 122. monitoring
- 123. code
- 124. across
- 125. spark
- 126. cluster,
- 127. using
- 128. SparkContext
- 139. https://github.com/Leemoonsoo/zeppelin-sparkmon -
- 140. ex)
- 141. display
- 142. result
- 143. data
- 144. as
- 145. a
- 146. wordcloud
- 157. https://github.com/Leemoonsoo/zeppelin-wordcloud
- 159. Server Web
- 162. pool Resource
- 163. pool Resource
- 164. pools
- 165. are
- 166. connected “Algorithm
- 167. runs
- 168. where
- 169. resource
- 170. exists”
- 171. API class YourApplication extends org.apache.zeppelin.helium.Application { @Override public void run(ApplicationArgument arg, InterpreterContext context) { ….. } } Easy
- 172. API Just
- 173. extend
- 174. helium.Application
- 175. Application Spec { mavenArtifact : groupId:artifactId:version, className : your.helium.application.Class, icon : fa fa-cloud, name : My app name, description : “some description, consume : [ org.apache.spark.SparkContext ] } Simple Writing
- 176. a
- 177. spec
- 178. file
- 179. allow
- 180. Zeppelin
- 181. load
- 182. application
- 185. to
- 186. Jar
- 187. and
- 189. Distributed
- 190. through
- 191. Maven Downloaded
- 192. on
- 193. the
- 194. fly
- 195. and
- 196. run
- 197. when
- 198. user
- 199. selects
- 200. it
- 201. Thank you Q A Moon moon@nflabs.com NFLabs www.nflabs.com http://zeppelin.incubator.apache.org/