![#GDSC 2018
Reverse Proxy
JupyterHub
/hub
Jupyter server
Kernel (Python, R)
Jupyter server
Kernel (Python, R)
/user/[name]
Authenticator
Spawner
Notebook
Storage
(Filesystem, Git, etc)
LDAP,
OAuth,
etc
Docker, k8s
Zeppelin Server
LDAP,
OAuth,
etc
Notebook
Storage
(Filesystem, Git, etc)
Interpreter Manager
Auth / ACL
Interpreter (kernel)
Interpreter (kernel)
Interpreter (kernel)](https://image.slidesharecdn.com/collaborativedatasicence-globalbigdataconference2018santaclara-180402183918/85/Collaborative-environment-with-data-science-notebook-28-320.jpg)
Collaborative environment with data science notebook
- 1. What makes Data driven environments more efficient and how to build a data science toolchain around Notebook technologies Creator of Apache Zeppelin Co-Founder, CTO Moon soo Lee [email protected]
- 2. #GDSC 2018 Who am I A true believer that data science notebook changes how people collaborate Creator of Apache Zeppelin Co-founder https://github.com/Leemoonsoo
- 3. #GDSC 2018 It was 2013, really wanted to have interactive analytics interface for .
- 4. #GDSC 2018 Started an opensource project - Zeppelin http://zeppelin-project.org/ data science notebook.Became an project in 2016. http://zeppelin.apache.org
- 5. #GDSC 2018 Iterations REPL interface (2012) Editor / Result interface (2013) Notebook interface (2014)
- 6. #GDSC 2018 Pilot to Production in 1 day Hey, take a look I need an update every morning!
- 7. #GDSC 2018 More notebook consumers than producers
- 8. #GDSC 2018 At the same time Opensource project receiving contributions like Authentication Access control
- 9. #GDSC 2018 Realized that notebook is a great collaboration tool Why notebook?
- 10. #GDSC 2018 Notebook is - Interactive - Flexible - Visualized - Inline description - Contain a story - Shareable
- 11. #GDSC 2018 How to build collaborative environment with notebook technology Data sharing Multi-user environment Notebook sharing
- 12. #GDSC 2018 Data scientist Data engineer Data Analyst Marketing SW engineer Sales Executive You Notebook Sharing
- 13. #GDSC 2018 You’re using only half of its potential if not sharing
- 14. #GDSC 2018 Github nbviewer Zeppelin Airbnb/knowledge-repo Commercial services for notebook sharing VCS Open source Service
- 15. #GDSC 2018 Github ● Store notebook in github ● Versioning ● Github provides .ipynb viewer ● Fork / pull request / merge ● Private / Public / Team / Org ● Hard to apply Notebook level ACL ● Not easy for Non-engineers
- 16. #GDSC 2018 nbviewer ● Publishing notebook ● Share notebook by sharing link ● Easy use ● No access control Nbconvert (endering ipynb to static HTML) as a webservice
- 17. #GDSC 2018 Apache Zeppelin ● Share notebook with ACL, Read/Write/Execute ● In case of Jupyter notebook, need to convert .ipynb to zeppelin format in command line.
- 18. #GDSC 2018 Airbnb/knowledge-repo https://github.com/airbnb/knowledge-repo ● .ipynb, md as a post ● Git repo for version control ● Feeds ● Search ● No access control
- 19. #GDSC 2018 Commercial services for notebook sharing Google Colab ● Share notebook through google drive ● View/Edit/Run ipynb notebook using Colab ● Realtime collaboration ZEPL ● Notebook level ACL ● View/Edit/Run .ipynb and Zeppelin notebook ● Realtime collaboration ● Import existing notebook from git/s3 storage www.zepl.com
- 21. #GDSC 2018 DON’Ts ● Email attach ● Direct send ● Share through USB ● ... Email attach Local copy in laptop USB drive
- 22. #GDSC 2018 DO’s ● Provide access to the same dataset ● Access control capability ● Horizontal scalability
- 23. #GDSC 2018 Data catalog ● Provides location of data, what it means and how to load ○ e.g. ● Catalogue need to be accessible / searchable / annotatable ● Many different way to build depends on team / infra ○ Hive Metastore as a data catalog ○ Cloud infrastructure service (e.g. AWS glue data catalog, Azure data catalog) ○ Data catalog / publishing software (e.g. CKAN, DKAN) ○ Custom built on top of RDBMS, Nosql, Indexing engine ○ Build data catalog using Notebook Dataset Location Schema Note Activity s3://service/activity Date (DateTime), type (INT), action(String) Type is either RUN or STOP. …. Images s3://service/images 512x256 pixel images Images are collected from profile photo...
- 24. #GDSC 2018 Build data catalog using Notebook ● Flexible enough to describe data ● Searchable, shareable, annotatable ● Programmatic generation
- 26. #GDSC 2018 I like my notebook running on my laptop. No you don’t.
- 27. #GDSC 2018 Sign in and Run Install libraries and Install notebook and Configure driver, environments and Request access to data and Setup access to notebook repo and …. Run
- 28. #GDSC 2018 Reverse Proxy JupyterHub /hub Jupyter server Kernel (Python, R) Jupyter server Kernel (Python, R) /user/[name] Authenticator Spawner Notebook Storage (Filesystem, Git, etc) LDAP, OAuth, etc Docker, k8s Zeppelin Server LDAP, OAuth, etc Notebook Storage (Filesystem, Git, etc) Interpreter Manager Auth / ACL Interpreter (kernel) Interpreter (kernel) Interpreter (kernel)
- 29. #GDSC 2018 ● Easier to implement / manage ● Notebook sharing is decoupled with execution environment ● Usually notebook sharing is basic or restricted. (no notebook level ACL) ● e.g. ○ JupyterHub ○ AWS Sagemaker Reverse Proxy Single user Notebook server Kernel Single user Notebook server Kernel Notebook Storage Multi user Notebook server Notebook Storage Kernel Kernel Kernel Browser Browser ● More complex to implement / manage ● Notebook sharing is coupled with execution environment ● Usually notebook sharing is more advanced and fine grained ● e.g. ○ Apache Zeppelin ○ ZEPL ○ Google Colab
- 30. #GDSC 2018 Conclusion Notebook Share Data share Multi-user environment Collaboration