Data Science Salon: Deep Learning as a Product @ Scribd
Download as PPTX, PDF•1 like•794 views
The document discusses the use of deep learning from a product perspective, focusing on applications like spelling correction and various generative tasks such as image and music generation. It emphasizes the importance of selecting suitable frameworks and architectures, and outlines the development process, including challenges in achieving high accuracy with deep learning models. Additionally, the document highlights the need for reproducibility in data science and potential future projects related to query parsing and content summarization.
![Dictionary matching FTW
Build a dictionary where the keys are phrases with 1 word dropped.
{ [joy luck]: “joy luck club”, [luck club]: “joy luck club”}](https://image.slidesharecdn.com/kevinperko-171217195925/85/Data-Science-Salon-Deep-Learning-as-a-Product-Scribd-55-320.jpg)
![Why do we hard things
Hard Problem (Spelling) -> Knowledge accumulation
New problem -> Query parsing
[‘john grisham pelican brief’] -> [author,author,title,title]](https://image.slidesharecdn.com/kevinperko-171217195925/85/Data-Science-Salon-Deep-Learning-as-a-Product-Scribd-59-320.jpg)
Data Science Salon: Deep Learning as a Product @ Scribd
- 3. Who is the talk for? People interested in our experience using deep learning Product side of deep learning Not a technical talk on how to best utilize LSTM cells/activation functions/networks architectures/GANS/etc
- 4. Motivation Spelling correction to improve the user experience
- 5. Deep Learning as Product Does the user even care?
- 6. What can Deep Learning even do? Image Editing Object Detection Boundary Detection Music Generation Voice Generation Translate Languages
- 7. This will be fun! Style Transfer - Product - Visual - Imperfect - Visualize the errors
- 8. Classic Van Gogh Style Transfer Example Source: http://ideate.xsead.cmu.edu/gallery/projects/art-transitions-style-transfers-by-neural-networks
- 10. STYLE TRANSFER FTW - PRISMA Base Foreground Transfer Heisenberg
- 11. Can it solve our problem? If they can translate styles onto images...
- 12. Motivation Spelling correction to improve the user experience
- 13. Spelling correction Spelling correction is solved right? If you have enough data Google is ok, we are not They can take a probabilistic approach, good for them :P We need to get fancier
- 14. Let’s get started 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch videos 5. Deep Learn
- 17. Which one to pick? We did a comprehensive framework bakeoff… AND THE WINNER IS ?!?! J/K We picked Keras - Abstraction layer (easier to get started) - Familiarity with Python
- 18. Life as a Deep Learner http://everythingbrilliant.co.uk/wp-content/uploads/2017/01/emotional-journey-of-creating-anything-new.jpeg
- 21. GANS eh? BAD DOG GOOD DOG
- 22. Which architecture to use? 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch Videos 5. Deep Learn
- 23. TL;DR Certain problems require certain architectures For example: CNN -> Image parsing/Object Detection
- 24. Where are we? What do we have? Keras (framework) What do we need? Architecture/algorithm Constraints? Avoid implementing our own from scratch
- 25. Can we do even do this? Let’s code some boilerplate for a basic neural network
- 27. Training How long does model development take TBD -> 1-2 weeks How long to let the model train 7 days Is more gpu moar better? Not yet...
- 28. So many parameters... 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch videos 5. Deep Learn
- 29. Details (Hyperparameters) Activation Units Sigmoid, relu, prelu (yes, not even joking), lolu (actually joking) Optimization Functions Adam, sgd, etc, etc Dropout Rate Batch Size Continues on forever...
- 31. Let’s find our algorithm 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch Videos 5. Deep Learn
- 32. Sequence to Sequence mapping TL;DR Given a sequence of things turn them into another sequence Spelling Correction ‘Cande’ -> ‘candy’ English -> Spanish language translation ‘Hello’ -> ‘hola’
- 33. Nice viz of LSTM cells
- 34. Where are we? 1. We have a framework 2. We have an algorithm 3. Let’s find an example
- 35. Deep Spelling Example Blog post on Medium Github code 90%+ Accuracy!
- 36. ALL PROBLEMS SOLVED !!!
- 37. WRONG :(
- 40. Data science blog posts Not reproducible Proprietary data sets that are not shared Incorrect accuracy metrics Someone please solve this (Academic + Industry partnership on peer reviewed + reproducible data science algorithms)
- 42. Let’s find a new approach 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch videos 5. Deep Learn
- 43. What about seq2seq? New approach - Existing libraries/frameworks - OpenNMT - Open Source contribution
- 44. SUCCESS!!!
- 45. What’s edit distance? Number of letters (operations) to replace to transform a given word into another kitten -> sitting = edit distance of 3 1. k -> s (substitution) 2. e -> i (substitution) 3. _ -> g (insertion)
- 46. SHOW ME THE DISTANCE
- 48. Production Developed on AWS, but need to deploy on internally We can’t use OpenNMT Ops :(, they have their reasons but FML What’s next? Tensorflow to the rescue!!!
- 49. Seq2Seq in Tensorflow Tensorflow is pain Tensorflow is a collection of a code not a “library/framework”
- 50. Let’s figure out Tensorflow 1. Read blog posts 2. Read books 3. Read tutorials 4. Watch Videos 5. Deep Learn
- 51. Outcome Added a week of development time...BUT Simpler and Faster algorithm Basis for production deployment
- 54. SO CLOSE! We can properly correct spelling but… We also have to guess partial words correctly Query: ‘ub of the centry’ Corrected to: ‘club of the century’ ← Where we are Correct query: ‘pub of the century’ ← Where we need to go
- 55. Dictionary matching FTW Build a dictionary where the keys are phrases with 1 word dropped. { [joy luck]: “joy luck club”, [luck club]: “joy luck club”}
- 56. Does the user even care? YES
- 57. Seq2Seq + Dictionary = 90%
- 58. How much time does it add? ● Seq2Seq model adds 15ms ● Dictionary lookup adds .2ms
- 59. Why do we hard things Hard Problem (Spelling) -> Knowledge accumulation New problem -> Query parsing [‘john grisham pelican brief’] -> [author,author,title,title]
- 60. More Projects Query Parsing (Authors/Titles/Topics/Series/etc) Content Summarization (Books/Articles/Documents) Churn Prediction Document Classification (Study Guides/Court cases/etc)
- 61. Data Blog (coming in 2018) Posts on: Spelling Correction Query Tagging Multi-Armed Bandits A/B Test System Infrastructure
- 62. Work on hard data problems? Work at Scribd.
- 63. Questions Or we can all leave early!!!