Hadoop, Pig, and Python (PyData NYC 2012)
7 likes6,206 views
The document provides an overview of using Python with Hadoop, discussing tools such as Jython, MRJob, and Pig to manage processing large datasets efficiently. It addresses the challenges associated with setting up and debugging distributed systems and emphasizes that while Hadoop facilitates data processing, it does not inherently optimize algorithms. Additionally, it outlines a structured approach for deployment and validation within Python and Hadoop environments.
Hadoop, Pig, and Python (PyData NYC 2012)
- 1. Hadoop, Pig, and Python PyData NYC 2012
- 2. Overview OF THIS SESSION Why Python on Hadoop? Fast Hadoop overview Jython Python MrJob Pig (How they work, challenges, efficiency, how to start)
- 3. Too much data FOR ONE MACHINE Data doubles every 18 mo
- 4. ETL / Munging Cleanse Format Simple calculations
- 5. Social Graph
- 6. Predict
- 7. Detect
- 8. Genetics
- 9. Hadoop RAPID OVERVIEW MapReduce programming model from Google (Jeff Dean and Sanjay Ghemawat)
- 11. Hadoop RAPID OVERVIEW Hadoop implements MapReduce (Java) (Doug Cutting) Incubated at Yahoo Indexing, Spam detection, more
- 12. Hadoop PROBLEMS Difficult Not much Python Batch only (...or it was)
- 13. Hadoop FUTURE Yarn MapReduce optional Generic management + distributed apps Impala
- 16. Jython ON HADOOP (REDUCE; 1ST HALF)
- 17. Jython ON HADOOP (REDUCE; 2ND HALF)
- 18. Jython ON HADOOP
- 19. Python ON HADOOP Streaming (Works with any language, not just
- 20. MrJob (Python) ON HADOOP Streaming + local / EMR / your Hadoop
- 21. MrJob (Python) ON HADOOP Multi-step jobs
- 22. Pig ON HADOOP Less code Expressive code
- 23. Pig BRIEF, EXPRESSIVE (thanks: twitter hadoop world presentation)
- 24. The Same Script, In FOR SERIOUS
- 25. Pig ON HADOOP Less code Expressive code Compiles to MR Insulates from API Popular (LinkedIn, Twitter, Salesforce, Yahoo, Stanford
- 26. Pig ON HADOOP Works with Jython Not Python Stream, no types UDF read stdin UDF deserialize, no types Serialize for Pig Write to stdout Exceptions
- 27. Pig + Python ON HADOOP
- 30. Hadoop + Python NOT ACTUALLY MAGIC Hadoop won’t magically parallelize your algorithm
- 31. Hadoop + Python EFFICIENCY Don’t stream Java-based languages • Jython • Pig + Jython Streaming has ~30% overhead • Python • MrJob • Pig + Python
- 32. Hadoop + Python EXCITED? Well... 90-95% of time isn’t spent on algos
- 33. Hadoop + Python HARD STUFF: SETUP Get Hadoop running Software where it needs to be Processes communicating Data available
- 34. Hadoop + Python HARD STUFF: DEVELOP Learn Project structure, modularity Dev environment like Production
- 35. Hadoop + Python HARD STUFF: VALIDATE Syntax check Packages available Data readable Data writable Without long waits for failure
- 36. Hadoop + Python HARD STUFF: DEBUG Distributed execution is hard to debug
- 37. Hadoop + Python HARD STUFF: TEST Data processing is hard to test But critical
- 38. Hadoop + Python HARD STUFF: DEPLOY Environments identical Code correctly deployed Configuration changes Non-disruptive
- 39. Hadoop + Python HARD STUFF: HISTORY Stats about prior runs What code was run? What’s changed?
- 40. Hadoop + Python HARD STUFF: LOGS Distributed logs hard to make sense of Hadoop logs hard to understand Ephemeral clusters lose logs
- 41. Hadoop + Python HARD STUFF: MORTAR’S APPROACH Setup: PaaS, pip installation, connectors Develop: learning, structure, instant dev env Validate: fast validate Debug: printf, more coming Test: Rails-like test suites Deploy: one-button deploy
- 42. K Young @kky