Hadoop導入事例 in クックパッド
16 likes•5,424 views
1. Hadoop is a framework for distributed processing of large datasets across clusters of computers. 2. Hadoop can be used to perform tasks like large-scale sorting and data analysis faster than with traditional databases like MySQL. 3. Example applications of Hadoop include processing web server logs, managing user profiles for a large website, and performing machine learning on massive datasets.
Hadoop導入事例 in クックパッド
- 1. Hadoop in
- 2. • id:sasata299 ( ) • Hadoop • • http://blog.livedoor.jp/sasata299/
- 4. Hadoop
- 5. Hadoop MySQL
- 6. … Hadoop MySQL
- 7. :-)
- 9. Hadoop
- 10. 915 30 3 1
- 11. ( )
- 12. ( )
- 13. ‣ ‣ GROUP BY ( ( Д`) ‣ 7000 ( )
- 14. !!
- 15. Hadoop
- 16. ‣ Google MapReduce ‣ ‣
- 17. mapper reducer ( ) ( )
- 19. ‣ Hadoop Streaming ‣ Ruby ‣ Cloudera CDH1 (0.18.3) ‣ EC2 Hadoop ( 10 50 ) ‣ Hadoop S3
- 20. S3 Native FileSystem (s3n://) ‣ ‣ 5GB S3 Block FileSystem (s3://) ‣ ‣ HDFS ‣
- 22. ( ) mapper ( )
- 23. mapper, reducer
- 24. master ‣ -file master slave hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -file outdata ‣ mapper, reducer File.open File.open(‘outdata’) {|f| ...}
- 25. S3 ‣ hadoop cat ※ ‣ -cacheFile S3 slave ( File.open) ※ hadoop jar xxx.jar -mapper hoge.rb -reducer fuga.rb -file hoge.rb -file fuga.rb -cacheFile s3n://path/to/outdata#othername mapper reducer
- 27. 7000 ( )→
- 28. 7000 ( )→ 30
- 29. Hadoop++
- 31. Hadoop … mapper, reducer … Hadoop …
- 32. 1. 2. Hadoop 3. Hadoop 4. Hadoop 5. Hadoop ( ) 6. (Excel ) 7. Hadoop
- 33. Hadoop !!
- 34. • MySQL Hadoop • Hadoop EC2/S3 • Hadoop Streaming Hadoop