Data Science at the Command Line
5 likes•4,282 views
The document discusses the advantages of using the command line interface (CLI) for data science tasks, highlighting its speed and efficiency compared to tools like Hadoop. It presents various CLI tools useful for data scientists and emphasizes the importance of command line proficiency for tasks such as data processing, machine learning, and automation. The talk encourages data scientists to leverage CLI for data discovery and project initiation before scaling with larger frameworks.
![MapReduce at the command line
Word count
Mapper
grep –oE ‘[a-zA-Z]{2,}’
(Shuffle)
& Sort
sort
Reduce
uniq -c](https://image.slidesharecdn.com/2015-01-21heloisenonnedsatthecli-150123024905-conversion-gate02/85/Data-Science-at-the-Command-Line-8-320.jpg)
Data Science at the Command Line
- 1. January 21st, 2015 Data Science Consulting Héloïse Nonne @heloisenonne, [email protected] Data Science at the Command Line Paris Data Geek @ AXA PLEASE DO NOT PRINT THE BLACK SLIDES. THINK ABOUT THE ENVIRONMENT
- 2. Data Science at the Command Line at Paris Data Geek @ AXA
- 4. $ Let’s stop clicking and start typing CLI has 40 years and counting A modern tool!
- 5. • Agile and interactive read–eval–print loop (REPL) vs edit-compile-run-debug cycle • Close to the filesystem • A good tool for the lazy Automates repetitive tasks • Good for integration with other technologies C/C++, python, perl, R, ruby, etc. Create your own tools $ why use the command line?
- 6. Do we really need Hadoop to process a few GB of data? 1.75 GB – 2 million chess games • Hadoop: 26 minutes (1.14MB/sec) • Bash, local: 12 seconds (270MB/sec) Source: Adam Drake, aadrake.com 2014
- 7. Streaming at the command line Spout < Bolt | Sink >
- 8. MapReduce at the command line Word count Mapper grep –oE ‘[a-zA-Z]{2,}’ (Shuffle) & Sort sort Reduce uniq -c
- 9. $ Lets go parallel
- 10. We have many jobs to run and 4CPUs Naive parallelization GNU parallel spawns a new process when one finishes All CPUs remain active
- 11. $ Data Scientist Toolkit
- 13. $ Machine learning at the command line
- 14. mlpack $ linear_regression --input_file dataset.csv --test_file predict.csv -v
- 15. dbacl Don’t Be Afraid of the Command Line?
- 16. Online learning with Vowpal Wabbit
- 17. Command line is good for • Starting data project before going on Hadoop, Spark, … • Data discovery • Data cleaning • Do some efficient machine learning (online, C) • Model / Feature discovery
- 18. What next? • Online learning • Benchmark with bigger data • Hadoop (Hive) vs CLI • Benchmark of Machine learning at the CLI • CLI tools vs Python / R
- 19. William Shotts Jeroen Janssens The man pages! $ Bibliography
- 20. $ Questions?