CSB_community
- 1. Trends in Data Science July 07, 2016 Albert Gavino Talas Data Scientist
- 2. Trends in Data Science July 07, 2016 Albert Gavino Talas Data Scientist
- 4. Trends in Data Science Domains Data Science Domain Status Statistics traditional Natural Language Processing (NLP) Entered the market Predictive Analytics / Machine Learning Entered the market Visualization / Dashboards Entered the market Image Processing (openCV) exploration Internet of Things (IoT) exploration Artificial Intelligence/ Deep Learning exploration
- 5. VOLUME VARIETY VELOCITY 3 V’s of BIG DATA
- 6. Trends on Programming Languages
- 7. Comparing R, Python and SQL
- 8. What Is Apache Hadoop? The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing. The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures. The project includes these modules: ● Hadoop Common: The common utilities that support the other Hadoop modules. ● Hadoop Distributed File System (HDFS™): A distributed file system that provides high-throughput access to application data. ● Hadoop YARN: A framework for job scheduling and cluster resource management. ● Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.
- 10. Sample methodology Problem Statement of Client: e.g. We want to know how many clusters we have We choose a Machine Learning or an algorithm appropriate for the problem (e.g. we use K-means) Visualization using Tableau Big data (Hadoop) Running R libraries, packages on your machine/server INSIGHTS 1 2 3 4 5 6
- 11. Things to get started ● Install R (open source) ● Install RStudio on top of R (this is AGPL) ● Install packages needed (RandomForest, ggplot etc) ● Create a Kaggle account ● Try learning on the competition use cases (see Image processing) ● Exchange learnings with other Data Science users/Data Scientists (meetups)