From Rocket Science to Data Science
1 like680 views
The document discusses the rapid growth and demand for data scientists, highlighting essential skills such as data management, analytics, and soft skills like communication and creativity. It provides guidance on how to start in data science, including resources for learning, data exploration, and project development, as well as insights into job roles and interview processes. The document emphasizes the importance of hands-on experience and understanding real-world problems across various sectors.
![Regression
• [(u'segmentation', 0.7036155462265015),
• (u'statistical', 0.6883552670478821),
• (u'mining', 0.6801210045814514),
• (u'graph', 0.6701105237007141),
• (u'algorithm', 0.6695878505706787),
• (u'theory', 0.6563447713851929),
• (u'predictive', 0.6474782228469849),
• (u'matlab', 0.6356837749481201),
• (u'recommendation', 0.6203793287277222),
• (u'analyses', 0.6119924783706665)]
Algorithm: word2vec synonym](https://image.slidesharecdn.com/dstalk-160306235503/85/From-Rocket-Science-to-Data-Science-18-320.jpg)
![Visualization
• [(u'tableau', 0.7196237444877625),
• (u'graph', 0.7084065675735474),
• (u'matlab', 0.6993618011474609),
• (u'libraries', 0.6821463108062744),
• (u'visualizations', 0.6746233701705933),
• (u'mining', 0.6517949104309082),
• (u'spss', 0.651625394821167),
• (u'text', 0.6145033836364746),
• (u'qlikview', 0.6053836345672607),
• (u'js', 0.5960412621498108)]
Algorithm: word2vec synonym](https://image.slidesharecdn.com/dstalk-160306235503/85/From-Rocket-Science-to-Data-Science-20-320.jpg)
From Rocket Science to Data Science
- 1. From Rocket Science to Data Science Sanghamitra Deb Data Scientist, Accenture Tech Lab
- 2. Sexiest Job of the 21st century Nate Silver predicted correctly how all 50 states would go in the presidential election 2012 Target predicted teen pregnancy from retail data.
- 3. The Big Data Challenge “With the need for data scientists growing at about 3x those for statisticians and BI analysts…. and an anticipated 100,000+ person analytic talent shortage through 2020… “ gartner article “… three core data science skills: data management, analytics modeling and business analysis. But beyond these, there’s an art to data science. We detail several soft skills that our research showed are also critical to success, i.e., communication, collaboration, leadership, creativity, discipline and passion (for information and truth).”
- 4. Who are you? • Front Engineer UX/UI • Backend engineer • Project Manager • Academic (PhD, physics, neuroscience, economics,CS … ) trying to find a niche in tech industry • Quantitative background , curiosity and ability to understand business needs. Start a data driven project relevant to the industry you want to join
- 5. Where to start blogs: yhat, data robot, datatau, upshot … twitter: follow data science news… Data Exploration/Discovery … open a dataset in your favorite coding language: Python, R , scala, julia, … Learn to pipe data in to a database such as MySQL/ MongoDB Kaggle competitions, live and older ones… e.g.: digit recognition, titanic Data Frameworks: Apache Spark. Do a few online courses on data science, big data, machine learning, python, R, … from coursera, udemy, khan academy, … form study groups, go to meetups. pros: DIY , bite size videos, flexibility, discussion forums, interactivity, great way to figure out if a new field is interesting. cons: DIY, choosing the correct course, signing up and not participating after first few weeks.
- 6. Small Data Project Flow Get open source data. Sources: city data (SanFrancisco, LA, Seattle, Chicago, transit data,…) Load it up on Python, if the data is too big I will put in MySQL (for structured data) or mongodb for free form json. Machine Learning, Statistics , counting statistics and histogram are very powerful. If you are a python user data frameworks such as “GraphLab” is open source & easy to learn. Create a dashboard/ viz/app Ask the right Question!!!
- 7. Data Wrangling/Cleaning • Open your data set and profile it • Look for missing data, bad data points vs true outliers • Pattern of your data, is it a phone number, timestamps or a social security number? is it structured data or unstructured text • Prep your data, identify the features that influence your outcome, feature selection and feature engineering.
- 8. Lets start … Question: What is a Data Scientist? Data : scraped indeed.com for all jobs containing “data” in the title. ~5000 jobs … Meta Data: Job title, job description, city, state job description: unstructured text…
- 9. Job Title Job Description text cleaning+ Bag of words
- 10. What are the data jobs? participates in evaluation of hardware and software platforms and integrating systems as they relate to the data architecture participates in selection of application packages, agency services, and technology/infrastructure capabilities to ensure alignment to data architecture works in an environment, which includes data modeling, data design, metadata and repository creation reviews object and data models and the metadata repository to structure the data for better management and quicker access plays a liaison role with business data owner/stewards 'data_architect'+ description title job title disambiguation {‘data_integration_architect', 'data_architect', 'data_warehouse_architect', 'data_warehouse_lead', ‘sr_data_architect'} data architect data scientist data engineer data entry database developer data analyst Algorithm: word2vec synonym
- 11. Where are the data jobs?
- 12. Job Title Job Description text cleaning+ Bag of words
- 14. What do the job descriptions mean? Algorithm: word2vec synonym degree report team written
- 17. Statistics • (u'mathematics', 0.8544293642044067), • (u'economics', 0.8378890752792358), • (u'applied', 0.8295730948448181), • (u'physics', 0.8211749792098999), • (u'math', 0.8039191961288452), • (u'quantitative', 0.8003592491149902), • (u'phd', 0.795414388179779), • (u'fields', 0.7486724257469177), • (u'science', 0.7226663827896118), • (u'masters', 0.7045900225639343) Algorithm: word2vec synonym
- 18. Regression • [(u'segmentation', 0.7036155462265015), • (u'statistical', 0.6883552670478821), • (u'mining', 0.6801210045814514), • (u'graph', 0.6701105237007141), • (u'algorithm', 0.6695878505706787), • (u'theory', 0.6563447713851929), • (u'predictive', 0.6474782228469849), • (u'matlab', 0.6356837749481201), • (u'recommendation', 0.6203793287277222), • (u'analyses', 0.6119924783706665)] Algorithm: word2vec synonym
- 19. Graph • (u'text', 0.7591882944107056), • (u'manipulating', 0.716569185256958), • (u'visualization', 0.7084065675735474), • (u'matlab', 0.7055898904800415), • (u'mining', 0.700824499130249), • (u'unstructured', 0.6868686676025391), • (u'regression', 0.6701105833053589), • (u'algorithms', 0.6691791415214539), • (u'natural', 0.6633298397064209), • (u'engines', 0.6632224321365356) Algorithm: word2vec synonym
- 20. Visualization • [(u'tableau', 0.7196237444877625), • (u'graph', 0.7084065675735474), • (u'matlab', 0.6993618011474609), • (u'libraries', 0.6821463108062744), • (u'visualizations', 0.6746233701705933), • (u'mining', 0.6517949104309082), • (u'spss', 0.651625394821167), • (u'text', 0.6145033836364746), • (u'qlikview', 0.6053836345672607), • (u'js', 0.5960412621498108)] Algorithm: word2vec synonym
- 21. Machine Learning • (u'learning', 0.8338875770568848), • (u'algorithms', 0.7662283182144165), • (u'natural', 0.7161275744438171), • (u'physics', 0.707731306552887), • (u'mining', 0.6965328454971313), • (u'ideally', 0.6682661175727844), • (u'graph', 0.6596766710281372), • (u'predictive', 0.656450629234314), • (u'applied', 0.6529620885848999), • (u'statistics', 0.6500071883201599) Algorithm: word2vec synonym
- 22. Fun with words data + engineer-software = {cleansing, analyst, modeler, scientist} python + ruby - html = {perl,scala,bash, scripting} storm + hadoop - scripting = {hive,hbase,spark,pig} visualizations +algorithms-predictive = {backend,libraries,js,jquery}
- 24. Digging deeper … • Create a data story, i.e put all the visualizations and insights in a dashboard create an infographic using tableau, d3 , … • Get data (say from crunchbase) on the companies that are hiring and figure out which industries dominate in the data world • Get data for atleast the past 6 months and have exact statistics for skills in the data world. Advanced text analytics (bi-gram, tri-gram modeling, topic modeling) • Create an app that gives tells you how “hot” your skills are and what skills are easiest for you to acquire to become “hotter”.
- 25. Right questions? Take different slices of the data and look for patterns that might be interesting to you? Retail: What effects customers shopping habits? what are the control variables? are promos, discounts influencing any of this habits? Crime: What are the sequence of crimes that happen every day? Do initiatives led by government or non-profit organizations have an effect on certain crime rates? Education: Does regular feedback to parents about their children’s education have an effect on the grades or engagement of the children? Healthcare: Does sending preventive care emails reduce knee surgeries?
- 26. Managing
- 27. Interview Process • 3-5 hours long • Depending on company size 4-6 people • Statistics white boarding … A/B testing calculations, • Formulation of a machine learning use case with parameter tuning, edge cases relevant to the company • Open question that the team is trying to solve • CS Algorithms … cracking the coding interview. • Databases, SQL queries … http://deblivingdata.net/wp-content/uploads/2014/05/DSTalk.slides.html
- 28. Now that you have landed the job …
- 29. References • Data Sources: data.gov, kaggle, open city data • Volunteering opportunities: Datakind, BayesImpact, Data For good • DS Schools: Insight Data Sciences, Zipfian Academy, … • sqlzoo.net • meetup.com