Engineering a Data Scientist

Editor's Notes

• #3: Data Scientist: The Sexiest Job of the 21st Century - 2012 report in Harvard Business Review by Thomas Davenport and DJ Patil
• #5: Definition lifted from Wikipedia
• #6: So what do data scientists do? This is Megan Price. She’s the co-founder and director of research of Human Rights Data Analysis Group (HRDAG). Simply put HRDAG is a non-profit, non-partisan organization that applies rigorous science to the analysis of human rights violations around the world. The organization has published findings on conflicts in Syria, Colombia, Chad, Kosovo, Guatemala, Perú, East Timor, India, Liberia, Bangladesh, and Sierra Leone. The organization provided testimony in the war crimes trials of Slobodan Milošević and Milan Milutinović at the International Criminal Tribunal for the former Yugoslavia, and in Guatemala's Supreme Court in the trial of General José Efraín Ríos Montt, the de facto president of Guatemala in 1982-1983. Gen. Ríos was found guilty of genocide and crimes against humanity. Most recently, the organization has published on police violence in the United States.
• #7: (From 2011 article on data science for Guatemala case) Daniel Guzman, working for HRDAG, performed an analysis of random samples drawn from the millions of documents in the Guatemalan National Police Archive. Held what archivists estimate to be 8 kilometers, or approximately 80 million sheets, of paper. Many of the police documents were created during the country’s internal armed conflict from 1960 to 1996, during which tens of thousands of Guatemalans disappeared.
• #8: Another example, the DataShader project from Continuum Analytics. Pickups and drop-offs of around ten million taxi cab rides in New York City. Red lines are pickups, green lines are dropoffs, the image has been dynamically ranged to maximize the amount of information your eyes can perceive
• #9: Tony Hey - Author of The Fourth Paradigm: Data-Intensive Scientific Discovery, published in 2009 (one of original authors of MPI for those who care about these things :) The fourth paradigm is established as “data-intensive” science, instead of the traditional paradigms of theory, experiment, and computation.
• #10: DJ Patil - Chief Data Scientist at United States Office of Science and Technology Policy, first to call himself “data scientist” (while at LinkedIn), Jeff Hammerbacher in 2008 (was at Facebook and did the same)
• #11: Jeff Wu, popularized the term “data science” in his inaugural talk to University of Michigan to the H.C Carver Professorship, in November of 1997, where he argued that statistics is data science (and should start treating itself that way).
• #12: Richard Fairbank - Devised information-based strategy in 1994, launched Capital One, which completely disrupted the credit card industry and now has a market capitalization of 36 billion dollars.
• #13: This is a quote from the 2011 MGI report on Big Data regarding a forecast for needing data scientists.
• #14: In 2018, there will be a shortage of between 140 and 190 thousand graduates with “deep analytical talent” By the way, 1.5 million jobs? That’s a misinterpretation of a later quote that 1.5 million analysts and managers will need to be retrained to *consume* this data.
• #15: Google doesn’t want engineers as data scientists.
• #16: Neither does the World Bank :(
• #18: There are other forms of this diagram, but the key point is that data science is fundamentally interdisciplinary and requires non-trivial experience in multiple domains. Look in the middle! The unicorn! That’s you!
• #19: Tesla career posting for data scientist
• #20: Facebook career posting for data scientist.
• #21: Capital One posting for a data scientist.
• #22: An example course schedule from Columbia University.
• #25: PYPL - Uses Google tutorial search trends to score languages
• #26: When assessing these indices, remember that established, general languages (Java, Python, C++) enjoy greater popularity due to their versatility and venerability. But there is nothing wrong with smaller, “boutique” languages, and they are often more pleasant to work with, if a little harder to find answers What’s older, Python or Java? Java was born in 1995, Python 1991 (Numeric in 1995, NumPy in 2006), R is from 1993, but based on S, which first appeared in 1976, Julia was born in 2012.
• #27: “I started programming when I was five, first with Logo and then BASIC. By the time this photo was taken, I had already written several BASIC games that I distributed as shareware on our local BBS. I was fast growing bored, so my parents (both software engineers) gave me the original dragon compiler textbook from their grad school days. That's when I started learning C and writing my own simple interpreters and compilers. My early interpreters were for BASIC, but by the time I entered high school, I had already created a self-hosting compiler for a non-trivial subset of C (no preprocessor, though). Throughout most of high school, I spent weekends coding in x86 assembly, obsessed with hand-tuning code for the newly-released Pentium II chips. When I started my freshman year at MIT as a Computer Science major, I already had over ten years of programming experience. So I felt right at home there.” Is this a data scientist? Nope! It’s Philip Guo, Professor of Computer Science. And he had almost no computer programming experience before he got to MIT, where he majored in Computer Science! It’s okay if you don’t know how to program yet, but if you want to be a data scientist…
• #28: You need hands-on experience.
• #29: And deeper experience if you can get it. See the September 2016 Cortana Intelligence Competition for a motivating problem with non-trivial structure.
• #30: Here’s an example of the skills and experiences people on the Machine Intelligence team use at Capital One.
• #31: Some more non-course resources.
• #32: A real-life example of the working with messy data.
• #33: This is Travis Oliphant, cofounder of Continuum Analytics, and his suitcase.
• #34: Travis left his suitcase on a Chicago city bus while visiting David Beazley.
• #35: This is a story about a time when Travis came to visit David Beazley. Travis flew into Chicago where David's office is, and took public transportation to David's office. The last leg of his trip was on bus route 22. When Travis arrived, he realized he left his suitcase on the bus. A Chicago city bus. Mere mortals would kiss that bag goodbye, but not David and Travis. They went up to the office and wrote a quick Python program to poll the city data for all of the CTA buses, track the location of all the buses on route 22, and pop open a browser window whenever one of the route 22 buses approached within a half mile of David's office. Then they'd run outside and check for Travis' suitcase. It took about eight hours until the right bus came by again, but there was his bag. The driver didn't even blink when these two guys got on the bus, grabbed a suitcase, and got back off.