Continuum Analytics and Python
12 likes4,690 views
Continuum Analytics, co-founded by Travis Oliphant, focuses on utilizing Python for data science and analytics, addressing challenges posed by the exponential growth of data. The Anaconda platform serves as an enterprise solution for data exploration, advanced analytics, and rapid application deployment, gaining massive popularity with over 2 million downloads in two years. The document discusses the importance of powerful, accessible languages for data science and outlines various tools and technologies that facilitate effective data analysis and visualization.
![Science led to Python
5
Raja Muthupillai
Armando Manduca
Richard Ehman
Jim Greenleaf
1997
ā¢0 (2ā”f)
2
Ui (a, f) = [Cijkl (a, f) Uk,l (a, f)],j
ā
= r ā„ U](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-5-320.jpg)
![58
Blaze
+ - / * ^ []
join, groupby, filter
map, sort, take
where, topk
datashape,dtype,
shape,stride
hdf5,json,csv,xls
protobuf,avro,...
NumPy,Pandas,R,
Julia,K,SQL,Spark,
Mongo,Cassandra,...](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-58-320.jpg)
![62
Blaze Server
Provide
Ā RESTful
Ā web
Ā API
Ā over
Ā any
Ā data
Ā supported
Ā by
Ā Blaze.
Ā
Server
Ā side:
Ā
>>> my_spark_rdd = ā¦
>>> from blaze import Server
>>> Server(my_spark_rdd).run()
Hosting computation on localhost:6363
Client
Ā Side:
Ā
$ curl -H "Content-Type: application/json"
-d ā{"expr": {"op": "sum", "args": [ ... ] }ā
my.domain.com:6363/compute.json
ā¢ Quickly share local data to collaborators
on the web.
ā¢ Expose any system (Mongo, SQL, Spark,
in-memory) simply
ā¢ Share local computation as well, sending
computations to server to run remotely.
ā¢ Conveniently drive remote server with
interactive Blaze client](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-62-320.jpg)
![dask.array: OOC, parallel, ND array
69
Arithmetic: +, *, ...
Reductions: mean, max, ...
Slicing: x[10:, 100:50:-2]
Fancy indexing: x[:, [3, 1, 2]]
Some linear algebra: tensordot, qr, svd
Parallel algorithms (approximate quantiles, topk, ...)
Slightly overlapping arrays
Integration with HDF5](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-69-320.jpg)
![dask.dataframe: OOC, parallel, ND array
70
Elementwise operations: df.x + df.y
Row-wise selections: df[df.x > 0]
Aggregations: df.x.max()
groupby-aggregate: df.groupby(df.x).y.max()
Value counts: df.x.value_counts()
Drop duplicates: df.x.drop_duplicates()
Join on index: dd.merge(df1, df2, left_index=True,
right_index=True)](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-70-320.jpg)
![73
from dask import dataframe as dd
columns = ["name", "amenity", "Longitude", "Latitude"]
data = dd.read_csv('POIWorld.csv', usecols=columns)
with_name = data[data.name.notnull()]
with_amenity = data[data.amenity.notnull()]
is_starbucks = with_name.name.str.contains('[Ss]tarbucks')
is_dunkin = with_name.name.str.contains('[Dd]unkin')
starbucks = with_name[is_starbucks]
dunkin = with_name[is_dunkin]
locs = dd.compute(starbucks.Longitude,
starbucks.Latitude,
dunkin.Longitude,
dunkin.Latitude)
# extract arrays of values fro the series:
lon_s, lat_s, lon_d, lat_d = [loc.values for loc in locs]
%matplotlib inline
import matplotlib.pyplot as plt
from mpl_toolkits.basemap import Basemap
def draw_USA():
"""initialize a basemap centered on the continental USA"""
plt.figure(figsize=(14, 10))
return Basemap(projection='lcc', resolution='l',
llcrnrlon=-119, urcrnrlon=-64,
llcrnrlat=22, urcrnrlat=49,
lat_1=33, lat_2=45, lon_0=-95,
area_thresh=10000)
m = draw_USA()
# Draw map background
m.fillcontinents(color='white', lake_color='#eeeeee')
m.drawstates(color='lightgray')
m.drawcoastlines(color='lightgray')
m.drawcountries(color='lightgray')
m.drawmapboundary(fill_color='#eeeeee')
# Plot the values in Starbucks Green and Dunkin Donuts Orange
style = dict(s=5, marker='o', alpha=0.5, zorder=2)
m.scatter(lon_s, lat_s, latlon=True,
label="Starbucks", color='#00592D', **style)
m.scatter(lon_d, lat_d, latlon=True,
label="Dunkin' Donuts", color='#FC772A', **style)
plt.legend(loc='lower left', frameon=False);](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-73-320.jpg)
![77
@jit('void(f8[:,:],f8[:,:],f8[:,:])')
def filter(image, filt, output):
M, N = image.shape
m, n = filt.shape
for i in range(m//2, M-m//2):
for j in range(n//2, N-n//2):
result = 0.0
for k in range(m):
for l in range(n):
result += image[i+k-m//2,j+l-n//2]*filt[k, l]
output[i,j] = result
~1500x speed-up](https://image.slidesharecdn.com/continuumandpython-151002015905-lva1-app6891/85/Continuum-Analytics-and-Python-77-320.jpg)
Continuum Analytics and Python
- 1. Continuum Analytics and Python Travis Oliphant āØ CEO, Co-Founder Continuum Analytics
- 3. Travis Oliphant - CEO 3 ā¢ PhD 2001 from Mayo Clinic in Biomedical Engineering ā¢ MS/BS degrees in Elec. Comp. Engineering ā¢ Creator of SciPy (1999-2009) ā¢ Professor at BYU (2001-2007) ā¢ Author of NumPy (2005-2012) ā¢ Started Numba (2012) ā¢ Founding Chair of Numfocus / PyData ā¢ Previous PSF Director SciPy
- 4. Started as a Scientist / Engineer 4 Images from BYU CERS Lab
- 5. Science led to Python 5 Raja Muthupillai Armando Manduca Richard Ehman Jim Greenleaf 1997 ā¢0 (2ā”f) 2 Ui (a, f) = [Cijkl (a, f) Uk,l (a, f)],j ā = r ā„ U
- 6. āDistractionsā led to my calling 6
- 8. 8 ā¢ Data volume is growing exponentially within companies. Most don't know how to harvest its value or how to really compute on it. ā¢ Growing mess of tools, databases, and products. New products increase integration headaches, instead of simplifying. ā¢ New hardware & architectures are tempting, but are avoided or sit idle because of software challenges. ā¢ Promises of the "Big Red Solve" button continue to disappoint.āØ (If someone can give you this button, they are your competitor.) Data Scientist Challenges
- 9. Our Solution 9 ā¢ A language-based platform is needed. No simple point-and-click app is enough to solve these business problems, which require advanced modeling and data exploration. ā¢ That language must be powerful, yet still be accessible to domain experts and subject matter experts. ā¢ That language must leverage the growing capability and rapid innovation in open-source. Anaconda Platform: Enterprise platform for Data Exploration, Advanced Analytics, and Rapid Application Development and Deployment (RADD) Harnesses the exploding popularity of the Python ecosystem that our principals helped create.
- 10. Why Python? 10 Analyst ā¢ Uses graphical tools ā¢ Can call functions, cut & paste code ā¢ Can change some variables Gets paid for: Insight Excel, VB, Tableau, Analyst / Data Developer ā¢ Builds simple apps & workļ¬ows ā¢ Used to be "just an analyst" ā¢ Likes coding to solve problems ā¢ Doesn't want to be a "full-time programmer" Gets paid (like a rock star) for: Code that produces insight SAS, R, Matlab, Programmer ā¢ Creates frameworks & compilers ā¢ Uses IDEs ā¢ Degree in CompSci ā¢ Knows multiple languages Gets paid for: Code C, C++, Java, JS, Python Python Python
- 11. Python Is Sweeping Education 11
- 12. Tools used for Data 12 Source: OāReilly Strata attendee survey 2012 and 2013
- 13. Python for Data Science 13 http://readwrite.com/2013/11/25/python-displacing-r-as-the-programming-language-for-data-science
- 14. Python is the top language in schools! 14
- 15. OUR CUSTOMERS & OUR MARKET 15
- 16. Some Users 16
- 17. Anaconda: Game-changing Python distribution 17 "Hands down, the best Scientiļ¬c Python distribution products for analytics and machine learning." "Thanks for such a great product, been dabbling in python for years (2002), always knew it was going to be massive as it hit the sweet spot in so many ways, with llvm-numba its magic!!!" "It was quick and easy and had everything included, even a decent IDE. I'm so grateful it brings tears to my eyes." "I love Anaconda more than the sun, moon, and starsā¦"
- 18. Anaconda: Game-changing Python distribution 18 ā¢ 2 million downloads in last 2 years ā¢ 200k / month and growing ā¢ conda package manager serves up 5 million packages per month ā¢ Recommended installer for IPython/Jupyter, Pandas, SciPy, Scikit-learn, etc.
- 19. Conferences & Community 19 ā¢ PyData: London, Berlin, New York, Bay Area, Seattle ā¢ Strata: Tutorials, PyData Track ā¢ PyCon, Scipy, EuroScipy, EuroPython, PyCon Brasil... ā¢ Spark Summit ā¢ JSM, SIAM, IEEE Vis, ICML, ODSC, SuperComputing
- 20. Observations & Off-the-record 20 ā¢ Hype is out of whack with reality ā¢ Dashboards are "old school" BI, but still important for narrative/confirmatory analysis ā¢ Agility of data engineering and exploration is critical ā¢ Get POCs out faster; iterate faster on existing things ā¢ Need cutting-edge tools, but production is hard ā¢ Notebooks, reproducibility, provenance - all matter
- 22. Data Science Platforms 22 ā¢ All-in-one new "platform" startups are walled gardens ā¢ Cloud vendor native capabilities are all about lock-in: "Warehouse all your data here!" ā¢ Machine Learning and Advanced Analytics is too early, and disrupting too fast, to place bets on any single walled garden. ā¢ Especially since most have no experience with "exotic" regulatory and security requirements
- 23. Good News 23 You can have a modern, advanced analytics system that integrates well with your infrastructure Bad News It's not available as a SKU from any vendor. META-PLATFORM CONSTRUCTION KIT
- 24. Great News 24 ā¢ If done well, it adds deep, fundamental business capability ā¢ Many wall street banks and firms using this. ā¢ All major Silicon Valley companies know this ā¢ Facebook, LinkedIn, Uber, Tesla, SpaceX, Netflix, ...
- 26. 26 Bitcoin is a digital currency invented in 2008 and operates on a peer-to-peer system for transaction validation. This decentralized currency is an attempt to mimic physical currencies in that there is limited supply of Bitcoins in the world, each Bitcoin must be āminedā, and each transaction can be verified for authenticity. Bitcoins are used to exchange every day goods and services, but it also has known ties to black markets, illicit drugs, and illegal gambling transactions. The dataset is also very inclined towards anonymization of behavior, though true anonymization is rarely achieved. The Bitcoin Dataset The Bitcoin dataset was obtained from http://compbio.cs.uic.edu/data/bitcoin/ and captures transaction-level information. For each transaction, there can be multiple senders and multiple receivers as detailed here: https://en.bitcoin.it/wiki/Transactions. This dataset provides a challenge in that multiple addresses are usually associated with a single entity or person. However, some initial work has been done to associated keys with a single user by looking at transactions that are associated with each other (for example, if a transaction has multiple public keys as input on a single transaction, then a single user owns both private keys). The dataset provided provides these known associations by grouping these addresses together under a single UserId (which then maps to a set of all associated addresses). Key Challenge Questions # Transactions: 15.8 Million+ # Edges: 37.4 Million + # Senders: 5.4 Million+ # Receivers: 6.3 Million+ # Bitcoins Transacted: 1.4 Million + Bitcoin Data Set Overview (May 15, 2013) Figure 1: Bitcoin Transactions Over Time Bitcoin Blockchain
- 28. Memex Dark/Deep Web Analytics 28
- 29. TECH & DEMOS 29
- 30. Data Science @ NYT 30 @jakevdp eSciences Institute, Univ. Washington
- 31. 31 conda cross-platform, multi-language package & container tool bokeh interactive web plotting for Python, R; no JS/HTML required numba JIT compiler for Python & NumPy, using LLVM, supports GPU blaze deconvolve data, expression, and computation; data-web dask lightweight, fast, Pythonic scheduler for medium data xray easily handle heterogeneously-shaped dense arrays holoviews slice & view dense cubes of data in the browser seaborn easy, beautiful, powerful statistical plotting beaker polyglot alternative Notebook-like project
- 32. 32 ā¢ Databricks Canvas ā¢ Graphlab Create ā¢ Zeppelin ā¢ Beaker ā¢ Microsoft AzureML ā¢ Domino ā¢ Rodeo? Sense? ā¢ H2O, DataRobot, ... Notebooks Becoming Table Stakes
- 33. 33
- 34. 34 "With Ā more Ā than Ā 200,000 Ā Jupyter Ā notebooks Ā already Ā on Ā GitHub Ā we're Ā excited Ā to Ā level-Āāup Ā the Ā GitHub-ĀāJupyter Ā experience."
- 35. Anaconda 35 ā¦ Centralized analytics environment ā¢ browser-based interface ā¢ deploys on existing infrastructure ā¦ Collaboration ā¢ cross-functional teams using same data and software ā¦ Publishing ā¢ code ā¢ data ā¢ visualizations
- 36. Bokeh 36 http://bokeh.pydata.org ā¢ Interactive visualization ā¢ Novel graphics ā¢ Streaming, dynamic, large data ā¢ For the browser, with or without a server ā¢ No need to write Javascript
- 39. Previous: Javascript code generation 39 server.py Browser js_str = """ <d3.js> <highchart.js> <etc.js> """ plot.js.template App Model D3 highcharts ļ¬ot crossļ¬lter etc. ... One-shot; no MVC interaction; no data streaming HTML
- 40. bokeh.py & bokeh.js 40 server.py BrowserApp Model BokehJS object graph bokeh-server bokeh.py object graph JSON
- 41. 41
- 42. 42 4GB Interactive Web Viz
- 44. 44
- 45. 45
- 46. 46
- 48. Additional Demos & Topics 48 ā¢ Airline flights ā¢ Pandas table ā¢ Streaming / Animation ā¢ Large data rendering
- 50. 50 Dark Data: CSV, hdf5, npz, logs, emails, and other files in your company outside a traditional data store
- 51. 51 Dark Data: CSV, hdf5, npz, logs, emails, and other files in your company outside a traditional data store
- 53. 53 Bring the Database to the Data Data Sources Data Sources Clients Blaze (datashape,dask) NumPy,Pandas,SciPy, sklearn,etc. (for analytics)
- 54. Anaconda ā portable environments 54 PYTHON'&'R'OPEN'SOURCE'ANALYTICS NumPy, SciPy, Pandas, Scikit=learn, Jupyter / IPython, Numba, Matplotlib, Spyder, Numexpr, Cython, Theano, Scikit=image, NLTK, NetworkX, IRKernel, dplyr, shiny, ggplot2, tidyr, caret, nnet and 330+ packages conda Easy to install Quick & agile data exploration Powerful data analysis Simple to collaborate Accessible to all
- 55. 55 ā¢ Infrastructure for meta-data, meta-compute, and expression graphs/dataflow ā¢ Data glue for scale-up or scale-out ā¢ Generic remote computation & query system ā¢ (NumPy+Pandas+LINQ+OLAP+PADL).mashup() Blaze is an extensible interface for data analytics. It feels like NumPy/Pandas. It drives other data systems. Blaze expressions enable high-level reasoning http://blaze.pydata.org Blaze
- 56. 56 Blaze ?
- 58. 58 Blaze + - / * ^ [] join, groupby, filter map, sort, take where, topk datashape,dtype, shape,stride hdf5,json,csv,xls protobuf,avro,... NumPy,Pandas,R, Julia,K,SQL,Spark, Mongo,Cassandra,...
- 59. 59 numpy pandas sql DB Data Runtime Expressions spark datashape metadata storage odo paralleloptimized dask numbaDyND blaze castra bcolz
- 60. 60 Data Runtime Expressions metadata storage/containers compute APIs, syntax, language datashape blaze dask odo parallelize optimize, JIT
- 61. 61
- 62. 62 Blaze Server Provide Ā RESTful Ā web Ā API Ā over Ā any Ā data Ā supported Ā by Ā Blaze. Ā Server Ā side: Ā >>> my_spark_rdd = ā¦ >>> from blaze import Server >>> Server(my_spark_rdd).run() Hosting computation on localhost:6363 Client Ā Side: Ā $ curl -H "Content-Type: application/json" -d ā{"expr": {"op": "sum", "args": [ ... ] }ā my.domain.com:6363/compute.json ā¢ Quickly share local data to collaborators on the web. ā¢ Expose any system (Mongo, SQL, Spark, in-memory) simply ā¢ Share local computation as well, sending computations to server to run remotely. ā¢ Conveniently drive remote server with interactive Blaze client
- 63. 63 Dask: Out-of-Core Scheduler ā¢ A parallel computing framework ā¢ That leverages the excellent Python ecosystem ā¢ Using blocked algorithms and task scheduling ā¢ Written in pure Python Core Ideas ā¢ Dynamic task scheduling yields sane parallelism ā¢ Simple library to enable parallelism ā¢ Dask.array/dataframe to encapsulate the functionality ā¢ Distributed scheduler coming
- 64. Example: Ocean Temp Data 64 ā¢ http://www.esrl.noaa.gov/psd/data/gridded/ data.noaa.oisst.v2.highres.html ā¢ Every 1/4 degree, 720x1440 array each day
- 65. Bigger data... 65 36 years: 720 x 1440 x 12341 x 4 = 51 GB uncompressed If you don't have this much RAM... ... better start chunking.
- 68. Core Concepts 68
- 69. dask.array: OOC, parallel, ND array 69 Arithmetic: +, *, ... Reductions: mean, max, ... Slicing: x[10:, 100:50:-2] Fancy indexing: x[:, [3, 1, 2]] Some linear algebra: tensordot, qr, svd Parallel algorithms (approximate quantiles, topk, ...) Slightly overlapping arrays Integration with HDF5
- 70. dask.dataframe: OOC, parallel, ND array 70 Elementwise operations: df.x + df.y Row-wise selections: df[df.x > 0] Aggregations: df.x.max() groupby-aggregate: df.groupby(df.x).y.max() Value counts: df.x.value_counts() Drop duplicates: df.x.drop_duplicates() Join on index: dd.merge(df1, df2, left_index=True, right_index=True)
- 71. More Complex Graphs 71 cross validation
- 73. 73 from dask import dataframe as dd columns = ["name", "amenity", "Longitude", "Latitude"] data = dd.read_csv('POIWorld.csv', usecols=columns) with_name = data[data.name.notnull()] with_amenity = data[data.amenity.notnull()] is_starbucks = with_name.name.str.contains('[Ss]tarbucks') is_dunkin = with_name.name.str.contains('[Dd]unkin') starbucks = with_name[is_starbucks] dunkin = with_name[is_dunkin] locs = dd.compute(starbucks.Longitude, starbucks.Latitude, dunkin.Longitude, dunkin.Latitude) # extract arrays of values fro the series: lon_s, lat_s, lon_d, lat_d = [loc.values for loc in locs] %matplotlib inline import matplotlib.pyplot as plt from mpl_toolkits.basemap import Basemap def draw_USA(): """initialize a basemap centered on the continental USA""" plt.figure(figsize=(14, 10)) return Basemap(projection='lcc', resolution='l', llcrnrlon=-119, urcrnrlon=-64, llcrnrlat=22, urcrnrlat=49, lat_1=33, lat_2=45, lon_0=-95, area_thresh=10000) m = draw_USA() # Draw map background m.fillcontinents(color='white', lake_color='#eeeeee') m.drawstates(color='lightgray') m.drawcoastlines(color='lightgray') m.drawcountries(color='lightgray') m.drawmapboundary(fill_color='#eeeeee') # Plot the values in Starbucks Green and Dunkin Donuts Orange style = dict(s=5, marker='o', alpha=0.5, zorder=2) m.scatter(lon_s, lat_s, latlon=True, label="Starbucks", color='#00592D', **style) m.scatter(lon_d, lat_d, latlon=True, label="Dunkin' Donuts", color='#FC772A', **style) plt.legend(loc='lower left', frameon=False);
- 74. 74 ā¢ Dynamic, just-in-time compiler for Python & NumPy ā¢ Uses LLVM ā¢ Outputs x86 and GPU (CUDA, HSA) ā¢ (Premium version is in Accelerate product) http://numba.pydata.org Numba
- 75. Python Compilation Space 75 Ahead Of Time Just In Time Relies on CPython / libpython Cython Shedskin Nuitka (today) Pythran Numba HOPE Theano Replaces CPython / libpython Nuitka (future) Pyston PyPy
- 76. Example 76 Numba
- 77. 77 @jit('void(f8[:,:],f8[:,:],f8[:,:])') def filter(image, filt, output): M, N = image.shape m, n = filt.shape for i in range(m//2, M-m//2): for j in range(n//2, N-n//2): result = 0.0 for k in range(m): for l in range(n): result += image[i+k-m//2,j+l-n//2]*filt[k, l] output[i,j] = result ~1500x speed-up
- 78. Features 78 ā¢ Windows, OS X, and Linux ā¢ 32 and 64-bit x86 CPUs and NVIDIA GPUs ā¢ Python 2 and 3 ā¢ NumPy versions 1.6 through 1.9 ā¢ Does not require a C/C++ compiler on the userās system. ā¢ < 70 MB to install. ā¢ Does not replace the standard Python interpreterāØ (all of your existing Python libraries are still available)
- 79. How Numba Works 79 Bytecode Analysis Python Function (bytecode) Function Arguments Type Inference Numba IR Rewrite IR Lowering LLVM IRLLVM JIT Machine Code @jit def do_math(a,b): ā¦ >>> do_math(x, y) Cache Execute!
- 81. Anaconda ā portable environments 81 PYTHON'&'R'OPEN'SOURCE'ANALYTICS NumPy, SciPy, Pandas, Scikit=learn, Jupyter / IPython, Numba, Matplotlib, Spyder, Numexpr, Cython, Theano, Scikit=image, NLTK, NetworkX, IRKernel, dplyr, shiny, ggplot2, tidyr, caret, nnet and 330+ packages conda Easy to install Quick & agile data exploration Powerful data analysis Simple to collaborate Accessible to all
- 82. 82 ā¢ cross platform package manager ā¢ can create sandboxes ("environments"), akin toāØ Windows Portable Applications or WinSxS ā¢ "un-container" for deploying data science/data āØ processing workflows http://conda.pydata.org Conda
- 83. System Package Managers 83 yum (rpm) apt-get (dpkg) Linux OSX macports homebrew ļ¬nk Windows chocolatey npackd Cross-platform conda
- 84. 84 ā¢ Excellent support for āsystem-levelā environments ā like having mini VMs but much lighter weight than docker (micro containers) ā¢ Minimizes code-copies (uses hard/soft links if possible) ā¢ Simple format: binary tar-ball + metadata ā¢ Metadata allows static analysis of dependencies ā¢ Easy to create multiple āchannelsā which are repositories for packages ā¢ User installable (no root privileges needed) ā¢ Integrates very well with pip ā¢ Cross Platform Conda features
- 85. Anaconda Cloud: analytics repository 85 ā¢ Commercial long-term support ā¢ Licensed for redistribution ā¢ Private, on-premises available ā¢ Proprietary tools for building custom distribution, like Anaconda ā¢ Enterprise tools for managing custom packages and environments ā¢ http://anaconda.org
- 86. Anaconda Cluster: Anaconda + Hadoop + Spark 86 For data scientists: ā¢ Rapidly, easily create clusters on EC2, DigitalOcean, on-prem cloud/provisioner ā¢ Manage Python, R, Java, JS packages across the cluster For operations & IT: ā¢ Robustly manage runtime state across the cluster ā¢ Outside the scope of rpm, chef, puppet, etc. ā¢ Isolate/sandbox packages & libraries for different jobs or groups of users ā¢ Without introducing complexity of Docker / virtualization ā¢ Cross platform - same tooling for laptops, workstations, servers, clusters
- 87. Cluster Creation 87 $ conda cluster create mycluster --profile=spark_profile $ conda cluster submit mycluster mycode.py $ conda cluster destroy mycluster spark_profile: provider: aws_east num_nodes: 4 node_id: ami-3c994355 node_type: m1.large aws_east: secret_id: <aws_access_key_id> secret_key: <aws_secret_access_key> keyname: id_rsa.pub location: us-east-1 private_key: ~/.ssh/id_rsa cloud_provider: ec2 security_group: all-open http://continuumio.github.io/conda-cluster/quickstart.html
- 88. 88 $ conda cluster manage mycluster list ... info -e ... install python=3 pandas flask ... set_env ... push_env <local> <remote> $ conda cluster ssh mycluster $ conda cluster run.cmd mycluster "cat /etc/hosts" Package & environment management: Easy SSH & remote commands: http://continuumio.github.io/conda-cluster/manage.html Cluster Management
- 89. Anaconda Cluster & Spark 89 # example.py conf = SparkConf() conf.setMaster("yarn-client") conf.setAppName("MY APP") sc = SparkContext(conf=conf) # analysis sc.parallelize(range(1000)).map(lambda x: (x, x % 2)).take(10) $ conda cluster submit MY_CLUSTER /path/to/example.py
- 90. Python & Spark in Practice 90 Challenges of real-world usage ā¢ Package management (perennial popular topic in Python) ā¢ Python (& R) are outside the "normal" Java build toolchain ā¢ bash scripts, spark jobs to pip install or conda install <foo> ā¢ Kind of OK for batch; terrible for interactive ā¢ Rapid iteration ā¢ Production vs dev/test clusters ā¢ Data scientist needs vs Ops/IT concerns
- 91. Fix it twiceā¦ 91 PEP 3118: Revising the buffer protocol Basically the āstructureā of NumPy arrays as a protocol in Python itself to establish a memory-sharing standard between objects. It makes it possible for a heterogeneous world of powerful array-like objects outside of NumPy that communicate. Falls short in not deļ¬ning a general data description language (DDL). http://python.org/dev/peps/pep-3118/
- 92. Putting Rest of NumPy in Std Lib 92 ā¢ memtype ā¢ dtype system on memory-views ā¢ extensible with Numba and C ā¢ extensible with Python ā¢ gufunc ā¢ generalized low-level function dispatch on memtype ā¢ extensible with Numba and C ā¢ usable by any Python Working on now with a (small) team ā could use funding
- 93. 93 ā¢ Python has had a long and fruitful history in Data Analytics ā¢ It will have a long and bright future with your help! ā¢ Contribute to the PyData community and make the world a better place! The Future of Python
- 94. Ā© 2015 Continuum Analytics- Confidential & Proprietary Thanks October1, 2015 ā¢SIG for hosting tonight and inviting me to come ā¢DARPA XDATA program (Chris White and Wade Shen) which helped fund Numba, Blaze, Dask and Odo. ā¢Investors of Continuum. ā¢Clients and Customers of Continuum who help support these projects. ā¢Numfocus volunteers ā¢PyData volunteers