Pandas & Cloudera: Scaling the Python Data Experience
1 like654 views
Ibis is a new open source project that allows Python data analysts and scientists to analyze large datasets using the same Python tools and APIs they already use for smaller datasets. Ibis provides a high-level Python interface for describing analytics and ETL processes that can be executed using Impala for scalability. The goal of Ibis is to enable analyzing big data using Python with no compromises to functionality or usability, at native hardware speeds. The first public release of Ibis is now available through Cloudera Labs.
Pandas & Cloudera: Scaling the Python Data Experience
- 1. 1  Š  Cloudera,  Inc.  All  rights  reserved.  Ibis:  Scaling  the  Python  Data  Experience  Wes  McKinney           Marcel  Kornacker  JusFn  Erickson   Silvius  Rus Â
- 2. 2  Š  Cloudera,  Inc.  All  rights  reserved.  Wes  McKinney  â˘âŻ A  key  person  in  building  todayâs  open  source  Python  data  community  â˘âŻ Creator  of  pandas,  a  standard  Python  data  wrangling  and  analyFcs  toolkit  used  by  data  scienFsts  â˘âŻ Author  of  best-Ââselling  canonical  text  Python  for  Data  Analysis  (2012)  â˘âŻ Formerly  Founder/CEO  of  DataPad  (acquired  by  Cloudera  in  2014) Â
- 3. 3  Š  Cloudera,  Inc.  All  rights  reserved.  Python  is  popular⌠ â˘âŻ Python  has  become  a  standard  language  of  data  science  â˘âŻ Why  is  it  popular?  â˘âŻMaximizes  producFvity  for  data  engineers  and  data  scienFsts  â˘âŻBuild  robust  so[ware  and  do  interacFve  data  analysis  with  100%  Python  code   â˘âŻEasy-Ââto-Ââlearn  and  makes  happy  and  producFve  data  teams   â˘âŻLarge,  diverse  open  source  development  community  â˘âŻComprehensive  libraries:  data  wrangling,  ML,  visualizaFon,  etc.  â˘âŻ Main  use  case:  data  science  &  engineering  swiss  army  knife  on  small-Ââto-Ââmedium  size  data Â
- 4. 4  Š  Cloudera,  Inc.  All  rights  reserved.  âŚbut  Python  does  not  scale  today  â˘âŻ Python  ecosystem  conďŹned  to  single-Âânode  analysis  â˘âŻGreat  for  smaller  data  sets  â˘âŻRequires  sampling  or  aggregaFons  for  larger  data  â˘âŻDistributed  tools  compromise  in  various  ways  â˘âŻ ExtracFng  samples  or  aggregaFons  for  larger  data  means:  â˘âŻâScalesâ  by  losing  more  ďŹdelity  â˘âŻAddiFonal  ETL  overhead  to  extract  samples/aggregaFons  â˘âŻLoss  of  producFvity  with  mulFple  languages,  tools,  etc  â˘âŻBlocks  certain  analysis  and  use  cases Â
- 5. 5  Š  Cloudera,  Inc.  All  rights  reserved.  Ibis:  Same  Python,  now  at  scale  â˘âŻ Target  user:  â˘âŻData  scienFsts  and  data  engineers  (âPython  data  usersâ)  â˘âŻ Goals:  â˘âŻMirrors  single-Âânode  Python  experience  â˘âŻScales  to  any  node  and  data  size  â˘âŻNo  compromise  in  funcFonality  or  usability  â˘âŻInteracFve  experience  at  naFve  hardware  speeds Â
- 6. 6  Š  Cloudera,  Inc.  All  rights  reserved.  Whatâs  announced?  â˘âŻ First  public  release  of  Ibis  â˘âŻhgp://ibis-Ââproject.org  â˘âŻ Beta  release  to  Cloudera  Labs  â˘âŻ InviFng  usage  and  community  development  â˘âŻ Apache-Ââlicensed  open-Ââsource Â
- 7. 7  Š  Cloudera,  Inc.  All  rights  reserved.  Ibisâs  Vision  â˘âŻ Uncompromised  Python  experience  â˘âŻ100%  Python  end-Ââto-Ââend  user  workďŹows   â˘âŻEnable  integraFon  with  the  exisFng  Python  data  ecosystem  (pandas,  scikit-Ââ learn,  NumPy,  etc)  â˘âŻ InteracFve  at  big  data  scale  â˘âŻFull-ÂâďŹdelity  analysis  without  extracFons  â˘âŻScalability  for  big  data  â˘âŻNaFve  hardware  speeds  for  a  broad  set  of  use  cases Â
- 8. 8  Š  Cloudera,  Inc.  All  rights  reserved. Â
- 9. 9  Š  Cloudera,  Inc.  All  rights  reserved.  Advantages  of  our  approach  â˘âŻ Analyze  big  data  100%  in  Python,  with  the  same  ease  as  small/medium  data  on  the  local  ďŹlesystem  â˘âŻ Full-ÂâďŹdelity  data  access  â˘âŻ Familiar  Python  experience  and  integraFon  with  exisFng  Python  data  libraries  â˘âŻ Provide  a  means  for  Python  high  performance  compuFng  tools  to  be  leveraged  at  Hadoop-Ââscale Â
- 10. 10  Š  Cloudera,  Inc.  All  rights  reserved.  Beta  0.3  release   â˘âŻ High  level  Python  API  for  describing  analyFcs  and  ETL  that  can  be  executed  by  Impala  â˘âŻFamiliar  API  for  users  of  pandas  â˘âŻComprehensive  coverage  of  operaFons  expressible  as  relaFonal  data  ďŹows  â˘âŻ Integrated  tools  for  managing  data  in  HDFS  â˘âŻ Simple  workďŹows  to  query  data  ďŹles  in  several  formats  (Parquet,  Avro,  Text)  â˘âŻ pandas  data  interchange Â
- 11. 11  Š  Cloudera,  Inc.  All  rights  reserved.  Ibis/Impala  Joint  Roadmap  â˘âŻ More  natural  data  modeling  â˘âŻComplex  types  support  â˘âŻ IntegraFon  with  full  Python  data  ecosystem  â˘âŻAdvanced  analyFcs  +  machine  learning  â˘âŻEnable  use  of  performance  compuFng  tools  â˘âŻ User  extensibility  with  naFve  performance  â˘âŻIn-Ââmemory  columnar  format  â˘âŻPython-Ââto-ÂâLLVM  IR  compilaFon  â˘âŻ WorkďŹow  and  usability  tools Â
- 12. 12  Š  Cloudera,  Inc.  All  rights  reserved.  BeneďŹts  of  Ibis  â˘âŻ Maximize  developer  producFvity  â˘âŻMirrors  single-Âânode  Python  experience  â˘âŻSolve  big  data  problems  without  leaving  Python  â˘âŻLeverage  Python  skills,  ecosystem,  and  tools  â˘âŻ Python  as  ďŹrst-Ââclass  language  for  Hadoop  â˘âŻFull-ÂâďŹdelity  analysis  without  extracFons  â˘âŻPython  analysis  at  any  scale  â˘âŻNaFve  hardware  speeds  for  a  broad  set  of  use  cases Â
- 13. 13  Š  Cloudera,  Inc.  All  rights  reserved.  Thank  you  [email protected] Â