Pig, Making Hadoop Easy
Download as PPTX, PDF48 likes84,756 views
This document introduces Pig, an open source platform for analyzing large datasets that sits on top of Hadoop. It provides an example of using Pig Latin to find the top 5 most visited websites by users aged 18-25 from user and website data. Key points covered include who uses Pig, how it works, performance advantages over MapReduce, and upcoming new features. The document encourages learning more about Pig through online documentation and tutorials.
Pig, Making Hadoop Easy
- 1. Alan F. GatesYahoo!Pig, Making Hadoop Easy
- 2. Who Am I?Pig committerHadoop PMC MemberAn architect in Yahoo!grid teamOr, as one coworker put it, “the lipstick on the Pig”
- 3. Who are you?
- 4. Motivation By Example Suppose you have user data in one file, website data in another, and you need to find the top 5 most visited pages by users aged 18 - 25.Load UsersLoad PagesFilter by ageJoin on nameGroup on urlCount clicksOrder by clicksTake top 5
- 6. In Pig LatinUsers = load‘users’as (name, age);Fltrd = filter Users by age >= 18 and age <= 25; Pages = load ‘pages’ as (user, url);Jnd = joinFltrdby name, Pages by user;Grpd = groupJndbyurl;Smmd = foreachGrpdgenerate group,COUNT(Jnd) as clicks;Srtd = orderSmmdby clicks desc;Top5 = limitSrtd 5;store Top5 into‘top5sites’;
- 8. Why not SQL?Data FactoryPigPipelinesIterative ProcessingResearchData WarehouseHiveBI ToolsAnalysisData Collection
- 9. Pig HighlightsUser defined functions (UDFs) can be written for column transformation (TOUPPER), or aggregation (SUM)UDFs can be written to take advantage of the combinerFour join implementations built in: hash, fragment-replicate, merge, skewedMulti-query: Pig will combine certain types of operations together in a single pipeline to reduce the number of times data is scannedOrder by provides total ordering across reducers in a balanced wayWriting load and store functions is easy once an InputFormat and OutputFormat existPiggybank, a collection of user contributed UDFs
- 10. Who uses Pig for What?70% of production jobs at Yahoo (10ks per day)Also used by Twitter, LinkedIn, Ebay, AOL, …Used toProcess web logsBuild user behavior modelsProcess imagesBuild maps of the webDo research on raw data sets
- 11. Accessing PigSubmit a script directlyGrunt, the pig shellPigServer Java class, a JDBC like interface
- 12. ComponentsJob executes on clusterHadoop ClusterPig resides on user machineUser machineNo need to install anything extra on your Hadoop cluster.
- 13. How It WorksPig LatinA = LOAD ‘myfile’ AS (x, y, z);B = FILTER A by x > 0; C = GROUP B BY x;D = FOREACH A GENERATEx, COUNT(B);STORE D INTO ‘output’;pig.jar:parses
- 14. checks
- 15. optimizes
- 16. plans execution
- 17. submits jar to Hadoop
- 18. monitors job progressExecution PlanMap:Filter CountCombine/Reduce:Sum
- 20. Upcoming FeaturesIn 0.8 (plan to branch end of August, release this fall):Runtime statistics collectionUDFs in scripting languages (e.g. python)Ability to specify a custom partitionerAdding many string and math functions as Pig supported UDFsPost 0.8Adding branches, loops, functions, and modulesUsabilityBetter error messagesFix ILLUSTRATEImproved integration with workflow systems
- 21. Learn MoreRead the online documentation: http://hadoop.apache.org/pig/On line tutorialsFrom Yahoo, http://developer.yahoo.com/hadoop/tutorial/From Cloudera, http://www.cloudera.com/hadoop-trainingUsing Pig on EC2: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2728A couple of Hadoop books available that include chapters on Pig, search at your favorite bookstoreJoin the mailing lists:[email protected] for user [email protected] for developer [email protected] for Howl
Editor's Notes
- #4: How many have used Pig? How many have looked at it and have a basic understanding of it?
- #15: Demo script:Show group query first, talk about: load and schema (none, declared, from data) data types data sources need not be from HDFS or even from files parallel clause, how parallelism is determined on maps how grouping works in Pig LatinSo far what I’ve shown you is a simple join/group query. Now let’s look at something less straight forward in SQLOften people want to group data a number of different ways. Look at multiquery script: Note how there’s a branch in the logic nowOften want to operate on the result of each record in a previous statement. Look at top5 query Note nested foreach allows you to operate on each record coming out of group by Since result of group by is a bag in each record, can apply operators to that bag Currently support order, distinct, filter, limit Use of flatten at the end Use of positional parametersThere will always be logic you need to write that you can’t get from Pig Latin. This is where rich support of UDFs come in. Look at session query Note registering UDF UDF now called like any other Pig builtin function (in fact Pig builtins implemented as UDFs)Look at SessionAnalysis.java Class name is UDF name Input to UDF is always a Tuple, avoids need to declare expected input, means UDF has to check what it gets Talk about how projection of bags works Talk about how EvalFunc is templatized on return typeAlso easy to write load and store functions to fit your data needs