Apache pig presentation_siddharth_mathur
- 1. CSC 5800: Pig Latin: A Not-So-Foreign Language for Data Intelligent Systems: Processing Algorithms and Tools By Siddharth Mathur 1
- 2. What we will be covering Introduction MapReduce Overview Pig Overview Pig Features Pig Latin Pig Debugger Demo 2
- 3. Introduction Enormous data Innovation critically depends upon analyzing terabytes of data collected everyday SQL can resolve the structure data problems Parallel Database processing – Data is enormous can‟t be analyzed serially. – Has to be analyzed in parallel. – Shared nothing clusters are the way to go. 3
- 4. Parallel DB Products Teradata, Oracle RAC, Netezza Expensive at web scale Programmers have to write complex SQL queries because of this declarative programming is not preferred 4
- 5. Procedural programming Map-Reduce programming model It can easily perform a group by aggregation in parallel over a cluster of machines The programmer provides map functions which is used as a filter or transforming method The reduce function performs the aggregation Appealing to the programmer because there are only 2 high level declarative functions to enable parallel processing 5
- 6. MapReduce Overview Programming Model – To cater large data analytics – Works over Hadoop – Java based – Splits data into independent chunks and process them in-parallel Program structure – Mapper – Reducer – Driver Program 6
- 7. MapReduce Driver Program Works as „Main‟ function for MR job Takes care of – Number of arguments – Input Data Location – Input Data Types – Output Data Location – Output Data types – Number of Mappers – Number of Reducers 7
- 8. Mapper and Reducer Class Mapper Class – Main task is to perform any function logic – Computes tasks like: • Filtering • Splitting • Tokenizing • Transforming Reducer Class – Works as an aggregator – Aggregates the intermediate results gathered from Mapper 8
- 9. Word Count Execution Input the quick brown fox Map Shuffle & Sort Reduce the, 1 brown, 1 fox, 1 Output Reduce Map brown, 2 fox, 2 how, 1 now, 1 the, 3 Reduce ate, 1 cow, 1 mouse, 1 quick, 1 the, 1 fox, 1 the, 1 the fox ate the mouse Map quick, 1 how, 1 now, 1 brown, 1 how now brown cow Map ate, 1 mouse, 1 cow, 1 9
- 10. MapReduce Word Count Program public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer tokenizer = new StringTokenizer(line); while (tokenizer.hasMoreTokens()) { word.set(tokenizer.nextToken()); context.write(word, one); } } } public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } 10
- 11. Map Reduce Limitations 1 input – 2 stage data flow is extremely rigid. – To perform a task like join or sum iteration task, workaround has to be devised. – Custom code for common task like filtering or transforming or projection – The code is difficult to reuse and maintain Moreover, because of its own data types, workflow and the fact that people have to learn java, makes it‟s a tough choice to take. 11
- 12. Pig An Apache open source project. Provides an engine for executing data flows in parallel on Hadoop. Includes a language called „Pig Latin‟ for expressing these data flows. High level declarative data workflow language. It has best of both worlds: – High Level declarative querying like SQL – Low Level procedural like Map Reduce 12
- 13. Hadoop Stack Hive … HBase Data Processing Layer Pig Hadoop MR Hadoop Yarn Resource Management Layer HDFS Storage Layer 13
- 14. Why Choose Pig Written like SQL, compiled into MapReduce Fully nested data model Extensive support for UDFs Can answer multiple questions in one single workflow. A = load './input.txt'; B = foreach A generate flatten(TOKENIZE((chararray)$0)) as word; C = group B by word; D = foreach C generate COUNT(B), group; store D into './output'; 14
- 15. Features and Motivation Design goal of pig is to provide programmers with appealing experience for performing ad-hoc analysis of extremely large data sets. – DataFlow Language – QuickStart and Interoperability – Nested Data Model – UDF‟s – Debugging Environment 15
- 16. Data Flow Language Each step specifies a single high level data transformation Different from SQL where all these results are a single output. The system has given opportunity to provide optimization function. – Example: A= Load „input.txt‟; B= Filter A by UDF (Column1); C= Filter B by Column1 > 0.8; 16
- 17. Quick start and Interoperability Data Load – Capability of Ad-Hoc analysis – Can run queries directly on Data from dump of search engines – Just have to provide a function that tells Pig how to parse the content of file into tuple. – Similarly for output • Any output format. • These function can be reused. • Used for visualization or dumped to excel directly 17
- 18. Pig as part of workflow Pig easily becomes a part of workflow eco-system – Can take most of the input types – Can output in many of the forms – Doesn‟t take over the data, i.e., it does not lock the data that is being processed. – Read only data analysis 18
- 19. Optional data schemas Schema can be provided by the user : – In the beginning – On the fly – Example: • A= LOAD „input.txt‟ as (Column1;Column2); • B= Filter A by Column1>5; If the schema is not provided then the columns can be referred by „$0‟, „$1‟, „$2‟…. for the 1st, 2nd, 3rd column etc. Example: A= LOAD „input.txt‟; B= Filter A by $0>5; 19
- 20. Nested Data Model Suppose, for a document, we want to extract the term and its position. Format of output : Map<document Set<position>> SQL data model: Term Document ID Position Hi 1 2 Hi 1 5 Or keep in normalized form, i.e., – term_info(termid, String) – position_info(termid, position, document) 20
- 21. Problem resolved using Pig In pig we have complex data types like map, tuple or bag to occur as a field of a table itself. Example: Term Document ID Position Hi 1 (2,5,8..) This approach is good because its more closer to what a programmer thinks. Data is stored on disk in a nested fashion only It gives user an ease in writing UDFs. 21
- 22. UDFs Significant part of data analysis is custom processing For example, user might want to process natural language stemming Or checking if the page is spam or not, or many other tasks To work on this, Pig Latin has extensive support for UDFs, most of the tasks can be resolved using the UDFs It can take non-atomic input and can provide a nonatomic output also Currently the UDFs can be written in java or python 22
- 23. Debugging Environment In any language, getting a data processing program work correctly usually takes many iterations First few iterations mostly produce errors With a large scale data this would result in serious time and resource wastage Debuggers can help Pig has a novel debugging environment Generates concise examples from input data Data samples are carefully chosen to resemble real data as far as possible Sample data is carved specially 23
- 24. Pig Latin Language in which data workflow statements are written It runs on the shell called „Grunt‟ It has a shared repository name Piggybank We can create our custom UDFs and add them to Piggybank 24
- 25. Data Model Rich, yet simple data models Atoms – Simple atomic values like string or number Tuple – A collection of fields each of which can be of any data type – Analogous to rows in SQL Bag – Collection of tuples or both tuples and atoms – Can also be heterogeneous 25
- 26. Data Model (cont.) Example of a relation Atom Tuple Bag T= „alice‟, (labours,1), {(„ipod‟, 2),„james‟} Tuple is represented with round braces Bag is represented with curly braces 26
- 27. Specifying Input Data : LOAD Its the first step in Pig Latin program Specifying what the input files are How are its contents to be deserialized, i.e., converted to pig data model. LOAD command – Example queries= LOAD „query_log.csv‟ USING PigStorage(„,‟) AS (userId,queryString,timestamp); 27
- 28. LOAD (cont.) Both the „USING‟ clause and the „AS‟ clause are optional We can work without them as shown earlier ($0 for first field) Pig Storage is a pre-defined function Can use custom function instead of Pig Storage 28
- 29. Per Tuple Processing : FOREACH Similar to FOR statements Its used for applying special processing to each tuple of the dataset Example – Expanded_query = FOREACH queries GENERATE UserId, Expand(queryString), timeStamp; Its not a FILTERING command „Expand‟ can take atomic input and can generate a bag of outputs 29
- 30. Per Tuple Processing : FOREACH(cont.) The semantics of FOREACH is such that there is no dependency between different tuples of input, therefore permitting efficient parallel implementation 30
- 31. Discarding Unwanted Data : FILTER Used as a where clause Can provide anything in the expression – Query = FILTER queries By user_id neq „bot‟; We can provide a UDF also, like – Query = FILTER queries by Isbot(user_id); 31
- 32. COGROUP Similar to Join Groups bags of different inputs together Ease of use for UDF‟s – Grouped_data = COGROUP results by querystring, revenue by querystring; 32
- 33. JOIN Not all users want to use COGROUP Simple equi-join is all that is required – Example Join_result = JOIN results by querystring, revenue by querystring; Other types of join are also supported: – Left outer – Right outer – Full outer 33
- 34. Other Commands Relational Operators – UNION – CROSS – ORDER – DISTINCT – LIMIT Eval Functions – Concat – Count – Diff 34
- 35. PARALLEL clause It is used to increase the parallelization of the job We can specify the number of reduce tasks of the MR jobs created by Pig It only effects the reduce task No control over map The system also can figure out number of reducers Mostly one reduce task is required 35
- 36. PARALLEL clause (cont.) Can be applied to only those commands which come under reduce phase – COGROUP – CROSS – DISTINCT – GROUP – JOINS – ORDER A = LOAD „ File1‟; B = LOAD „ File2‟; C = CROSS A, B PARALLEL 10; 36
- 37. Split Clause We can split the input record into many by providing condition A = LOAD „data‟ AS (F1:int, F2:int, F3:int) (1,2;3) (2,3;7) SPLIT A INTO B IF F1>7, C IF F2==5; B (1,2,3) C (2,5,7) (2,5,7) Any expression can be written UDFs can be used It is not partitioning 37
- 38. Output There are two ways to display – STORE • If you want to store the output in any location STORE output_1 INTO „hadoopuser/output‟ – DUMP • Basically used to display the result in the GRUNT shell itself • Dumping doesn‟t store the output anywhere DUMP query_result; 38
- 39. Building a Logical Plan Pig interpreter first parses all the commands which the client issues Verifies that the input files, bags or columns referred by the command are valid Builds a logical plan for every bag the user defines No processing is carried out Processing triggers where a user invokes STORE/DUMP command Called as a Lazy execution approach Helps in FILTER reordering 39
- 40. Debugging Environment This is used to avoid running the complete code on the entire dataset User can create a sample data Difficult to tailor these datasets and end up in self cooked data Pig Pen is Pig‟s debugging environment Creates side dataset automatically, called as sandbox dataset Pig Pen has its own user interface 40
- 41. Pig Pen Outputs can be easily analyzed Errors can be rectified earlier 41
- 42. Future Work User Interface – Drag-Drop style would help – Logical plan diagram create made easy UDF support for other languages Unified Environment – Currently, lacks in control structures like loops – Has to embedded for all iterative tasks 42
- 43. Summary Not So Foreign Language Aims a sweet spot between SQL and MapReduce Reusable and easy to use Novel Debugging Environment: Pig Pen Pig has an active and growing user base in Yahoo! Pigs – Eats anything – Live anywhere – Are domestic 43
- 44. 44
- 45. Based on “Pig Latin: A Not-So-Foreign Language for Data Processing” SIGMOD‟08,June 9–12, 2008, Vancouver,BC,Canada Christopher Olston Yahoo! Research Benjamin Reed Yahoo! Research Utkarsh Srivastava Yahoo! Research Ravi Kumar Yahoo! Research Andrew Tomkins Yahoo! Research 45
- 46. References http://infolab.stanford.edu/~usriv/papers/pig-latin.pdf http://pig.apache.org/docs/r0.7.0/piglatin_ref2.html Book: Programming pig http://www.brentozar.com/archive/2011/11/good-pig/ http://hortonworks.com/hadoop/pig/ 46