Poster Hadoop summit 2011: pig embedding in scripting languages
Download as PPTX, PDF1 like949 views
Pig Scripting allows Pig to be used for iterative algorithms and user defined functions by embedding Pig within a scripting language. This allows transitive closure, an iterative process, to be computed with one file using Python functions as UDFs and Python variables in Pig scripts, rather than requiring seven files and Java UDFs as plain Pig would.
Poster Hadoop summit 2011: pig embedding in scripting languages
- 1. PIG ScriptingMaking Pig Turing-complete through embedding in a scripting languageJulien Le Dem - YahooOverviewPig execution model - Map/Reduce: a solution to process big data. - Pig: makes it easier to manipulate data on the grid. - Pig scripting: makes Pig easier with iterative algorithms and User Defined Functions.Jiras: PIG-1479, PIG-1794 (in 0.9), PIG-928 (in 0.8)Example: Transitive closure - Iterative process: requires a loop and a termination condition - Requires multiple join/group by: typical Pig usage - Requires User Defined FunctionsSolution 2: using Pig scriptingSolution 1: plain Pig1 file required:7 files required:UDFs take the elements of the tuple as parameters, not a tuple.The output schema is specified using a decorator (it can be a function if you need to manipulate the input schema)Modification flow:Modification flow:UDFs use standard Python constructs (tuple/list/dictionary) automatically converted to Pig.Python functions are automatically available as UDFsEmbedded Pig calls in PythonPython variables can be used in the pig scripts $n, $i, ...Unfortunately this solution does not fit here.It requires 7 artifacts: 3 Java UDFs: 3 classes must be compiled and packaged in a jar. The average UDF size is 50 to 100 lines of code.
- 2. 3 Pig scripts in 3 separate files: init, main loop content, finalization. The average Pig script size is 5 to 10 lines.
- 3. Main program: executes and coordinates the Pig scripts. It contains about 50 lines of code.This solution does fit on the poster.It requires 1 artifact, containing: 3 UDFs provided as functions: The average UDF size is 8 to 12 lines of code.
- 4. 3 Pig queries: init, main loop content, finalization. Each Pig query adds an average 5 to 10 lines.
- 5. Main function: executes and coordinates the Pig scripts. It adds about 10 lines to the script.Access to the output of pig scripts