ETL Metadata Injection with Pentaho Data Integration
5 likes5,644 views
This document discusses using ETL Metadata Injection in Pentaho to dynamically load metadata at runtime. Specifically, it describes how to use a template transformation to load cost files with a dynamic header structure, where budget files contain 12 months of data but forecast files can contain between 1-12 months. It provides examples of injecting metadata into text file input and row normalization steps in the template using the Metadata Injection step, and how to programmatically inject metadata into steps that don't natively support injection using the Pentaho API.
ETL Metadata Injection with Pentaho Data Integration
- 1. ETL Metadata Injection David Fombella Pombal http://anonymousbi.wordpress.com @Pentaho_Fan
- 2. ETL Metadata Injection 2 David Fombella Pombal BI Consultant, Technical Developmental Editor and Pentaho Community Guy About me
- 3. ETL Metadata Injection 3 Introduction •The ETL Metadata Injection step is capable of injecting step metadata into a template transformation. So instead of statically entering ETL metadata in a step dialog, you pass it at run-time. It is possible to solve repetitive ETL workloads like loading of text files, data migration and so on.
- 4. ETL Metadata Injection 4 Problem description •We need to load cost files containing a dynamic header •Budget files contain 12 months information •Forecast files contain between 1 and 12 months •3 fixed columns containing Entity, Account and Cost Centre
- 5. ETL Metadata Injection 5 File Types •Budget file containing 2015 (12 months) •Forecast file October 2014 (3 months)
- 6. ETL Metadata Injection 6 Easy to use + Open Source + BI Solution = Pentaho and Mondrian! Template transformation (Text file input)
- 7. ETL Metadata Injection 7 Easy to use + Open Source + BI Solution = Pentaho and Mondrian! Template transformation (Row normaliser)
- 8. ETL Metadata Injection 8 Load file Metadata Injection 1
- 9. ETL Metadata Injection 9 Load file Metadata Injection 2
- 10. ETL Metadata Injection 10 Load file Metadata Injection 3
- 11. ETL Metadata Injection 11 Load file after Injection
- 12. ETL Metadata Injection 12 Non-Native Metadata Injection Metadata Injection is one of the single most powerful features of PDI. However it only some steps support it. But as PDI has 100s of steps, more being added weekly, there’s always going to be a scenario where the step you want to Inject doesn't support it. So what do you do? Use the API. The API is how PDI itself sets the metadata for a step. So the steps are: 1. Open the transformation file 2. Find the step(s) we want to change 3. Configure it accordingly 4. Save the file
- 13. ETL Metadata Injection 13 Non-Native Metadata Injection (template)
- 14. ETL Metadata Injection 14 Non-Native Metadata Injection (Inject 1)
- 15. ETL Metadata Injection 15 Non-Native Metadata Injection (Inject 2)
- 16. ETL Metadata Injection 16 Non-Native Metadata Injection (Inject 3)
- 17. ETL Metadata Injection 17 Non-Native Metadata Injection Thanks to…