HPCC Systems - ECL for Programmers - Big Data - Data Scientist
0 likes1,353 views
The document is an introduction to Enterprise Control Language (ECL) used in HPCC Systems for big data querying, emphasizing its simplicity and effectiveness compared to other programming languages. It highlights how ECL allows for straightforward data loading and querying, likening it to common operations performed on text files. Additionally, the document provides various examples of ECL syntax and functionalities, demonstrating its capabilities for data manipulation and analysis.
![Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
General ECL](https://image.slidesharecdn.com/eclforprogrammersslideshare-141006170408-conversion-gate02/85/HPCC-Systems-ECL-for-Programmers-Big-Data-Data-Scientist-10-320.jpg)
![Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
Filter Data By
General ECL](https://image.slidesharecdn.com/eclforprogrammersslideshare-141006170408-conversion-gate02/85/HPCC-Systems-ECL-for-Programmers-Big-Data-Data-Scientist-11-320.jpg)
![Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
Filter Data By
Output
General ECL](https://image.slidesharecdn.com/eclforprogrammersslideshare-141006170408-conversion-gate02/85/HPCC-Systems-ECL-for-Programmers-Big-Data-Data-Scientist-12-320.jpg)
![Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
sara := d(Name = ‘Sara’);
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
Filter Data By
Output
General ECL](https://image.slidesharecdn.com/eclforprogrammersslideshare-141006170408-conversion-gate02/85/HPCC-Systems-ECL-for-Programmers-Big-Data-Data-Scientist-13-320.jpg)
![Find “Sara”: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
Split Data by Column
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = split(row,“ ”)!
! if(new_row[0] == ‘Sara’){!
! ! print ”Found Sara”!
! }!
}
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
sara := d(Name = ’Sara’);
OUTPUT(sara);
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
Filter Data By
Output
General ECL](https://image.slidesharecdn.com/eclforprogrammersslideshare-141006170408-conversion-gate02/85/HPCC-Systems-ECL-for-Programmers-Big-Data-Data-Scientist-14-320.jpg)
![Find “Sara” & Older then 50: general programming vs ECL
cs := RECORD!
! STRING20 Name;!
! STRING2 State;!
! INT3 Age;!
END
d = fopen(‘~/cdata_2010.txt’)
new_d = split( d ,“rn”)
for(x = 0; x< 3; x++){!
! row = new_d[x]!
! new_row = row.split(“ ”)!
! if(new row[0] == ‘Sara’ and row[2] >50){!
! ! print ”Found Sara”!
! }!
}
d := DATASET(‘~hpcc::cdata_2010’,cs,THOR);
sara := d(Name = ‘Sara’ AND Age > 50);
OUTPUT(sara);
0 1 2
Kevin CA 45
Mark MI 27
Sara FL 64
General ECL](https://image.slidesharecdn.com/eclforprogrammersslideshare-141006170408-conversion-gate02/85/HPCC-Systems-ECL-for-Programmers-Big-Data-Data-Scientist-15-320.jpg)
HPCC Systems - ECL for Programmers - Big Data - Data Scientist
- 1. HPCC Systems - ECL Intro Big Data Querying Made EZ By Fujio Turner Enterprise Control Language explained for Programmers @FujioTurner
- 2. Comparison Block Based File Based JAVA C++ Petabytes 1-80,000 Jobs/day Since 2005 Exabytes Non-Indexed 4X-13X Indexed: 2K-3K Jobs/sec Since 2000 ? ? ? ? ? ? Thor Roxie
- 3. What Is ECL? ECL (Enterprise Control Language) is a C++ based query language for use with HPCC Systems Big Data platform. ECLs syntax and format is very simple and easy to learn.! ! Note - ECL is very similar to Hadoop’s pig ,but! more expressive and feature rich.
- 4. Comparing ECL to General Programming In this presentation you will see how in ECL loading and querying data is just like reading and finding data in a plain text file.! general programming (general common logic)! vs.! ECL General Code HERE ECL Code HERE General ECL
- 5. Example Text File Name State Age Kevin CA 45 Mark MI 27 Sara FL 64 Customer Data May 2010 ~/cdata_2010.txt! example file name = ~/hpcc::cdata_2010.txt ECL example file distributed in HPCC cluster
- 6. Opening File: general programming vs ECL d = fopen(‘~/cdata_2010.txt’) File Location d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); General ECL
- 7. Opening File: general programming vs ECL d = fopen(‘~/cdata_2010.txt’) File Location Open File Function d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); General ECL
- 8. Organizing: general programming vs ECL d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) Split Data(d) by Row Kevin CA 45 Mark MI 27 Sara FL 64 d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); General ECL
- 9. Organizing: general programming vs ECL d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END Split Data(d) by Row d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); Use This Schema on this file! to Give Structure to Data Kevin CA 45 Mark MI 27 Sara FL 64 General ECL
- 10. Find “Sara”: general programming vs ECL cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) Split Data by Column for(x = 0; x< 3; x++){! ! row = new_d[x]! ! new_row = split(row,“ ”)! ! if(new_row[0] == ‘Sara’){! ! ! print ”Found Sara”! ! }! } 0 1 2 Kevin CA 45 Mark MI 27 Sara FL 64 d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); General ECL
- 11. Find “Sara”: general programming vs ECL cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) Split Data by Column for(x = 0; x< 3; x++){! ! row = new_d[x]! ! new_row = split(row,“ ”)! ! if(new_row[0] == ‘Sara’){! ! ! print ”Found Sara”! ! }! } 0 1 2 Kevin CA 45 Mark MI 27 Sara FL 64 d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); Filter Data By General ECL
- 12. Find “Sara”: general programming vs ECL cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) Split Data by Column for(x = 0; x< 3; x++){! ! row = new_d[x]! ! new_row = split(row,“ ”)! ! if(new_row[0] == ‘Sara’){! ! ! print ”Found Sara”! ! }! } 0 1 2 Kevin CA 45 Mark MI 27 Sara FL 64 d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); Filter Data By Output General ECL
- 13. Find “Sara”: general programming vs ECL cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) Split Data by Column for(x = 0; x< 3; x++){! ! row = new_d[x]! ! new_row = split(row,“ ”)! ! if(new_row[0] == ‘Sara’){! ! ! print ”Found Sara”! ! }! } d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); sara := d(Name = ‘Sara’); 0 1 2 Kevin CA 45 Mark MI 27 Sara FL 64 Filter Data By Output General ECL
- 14. Find “Sara”: general programming vs ECL cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) Split Data by Column for(x = 0; x< 3; x++){! ! row = new_d[x]! ! new_row = split(row,“ ”)! ! if(new_row[0] == ‘Sara’){! ! ! print ”Found Sara”! ! }! } d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); sara := d(Name = ’Sara’); OUTPUT(sara); 0 1 2 Kevin CA 45 Mark MI 27 Sara FL 64 Filter Data By Output General ECL
- 15. Find “Sara” & Older then 50: general programming vs ECL cs := RECORD! ! STRING20 Name;! ! STRING2 State;! ! INT3 Age;! END d = fopen(‘~/cdata_2010.txt’) new_d = split( d ,“rn”) for(x = 0; x< 3; x++){! ! row = new_d[x]! ! new_row = row.split(“ ”)! ! if(new row[0] == ‘Sara’ and row[2] >50){! ! ! print ”Found Sara”! ! }! } d := DATASET(‘~hpcc::cdata_2010’,cs,THOR); sara := d(Name = ‘Sara’ AND Age > 50); OUTPUT(sara); 0 1 2 Kevin CA 45 Mark MI 27 Sara FL 64 General ECL
- 16. ECL is EZ •Make your own functions & libraries in ECL.! •Modularize your code with “Import”: reuse old code Machine Learning Built-in http://hpccsystems.com/ml
- 17. ECL Plugin for Eclipse IDE http://hpccsystems.com/products-and-services/products/plugins/eclipse-ide
- 18. ECL + Others Languages ECL is C++ based so all your C/C++ code can be used in ECL.! &! Use other languages and methods like below to query too.
- 19. ECL GUIDE http://hpccsystems.com/download/docs/ecl-language-reference JOIN! MERGE! LENGTH! REGEX! ROUND! SUM! COUNT! TRIM! WHEN! AVE! ABS! CASE! DEDUP! NORMALIZE! DENORMALIZE! IF! SORT! GROUP! more ….
- 20. For More HPCC “How To’s” Go to Query with Plain SQL http://www.slideshare.net/hpccsystems/jdbc-hpcc or SQL TO ECL http://www.slideshare.net/FujioTurner/meet-up-sqldemopp
- 21. Watch how to install HPCC Systems in 5 Minutes Download HPCC Systems Open Source Community Edition http://hpccsystems.com/download/ http://www.youtube.com/watch?v=8SV43DCUqJg or Source Code https://github.com/hpcc-systems