Meetup - Exabyte Big Data - HPCC Systems - SQL to ECL
0 likes•1,426 views
ECL (Enterprise Control Language) is a C++ based query language for HPCC Systems, designed for big data processing, and is similar yet more expressive than Hadoop's Pig. The document outlines practical examples of using ECL to perform SQL-like queries including creating tables, applying schemas, and operations such as selection, grouping, and counting data. Additionally, resources for further learning, such as installation guides and sample datasets, are provided.
![DATA
SQL ECL
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
USE ‘layout_person’;
or
Inline Data
Data from HPCC Systems Cluster
allPeople := DATASET(‘~file’,Layout_Person,THOR);](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-6-320.jpg)
![WHERE
SQL ECL
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
END;
!
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
!
somePeople := allPeople(LastName = 'Smith');
!
OUTPUT(somePeople);
SELECT * FROM `allPeople`
WHERE `LastName` = ‘Smith’](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-7-320.jpg)
![SELECT
SQL ECL
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
END;
!
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
!
//somePeople := allPeople(LastName = 'Smith');
//OUTPUT(somePeople);
!
OUTPUT(allPeople);
SELECT * FROM `allPeople`](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-8-320.jpg)
![SELECT WITH
SQL ECL
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
END;
!
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
!
OUTPUT(allPeople,{LastName,FirstName});
SELECT ‘LastName’,’FirstName’
FROM `allPeople`](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-9-320.jpg)
![ORDER BY
SQL ECL
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
END;
!
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
!
!
OUTPUT(SORT(allPeople,LastName));
SELECT * FROM `allPeople`
ORDER BY ‘LastName’](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-10-320.jpg)
![GROUP BY
SQL ECL
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
END;
!
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
!
!
OUTPUT(GROUP(SORT(allPeople,LastName)));
SELECT * FROM `allPeople`
GROUP BY ‘LastName’](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-11-320.jpg)
![GROUP BY & COUNT
SQL ECL
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
END;
!
allPeople := DATASET([
{1,'Fred','Smith'},
{2,'Joe','Blow'},
{3,’Jane','Smith'}
],Layout_Person);
!
!
OUTPUT(TABLE(allPeople,
{LastName,
LastNameC := COUNT(GROUP)},
LastName)
);
SELECT LastName,COUNT(*)
FROM `allPeople`
GROUP BY ‘LastName’](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-12-320.jpg)
![LIKE
SQL ECL
IMPORT Std;
!
Layout_Person := RECORD
UNSIGNED1 PersonID;
STRING15 FirstName;
STRING25 LastName;
STRING33 Email;
END;
!
allPeople := DATASET([
{1,’Fred’,’Smith’,’fred@yahoo.com’},
{2,'Joe','Blow',’joe@hotmail.com’},
{3,’Jane’,’Smith',’jane@gmail.com’}
],Layout_Person);
!
emails := allPeople(
Std.Str.EndsWith(Email, ‘@gmail.com’)
);
OUTPUT(emails);
SELECT * FROM `allPeople`
WHERE `Email` LIKE ‘%@gmail.com’
Import C++ libraries
and/or make
your own functions &
libraries](https://image.slidesharecdn.com/pxgq43srqsqusggsmplo-140617000157-phpapp01/85/Meetup-Exabyte-Big-Data-HPCC-Systems-SQL-to-ECL-13-320.jpg)
Meetup - Exabyte Big Data - HPCC Systems - SQL to ECL
- 1. SQL TO ECL (ENTERPRISE CONTROL LANGUAGE) By Fujio Turner @FujioTurner
- 2. WHAT IS ECL ECL (Enterprise Control Language) is a C++ based query language for use with HPCC Systems Big Data platform. ECLs syntax and format is very simple and easy to learn.! ! Note - ECL is very similar to Hadoop’s pig ,but! more expressive and feature rich.
- 3. JDBC CONNECTOR HPCC Systems Can Do SQL SELECT * FROM `allPeople` Learn ECL to Do Complex Queries SQL OR Graph http://www.slideshare.net/hpccsystems/jdbc-hpcc
- 4. 2. 1. 3. 4. Lets Practice ECL
- 5. SCHEMA SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; CREATE TABLE layout_person ( PersonID INT(15) NOT NULL, FirstName VARCHAR(15) NOT NULL, LastName VARCHAR(25) NOT NULL, PRIMARY KEY (PersonID) ); Apply Schema on Read
- 6. DATA SQL ECL allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); USE ‘layout_person’; or Inline Data Data from HPCC Systems Cluster allPeople := DATASET(‘~file’,Layout_Person,THOR);
- 7. WHERE SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; ! allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); ! somePeople := allPeople(LastName = 'Smith'); ! OUTPUT(somePeople); SELECT * FROM `allPeople` WHERE `LastName` = ‘Smith’
- 8. SELECT SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; ! allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); ! //somePeople := allPeople(LastName = 'Smith'); //OUTPUT(somePeople); ! OUTPUT(allPeople); SELECT * FROM `allPeople`
- 9. SELECT WITH SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; ! allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); ! OUTPUT(allPeople,{LastName,FirstName}); SELECT ‘LastName’,’FirstName’ FROM `allPeople`
- 10. ORDER BY SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; ! allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); ! ! OUTPUT(SORT(allPeople,LastName)); SELECT * FROM `allPeople` ORDER BY ‘LastName’
- 11. GROUP BY SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; ! allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); ! ! OUTPUT(GROUP(SORT(allPeople,LastName))); SELECT * FROM `allPeople` GROUP BY ‘LastName’
- 12. GROUP BY & COUNT SQL ECL Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; END; ! allPeople := DATASET([ {1,'Fred','Smith'}, {2,'Joe','Blow'}, {3,’Jane','Smith'} ],Layout_Person); ! ! OUTPUT(TABLE(allPeople, {LastName, LastNameC := COUNT(GROUP)}, LastName) ); SELECT LastName,COUNT(*) FROM `allPeople` GROUP BY ‘LastName’
- 13. LIKE SQL ECL IMPORT Std; ! Layout_Person := RECORD UNSIGNED1 PersonID; STRING15 FirstName; STRING25 LastName; STRING33 Email; END; ! allPeople := DATASET([ {1,’Fred’,’Smith’,’[email protected]’}, {2,'Joe','Blow',’[email protected]’}, {3,’Jane’,’Smith',’[email protected]’} ],Layout_Person); ! emails := allPeople( Std.Str.EndsWith(Email, ‘@gmail.com’) ); OUTPUT(emails); SELECT * FROM `allPeople` WHERE `Email` LIKE ‘%@gmail.com’ Import C++ libraries and/or make your own functions & libraries
- 14. MORE SAMPLES Install HPCC Systems and go to the admin playground and see more samples of ECL.
- 15. ECL LANGUAGE GUIDE http://hpccsystems.com/download/docs/ecl-language-reference JOIN! MERGE! LENGTH! REGEX! ROUND! SUM! COUNT! TRIM! WHEN! AVE! ABS! CASE! DEDUP! NORMALIZE! DENORMALIZE! IF! SORT! GROUP! more ….
- 16. SCHEMA MADE EZ Uploading data from a new source and want a quick schema? Click below http://hpccsystems.com/demos/data-profiling-demo CSV IN Schema OUT Click
- 17. HPCC SYSTEMS DEMO http://hpccsystems.com/demos/wikidemo
- 18. Watch how to install HPCC Systems in 5 Minutes Download HPCC Systems Open Source Community Edition http://hpccsystems.com/download/ http://www.youtube.com/watch?v=8SV43DCUqJg or Source Code https://github.com/hpcc-systems