Nov HUG 2009: Hadoop Record Reader In Python
4 likes782 views
The document discusses creating a Hadoop Record Reader in Python, highlighting the use of the native data storage format called Jute and a data definition language (DDL) for data structures. It touches on the challenges of parsing binary formats and provides insights on building a parser using lex and yacc. The author mentions future work, including developing a DDL converter and integrating feedback into the project.
![remember, to only use C++/Java$rcc--help Usage: rcc --language[java|c++] ddl-files](https://image.slidesharecdn.com/hadooprecordreaderinpython-091124154806-phpapp01/85/Nov-HUG-2009-Hadoop-Record-Reader-In-Python-6-320.jpg)
![and any time you feel the pain…Parsing the binary format is hardVector vsstruct???struct= "s{" record *("," record) "}"vector = "v{" [record *("," record)] "}"LazyString – don’t decode if not needed99% of my hadoop time was decoding strings I didn’t needBinary on disk -> CSV -> python == wastefulHadoopupacks zip files – name it .mod](https://image.slidesharecdn.com/hadooprecordreaderinpython-091124154806-phpapp01/85/Nov-HUG-2009-Hadoop-Record-Reader-In-Python-11-320.jpg)
Nov HUG 2009: Hadoop Record Reader In Python
- 1. Hadoop Record Reader in PythonHUG: Nov 18 2009Paul Tarjanhttp://paulisageek.com@ptarjanhttp://github.com/ptarjan/hadoop_record
- 2. Hey Jute…Tabs and newlines are good and allFor lots of data, don’t do that
- 3. don’t make it bad...Hadoop has a native data storage format called Hadoop Record or “Jute”org.apache.hadoop.recordhttp://en.wikipedia.org/wiki/Jute
- 4. take a data structure…There is a Data Definition Language!module links { class Link {ustringURL;booleanisRelative;ustringanchorText; };}
- 5. and make it better…And a compiler$ rcc -lc++ inclrec.jrtestrec.jr namespace inclrec { class RI : public hadoop::Record { private: int32_t I32; double D;std::string S;
- 6. remember, to only use C++/Java$rcc--help Usage: rcc --language[java|c++] ddl-files
- 7. then you can start to make it better…I wanted it in pythonNeed 2 parts:Parsing library and DDL translatorI only did the first partIf you need second part, let me know
- 8. Hey Jute don't be afraid…
- 9. you were made to go out and get her…http://github.com/ptarjan/hadoop_record
- 10. the minute you let her under your skin…I bet you thought I was done with “Hey Jude” references, eh?How I built itPly == lex and yaccParser == 234 lines including tests!Outputs generic data typesYou have to do the class transform yourselfYou can use my lex and yacc stuff in your language of choice
- 11. and any time you feel the pain…Parsing the binary format is hardVector vsstruct???struct= "s{" record *("," record) "}"vector = "v{" [record *("," record)] "}"LazyString – don’t decode if not needed99% of my hadoop time was decoding strings I didn’t needBinary on disk -> CSV -> python == wastefulHadoopupacks zip files – name it .mod
- 12. nananananaFuture workDDL ConverterIntegrate it officiallyRecord writer (should be easy)SequenceFileAsOutputFormatIntegrate your feedback