Command line Data Tools
2 likes290 views
This document lists and describes several command-line data tools. It discusses why command-line tools are useful for data work, and provides sources for finding more tools. Several specific command-line tools are called out, including jq for JSON, csvkit for CSV, and dt for various data formats. The document proposes an ideal command-line data tool that would support many file formats and data processing capabilities. It also mentions that Continuum is hiring for various roles related to their Anaconda distribution and data science products.
Command line Data Tools
- 1. Command-line Data Tools Peter Wang @pwang
- 2. Why? • Some times you just want to sling data • Text is still king; Lowest common denominator • Machines are pretty honking big now
- 3. This Presentation • List of some good collections of cmd-line tools • Call out and describe a few in particular • The PyDataTool of my desire
- 4. Sources • From author of “Data Science at the Command Line”: http://jeroenjanssens.com/2013/09/19/seven- command-line-tools-for-data-science.html (larger list at http://datascienceatthecommandline.com/) • HN discussion: https://news.ycombinator.com/ item?id=6412190 • https://github.com/bitly/data_hacks
- 5. Tools • JSON: • jq: https://stedolan.github.io/jq/ • RecordStream: https://github.com/benbernard/ RecordStream • csvkit: https://csvkit.readthedocs.io/en/1.0.2/ • dt: https://github.com/clarkgrubb/data-tools • XMLStarlet: http://xmlstar.sourceforge.net/overview.php
- 6. Honorable Mentions • Pythonic awk: https://github.com/alecthomas/ pawk • Google Crush Tools: https://github.com/google/ crush-tools • Xonsh: http://xon.sh/tutorial.html
- 7. The PyDataTool of My Desire • Support for csv, json, sql, xls, hdf5; image formats; network formats (pcap etc.) • Capability of: • csvkit, jq, dt, “cols” tool • unix tools: sed, sort, shuf, split, tr, tee, uniq, wc, head, tail, bc • netpbm, imagemagick for images • Work in streaming mode (netcat, wget, curl) • First-class support for dask, spark • Basic plotting via gnuplot, mpl, bokeh • Built-in SQLite to do in-memory support for queries
- 8. Continuum Is Hiring! • Creators of Anaconda, conda, bokeh, blaze, dask, holoviews, numba, phosphorJS • Maintainers/contributors to Jupyter, JupyterLab, Spyder, pandas, conda-forge, … • 150+ ppl, 80 in Austin • Venture backed • Enterprise product, OSS community innovation, consulting, training
- 9. Continuum Is Hiring • Enterprise Product Team: • Dev Manager (reports to CTO, runs product engineering) • QA Lead Engineer - creates test plans, coordinates with product mgmt, dev, and testing team • Senior Python Developer - enterprise product development; backend, web tech; full stack preferred • DevOps and Operations - enterprise product, anaconda.org, Anaconda build system • Email [email protected]