Word Count with a token scrub operation -- similar to "Impatient", Part 3:
rm -rf output
scald.rb --hdfs-local src/main/scala/Example3.scala --doc data/rain.txt --wc output/wc
Word Count with a stop word list based on HashJoin -- similar to "Impatient", Part 4:
rm -rf output
scald.rb --hdfs-local src/main/scala/Example4.scala --doc data/rain.txt --stop data/en.stop --wc output/wc
To build a fat jar with Gradle (version 1.3 or later) which uses the build.gradle script:
gradle clean jar
rm -rf output
hadoop jar build/libs/impatient.jar Example3 --hdfs --doc data/rain.txt --wc output/wc
rm -rf output
hadoop jar build/libs/impatient.jar Example4 --hdfs --doc data/rain.txt --stop data/en.stop --wc output/wc
To build a fat jar with Maven (version 3.0.4 or later) which uses the pom.xml script:
mvn clean package
rm -rf output
hadoop jar target/part8-1.0.0.jar Example3 --hdfs --doc data/rain.txt --wc output/wc
rm -rf output
hadoop jar target/part8-1.0.0.jar Example4 --hdfs --doc data/rain.txt --stop data/en.stop --wc output/wc
Example logs of running these apps are in https://gist.github.com/4371896
BTW, if you need to use any locally built JARs with this script, move them into the local lib directory.
To run this Scalding app on the Amazon AWS cloud, you'll need to have an AWS account.
Be sure to sign up for EMR, S3, SimpleDB. Also have your credentials set up locally --
for example, in your ~/.aws_cred/ directory.
Then install these two excellent AWS tools:
Next, edit the emr.sh shell script to update the BUCKET variable for one of your S3 buckets.
Finally, use the emr.sh shell script to upload your JAR and input data,
and run the app on Elastic MapReduce.
Many thanks to Sujit Pal, Chris Severs, Dean Wampler, Oscar Boykin, Hans Dockter -- for assistance with Scalding code samples, Gradle build for Scala, Maven build, etc.
For more discussion, see the cascading-user email forum.
Stay tuned for the next installments of our Cascading for the Impatient series.