Apache Spark part of Eindhoven Java Meetup
0 likes100 views
Apache Spark is a fast, general-purpose engine for large-scale data processing, utilizing concepts like RDDs and Spark SQL for data analysis. It showcases functionalities such as Spark Streaming for real-time data handling and GraphX for graph processing. The document includes code examples demonstrating the usage of Spark in various contexts, including data input/output and machine learning.
![Java example (accumulator)
JavaRDD<String> rdd = sc.textFile(args[1]);
final Accumulator<Integer> blankLines = sc.accumulator(0);
JavaRDD<String> callSigns = rdd.flatMap(
new FlatMapFunction<String, String>() {
public Iterable<String> call(String line) {
if (line.equals("")) {
blankLines.add(1);
}
return Arrays.asList(line.split(" "));
}});
callSigns.saveAsTextFile("output.txt")](https://image.slidesharecdn.com/sparkmeetup-160603153111/85/Apache-Spark-part-of-Eindhoven-Java-Meetup-10-320.jpg)
Apache Spark part of Eindhoven Java Meetup
- 2. Introducing myself… • Mylène Reiners • Architect @Atos • Focus innovation
- 3. Sketching the context • Big Data • New insights • Analytics • Data discovery
- 6. Short demo in Scala (shell) • Simple data analysis • Read “README.md” • Count the number of lines
- 8. RDD • Resilient Distributed Dataset • Creation • Transformations • Actions
- 10. Java example (accumulator) JavaRDD<String> rdd = sc.textFile(args[1]); final Accumulator<Integer> blankLines = sc.accumulator(0); JavaRDD<String> callSigns = rdd.flatMap( new FlatMapFunction<String, String>() { public Iterable<String> call(String line) { if (line.equals("")) { blankLines.add(1); } return Arrays.asList(line.split(" ")); }}); callSigns.saveAsTextFile("output.txt")
- 13. Hive example // Import Spark SQL import org.apache.spark.sql.hive.HiveContext; // Or if you can't have the hive dependencies import org.apache.spark.sql.SQLContext; // Import the JavaSchemaRDD import org.apache.spark.sql.SchemaRDD; import org.apache.spark.sql.Row; (...) JavaSparkContext ctx = new JavaSparkContext(...); SQLContext hiveCtx = new HiveContext(ctx);
- 14. Hive example (cont’d) SchemaRDD input = hiveCtx.jsonFile(inputFile); // Register the input schema RDD input.registerTempTable("tweets"); // Select tweets based on the retweetCount SchemaRDD topTweets = hiveCtx.sql( "SELECT text, retweetCount FROM tweets ORDER BY retweetCount LIMIT 10");
- 16. Example // Create a StreamingContext with a 1-second batch size from a SparkConf JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1)); // Create a DStream from all the input on port 7777 JavaDStream<String> lines = jssc.socketTextStream("localhost", 7777); // Filter our DStream for lines with "error" JavaDStream<String> errorLines = lines.filter(new Function<String, Boolean>() { public Boolean call(String line) { return line.contains("error"); }}); // Print out the lines with errors errorLines.print();
- 17. Example // Start our streaming context and wait for it // to "finish" jssc.start(); // Wait for the job to finish jssc.awaitTermination();
- 19. Example // Load the edges as a graph val graph = GraphLoader.edgeListFile(sc, "followers.txt") // Run PageRank val ranks = graph.pageRank(0.0001).vertices // Join the ranks with the usernames val users = sc.textFile("users.txt").map { line => val fields = line.split(",") (fields(0).toLong, fields(1)) }
- 20. Example val ranksByUsername = users.join(ranks) .map {case (id, (username, rank)) => (username, rank) } // Print the result println(ranksByUsername.collect().mkString("n"))
- 22. Thank you