4developers2016- Strumieniowanie danych w Sparku- Bartosz Kowalik
0 likes182 views
The document is a presentation by Bartosz Kowalik on data streaming with Apache Spark, including an overview of Spark features and operations like transformations and actions. It briefly touches on architecture and monitoring, as well as recommended resources for learning Spark. The presentation aims to engage the audience with questions about their familiarity with Scala and Spark.
![TRANSFORMACJA
map(func) reduceByKey(func, [numTasks])
filter(func) aggregateByKey(zeroValue)(seqOp, combOp,
[numTasks])
flatMap(func) sortByKey([ascending], [numTasks])
mapPartitions(func) join(otherDataset, [numTasks])
mapPartitionsWithIndex(func) cogroup(otherDataset, [numTasks])
sample(withReplacement,
fraction, seed)
cartesian(otherDataset)
union(otherDataset) pipe(command, [envVars])
intersection(otherDataset) coalesce(numPartitions)
distinct([numTasks])) repartition(numPartitions)
groupByKey([numTasks]) repartitionAndSortWithinPartitions(partition](https://image.slidesharecdn.com/strumieniowaniedanychwsparku-160516080751/85/4developers2016-Strumieniowanie-danych-w-Sparku-Bartosz-Kowalik-8-320.jpg)
![AKCJE
reduce(func) takeSample(withReplacement, num, [seed])
collect() takeOrdered(n, [ordering])
count() saveAsTextFile(path)
first() saveAsSequenceFile(path)
take(n) saveAsObjectFile(path)
https://spark.apache.org/docs/latest/programmingguide.html#actions](https://image.slidesharecdn.com/strumieniowaniedanychwsparku-160516080751/85/4developers2016-Strumieniowanie-danych-w-Sparku-Bartosz-Kowalik-10-320.jpg)
4developers2016- Strumieniowanie danych w Sparku- Bartosz Kowalik
- 6. PRZYKŁAD
- 8. TRANSFORMACJA map(func) reduceByKey(func, [numTasks]) filter(func) aggregateByKey(zeroValue)(seqOp, combOp, [numTasks]) flatMap(func) sortByKey([ascending], [numTasks]) mapPartitions(func) join(otherDataset, [numTasks]) mapPartitionsWithIndex(func) cogroup(otherDataset, [numTasks]) sample(withReplacement, fraction, seed) cartesian(otherDataset) union(otherDataset) pipe(command, [envVars]) intersection(otherDataset) coalesce(numPartitions) distinct([numTasks])) repartition(numPartitions) groupByKey([numTasks]) repartitionAndSortWithinPartitions(partition
- 10. AKCJE reduce(func) takeSample(withReplacement, num, [seed]) collect() takeOrdered(n, [ordering]) count() saveAsTextFile(path) first() saveAsSequenceFile(path) take(n) saveAsObjectFile(path) https://spark.apache.org/docs/latest/programmingguide.html#actions
- 12. PRZYKŁAD
- 14. ARCHITEKTURA
- 15. KOD!
- 17. MONITORING
- 18. MONITORING