Chris Hillman – Beyond Mapreduce Scientific Data Processing in Real-time
Download as PPTX, PDF2 likes6,966 views
This document discusses processing scientific mass spectrometry data in real-time using parallel and distributed computing techniques. It describes how a mass spectrometry experiment produces terabytes of data that currently takes over 24 hours to fully process. The document proposes using MapReduce and Apache Flink to parallelize the data processing across clusters to help speed it up towards real-time analysis. Initial tests show Flink can process the data 2-3 times faster than traditional Hadoop MapReduce. Finally, it discusses simulating real-time streaming of the data using Kafka and Flink Streaming to enable processing results within 10 seconds of the experiment completing.
![public class PeakPick extends Configured implements Tool {
Job job=new Job(getConf(), "peakpick");
job.setJarByClass(PeakPick.class);
job.setNumReduceTasks(104);
job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(Text.class);
job.setMapperClass(MapHDFS.class);
job.setReducerClass(ReduceHDFS.class);
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setPartitionerClass(MZPartitioner.class);
FileInputFormat.setInputPaths(job, new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));
job.waitForCompletion(true);
return job.isSuccessful() ? 0 : 1;
} public static void main(String[] args) throws Exception{
int res = ToolRunner.run(new Configuration(), new PeakPick(), args);
System.exit(res); }
MR Job](https://image.slidesharecdn.com/chflinkscientificdata-151019140946-lva1-app6892/85/Chris-Hillman-Beyond-Mapreduce-Scientific-Data-Processing-in-Real-time-17-320.jpg)
![MR Read
public class MapHDFS extends Mapper<LongWritable, Text, IntWritable, Text> {
public void map(LongWritable key, Text value, Context context {
String inputLine = value.toString();
tempStr = inputLine.split("t"); scNumber = tempStr[1];
……
intensityString = tempStr[8]
}
public class MapCassandra extends Mapper<ByteBuffer, SortedMap<ByteBuffer, Cell>,
Text, IntWritable> {
public void map(ByteBuffer key, SortedMap<ByteBuffer, Cell> columns, Context conte
{
scNumber = String.valueOf(key.getInt());
for (Cell cell : columns.values())
{String name = ByteBufferUtil.string(cell.name().toByteBuffer());
if (name.contains("scan")) scNumber String.valueOf(ByteBufferUtil.toInt(cell.value()));
if (name.contains("mslvl")) scLevel = String.valueOf(ByteBufferUtil.toInt(cell.value()));
if (name.contains("rettime")) RT = String.valueOf(ByteBufferUtil.toDouble(cell.value()));
}](https://image.slidesharecdn.com/chflinkscientificdata-151019140946-lva1-app6892/85/Chris-Hillman-Beyond-Mapreduce-Scientific-Data-Processing-in-Real-time-18-320.jpg)
![Flink Job
public class PeakPickFlink_MR {
final ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
Job job = Job.getInstance();
//Setup input format
HadoopInputFormat<LongWritable, Text> hadoopIF =
new HadoopInputFormat<LongWritable, Text>(
new TextInputFormat(), LongWritable.class, Text.class, job );
TextInputFormat.addInputPath(job, new Path(args[0]));
//Read HDFS Data
DataSet<Tuple2<LongWritable, Text>> text = env.createInput(hadoopIF);](https://image.slidesharecdn.com/chflinkscientificdata-151019140946-lva1-app6892/85/Chris-Hillman-Beyond-Mapreduce-Scientific-Data-Processing-in-Real-time-20-320.jpg)
![Flink Job
// Set up the Hadoop TextOutputFormat.
HadoopOutputFormat<IntWritable, Text> hadoopOF =
new HadoopOutputFormat<IntWritable, Text>(
new TextOutputFormat<IntWritable, Text>(), job
);
//Write results back to HDFS
hadoopOF.getConfiguration().set("mapreduce.output.textoutputformat.separator"
, " ");
TextOutputFormat.setOutputPath(job, new Path(args[1]));
// Emit data using the Hadoop TextOutputFormat.
result.output(hadoopOF).setParallelism(1);
// Execute Code
env.execute("Hadoop PeakPick");
}
}](https://image.slidesharecdn.com/chflinkscientificdata-151019140946-lva1-app6892/85/Chris-Hillman-Beyond-Mapreduce-Scientific-Data-Processing-in-Real-time-22-320.jpg)
Chris Hillman – Beyond Mapreduce Scientific Data Processing in Real-time
- 1. beyond mapreduce scientific data processing in real-time Chris Hillman October 13th 2015 [email protected]
- 3. Mass Spectrometry Each Experiment produces 7Gb XML file containing 40,000 scans 600,000,000 data points In approx 100 minutes Data processing can take over 24 hours • Pick 2D Peaks • De-isotope • Pick 3D Peaks • Match weights to known peptides
- 4. Mass Spectrometry New Lab has 12 Machines That’s a lot of data and a lot of data processing
- 6. Parallel Processing Amdahl’s Law Serial portion is fixed Gustafson’s Law Size of problem is not fixed Gunther’s Law linear scalability issues
- 8. Parallel Algorithm 2D peak picking fits well into a Map task – Read into memory – Decode base64 float array – Peak pick, isotopic envelope detection
- 9. Parallel Algorithm 3D peak picking fits well into a Reduce task – Receive partitions of 2D peaks – Detect 3D peaks – Isotopic envelopes – Output peak Mass Intensity
- 10. Issues XML is not a good format for parallel processing
- 11. Issues 38 38.5 39 39.5 40 40.5 41 372 372.2 372.4 372.6 372.8 373 373.2 373.4 373.6 373.8 374
- 12. Issues Data Shuffle and skew on the cluster 0 20000 40000 60000 80000 100000 120000 50 85 120 155 191 226 261 296 331 366 401 436 471 506 541 576 611 646 681 716 751 786 821 856 891 926 961 996 1031 1066 1101 1136 1172 1208 1243 1280 1316 1351 1386 1422 1457 1492 1527 1562 1597 Series2
- 13. Results
- 15. MapReduce Transforming the XML and writing the modified data to • HDFS • Hbase • Cassandra Executing the MapReduce code reading from the above Batch process shows potential of speeding up the current process by scaling the size of the cluster running it.
- 16. Flink Experiences so far Very easy to install Very easy to understand Good documentation Very easy to adapt current code I like it!
- 17. public class PeakPick extends Configured implements Tool { Job job=new Job(getConf(), "peakpick"); job.setJarByClass(PeakPick.class); job.setNumReduceTasks(104); job.setOutputKeyClass(IntWritable.class); job.setOutputValueClass(Text.class); job.setMapperClass(MapHDFS.class); job.setReducerClass(ReduceHDFS.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); job.setPartitionerClass(MZPartitioner.class); FileInputFormat.setInputPaths(job, new Path(args[1])); FileOutputFormat.setOutputPath(job, new Path(args[2])); job.waitForCompletion(true); return job.isSuccessful() ? 0 : 1; } public static void main(String[] args) throws Exception{ int res = ToolRunner.run(new Configuration(), new PeakPick(), args); System.exit(res); } MR Job
- 18. MR Read public class MapHDFS extends Mapper<LongWritable, Text, IntWritable, Text> { public void map(LongWritable key, Text value, Context context { String inputLine = value.toString(); tempStr = inputLine.split("t"); scNumber = tempStr[1]; …… intensityString = tempStr[8] } public class MapCassandra extends Mapper<ByteBuffer, SortedMap<ByteBuffer, Cell>, Text, IntWritable> { public void map(ByteBuffer key, SortedMap<ByteBuffer, Cell> columns, Context conte { scNumber = String.valueOf(key.getInt()); for (Cell cell : columns.values()) {String name = ByteBufferUtil.string(cell.name().toByteBuffer()); if (name.contains("scan")) scNumber String.valueOf(ByteBufferUtil.toInt(cell.value())); if (name.contains("mslvl")) scLevel = String.valueOf(ByteBufferUtil.toInt(cell.value())); if (name.contains("rettime")) RT = String.valueOf(ByteBufferUtil.toDouble(cell.value())); }
- 19. MR Write public class MapHDFS extends Mapper<LongWritable, Text, IntWritable, Text> { public void map(LongWritable key, Text value, Context context { ………… for (int i=0; i<outputPoints.size(); i++){ mzStringOut = scNumber + "t" + scLevel + "t" + RT + "t" + Integer.toString(outputPoints.get(i).getCurveID()) + "t" + Double.toString(outputPoints.get(i).getWpm()) context.write(new IntWritable(outputPoints.get(i).getKey()), peakOut); } public class ReduceHDFS extends Reducer<IntWritable, Text, IntWritable, Text> { public void reduce(IntWritable key, Iterable<Text> values, Context context){ ………… for (int k = 0; k<MonoISO.size();k++){ outText = MonoISO.get(k).getCharge() + "t" + MonoISO.get(k).getWpm() + "t" + MonoISO.get(k).getSumI() + "t" + MonoISO.get(k).getWpRT(); context.write(new IntWritable(0), new Text(outText)); }
- 20. Flink Job public class PeakPickFlink_MR { final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); Job job = Job.getInstance(); //Setup input format HadoopInputFormat<LongWritable, Text> hadoopIF = new HadoopInputFormat<LongWritable, Text>( new TextInputFormat(), LongWritable.class, Text.class, job ); TextInputFormat.addInputPath(job, new Path(args[0])); //Read HDFS Data DataSet<Tuple2<LongWritable, Text>> text = env.createInput(hadoopIF);
- 21. Flink Job // use Hadoop Mapper as MapFunction DataSet<Tuple2<IntWritable, Text>> result = text .flatMap(new HadoopMapFunction<LongWritable, Text, IntWritable, Text>( new MapHDFS() )) .groupBy(0) // use Hadoop Reducer .reduceGroup(new HadoopReduceFunction<IntWritable, Text, IntWritable, Text>(new ReduceHDFS() ));
- 22. Flink Job // Set up the Hadoop TextOutputFormat. HadoopOutputFormat<IntWritable, Text> hadoopOF = new HadoopOutputFormat<IntWritable, Text>( new TextOutputFormat<IntWritable, Text>(), job ); //Write results back to HDFS hadoopOF.getConfiguration().set("mapreduce.output.textoutputformat.separator" , " "); TextOutputFormat.setOutputPath(job, new Path(args[1])); // Emit data using the Hadoop TextOutputFormat. result.output(hadoopOF).setParallelism(1); // Execute Code env.execute("Hadoop PeakPick"); } }
- 23. Interim Results Mapper only Hadoop 12m25s Flink 4m50s Mapper and Reducer Hadoop 28m32s Flink 10m20s Existing code 35m22s
- 24. Near real-time? Still not fast enough to be called near real-time Processing x scans per second, if the cluster was big enough then maybe… But… the mass spectrometer takes 100 minutes to complete its processing for one experiment so in fact we have more than enough time to process the data if we are streaming the results and processing the data as it is produced…
- 25. Streaming the data Simulate streaming data using an existing data file and kafka Ingest the data using Flink Streaming API and process the scans using the existing mapper code Existing Data File Kafka Flink Streaming API
- 26. Streaming the results A peptide elutes over a period of time, this means the data from many scans needs to be compared at the same time. A safe window to measure the quantity of a peptide is 10 seconds 10 seconds
- 27. Interim Results Overlapping 10 second windows captures 3D peaks from the 2D scans
- 28. Interim Results Processing the entire scan in the 10 second window means that we don’t need the overlapping window and the de-duplication step
- 29. All this means that the data will be fully pre-processed just over 10 seconds after the mass spectrometer completes the experiment. Near real-time? Stream Processing
- 30. Complete stable working system Contrast with Spark and Storm Hookup previous research on database lookup to create a complete system Pay for some EC2 system time to complete testing Write a thesis… To Do list
Editor's Notes
- #2: Who am I Teradata Working abroad a lot – time in hotels Part-time PhD University of Dundee Stubborn-ness
- #3: Proteomics intro, Genome project proteomics studying proteins in cells (treated vs non-treated) Reasons for disease cures for disease complexity, current processing methods and times Experiment digesting proteins using an enzyme into smaller pieces called peptides… Full process including database search, area of research pre-processing linking to work already done.
- #4: Peptides detected in a mass spectrometer…. Current problem space, description of data files. Issues with lag between experiment and data output, Biological and technical replicates QC data between experiments
- #5: 140Gb XML data 20 billion data points That’s big data to me Parallel algorithm, test on various distributed computing environments
- #6: Area of research is to write a parallel algorithm for processing the data and test it on different systems HDFS, Cassandra and Hbase for data storage Hadoop MapReduce, Spark, Storm for processing Doing some work for the EU commission and then flink came along…
- #7: Because this is a research project, there needs to be a formal approach Working in IT you get what works etc. These 3 laws are often quoted in conjunction with parallel processing
- #9: The experiment cuts the proteins into pieces (called peptides) Peaks are the molecular weight of the protein piece that we are trying to measure
- #10: The 3d peak matching is a complex algorithm
- #11: XML poor format for parallel processing Code to decode XML as it is copied on to the cluster Write to local, HDFS, Cassandra, Hbase Describe fields, scan number, retention time, Base64 fields containing float arrays
- #12: Overlapping regions required to process the reducers We know the details of how wide a 3D peak could be based on the biological properties of the peptides.. De-duplication may be needed
- #13: Custom partitioner needed to balance out the skew and produce the overlapping regions Beware coding always on a single node pseudo cluster – no distribution Beware test datasets – too small can hide O(N2) problems
- #14: It works!
- #15: Quick recap - what has been done so far using Hadoop ver 2 The map and reduce code produce correct results and the potential is there to decrease the time taken to process the data I know how many scans per second can be processed and the relationship between the size of the reduce mass window and time to completion.
- #16: The Hadoop MapReduce code works well, reading and writing from HDFS, Hbase and cassandra (writing to Cassandra issue) Current widely used software takes around 35mins, running bare metal Speed, little disappointing around 28 minutes to decode the test file – virtual machine issues and only 8 Map Slots Started looking at Spark but then doing some reviewing work and kept reading about Flink. So had to try it and the rest of the presentation is about my experiences with Flink
- #17: To Begin with I looked at how to port my existing code
- #18: Looking at some code to compare and contrast hadoop mapreduce to flink
- #19: Hadoop interfaces versions!! HDFS, read in key value and use string split to parse the tab separated values in the value part, Cassandra read uses the Cassandra Hadoop jars to read a cassandra column family (or table)
- #20: Hadoop interfaces versions!! Writing requires concatenating the output values into the value and writing back to HDFS Which the Reducer then reads back in and splits back into parts. After all the calcs are done the reducer concatenates all the results and writes back to HDFS
- #21: Flink Code
- #22: Using the example code I could put in my mapper and reducer from the MapReduce code and Run straight away… the map test worked perfectly and it’s fast (more later) The reducer worked much faster than expected (but results wrong!) (again more later) Note no write out to HDFS between map and reduce for the shuffle
- #23: Hadoop interfaces versions!! Writing requires concatenating the output values into the value and writing back to HDFS Straight from the Example code- mapper worked first time perfectly
- #24: Running just the map task is faster in flink, more so than I expected – especially considering how little effort it took in getting the mapper running. The full task with map and reduce was again faster but… the results were wrong. Slow and correct or fast and wrong… The issue is to do with my custom partitioner and the way results were sorted – more on that later Rewriting the controlling job and making small changes to the map and reduce code I could use data types such as the Tuple8 to store and pass my values Which is nice…
- #25: The point of the research was to produce the results in near real-time Is 10 minutes good enough? What about when the next part of the processing is added to the batch. The current database lookup process can take hours to complete.
- #26: Use Kafka to simulate streaming the output directly from the mass spec. In reality this would mean a major change by the manufacturer But this can show the benefits of making such a change. The map code which processes each scan is fairly easy to incorporate into a streaming pattern. You just treat each scan as a single unit of data and use exactly the same code as already exists. The reducer is more of a problem though.
- #27: The reduce code is more complicated
- #28: Need to process the 2D results in overlapping 10 second windows to find the 3D peaks. Sliding window, 10 seconds is roughly 40 scans, so 40 scans every 1 scan gives me what I need From the profiled of the 3D curve created by matching up the 2D peaks we can tell if we are have got the start and end of the peak in the window so no need for de-duplication. It still seems to be a map and a reduce step with the mapper decoding each scan and producing peaks, the reducer then aggregates the 2D peaks over time.
- #29: Removing the need to split the 2D scan by time removes the need for the overlapping window and the de-duplication. Previous methods was constrained by the way MapReduce makes you think… ended up with complex system of overlapping data and de-duplication to produce a way of running in parallel. The streaming method is still parallel but it is each 10 second window that can run on a separate machine not a mass window… Also removes the problem of having to decide how many mass windows to define as this determines how many reduce slots we need.
- #30: Using kafka to simulate the streaming of data and Flink to ingest the stream and process it show that would be possible to build and information factory for the life sciences data that could produce fully pre-processed mass spectrometer data (peak picked) data in near real-time
- #31: Using kafka to simulate the streaming of data and Flink to ingest the stream and process it show that would be possible to build and information factory for the life sciences data that could produce fully pre-processed mass spectrometer data (peak picked) data in near real-time