Fault-Tolerant File Input & Output
Download as PPTX, PDF0 likes375 views
The document outlines the fault-tolerant file input and output mechanisms in Apache Apex, detailing features such as window management, various file reading techniques, and scalability solutions. It describes how different operators, such as AbstractFileInputOperator and AbstractFileOutputOperator, ensure data integrity and fault tolerance through state checkpointing and record management. Additionally, it provides insights into custom implementations and mentions the contributions of the Apache Apex community.
![FileSplitterInput & AbstractFSBlockReader: Implementations
- FileSplitterInput is concrete. Default behavior can be overridden.
- FS Block Readers
- FSSliceReader : record is a slice
- AbstractFSLineReader and AbstractFSReadAheadLineReader: record is a line
- Custom FS Block Reader
public class CustomFSBlockReader<RECORD> extends AbstractFSBlockReader<RECORD>
{
public CustomFSBlockReader()
{
//initialize reader context
this.readerContext = new RecordReaderContext();
}
@Override
protected RECORD convertToRecord(byte[] bytes)
{
//convert bytes to RECORD
return RECORD.from(bytes);
}
}](https://image.slidesharecdn.com/fault-tolerantfileinputoutput-160505154611/85/Fault-Tolerant-File-Input-Output-14-320.jpg)
![AbstractFileOutputOperator : Custom Implementation
public class CustomFileOutputOperator<RECORD> extends AbstractFileOutputOperator<RECORD>
{
public CustomFileOutputOperator()
{
setMaxLength(1024 * 1024);
setRotationWindows(600);
}
@Override
protected String getFileName(RECORD tuple)
{
//file name
return tuple.getFileName();
}
@Override
protected byte[] getBytesForTuple(RECORD tuple)
{
//bytes from record
return tuple.toBytes();
}
}](https://image.slidesharecdn.com/fault-tolerantfileinputoutput-160505154611/85/Fault-Tolerant-File-Input-Output-20-320.jpg)
Fault-Tolerant File Input & Output
- 1. Fault-tolerant File Input & Output Chandni Singh - Committer Apache Apex May 4, 2016
- 2. Background- Windows in Apex - Window: finite piece of a data set along temporal boundaries* - Apex assigns an id to each window which helps with fault-tolerance. - An operator is provided hooks to know which window id it is on.
- 3. File Input - AbstractFileInputOperator for reading approx. equal sized files. - Out of box supported file formats include txt, Parque and Avro. - FileSplitterInput & AbstractFSBlockReader - reading different sized files - parallelizing read on a single file.
- 4. AbstractFileInputOperator - Scans a folder periodically for new files. - Parses the file for records. - Fault-tolerant and scalable.
- 5. AbstractFileInputOperator : Fault tolerance - A record is not lost. - A record is associated with only one window id irrespective of failures. - If a window is replayed then all the records associated with it will be replayed.
- 6. AbstractFileInputOperator : Fault tolerance cont’d Failure Window 0: committed
- 7. AbstractFileInputOperator : Fault tolerance cont’d Fault tolerance is achieved by - Support from platform - Automatic checkpointing of the state of every operator in the dag. - Automatic restoring a failed operator in another container. - WindowDataManager - Saves incremental state every window. - Helps with replaying windows that were completed by this operator.
- 8. AbstractFileInputOperator : Scalability - Operator partitions read different subset of files. - Files are distributed between partitions based on their hash. - Number of partitions can be changed at run time by changing a property. - For advanced use cases, subclasses can override the directory scanner to customize behavior such as having each partition scan a different directory. - Auto-scalability supported as well in AbstractThroughputFileInputOperator.
- 9. AbstractFileInputOperator : Implementations - LineByLineFileInputOperator in Malhar library - Custom implementation public class CustomFileInputOperator<RECORD> extends AbstractFileInputOperator<RECORD> { public final transient DefaultOutputPort<RECORD> output = new DefaultOutputPort<RECORD>(); @Override protected RECORD readEntity() throws IOException { //read record from input stream RECORD record= inputStream.read(...); return record; } @Override protected void emit(RECORD tuple) { output.emit(tuple); } }
- 10. FileSplitterInput & AbstractFSBlockReader - Task of discovering files and reading them is separated into different logical operators. - File splitter discovers files asynchronously and creates task descriptions- FileBlockMetadata. - Block readers use FileBlockMetada to read a particular block of file. - Fault-tolerant, parallelizes reading on a single file and is auto-scalable.
- 11. FileSplitterInput & AbstractFSBlockReader: Fault tolerance - Platform supports checkpointing state and re-deployment automatically. - FileSplitterInput uses WindowDataManager to replay tuples of completed windows. - AbstractFSBlockReader relies on the upstream buffer-server to replay tuples from a given window. - Buffer-server is a buffer associated with each output port of an operator which holds tuples emitted by that port.
- 12. FileSplitterInput & AbstractFSBlockReader: Fault tolerance cont’d
- 13. FileSplitterInput & AbstractFSBlockReader: Scalability - FileSplitterInput is a simple operator which does not take much resources. - Block reader does the actual work of reading files and is auto-scalable (in beta). - Min and max partitions are configurable. - Frequency of re-partition is controlled by a time interval property. - Scales up/down based on the pending FileBlockMetadata in the input port queue.
- 14. FileSplitterInput & AbstractFSBlockReader: Implementations - FileSplitterInput is concrete. Default behavior can be overridden. - FS Block Readers - FSSliceReader : record is a slice - AbstractFSLineReader and AbstractFSReadAheadLineReader: record is a line - Custom FS Block Reader public class CustomFSBlockReader<RECORD> extends AbstractFSBlockReader<RECORD> { public CustomFSBlockReader() { //initialize reader context this.readerContext = new RecordReaderContext(); } @Override protected RECORD convertToRecord(byte[] bytes) { //convert bytes to RECORD return RECORD.from(bytes); } }
- 15. AbstractFileOutputOperator - Persists data to a single file or multiple files. - Automatic rotation of files (optional) based on - file size - window count - Optional compression and encryption of data. - Fault-tolerant - Scalable as long as different partitions write to different files. Subclasses can achieve this by appending the operator id to the file name.
- 16. AbstractFileOutputOperator : Fault tolerance Record is persisted exactly once. - A record is never missed. - A record is not duplicated. Example application that persists data exactly once: AtomicFileOutputApp
- 17. AbstractFileOutputOperator : Fault tolerance cont’d
- 18. AbstractFileOutputOperator : Fault tolerance cont’d To write exactly once - Assumes idempotent processing - Checkpoint consists of size of each file the operator has written so far. - Truncation of files to the size saved in the restoration checkpoint.
- 19. AbstractFileOutputOperator : Fault tolerance cont’d To avoid dangling lease issues in HDFS - Data is always written to temporary files - Renaming temp files to actual files when a file is finalized, that is, closed for writing. - User can choose when the files get finalized. Rotation handles finalization automatically.
- 20. AbstractFileOutputOperator : Custom Implementation public class CustomFileOutputOperator<RECORD> extends AbstractFileOutputOperator<RECORD> { public CustomFileOutputOperator() { setMaxLength(1024 * 1024); setRotationWindows(600); } @Override protected String getFileName(RECORD tuple) { //file name return tuple.getFileName(); } @Override protected byte[] getBytesForTuple(RECORD tuple) { //bytes from record return tuple.toBytes(); } }
- 21. Acknowledgements - Apex dev team Munagala Ramanath Pramod Immaneni Sasha Parfenov Thomas Weise Timothy Farkas - Meetup organizers Amol Kekre Qybare Pula Ian Gomez - Apache Apex Community
- 22. © 2016 DataTorrent Resources 22 * https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 • Apache Apex website - http://apex.apache.org/ • Subscribe - http://apex.apache.org/community.html • Download - http://apex.apache.org/downloads.html • Twitter - @ApacheApex; Follow - https://twitter.com/apacheapex • Facebook - https://www.facebook.com/ApacheApex/ • Meetup - http://www.meetup.com/topics/apache-apex • Free Enterprise License for Startups - https://www.datatorrent.com/product/startup- accelerator/
- 23. © 2016 DataTorrent We Are Hiring 23 • [email protected] • Developers/Architects • QA Automation Developers • Information Developers • Build and Release • Community Leaders