Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.1
Description
HADOOP-19072 shows we want to add more optimisations than that of HADOOP-18930.
- Extending the new optimisations to the existing option is brittle
- Adding explicit options for each feature gets complext fast.
Proposed
- A new class S3APerformanceFlags keeps all the flags
- it build this from a string[] of values, which can be extracted from getConf(),
- and it can also support a "*" option to mean "everything"
- this class can also be handed off to hasPathCapability() and do the right thing.
Proposed optimisations
- create file (we will hook up
HADOOP-18930) - mkdir (
HADOOP-19072) - delete (probe for parent path)
- rename (probe for source path)
We could think of more, with different names, later.
The goal is make it possible to strip out every HTTP request we do for safety/posix compliance, so applications have the option of turning off what they don't need.
Attachments
Issue Links
- blocks
-
HADOOP-19072 S3A: expand optimisations on stores with "fs.s3a.performance.flags" for mkdir
-
- Resolved
-
- relates to
-
HADOOP-18544 S3A: add option to disable probe for dir marker recreation on delete/rename.
-
- Resolved
-
-
HADOOP-18930 S3A: make fs.s3a.create.performance an option you can set for the entire bucket
-
- Resolved
-
- links to