Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18477 Über-jira: S3A Hadoop 3.3.9-3.4.1 features
  3. HADOOP-18679

Add BulkDelete API for paged delete of files and objects

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.5
    • 3.4.1
    • fs/s3

    Description

      iceberg and hbase could benefit from being able to give a list of individual files to delete -files which may be scattered round the bucket for better read peformance.

      Add some new optional interface for an object store which allows a caller to submit a list of paths to files to delete, where
      the expectation is

      • if a path is a file: delete
      • if a path is a dir, outcome undefined
        For s3 that'd let us build these into DeleteRequest objects, and submit, without any probes first.

      Cherrypicking

      when cherrypicking, you must include

      Attachments

        Issue Links

          causes

          Sub-task - The sub-task of the issue HADOOP-19184 TestStagingCommitter.testJobCommitFailure failing

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-19188 TestHarFileSystem and TestFilterFileSystem failing after bulk delete API added

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-19196 Bulk delete api doesn't take the path to delete as the base path

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue HADOOP-17881 S3A DeleteOperation to parallelize POSTing of bulk deletes

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-19131 WrappedIO to export modern filesystem/statistics APIs in a reflection friendly form

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-19191 Batch APIs for delete

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue SPARK-49550 Upgrade Hadoop to 3.4.1

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-18722 Optimise S3A delete objects when multiObjectDelete is disabled

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-19254 Implement bulk delete command as hadoop fs command operation

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-19203 WrappedIO BulkDelete API to raise IOEs as UncheckedIOExceptions

          • Major - Major loss of function.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-18420 Optimise S3A’s recursive delete to drop successful S3 keys on retry of S3 DeleteObjects

          • Major - Major loss of function.
          • Open
          links to

          Web Link GitHub Pull Request #5993

          Web Link GitHub Pull Request #6494

          Web Link GitHub Pull Request #6726

          Web Link GitHub Pull Request #6738

          Web Link GitHub Pull Request #6854

          Web Link GitHub Pull Request #10233

          Activity

            People

              mukund-thakur Mukund Thakur
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: