Description
Describe the bug
If I create a log bucket. Then a content bucket, using log bucket for server access logs, when I do a "cdk destroy" the access logs are not removed and the stack fails to destroy.
Expected Behavior
The destroy to complete successfully! (The log bucket and content to be removed.)
Current Behavior
Cloudwatch logs for the delete objects Lambda suggests it was successful :
START RequestId: 72426a95-6f49-4d26-9642-2e15450b205a Version: $LATEST
2022-08-26T10:14:21.449Z 72426a95-6f49-4d26-9642-2e15450b205a INFO
{
"RequestType": "Delete",
"ServiceToken": "arn:aws:lambda:eu-west-2:123492961234:function:CdkBucketStack-CustomS3AutoDeleteObjectsCustomReso-kArk0g4I9Il2",
"ResponseURL": "...",
"StackId": "arn:aws:cloudformation:eu-west-2:123492961234:stack/CdkBucketStack/0531d0e0-2522-11ed-abbb-0aa24eb467d4",
"RequestId": "b81d7b6f-c28c-455f-83cf-8cf4be357191",
"LogicalResourceId": "LogBucketAutoDeleteObjectsCustomResource7762F42C",
"PhysicalResourceId": "f9dbbbc0-641f-4026-a15e-df345405a307",
"ResourceType": "Custom::S3AutoDeleteObjects",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:eu-west-2:123492961234:function:CdkBucketStack-CustomS3AutoDeleteObjectsCustomReso-kArk0g4I9Il2",
"BucketName": "cdkbucketstack-logbucketcc3b17e8-79kbay52r42q"
}
}
2022-08-26T10:14:27.491Z 72426a95-6f49-4d26-9642-2e15450b205a INFO submit response to cloudformation {
Status: 'SUCCESS',
Reason: 'SUCCESS',
StackId: 'arn:aws:cloudformation:eu-west-2:123492961234:stack/CdkBucketStack/0531d0e0-2522-11ed-abbb-0aa24eb467d4',
RequestId: 'b81d7b6f-c28c-455f-83cf-8cf4be357191',
PhysicalResourceId: 'f9dbbbc0-641f-4026-a15e-df345405a307',
LogicalResourceId: 'LogBucketAutoDeleteObjectsCustomResource7762F42C',
NoEcho: undefined,
Data: undefined
}
END RequestId: 72426a95-6f49-4d26-9642-2e15450b205a
REPORT RequestId: 72426a95-6f49-4d26-9642-2e15450b205a Duration: 6312.29 ms Billed Duration: 6313 ms Memory Size: 128 MB Max Memory Used: 81 MB Init Duration: 174.79 ms
But the result of the cdk destroy is a fail because there is content :
CdkBucketStack: destroying...
11:15:21 | DELETE_FAILED | AWS::S3::Bucket | LogBucketCC3B17E8
The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: HJVZC3J1DP4N2QSM; S3 Extended Request ID: /OcCq7vdx6Yxt6bShdNQhgr8fUI3/pG1gmCuC4cu7dBm3juB+Sa9U+FM+O41vk+FVCV8Xwk49YA=; Proxy: null)
❌ CdkBucketStack: destroy failed Error: The stack named CdkBucketStack is in a failed state. You may need to delete it from the AWS console : DELETE_FAILED (The following resource(s) failed to delete: [LogBucketCC3B17E8]. ): The bucket you tried to delete is not empty (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID: HJVZC3J1DP4N2QSM; S3 Extended Request ID: /OcCq7vdx6Yxt6bShdNQhgr8fUI3/pG1gmCuC4cu7dBm3juB+Sa9U+FM+O41vk+FVCV8Xwk49YA=; Proxy: null)
Reproduction Steps
Deploy the template below. Upload your current directory to "bucket" and wait up to an hour until there are some logs in logBucket. Then run the destroy.
(Yes the code is a bit janky, I did a lazy copy/paste and removed some logic that isn't relevant etc...)
import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as iam from 'aws-cdk-lib/aws-iam';
export class CdkBucketStack extends cdk.Stack {
public readonly Bucket: cdk.CfnOutput;
public readonly logBucket: cdk.CfnOutput;
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create a bucket for all the different logs.
let logBucketProps: any = {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
accessControl: s3.BucketAccessControl.LOG_DELIVERY_WRITE,
};
logBucketProps["enforceSSL"] = true;
logBucketProps["encryption"] = s3.BucketEncryption.S3_MANAGED;
logBucketProps["blockPublicAccess"] = s3.BlockPublicAccess.BLOCK_ALL;
const logBucket = new s3.Bucket(this, 'LogBucket', logBucketProps);
logBucket.addToResourcePolicy(
new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
principals: [new iam.ServicePrincipal('logging.s3.amazonaws.com')],
actions: ['s3:PutObject'],
resources: [`${logBucket.bucketArn}/*`],
}),
);
let BucketProps: any = {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
};
BucketProps["enforceSSL"] = true;
BucketProps["encryption"] = s3.BucketEncryption.S3_MANAGED;
BucketProps["serverAccessLogsPrefix"] = "output-access-logs/";
BucketProps["serverAccessLogsBucket"] = logBucket;
BucketProps["blockPublicAccess"] = s3.BlockPublicAccess.BLOCK_ALL;
const bucket = new s3.Bucket(this, 'OutputBucket', BucketProps);
this.Bucket = new cdk.CfnOutput(this, 'S3Bucket', {
value: bucket.bucketName,
description: "The bucket",
});
this.logBucket = new cdk.CfnOutput(this, 'logS3Bucket', {
value: logBucket.bucketName,
description: "The log bucket",
});
}
}
Possible Solution
I imagine the "delete content" lambda is not checking some error condition it isn't expecting and reporting success when it hasn't succeeded. The Lambda does get deleted and the code in the Lambda editor is minified so I couldn't easily retry or debug it.
Additional Information/Context
The logic I removed in the code example avoids deleting the content in our "production" deployments. But in dev, we have no need to keep the access logs. So if someone was thinking "but it's log data, we can't delete that", please : do delete any data. Or at least provide a "actuallyAutoDeleteAllObjects: true" option.
CDK CLI Version
2.38.1 (build a5ced21)
Framework Version
No response
Node.js Version
Node.js v18.8.0
OS
OSX
Language
Typescript
Language Version
No response
Other information
No response