(pipelines): view `cdk diff` output in CodePipeline

[blimmer]

It would be awesome to be able to see the results of cdk diff in the CodePipeline UI along with a manual approval step before infrastructure changes are deployed by CDK Pipelines.

Use Case

CDK Pipelines are great. I love that there's an officially supported way to deploy CDK from a CI-environment vs. people's local machines. In fact, I created a system for a client that does this with Circle CI before pipelines were available. Eventually, it would be great to move clients over to the officially supported pipelines module from the custom solution.

The one thing that's missing from the current solution, IMO, is the ability to view the cdk diff output in CodePipeline before mutating infrastructure.

Imagine that I'm a developer working on a CDK-managed RDS cluster. Some CloudFormation properties, when updated, require replacement of the Database instance and cannot be performed in-place. Imagine a developer gets their PR approved and merges without running a cdk diff locally, first. Suddenly, they could be in for a big surprise when the RDS cluster is being recreated by CloudFormation.

This situation could be prevented if the developer would have checked out the cdk diff before mutating the infrastructure. The red REQUIRES REPLACEMENT message would have tipped them off that they should look into other ways of accomplishing their update.

This is just one example. It's always a good idea to check out the cdk diff before updating your infrastructure.

Proposed Solution

One way to accomplish this would be to create a CodePipeline stage for running cdk diff. It could use CodeBuild to execute cdk diff, just like it does to run cdk synth. Then, a manual approval step could optionally be added, linking to the CodeBuild output with the cdk diff. If the diff looks OK, the user approves and the applicationStage is deployed. If something looks wrong, the user could reject and fix the problem.

Other

The CodePipeline would look something like this:

Source
|
Build
|
Diff
|-- optional manual approval
|
Update Pipeline
|
Application Stage

• 👋 I may be able to implement this feature request
• ⚠️ This feature might incur a breaking change

---

This is a 🚀 Feature Request

[rix0rrr]

We totally want to do something like this eventually. The problem is, where do we put this diff, and what are we diffing? These are not easy things to answer and any answer we pick will leave some folks unhappy.

• Keep in mind that a pipeline can have N stages in them (could be 1, 2, 5, 20).
• Keep in mind that a pipeline can have multiple changes going through it at once.

So:

Do we put an approval gate before every stage? Most accurate but kinda defeats the purpose of CI/CD. People WILL forget to hit that button, your pipeline WILL stall.

Do we put a single approval gate at the start? But then what are we diffing against what? Just the first stage? All stages? If all stages, what about changes that are still percolating through the pipeline, the diff will look wrong. What if looking up a diff for a single stage fails (for whatever reason). Maybe it's the last stage in your pipeline of 20. Are we now not going to deploy anywhere?

And then there's the question of how we're going to publish a UI for this. CodePipeline doesn't really give us a way to host a good UI so we're going to have to do that somewhere else, maybe a serverless website. Now we need to do auth to that website. What are we going to auth with?

---

All this is to say, it's definitely on the radar but there are some questions that need answering here, and they're all pretty tough.

If you have any clever ideas, I'd love to hear them!

[simon-dk]

@blimmer @rix0rrr
Stumpled upon this request today as I have been working with pipelines as well. Maybe I am a bit daft, but it does not seem possible to run "cdk diff" locally either, or is there something I am completely missing?

Cdk diff only shows me the diff for the pipeline, not for each stack (if you know a way to show the diffs for each stack, please let me know!), but I agree with @blimmer that a way to add a "diff stage" in pipelines would be very useful for teams.

[blimmer]

@Simon-SDK - to see the diffs for your stages, you have to run something like:

> cdk diff -a cdk.out/assembly-MyPipeline-MyStage

It'll be different based on the names in your project. This is also reported in #8676, which is marked for GA.

[mrpackethead]

Just adding my 5c to this problem. Its a wee bit of a stumbling block for one of my clients, who woudl really like to see whats going to change. before they commit to it.

cdk diff provides the information thats needed.

I have a couple of use cases.. The simple one is what i'd call an 'infrastructure' change... An example being that I'd typically deploy a VPC via cdk for an account.. It doesnt have a 'dev/test/prod' type set up.. So in effect its only one stage, but my stages could well have multiple stacks across accounts.

A second use case is deploying some kind of applicaiton that may have a dev/test/prod.. Those have multiple stages.. and and a few different ways of being progressed through the pipeline.. Some will have manual approvals, and some will need to pass some automated testing..

I agree with @rix0rrr comment above,.. 'These are not easy things to answer and any answer we pick will leave some folks unhappy.'

My own needs require several different approaches, depending on what i'm doing.. Whatever the approach is, its got to be very flexible to cover all the use-cases....

The first problem to resolve might just be a way to to create 'external' ( to the pipeline ) approvals.. something to replace the manual approval.

[mrpackethead]

Did some more thinking since the last comment. :-) ( and sleeping zzzzzz ). It probably helps.
Firstly i'm not sure that this is a problem that can be universally solved, for everyone with a L3 type construct. I think we've learned that with ec2.VPC, that there are times that such highly opinionated constructs actually lead to more problems than they solve. ( thats not a criticism, thats just learning we are all making. ).

I'm going to have a go and trying to resolve this for my own situation..

I've been using Manual Approvals for some time. However what i did'nt really appreicate is that 'manual' approvals dont' need to be 'manual' at all. ( and that leads to the uestion why are they called 'Manual Approvals'.. should they just be 'Approval' ? ).

Currently i'm doing this, where i need to ( sorry I am a python guy, but hopefully its reasonably clear to the 74% of CDK users who are not )

this_stage.add_manual_approval_action(
    action_name = 'Release_this_Stage',
    run_order = 1
)

This just adds a simple 'push the button'.. While it might stop an 'accidental' deployment if you accepted a PR that you really did'tn mean to, it does not provide any useful info..

So, I'm thinking i need some smarts. It turns out that theres a an 'add_actions' method for stages, which lets me add codepipeline actions. Lets use a 'ManualApprovalAction'.. This will send an SNS message..

this_stage.add_actions(
     codepipeline_actions.ManualApprovalAction(
         action_name = 'CreateDiffForApproval',
         notification_topic = creatediff    # this is an sns topic. 
) 

Receving the sns message can be a lambda.. The lambdas function essentially would be to

(a) Create a diff using cdk diff.. The result would likely be placed in an S3 bucket.
(b) notify the approvers that a change is needing approval. This coudl be email/slack/teams what ever... They would provide a link to an API...

the API ( and a bit of web-based smarts or maybe it will just be CLI) ( which would need some authentication ) would display the diff, and then provide an approve button... The approve button triggers another lambda which then uses the PutApprovalResult API call ( codepipeline ) to 'approve' the stage and it can move on.

Depending on the need, you could set up differnet approval rules.. For example, if you are making some kind of changes that change some security senstive thigns, you might need a 'security team' approval.... This of course really gets very specific to your situation.

This will take a bit of experimentation, but it looks fesible..

Credit I got a bit of inspiration from this blog, which is not cdk specific, but describes the codepipeline "manualApproval'
https://www.eliasbrange.dev/posts/aws-auto-approve-codepipeline/

[polothy]

The way I went about doing this is using GitHub actions because I wanted to see the CloudFormation changes at the PR, before merging.

See https://github.com/blackboard-innersource/gh-action-cdk-diff (use at your own risk, I'll freely modify this to suite our needs).

It's all bash scripting stuff, so you could easily do the same via CodeBuild on PR (or any other build system).

The tl;dr of the action:

• Run synth on the main branch.
• Run synth on the PR branch.
• Diff the two cdk.out directories.
• Post the diff to a comment on the PR (the comment uses GH collapsable sections and diff syntax highlighting).

I realize this isn't the same as doing cdk diff against the deployed CloudFormation stack, but it seems to do the trick for us.

[ahammond]

I think that having an option to "generate the changeset then wait for manual approval" would be Good Enough. Ideally, I'd want to set it at the wave level:

So I'd be able to write something like

const manualApprovalAfterCreateChangeSet = new pipelines.ManualApprovalStep('Approve Changeset', {...});
const myWave = myPipeline.addWave('MyWaveName', { manualApprovalAfterCreateChangeSet });
// add stages to wave

And... almost certainly this should be part of an interface which is common to wave and stage.