(@aws-cdk/aws-lambda-nodejs): NodejsFunction - Enable parallel building

[AllanOricil]

Describe the feature

Currently CDK builds one Function/NodeJsFunction at time but I believe it could try to build all lambda function on a given stack in parallel. This would reduce the amount of time the build phase takes, and make a better use of idle resources.

Use Case

I have an API which has one lambda per method. I did it this way because each method has different configs. This API has like 100 lambdas and it is a single stack. It takes a huge amount of time to build the entire stack because my lambdas are all built sequentially. It takes a lot of time to validate the api and run integration tests because of the build phase.

Proposed Solution

• synth the stack
• read its template to determine all lambdas that need to be built
• spawn N docker containers (if forceDocker = true) or processes to build lambdas in parallel. Allow N (min) to be configurable as an env variable. N can be defined based on how many resources are available for docker in the machine used to build.
• continue with the cdk flow

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.114.0

Environment details (OS name and version, etc.)

MacOS Sonoma Developer Beta 3

[tmokmss]

Maybe we can allow to configure asset-build concurrency parameter here?

aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts

Lines 390 to 394 in 1e69cc6

	const graphConcurrency: Concurrency = {
	'stack': concurrency,
	'asset-build': 1, // This will be CPU-bound/memory bound, mostly matters for Docker builds
	'asset-publish': (options.assetParallelism ?? true) ? 8 : 1, // This will be I/O-bound, 8 in parallel seems reasonable
	};

Although it's true that build process is cpu-bound, we can utilize multiple cores even with Node.js since usually we are just spawning external processes like docker or esbuild to build those assets.

That said, when I tested only to change the asset-build concurrency parameter it didn't work (builds seemed executed serially). I think we have to understand how this parameter is used.

[AllanOricil]

@tmokmss I dont get this part of the workGraph code

https://github.com/aws/aws-cdk/blob/main/packages%2Faws-cdk%2Flib%2Futil%2Fwork-graph.ts#L150-L173

Shouldnt startOne be called with n nodes to enable parallel building?

There seem to exist a parallelPromises method which isn't being used on any of the asset types. I see a single promise being executed at a time inside the switch

[tmokmss]

So I found that asset bundling happens in a totally different part: the constructor of AssetStaging class in the below code.

aws-cdk/packages/aws-cdk-lib/aws-s3-assets/lib/asset.ts

Lines 143 to 148 in 1e69cc6

	const staging = new cdk.AssetStaging(this, 'Stage', {
	...props,
	sourcePath: path.resolve(props.path),
	follow: props.followSymlinks ?? toSymlinkFollow(props.follow),
	assetHash: props.assetHash ?? props.sourceHash,
	});

And afaik CDK runs all the class constructors serially (they are not async functions after all), I guess it's currently difficult to naively parallelize them.

Shouldnt startOne be called with n nodes to enable parallel building

in startOne method an async function fn is called without await, so it will run in parallel if there're multiple nodes and they don't exceed the max concurrency. But this function does not actually process asset bundling :(

[AllanOricil]

Deploys happen in parallel. Just the builds that don't. It would be a really good improvement if you could enable parallelization.

[martinjlowm]

I was just brainstorming a similar case. We are introducing Nix throughout our projects and one possible way of supporting parallel builds (I believe) would be a combination of Lazy and Aspects.

An idea would be to support Lazy for the code property on the Lambda construct and traverse the stack tree looking for lazy-evaluated tokens that are assigned to the code property and concatenate all the build commands into a single one, e.g. (pseudo-code):

const entry = new Lambda.Function(this, 'func', {
    runtime: lambda.Runtime.NODEJS_18_X,
    handler: 'main.handler',
    code: lambda.Code.fromAsset(
        cdk.Lazy.any({
            produce: () => {
                return '.#bundleMainHandler';
            },
        }),
    ),
    ...
});

function mapInstallableToAssetPath(installable: string, context: IResolveContext) {
  // scope is expected to be a stack in this case
  return context.scope.assets[installable];
}

class NixBundler implements IAspect {
  installables: string[] = [];
  stack: Stack
  
  public visit(node: IConstruct): void {
    if (node instanceof Stack) {
      node.assets = {};
      this.stack = node;
    }

    if (node instanceof CfnFunction) {
      if (Tokenization.isResolvable(node.code)) {
        this.installables.push(node.code.resolve({ registerPostProcessor: mapInstallableToAssetPath }));
      }
    }
  }

  build() {
    cp.execSync(`nix build ${this.installables.join(' ')}`);
    const outputs = fs.readDirSync('result/');
    for (const [installable, output] of zip(this.installables, outputs)) {
      this.stack.assets[installable] = output;
    }
  }
}

const stack = new Stack(...);
const bundler = new NixBundler();
Aspects.of(stack).add(bundler);
bundler.build();

So the idea is to,

1. store all the "bundle references" per Lambda function
2. collect all of them from an Aspect, NixBundler.
3. some finalization place I am yet to find to concatenate the bundle references (installables) and run nix build ... synchronously where Nix will handle the parallelization
4. Store the emitted bundles/artifacts paths in a map-like structure, so the installable can be used to lookup that path.

In terms of using Nix, we're also utilizing a binary cache for our artifacts and we're currently using https://www.cachix.org/ - however, S3, is also an option https://nixos.org/manual/nix/stable/store/types/s3-binary-cache-store - in which case, we may as well resolve S3 properties instead of uploading the artifact twice (Nix uploads it and CDK will try to do the same).

I noticed from
https://github.com/aws/aws-cdk/blob/main/packages/aws-cdk-lib/core/lib/resolvable.ts#L166-L167

that the postProcessor may be used as a final step to do the installable -> path transformation.

In the case of this issue, the concept would be the same if the finalization step would be asynchronous, then you could do the execution part as fs.exec(...) and await all of them in a Promise.all.

The aspect you can then attach whereever you wish and change the behavior as you see fit for different stacks.

I'm not at all an expert in the CDK internals, but on a high level I believe this should be doable with the current setup if only asset references would support lazy-evaluated tokens.

The big question apart from lazy support is there order of execution in these steps and there's a lot of error handling to consider.

I can see the lazy support was requested here as well: #28732

[TheDiggster13]

We are seeing a similar issue with dotnet lambda functions. We have about 30 in a project, and it's meant the synth step of our pipeline now takes 10-15 minutes. If we could parallelise these it would be a massive benefit.

[AllanOricil]

I noticed an issue when using the NodeJsFunction construct. The more builds you do in the same host, without deleting cdk.out, the longer the build gets. After purging cdk.out my local deploys were so much faster. Would you know why?