fix(stepfunctions-tasks): allow passing apiEndpoint as intrinsic function (under feature flag) #32139

MathieuGilbert · 2024-11-14T20:49:46Z

Issue #29925

Reason for this change

The apiEndpoint prop currently only works when it's a string (ie. TaskInput.fromText('some/text')), with the task failing when passed as a reference (ie. TaskInput.fromText(JsonPath.format('some/text/{}', '123')). This is needed to allow for dynamic parts in the path.

Description of changes

Change the ApiEndpoint task parameter to use the JsonPath.format intrinsic function to combine the apiRoot and apiEndpoint props, instead of basic string concatenation.
Update README entry with more complex example.

Description of how you validated changes

A unit test was added to cover passing formatted input.
An integration test was added using fromJsonPathAt for the endpoint.
A test stack was deployed with an API Gateway endpoint with basic auth Connection and was successfully invoked with dynamic payload:

    const httpInvokeTask = new HttpInvoke(this, 'HttpInvoke', {
      apiRoot: api.url,
      apiEndpoint: TaskInput.fromJsonPathAt('$.endpointName'),
      method: TaskInput.fromText('GET'),
      connection,
      outputPath: '$.ResponseBody',
    })

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

codecov · 2024-11-14T21:01:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.38%. Comparing base (e307404) to head (49d80dc).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #32139   +/-   ##
=======================================
  Coverage   82.38%   82.38%           
=======================================
  Files         120      120           
  Lines        6937     6937           
  Branches     1170     1170           
=======================================
  Hits         5715     5715           
  Misses       1119     1119           
  Partials      103      103

Flag	Coverage Δ
suite.unit	`82.38% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
packages/aws-cdk	`∅ <ø> (∅)`
packages/aws-cdk-lib/core	`82.38% <ø> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

MathieuGilbert · 2024-11-15T01:36:17Z

Thanks for your help in getting these changes to green @ePak. I think it was the npm test after running the old tests that got it to pass. Unfortunately I don't get much time to work on this, so it's been dragging on.

aws-cdk-automation · 2025-02-02T00:12:33Z

This PR has been in the MERGE CONFLICTS state for 3 weeks, and looks abandoned. To keep this PR from being closed, please continue work on it. If not, it will automatically be closed in a week.

…mic values in the path. Update README entry with more complex example.

…ded now.

…nt-integ

MathieuGilbert · 2025-02-10T16:09:25Z

I'm not sure what I'm supposed to do about generated snapshot files failing CodeQL checks.

avanderm-pfm · 2025-02-11T14:55:45Z

Workaround with JSONata in the mean time, tested to work with CDK 2.178.1 thanks to #28673 and #32343:

const callApi = HttpInvoke.jsonata(this, 'CallApi', {
  apiRoot: 'https://...',
  apiEndpoint: TaskInput.fromText("{% 'some/path/' & $states.input.someVariable %}"),
  connection,
  integrationPattern: IntegrationPattern.REQUEST_RESPONSE,
  method: TaskInput.fromText('GET'),
});

samson-keung

Left a question.

Additionally, I see that this fix changes all the apiEndpoint values to use State.Format(..). I am wondering if this may push some existing Step Function Definition over the length limit, hence, causing a breaking change to those CDK users. I understand that this may be a very edge case but I would like to get more input from the CDK team to see if we need a feature flag for this fix. I will get back asap. Thank you.

samson-keung · 2025-02-21T00:15:58Z

packages/aws-cdk-lib/aws-stepfunctions-tasks/test/http/invoke.test.ts

@@ -80,7 +84,7 @@ describe('AWS::StepFunctions::Tasks::HttpInvoke', () => {
      },
      End: true,
      Arguments: {
-        ApiEndpoint: 'https://api.example.com/path/to/resource',
+        ApiEndpoint: "States.Format('{}/{}', 'https://api.example.com', 'path/to/resource')",


I see on line 72 above that this state definition uses JSONata. Does States.Format(...) work with JSONata?

I'm not sure about how JSONata works, as that was all added in while I was trying to get the original fix in. Some brief research suggests it would not be compatible.

JsonPath and Jsonata should probably be handled separately in the construct, but that jsonPath static method was already added to just call the existing constructor, so it can't even be used for this fix now. Maybe the existing logic can be reworked without breaking newly-existing functionality.

samson-keung · 2025-02-21T22:54:16Z

Hi @MathieuGilbert, I have consulted other members in the CDK team. We think that this fix requires a feature flag because the current approach of the fix will introduce changes to existing CDK stacks and potentially cause problems. Other than the length limit problem, for stacks without this fix, the ApiEndpoint value would be static in the CFN template, after the fix, the value will be resolved at the Step Function run time. This has the potential of breaking customers who rely on the static value.

To move forward, I suggest implementing the fix behind feature flag. Thank you for looking into this bug and making contributions to the CDK repo.

MathieuGilbert · 2025-02-22T00:53:45Z

Thanks for looking at this @samson-keung. I'm not sure how much more effort I can put towards this. This is my 3rd PR since July trying to get the fix in, and attempting to contribute has been a very discouraging experience overall.

If I can justify the extra hours, I'll try figuring out feature flags, address the issue exposed by that jsonata test you commented on, and spin my wheels a bit longer on integration tests and the CI linter errors in the generated snapshot files.

samson-keung

@MathieuGilbert Understandable. We are currently improving our workflow to better track PRs that have been for a review for a long time.

For now, I will mark this PR as "Request changes" for tracking purposes. If there are team members available, we can pick this up.

Appreciate your effort on this. I am happy to take notes on what is most discouraging you from contributing if you want to share your thoughts. I cannot make guarantees but I will think about what I can do.

Thanks again!

aws-cdk-automation · 2025-03-19T00:17:48Z

This PR has been in the CHANGES REQUESTED state for 3 weeks, and looks abandoned. Note that PRs with failing linting check or builds are not reviewed, please ensure your build is passing

To prevent automatic closure:

Resume work on the PR
OR request an exemption by adding a comment containing 'Exemption Request' with justification e.x "Exemption Request: "
OR request clarification by adding a comment containing 'Clarification Request' with a question e.x "Clarification Request: "

This PR will automatically close in 14 days if no action is taken.

…tpinvoke-format-apiendpoint-integ

…ecific test case for when the flag is off.

…cking the format of the apiRoot and apiEndpoint props, to the tasks queryLanguage. This prevents it from treating it as JSONPath when neither prop is actually a JSONata expression.

Pull request has been modified.

MathieuGilbert · 2025-03-20T22:33:55Z

I am happy to take notes on what is most discouraging you from contributing if you want to share your thoughts. I cannot make guarantees but I will think about what I can do.

@samson-keung I guess a big part was around working with the integration test framework and the snapshots. I was relying more on other tests as examples than the documentation, and then I kept getting stuck in CI with failing snapshot tests with no clear way forward. I'm still not clear whether it's even possible to update an integration test, or we have to always create a new one.

…tpinvoke-format-apiendpoint-integ

mergify · 2025-05-28T12:27:50Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

Abogical · 2025-06-26T15:44:51Z

@Mergifyio requeue

mergify · 2025-06-26T15:44:58Z

requeue

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Abogical · 2025-06-26T16:19:44Z

@Mergifyio rebase

mergify · 2025-06-26T16:19:51Z

rebase

☑️ Nothing to do, the required conditions are not met

-conflict [📌 rebase requirement]
-closed [📌 rebase requirement]
queue-position = -1 [📌 rebase requirement]
any of:
- #commits > 1 [📌 rebase requirement]
- #commits-behind > 0 [📌 rebase requirement]
- -linear-history [📌 rebase requirement]

Abogical

Hi @MathieuGilbert ! Apologies for the delay, it seems @Mergifyio has not merged your PR. Not sure why it silently failed. Currently, the PR has merge conflicts that will need to fixed in order to get it merged.

Abogical · 2025-06-26T16:27:25Z

@Mergifyio update

mergify · 2025-06-26T16:27:35Z

update

☑️ Nothing to do, the required conditions are not met

-conflict [📌 update requirement]
#commits-behind > 0 [📌 update requirement]
-closed [📌 update requirement]
queue-position = -1 [📌 update requirement]

…tpinvoke-format-apiendpoint-integ

Pull request has been modified.

…lts" now.

aws-cdk-automation · 2025-07-04T22:58:07Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 347d517
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

github-actions bot added the beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK label Nov 14, 2024

aws-cdk-automation requested a review from a team November 14, 2024 20:49

github-actions bot added bug This issue is a bug. effort/small Small work item – less than a day of effort p2 labels Nov 14, 2024

MathieuGilbert force-pushed the awsstepfunctions-httpinvoke-format-apiendpoint-integ branch from de76858 to b278145 Compare November 15, 2024 00:19

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Nov 15, 2024

github-actions bot mentioned this pull request Dec 1, 2024

Monthly PR metrics report #32340

Closed

MathieuGilbert added 3 commits February 6, 2025 15:57

Fix concatenation of apiRoot and apiEndpoint props, to allow for dyna…

12be16c

…mic values in the path. Update README entry with more complex example.

Add new test for the JsonPath.

15b8a54

Add directive indicating which stack to test, which apparently is nee…

2d3c186

…ded now.

MathieuGilbert force-pushed the awsstepfunctions-httpinvoke-format-apiendpoint-integ branch from c512f0e to 2d3c186 Compare February 7, 2025 01:39

MathieuGilbert added 2 commits February 7, 2025 07:46

Merge branch 'main' into awsstepfunctions-httpinvoke-format-apiendpoi…

d3126e3

…nt-integ

Fix new unit test.

4cfa4a1

samson-keung self-assigned this Feb 19, 2025

samson-keung reviewed Feb 21, 2025

View reviewed changes

samson-keung previously requested changes Feb 25, 2025

View reviewed changes

aws-cdk-automation removed the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Feb 25, 2025

godwingrs22 unassigned samson-keung Feb 27, 2025

MathieuGilbert added 3 commits March 20, 2025 10:54

Merge remote-tracking branch 'upstream/main' into awsstepfunctions-ht…

bc14d89

…tpinvoke-format-apiendpoint-integ

Add feature flag enabling dynamic JSONPath apiEndpoint.

ed021b0

Only apply dynamic apiEndpoint when feature flag is enabled. Add a sp…

6e50370

…ecific test case for when the flag is off.

Change the condition for treating the apiEndpoint as JSONata from che…

ded392f

…cking the format of the apiRoot and apiEndpoint props, to the tasks queryLanguage. This prevents it from treating it as JSONPath when neither prop is actually a JSONata expression.

MathieuGilbert force-pushed the awsstepfunctions-httpinvoke-format-apiendpoint-integ branch from a42d5ce to ded392f Compare March 20, 2025 22:22

MathieuGilbert changed the title ~~fix(stepfunctions-tasks): allow passing apiEndpoint as intrinsic function~~ fix(stepfunctions-tasks): allow passing apiEndpoint as intrinsic function (under feature flag) Mar 20, 2025

MathieuGilbert added 2 commits March 20, 2025 21:16

Add feature flag to features test.

d0fdf8f

Merge remote-tracking branch 'upstream/main' into awsstepfunctions-ht…

49d80dc

…tpinvoke-format-apiendpoint-integ

aws-cdk-automation added the pr/needs-community-review This PR needs a review from a Trusted Community Member or Core Team Member. label Mar 21, 2025

Abogical self-assigned this May 28, 2025

Abogical approved these changes May 28, 2025

View reviewed changes

Abogical previously requested changes Jun 26, 2025

View reviewed changes

MathieuGilbert added 2 commits July 4, 2025 14:31

revert conflicting files.

d135331

Merge remote-tracking branch 'upstream/main' into awsstepfunctions-ht…

e9e182a

…tpinvoke-format-apiendpoint-integ

MathieuGilbert requested a deployment to test-pipeline July 4, 2025 21:55 — with GitHub Actions Waiting

Feature flags seem to use "unconfiguredBehavesLike" instead of "defau…

347d517

…lts" now.

MathieuGilbert requested a deployment to test-pipeline July 4, 2025 22:38 — with GitHub Actions Waiting

fix(stepfunctions-tasks): allow passing apiEndpoint as intrinsic function (under feature flag) #32139

Are you sure you want to change the base?

fix(stepfunctions-tasks): allow passing apiEndpoint as intrinsic function (under feature flag) #32139

Conversation

MathieuGilbert commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue #29925

Reason for this change

Description of changes

Description of how you validated changes

Checklist

Uh oh!

codecov bot commented Nov 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

MathieuGilbert commented Nov 15, 2024

Uh oh!

aws-cdk-automation commented Feb 2, 2025

Uh oh!

MathieuGilbert commented Feb 10, 2025

Uh oh!

avanderm-pfm commented Feb 11, 2025

Uh oh!

samson-keung left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samson-keung Feb 21, 2025

Choose a reason for hiding this comment

Uh oh!

MathieuGilbert Feb 22, 2025

Choose a reason for hiding this comment

Uh oh!

samson-keung commented Feb 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MathieuGilbert commented Feb 22, 2025

Uh oh!

samson-keung left a comment

Choose a reason for hiding this comment

Uh oh!

aws-cdk-automation commented Mar 19, 2025

Uh oh!

MathieuGilbert commented Mar 20, 2025

Uh oh!

mergify bot commented May 28, 2025

Uh oh!

Abogical commented Jun 26, 2025

Uh oh!

mergify bot commented Jun 26, 2025

✅ The queue state of this pull request has been cleaned. It can be re-embarked automatically

Uh oh!

Abogical commented Jun 26, 2025

Uh oh!

mergify bot commented Jun 26, 2025 • edited by Abogical Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

☑️ Nothing to do, the required conditions are not met

Uh oh!

Abogical left a comment

Choose a reason for hiding this comment

Uh oh!

Abogical commented Jun 26, 2025

Uh oh!

mergify bot commented Jun 26, 2025

☑️ Nothing to do, the required conditions are not met

Uh oh!

aws-cdk-automation commented Jul 4, 2025

AWS CodeBuild CI Report

Uh oh!

Uh oh!

MathieuGilbert commented Nov 14, 2024 •

edited

Loading

codecov bot commented Nov 14, 2024 •

edited

Loading

samson-keung left a comment •

edited

Loading

samson-keung commented Feb 21, 2025 •

edited

Loading

mergify bot commented Jun 26, 2025 •

edited by Abogical

Loading