Skip to content

Lambda Go SDK: upload large inputProps to S3 - #8987

Merged
JonnyBurger merged 9 commits into
remotion-dev:mainfrom
iliya-vidrush:lambda-go-s3-upload
Jul 13, 2026
Merged

Lambda Go SDK: upload large inputProps to S3#8987
JonnyBurger merged 9 commits into
remotion-dev:mainfrom
iliya-vidrush:lambda-go-s3-upload

Conversation

@iliya-vidrush

Copy link
Copy Markdown
Contributor

Stacked on #8986 (merge that first). Until it lands, this PR's diff also shows the cleanup commit; it will collapse to just the S3 changes once #8986 is merged.

Motivation

The Go SDK currently rejects inputProps that exceed the inline Lambda payload limit, while the JS and PHP clients transparently upload oversized props to S3 and pass a reference. This brings the Go SDK in line with them.

Changes

  • Add s3.go with the bucket and upload helpers:
    • getOrCreateBucket: lists remotionlambda- buckets in the region, errors if more than one exists (linking to the multiple-buckets docs), otherwise reuses the single bucket or creates one (with a LocationConstraint outside us-east-1).
    • SHA256 object hashing, the remotionlambda-<region>-<hash> bucket naming, the input-props/<hash>.json key, and a private application/json PutObject.
  • Rewrite serializeInputProps to return a SerializedInputProps of type payload when the props fit inline, or bucket-url with the hash and bucket name after an upload. The inline threshold uses the same margin as the reference clients (200KB video, 5MB still).
  • Honor forceBucketName as the user-specified bucket and forcePathStyle on the S3 client.
  • Validate the render options before serializing, so an invalid request never triggers an S3 upload or bucket creation.

Testing

  • Added tests for inline serialization, the upload threshold, and the bucket and key naming (all AWS-free).
  • go build ./..., go vet ./..., go test ./... and gofmt -l pass. The S3 upload path itself requires live AWS credentials and was not exercised in CI.

- Return the serialization error from constructRenderInternals instead of
  calling log.Fatal, so callers stay in control of process exit.
- Wrap invocation, marshalling and response-parsing errors with %w and
  include the function name for context.
- Type the internal Crf, ColorSpace and GopSize fields as pointers instead
  of interface{} so an unset value serializes to null without a typed
  zero-value.
- Fix the example progress output that printed RenderId under the
  "bucketName" label and dropped the redundant print.
- Remove the stray semicolon and add the missing trailing newline in
  version.go.
Previously the Go SDK rejected inputProps that exceeded the inline Lambda
payload limit. Match the behavior of the JS and PHP clients by uploading
oversized props to S3 and passing a bucket-url reference instead.

- Add s3.go with the bucket resolution and upload helpers: getOrCreateBucket
  (list remotionlambda- buckets in the region, error on multiple, create one
  otherwise), SHA256 object hashing, the remotionlambda-<region>-<hash> bucket
  naming, the input-props/<hash>.json key and a private application/json
  PutObject.
- Rewrite serializeInputProps to return a SerializedInputProps of type
  "payload" when the props fit inline, or "bucket-url" with the hash and
  bucket name after an upload. The inline threshold uses the same margin as
  the reference clients (200KB video, 5MB still).
- Honor forceBucketName as the user-specified bucket and forcePathStyle on
  the S3 client.
- Validate the render options before serializing so an invalid request never
  triggers an S3 upload or bucket creation.
- Add tests for inline serialization, the upload threshold and the bucket and
  key naming.
@vercel

vercel Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
bugs Ready Ready Preview, Comment Jul 13, 2026 3:14am
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
remotion Skipped Skipped Jul 13, 2026 3:14am

Request Review

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

The new S3 upload path will fail on AWS accounts using the default S3 Object Ownership setting (ACLs disabled), because the bucket is created without enabling ACLs and the object upload requests a private ACL.

Reviewed changes — added S3 bucket discovery/creation and oversized inputProps upload to the Go Lambda SDK, plus some error-handling cleanup.

  • Add s3.go with bucket discovery, creation, and private object upload helpers.
  • Rewrite serializeInputProps to return a SerializedInputProps payload and upload to S3 when the inline limit is exceeded.
  • Move inputProps serialization after validation and remove the previous log.Fatal on serialization errors.
  • Clean up pointer serialization for Crf, ColorSpace, and GopSize, and fix the version.go package statement.
  • Fix the example output in lambda-go-example/main.go so it prints BucketName and RenderId.

Note: 1 inline comment(s) dropped because they did not anchor to lines inside the PR diff:

  • packages/lambda-go/lambda_go_sdk_test.go:22 (RIGHT) — line 22 (RIGHT) is not inside a diff hunk

Pullfrog  | Fix all ➔Fix 👍s ➔View workflow run | Using Kimi K2 (free via Pullfrog for OSS) | 𝕏

Comment thread packages/lambda-go/s3.go Outdated
Comment thread packages/lambda-go/s3.go Outdated
Comment thread packages/lambda-go/s3.go
Comment thread packages/lambda-go/s3.go Outdated
Comment thread packages/lambda-go/utils.go Outdated
Comment thread packages/lambda-go/utils.go
Comment thread packages/lambda-go/models.go
Comment thread packages/lambda-go/s3.go Outdated
Comment thread packages/lambda-go/lambda_go_sdk_test.go Outdated

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No new issues in the latest commits — the delta only resolves merge-time version/dependency drift.

Changes since last review — reviewed the merge commit from 06ba5fd to f32ea1e.

  • Version alignment: package.json and go.mod versions were bumped from 4.0.486 to 4.0.487, matching version.go.
  • Dependency refresh: go.mod/go.sum updated to aws-sdk-go v1.55.8, Go directive 1.26, toolchain go1.26.5, and refreshed indirect dependencies.
  • Source code: the feature code is unchanged; go test, go vet, and gofmt still pass.

Note: the feedback from the previous review still needs to be addressed.

Pullfrog  | Fix it ➔View workflow run | Using Kimi K2 (free via Pullfrog for OSS) | 𝕏

Make private uploads compatible with ACL-disabled buckets, propagate bucket location errors, align payload thresholds and warning rounding, preserve the exported PayloadData name, use unbiased bucket suffixes, and cover upload boundaries.
Add focused coverage for ACL-free uploads, bucket-location error handling, warning rounding, the PayloadData compatibility alias, wrapped upload failures, and random hash formatting.
Replace AWS scalar pointer helpers in the new S3 code with Go 1.26 new expressions.
Replace the end-of-support AWS SDK for Go v1 with supported v2 Lambda, S3, config, and Smithy modules. Preserve shared configuration, region, path-style S3 behavior, and service error handling.
@JonnyBurger JonnyBurger changed the title feat(lambda-go): upload large inputProps to S3 Lambda Go SDK: upload large inputProps to S3 Jul 11, 2026
Comment thread packages/lambda-go/invocations.go Outdated
Replace context.TODO with context.Background for AWS configuration and Lambda invocation calls where the existing public API provides no caller context.
# Conflicts:
#	packages/lambda-go/internalops.go
#	packages/lambda-go/version.go
@vercel
vercel Bot temporarily deployed to Preview – remotion July 13, 2026 03:14 Inactive
@JonnyBurger
JonnyBurger merged commit b6b3e51 into remotion-dev:main Jul 13, 2026
15 checks passed
@iliya-vidrush
iliya-vidrush deleted the lambda-go-s3-upload branch July 15, 2026 08:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants