Most product-manual search experiences are built around keyword matching. That works when a customer searches for an exact phrase, model number, or error code. It becomes less reliable when the customer describes a problem in natural language.

For example, a customer may search for:

Why does my device stop charging after the battery reaches 80%?

The manual might describe the same behavior using completely different wording, such as “battery protection mode” or “charge-limit configuration.” A keyword-only search can miss that connection.

I built a hybrid search assistant for product manuals to address this problem. The system combines lexical search with semantic vector search, then uses Claude on Amazon Bedrock to generate a grounded answer from the retrieved manual sections.

This article explains the architecture, the retrieval approach, and how I used AgentCore components to make the workflow more extensible.

The Tech Stack

The project used the following AWS services and components:

• Embeddings: Amazon Titan Text Embeddings, to convert manual sections and user queries into vectors
• Vector and lexical search: Amazon OpenSearch Serverless, to store embeddings and run keyword/vector retrieval
• Answer generation: Claude through Amazon Bedrock to generate a grounded response from retrieved context
• Agent runtime: AgentCore Runtime, to host and execute the agent workflow
• Tool integration: AgentCore Gateway, to expose search capabilities as agent tools
• Conversation state: AgentCore Memory, to preserve relevant interaction context
• Access control: AgentCore Identity, to manage identity and authorized tool access
• Monitoring: AgentCore Observability, to trace agent decisions, tool calls, and failures

The data source for this project was a collection of product manuals. The manuals were parsed, divided into sections, enriched with metadata such as product name and section title, and indexed in OpenSearch Serverless.

Why Keyword Search Alone Was Not Enough

Product manuals contain many exact values that are important to preserve:

• Model numbers
• Error codes
• Button names
• Configuration keys
• Firmware versions
• Safety warnings

For these terms, lexical search is often the right tool. A search for E-1042 should return the section containing E-1042, even if the surrounding language is not semantically similar to the user’s query.

However, customers do not always use the terminology found in the manual. They may ask:

The screen is on, but the device will not connect to Wi-Fi.

The manual may use terms such as “network pairing failure” or “wireless configuration.” Semantic search helps connect those expressions.

The key idea is not to choose keyword search or vector search. It is to use each one where it is strongest.

Architecture

The system follows this flow:

The agent receives the user’s question and decides which retrieval capabilities are needed. The search tools query the same OpenSearch Serverless collection, but they use different retrieval modes:

• Lexical search uses the indexed text fields.
• Vector search uses embeddings generated with Amazon Titan Text Embeddings.
• Metadata filters restrict results to the relevant product or manual version.

The retrieved sections are combined before they are passed to Claude. The model is instructed to answer only from the supplied context and to say when the manuals do not contain enough information.

Preparing Product Manuals for Search

The ingestion pipeline is:

Product manuals
    |
    v
Text extraction
    |
    v
Heading-aware chunking
    |
    v
Metadata enrichment
    |
    v
Amazon Titan Text Embeddings
    |
    v
Amazon OpenSearch Serverless

Each chunk contains the text of a manual section together with metadata such as:

{
  "product": "Example Product One",
  "manual_version": "2025.1",
  "section": "Battery and charging",
  "page": 42,
  "source": "product-one-manual.pdf"
}

Metadata is useful for two reasons. It improves relevance by allowing product-specific filtering, and it provides source information that can be shown with the final answer.

Choosing the Right Chunk Size

Chunking has a direct effect on retrieval quality. A chunk that is too large may contain several unrelated instructions. A chunk that is too small may remove the context needed to understand a warning or procedure.

For product manuals, heading-aware chunks are more useful than blindly splitting every fixed number of characters. A section such as “Resetting the network connection” should remain connected to its steps, prerequisites, and warnings where possible.

The practical rule I used was to preserve the manual’s logical structure first, then apply size limits and overlap only when required.

Combining Lexical and Vector Results with RRF

A simple weighted sum is tempting, but raw BM25 scores and vector similarity scores are not necessarily comparable. They can have different scales and distributions.

For this reason, I used Reciprocal Rank Fusion, or RRF, to combine ranked result lists. RRF uses the position of a document in each result list rather than assuming that the underlying scores are directly comparable.

A simplified form is:

RRF score(document) =
    1 / (k + lexical_rank)
    + 1 / (k + vector_rank)

Here, k is a smoothing constant. A document that appears near the top of both lists receives a strong combined score. Documents that appear in only one list can still be retained, but they are less likely to dominate the final ranking without strong evidence.

This approach is particularly useful for product manuals because exact error codes can rank highly in lexical search while natural-language descriptions can rank highly in vector search.

Routing Queries with AgentCore

The search assistant does not treat every question the same way. AgentCore helps coordinate the workflow and expose search operations as tools.

The routing behaviour can be described with three examples:

1. Exact error-code query: route primarily to keyword search.
2. Conceptual troubleshooting question: route primarily to vector search.
3. Complex question containing a product name and a natural-language problem: use both searches and fuse the results.

How the AgentCore Components Fit

AgentCore Runtime

Runtime hosts the agent execution flow. It provides the environment in which the assistant receives a query, selects tools, retrieves context, and prepares the final model request.

AgentCore Gateway

Gateway exposes the retrieval operations as tools. Instead of embedding every search operation directly in the agent logic, the workflow can call tools such as:

search_manual_keywords(query, product, manual_version)
search_manual_semantic(query, product, manual_version)
get_manual_section(source, page)

This separation makes it easier to add another data source or retrieval capability later.

AgentCore Memory

Memory preserves useful conversation context. For example, if a user first identifies a product and then asks three follow-up questions, the assistant can use that context without requiring the product name to be repeated in every query.

Memory should be used carefully. It should not replace authoritative retrieval from the manuals, and stale or irrelevant conversation history should not be allowed to override the current product or manual version.

AgentCore Identity

Identity controls who can access the agent and which tools or data sources the workflow is authorised to use. In a multi-product or multi-tenant environment, identity and metadata filtering are important safeguards against returning content from the wrong product or customer context.

AgentCore Observability

Observability makes the agent workflow inspectable. Useful trace information includes:

• The original user query
• The selected tools
• Search latency
• Retrieved document IDs
• Ranking and fusion results
• Model latency
• Errors and fallback paths

Without this information, it is difficult to tell whether a poor answer came from the model, retrieval, chunking, routing, or missing data.

A Bedrock Request Example

The following is a simplified implementation example for the product-manual assistant. It demonstrates the shape of a Bedrock Runtime request after the search layer has retrieved relevant manual sections.

import json
import boto3
 
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
 
MODEL_ID = "anthropic.claude-sonnet-4-6-v1"
 
manual_context = """
Product: Example Product One
Source: product-one-manual.pdf, page 42
Section: Battery and charging
The device may limit charging to 80% when Battery Protection Mode is enabled.
"""
 
user_question = "Why does my device stop charging at 80%?"
 
prompt = f"""
You are a product support assistant.
Answer the question only from the manual context below.
If the context is insufficient, say that the manual does not provide enough information.
Include the source page in your answer.
 
Manual context:
{manual_context}
 
Question:
{user_question}
"""
 
response = bedrock.invoke_model(
    modelId=MODEL_ID,
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 500,
        "temperature": 0.1,
        "messages": [
            {
                "role": "user",
                "content": [{"type": "text", "text": prompt}]
            }
        ]
    })
)
 
result = json.loads(response["body"].read())
print(result["content"][0]["text"])

Implementation note: Confirm the exact Claude model ID, request schema, and API version for the model and AWS Region you deploy. The snippet is intentionally simplified and is an implementation example, not a copy-paste production integration.

Evaluation: What to Measure

I did not want to evaluate the system only by asking whether an answer sounded good. A useful evaluation set should contain different query types:

• Exact model-number queries
• Exact error-code queries
• Natural-language troubleshooting questions
• Questions requiring metadata filtering
• Questions with no answer in the manuals

A practical first benchmark could contain 20 manually reviewed questions: 10 exact-term questions and 10 natural-language questions. For each query, compare keyword-only retrieval with the hybrid pipeline using the same top-k value.

Each configuration (keyword only, vector only, and hybrid with RRF) runs against the same labelled query set. For every query, the evaluation records the retrieved document IDs and their ranks, whether the answer is faithful to the retrieved sections, latency, and token usage.

Two metrics summarise the comparison: the share of queries with a relevant section in the top 3 results, and the share of answerable questions that are grounded correctly. Questions with no answer in the manuals are scored separately, on whether the assistant correctly says the information is not available.

What I Learned

The project produced several practical lessons.

First, retrieval quality often matters more than changing the generation model. If the correct manual section is not retrieved, the model cannot reliably answer from it.

Second, hybrid search needs a principled fusion method. BM25 and vector scores should not be added together without checking their scales. Rank-based fusion such as RRF is a safer starting point when the retrieval systems produce incompatible scores.

Third, agents should coordinate capabilities rather than hide the system’s behaviour. The routing decision, tool calls, retrieved context, and fallback path should be observable.

Finally, product support assistants need to be conservative. A confident but unsupported troubleshooting instruction can be worse than a clear “I could not find that in the manual.”

Limitations and Next Steps

This project is an implementation study rather than a complete production support platform. The next steps would be to:

• Build a larger labelled evaluation set
• Measure recall, precision, faithfulness, latency, and cost
• Add a real user-interface screenshot and trace examples
• Test prompt-injection content inside uploaded manuals
• Add document version and access-control tests
• Compare different chunking strategies
• Add automated regression tests for retrieval changes

Conclusion

A product-manual assistant becomes more useful when it combines exact retrieval with semantic understanding. Amazon Titan Text Embeddings and Amazon OpenSearch Serverless provide the search foundation, while Claude on Amazon Bedrock turns retrieved manual sections into a conversational answer.

AgentCore adds the orchestration layer: Runtime executes the workflow, Gateway exposes tools, Memory preserves useful context, Identity controls access, and Observability makes the decisions and failures traceable.

The main lesson from this project is that a reliable AI search assistant is not just a model call. It is a system of ingestion, metadata, retrieval, ranking, orchestration, grounding, access control, and evaluation.

References

• Amazon Titan Text Embeddings models
• Working with vector search collections (Amazon OpenSearch Serverless)
• Anthropic Claude Messages API on Amazon Bedrock
• What is Amazon Bedrock AgentCore?
• Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (Cormack et al., SIGIR 2009)

If this helped, give it a clap and follow for more practical breakdowns on cloud and AI.

Building a Hybrid Search Assistant for Product Manuals with Amazon Bedrock and AgentCore was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

The moment a product manager asks for “search functionality”, the default engineering reflex is to provision a dedicated search cluster. We automatically reach for Amazon OpenSearch, Elasticsearch, or a managed vector database.

But if you are dealing with internal documentation, operational manuals, or specialized internal datasets, provisioning a cluster is often an expensive operational overkill. OpenSearch instances must run 24/7, requiring minimum node instances, networking overhead, index management, and a chunk of your monthly AWS budget, even when idle. What if you could build a production-grade, highly resilient hybrid (keyword + semantic) search engine entirely inside a serverless AWS Lambda function for pennies on the dollar?

Here is the blueprint of a “zero-cluster” search architecture that combines lightning-fast, in-memory JavaScript substring matching with Amazon Bedrock embeddings.

The Architecture Overview

Instead of maintaining a massive external index, this architecture shifts the heavy lifting to smart application logic and managed serverless primitives. The system uses a two-pass strategy executed inside an AWS Lambda function:

[ User Query ]
      │
      ├──> Pass 1: Local In-Memory Substring Match (DynamoDB Cache)
      │             └── (Exact Match Found? -> Return Immediately & Save Cost)
      │
      └──> Pass 2: Semantic Vector Match (If Keyword Pass Isn't Absolute)
                    ├── Embed query via Amazon Bedrock (Titan V2)
                    ├── Query S3 Global/Org Vector Indexes
                    └── Map hits to DB records via Custom Resolver

Pass 1: The High-Speed Keyword Bypass

The first layer of defense is pure, deterministic execution. The system fetches manual data from Amazon DynamoDB via a Global Secondary Index ( manualsByIsLatest ). Instead of querying the database on every single keystroke, the Lambda function implements dual-tier in-memory caching which are;

• Metadata Cache (5 Seconds): Names, normalized names, and categories are cached briefly to capture fast operational updates.
• Content Cache (5 Minutes): Heavy text bodies are cached longer to protect the Lambda from data-transfer bottlenecks and eliminate DynamoDB read-capacity spikes.

When a query comes in, the code lowercases the input and uses native JavaScript .includes() to check against the manual’s attributes.

The Optimization Trick:

The content body check only triggers if the query is at least 3 characters long. Furthermore, if the query returns an exact match against a document’s name or category, the engine assumes absolute intent. It returns the result immediately and skips the semantic vector pass entirely, saving execution time and AI token costs.

Pass 2: The Serverless Semantic Layer

If the keyword match isn’t an exact, definitive shortcut, the engine seamlessly initiates the semantic pass.

1. Query Embedding: The search string is embedded using Amazon Bedrock with the amazon.titan-embed-text-v2:0 model.
2. S3 Vector Indexing: Instead of querying a running vector database, the application queries pre-computed vector files hosted on Amazon S3. It targets two distinct index scopes, a global index and an organization-specific index ( org-{organizationId} ), restricted by the caller's department permissions.
3. Distance Pruning: The engine retrieves the top 10 matches and applies a mathematical threshold, dropping any match with a cosine distance above 0.5.

The Identity Problem: Reverse-Mapping

Vectors don’t naturally understand your database primary keys, they return abstract file chunks or URLs. To solve this, a dedicated resolver ( manual-vector-resolver.ts ) maps the vector matches back to primary DynamoDB keys by cross-referencing deterministic fields like the PDF source URL, editor key, or normalized name.

Solution for Resiliency: The Graceful Fallback

Relying on external AI APIs introduces risk. Bedrock can hit rate limits, network blips happen, or S3 index reads can occasionally time out.

To handle this, the search layer wraps the entire hybrid pipeline in a robust fail-safe. If the Bedrock embedding or vector retrieval throws an error, the engine gracefully downgrades to the raw in-memory substring match. The user might lose semantic understanding for a split second, but they are never met with a blank screen or a 500 Internal Server Error.

Designing for Distinct Contexts

One size rarely fits all. In a production ecosystem, you might need this high-speed manual locator to coexist with a broader Document Library search ( document-library-search.ts ).

While your operational manual search prioritizes exact text containment and quick shortcuts, a deeper document library search can run an entirely different algorithmic blend, matching whole words with simple word stemming, treating multi-word phrases differently than single terms, and applying stricter distance cut-offs to keep massive archives clean.

What about DynamoDB’s native vector search?

While AWS’s introduction of native vector support inside Amazon DynamoDB eliminates the “Identity Problem” by allowing you to query embeddings directly inside your database records, it does not make this “zero-cluster” blueprint obsolete. Instead, it presents a classic trade-off between operational convenience and extreme cost-efficiency. Switching to native vectors simplifies your pipeline by removing custom S3 sync scripts, but it introduces persistent database indexing costs. More importantly, native database features cannot replace full-stack application logic, even with native vectors, Pass 1 (The Keyword Bypass) remains essential. Without this local, in-memory JavaScript layer protecting your pipeline, every single search would still force a costly network round-trip and API token charge to Amazon Bedrock just to embed the query. Stick to the S3 flat-file approach for absolute zero-dollar idle infrastructure and strict multi-tenant isolation, or adopt native DynamoDB vectors if you prefer maximum engineering simplicity and are willing to pay a minor baseline premium to completely offload index management.

Summary: When to Go Clusterless

This architecture proves that you don’t always need to inherit the operational baggage of a dedicated search engine cluster to deliver smart search results.

Go Serverless/Clusterless if:

• Your dataset is bounded (e.g., thousands of highly detailed manuals rather than millions of retail products).
• You are operating on a strict budget and want near-zero idle infrastructure costs.
• You want absolute control over caching and algorithmic fallbacks using standard JavaScript.

Stick to OpenSearch if:

• You are searching across millions of rapidly mutating database rows.
• You require complex native text analytics like tokenization, fuzzy phonetic matching, or complex multi-language stemming out of the box.

By letting application memory handle the obvious matches and using serverless embeddings for the conceptual ones, you can build a fast, incredibly cheap, and maintenance-free search layer that scales perfectly with your traffic.

Skip OpenSearch: Build a Serverless Hybrid Search Engine was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Custom business logic is one of the nicer parts of Amplify Gen 2. You add a placeOrder mutation to your schema, point it at a Lambda function, write a few lines of TypeScript, and the sandbox deploys it in a couple of minutes. The frontend calls it like any other operation.

Then the function needs to do a little more. Before it creates an order, it should check that the product exists and has stock. That means reading another model, so you give the function access to the data API with one line in the schema, save, and wait for the sandbox.

This time it does not deploy:

The CloudFormation deployment failed due to circular dependency found between nested stacks [data1234ABCD, function6789XYZ]

Nothing in your handler is wrong, and nothing in the schema is invalid. The two lines you wrote are both correct on their own. Together, they ask CloudFormation for something it cannot do.

In this article, we will look at what the Amplify Gen 2 circular dependency error is actually describing, why this particular pair of lines causes it, and the one-property fix that the Amplify docs recommend.

What a Nested Stack Is (In Simple Terms)

Amplify Gen 2 turns your amplify/ folder into a CloudFormation deployment. Each part of the backend lands in its own nested stack, which is why the error names stacks like data1234ABCD and function6789XYZ. The suffix changes from app to app. The prefix is the part that matters.

Functions have a default home. As the Configure Functions page puts it, “By default, functions are grouped together in a resource group named function." Your data API lives in the data group.

Nested stacks are not isolated. When one needs a value from another, such as an ARN or an endpoint URL, CloudFormation has to finish the stack that produces the value before it can start the one that consumes it. That gives every reference a direction, and the stacks a deploy order.

An Amplify CloudFormation circular dependency is what you get when those directions form a loop. If data must wait for function, and function must wait for data, neither can go first.

Two Arrows Pointing at Each Other

Here is the setup after that last change. The storage definition registers the function as its upload trigger and gives it access to two folders:

// amplify/storage/resource.ts
import { defineStorage } from '@aws-amplify/backend';
import { processUpload } from './process-upload/resource';

export const storage = defineStorage({
  name: 'uploads',
  triggers: {
    onUpload: processUpload, // storage → function
  },
  access: (allow) => ({
    'uploads/*': [allow.resource(processUpload).to(['read'])],
    'thumbnails/*': [allow.resource(processUpload).to(['write'])],
  }),
});

Then, in backend.ts, the function is given the bucket name so it knows where to write the thumbnail:

// amplify/backend.ts
import { defineBackend } from '@aws-amplify/backend';
import { storage } from './storage/resource';
import { processUpload } from './storage/process-upload/resource';
const backend = defineBackend({ storage, processUpload });
backend.processUpload.addEnvironment(
 'THUMBNAIL_BUCKET',
 backend.storage.resources.bucket.bucketName, // function → storage
);

Two lines create the two arrows.

onUpload: processUpload makes the function a storage trigger. S3 has to know which Lambda function to call when an object lands in the bucket, so the storage stack needs the function's ARN from the function stack.

addEnvironment(...) goes the other way. The variable holds a value that only exists once the bucket has been created, so the function stack now needs the bucket name from the storage stack.

The first deploy worked because only the trigger existed. Adding the environment variable closed the loop.

The access rules are not part of the problem. According to the storage authorization docs, granting access with allow.resource() already gives the function an environment variable named uploads_BUCKET_NAME, and Amplify wires that reference in one direction only. The hand-written THUMBNAIL_BUCKET duplicates something the function already had.

The Obvious Fix Isn’t the Right One

The quick way out is to remove one arrow. Drop allow.resource(), copy the GraphQL endpoint from amplify_outputs.json into an environment variable, and handle authorization in the function yourself.

It deploys, and it is a trap. The endpoint is different in every sandbox and every branch environment, so the value you pasted points at one specific backend. The next teammate who starts their own sandbox has a function quietly writing to yours. You have also traded access that Amplify scopes and grants for credentials you now manage by hand.

Deploying in two passes, first without the access rule and then with it, has the same weakness. The second pass still produces the loop, and a fresh environment has to repeat the dance every time.

Both arrows are legitimate. The fix is to stop them crossing a stack boundary.

The Fix: Put the Function in the Data Stack

Open the function’s resource.ts and add one property:

// amplify/data/place-order/resource.ts
import { defineFunction } from '@aws-amplify/backend';

export const placeOrder = defineFunction({
  name: 'place-order',
  resourceGroupName: 'data',
});

Setting the Amplify function resourceGroupName to 'data' is exactly what the circular dependency troubleshooting page recommends for this case: "if you are using the function as a query handler, but the function also needs access to the data (or AppSync) API, you might run into this issue. To resolve, group this function with other resources in the data stack."

The handler itself does not change. It configures the data client from the function’s environment and then works with models as usual:

// amplify/data/place-order/handler.ts
import type { Schema } from '../resource';
import { Amplify } from 'aws-amplify';
import { generateClient } from 'aws-amplify/data';
import { getAmplifyDataClientConfig } from '@aws-amplify/backend/function/runtime';
import { env } from '$amplify/env/place-order';

const { resourceConfig, libraryOptions } = await getAmplifyDataClientConfig(env);
Amplify.configure(resourceConfig, libraryOptions);
const client = generateClient<Schema>();
export const handler: Schema['placeOrder']['functionHandler'] = async (event) => {
  const { productId, quantity } = event.arguments;
  const { data: product } = await client.models.Product.get({ id: productId });
  if (!product || product.stock < quantity) {
    throw new Error('Product unavailable');
  }
  const { data: order, errors } = await client.models.Order.create({ productId, quantity });
  if (!order) {
    throw new Error(errors?.[0]?.message ?? 'Order was not created');
  }
  return order;
};

A few lines are worth reading closely:

• name: 'place-order' must match the import path $amplify/env/place-order, because that module is generated per function name.
• getAmplifyDataClientConfig(env) is where the function picks up the API details that allow.resource() granted. No endpoint is hard-coded anywhere.
• Schema['placeOrder']['functionHandler'] types both the arguments and the return value against the mutation in your schema.

Why does this work? Both references still exist. They just no longer cross between two nested stacks. With the function inside data, there is no second stack to wait for, so there is no stack-level loop to report.

To verify, save the file and let npx ampx sandbox redeploy. The deployment should complete without the circular dependency message. Then call the mutation from a signed-in client and confirm an Order comes back. Any error you hit now comes from your handler, which is a much better problem to have.

If a function serves a resource and also calls it, it belongs in that resource’s stack.

Reading the Error Before Changing Anything

resourceGroupName: 'data' fixes the data case. It is not a universal answer, and the error message tells you which case you are in. The names in the brackets are the stacks involved.

The same troubleshooting page covers the other shapes. A function used as an auth trigger can be grouped with resourceGroupName: 'auth'. A custom CDK resource that loops with an Amplify stack should be created inside that Amplify stack, for example new sqs.Queue(backend.storage.stack, 'MyCustomQueue'), rather than in a stack of its own. If the error names resources inside a single stack instead of nested stacks, the docs point to the AWS post on handling circular dependency errors in CloudFormation.

When One Group Is Not Enough

Because every function shares the function group by default, two unrelated functions can combine into a loop that neither causes alone. One is an auth trigger, another reads from data, and the shared stack now sits between auth and data in both directions. In that case, look at each function in the bracketed stack and move the ones with cross-stack references, either into the stack they depend on or into their own group, which resourceGroupName also allows.

If the error survives all of that, the docs ask you to open an issue on the amplify-backend repository with the full message.

It is also worth asking whether the logic needs a function at all. If the handler only reads or writes your own tables, an AppSync JavaScript resolver may do the job with no function stack involved. We covered that path in How to Get Started with JavaScript Resolvers in AWS Amplify Gen 2.

Final Thoughts

The Amplify Gen 2 circular dependency error looks like infrastructure going wrong, but it is describing your design accurately. A function that is both a handler for the data API and a client of it really does depend on data in both directions. Amplify cannot deploy that across two stacks, so you put both sides in one.

The rule carries over to every resource group: find the stacks named in the error, and group the function with the resource it serves. A function that only calls other resources, without serving any of them, can stay in the default function group, as long as it does not share that group with a function pointing the other way.

FAQ

What causes the circular dependency error in Amplify Gen 2?

A function that is both the handler for a custom query or mutation and granted access to the data API with allow.resource(). The data stack needs the function, the function stack needs the API, and CloudFormation cannot deploy either one first.

Can I hard-code the GraphQL endpoint instead?

It deploys, and it breaks later. The endpoint is different in every sandbox and branch environment, and you lose the access Amplify scopes and grants for you.

Do I need to change the handler code after moving the function?

No. The handler still reads the API details through getAmplifyDataClientConfig(env). Only the function's resource.ts changes.

What if the error names the auth stack instead of data?

Group the function with auth when it is an auth trigger, using resourceGroupName: 'auth'. For a custom CDK resource, create it inside the Amplify stack it depends on.

Next in this series: the “API key not found” error on a sandbox redeploy, and how sandbox identifiers keep environments from stepping on each other.

Fixing the Circular Dependency Error When a Function Calls Your Data API was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Serverless is great at small, event-driven jobs. It gets harder when the job turns into a workflow: several steps, calls to flaky external APIs, a wait of a few hours, and maybe a human approval in the middle. AWS Lambda Durable Functions let you write that kind of workflow as ordinary TypeScript or Python, while AWS handles checkpointing, replay, suspension, and recovery for you.

I’ve run Azure Durable Functions in production, and I’ve built scraping and data pipelines where something fails halfway through every day. So this isn’t a rewrite of the AWS docs. It covers how the model actually works, where it can catch you out, and how I’d use it on a real vehicle-data pipeline.

What a “workflow” used to look like on Lambda

Take an ordinary order flow:

Receive order → Validate → Reserve inventory → Take payment → Wait for shipment → Notify customer

Now add failure. Payment times out. The inventory service is down for two minutes. Shipping confirmation arrives six hours later. A manager has to approve a refund.

A single Lambda invocation can’t cope with that. It has a 15-minute maximum and forgets everything when it ends. So we usually build the durability ourselves:

• DynamoDB to store which step we’re on
• SQS or EventBridge to hand work between functions
• Several Lambdas, one per step
• Custom retry logic and dead-letter queues
• Scheduled events to “come back later”
• Idempotency checks so retries don’t double-charge anyone

None of these are bad building blocks. The problem is that by the end, you’ve written your own workflow engine, and the business logic is scattered across five services.

Durable functions turn that around. The workflow lives in one function’s code, and Lambda keeps track of where it got to.

The short version

• A durable function is still a Lambda function. You wrap the handler with the SDK and get a DurableContext.
• A durable execution can run for up to one year. Each invocation inside it still has the normal 15-minute limit.
• Steps are checkpointed. Once a step finishes, its result is saved.
• Recovery works by replay. The handler runs again from the top, and completed steps return their saved results instead of running again.
• Waits cost no compute. The function suspends, Lambda exits, and AWS resumes it later.
• Retries are built into steps, with backoff, and the retry delays don’t use execution time either.
• Your orchestration code must be deterministic. This is the main new thing to learn.
• Step Functions are still useful. The two tools fit different jobs, and they work well together.

What are AWS Lambda Durable Functions?

A durable function is a regular Lambda function with durable execution enabled. AWS provides the Durable Execution SDK for JavaScript/TypeScript, Python, and Java. For TypeScript:

npm install @aws/durable-execution-sdk-js

A minimal durable function looks like this:

import { DurableContext, withDurableExecution } from "@aws/durable-execution-sdk-js";

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const order = await context.step("validate-order", async () =>
      validateOrder(event.orderId)
    );

    const payment = await context.step("process-payment", async () =>
      processPayment(order)
    );

    await context.wait({ minutes: 30 });

    return await context.step("confirm-order", async () =>
      confirmOrder(order, payment)
    );
  }
);

It reads like normal async code, and that’s intentional. There’s no state machine JSON and no separate workflow designer. The workflow is the code.

One point people often miss: “runs for a year” does not mean a Lambda container stays alive for a year. A durable execution is a series of short invocations with saved progress in between:

Invocation 1 → run steps → checkpoint → exit
      ... hours pass (no compute billed) ...
Invocation 2 → replay → run next step → checkpoint → exit
      ... days pass ...
Invocation 3 → replay → finish → result

Steps: the unit of durable work

A step wraps work that should happen once and be remembered: an API call, a database write, a calculation you don’t want to repeat.

const customer = await context.step("load-customer", async () =>
  loadCustomer(event.customerId)
);

When the step finishes, the SDK checkpoints its result. If the function is replayed later, context.step("load-customer") doesn't call loadCustomer again. It returns the stored value straight away.

Two habits that pay off:

• Name every step. Names make execution history readable, and they help the runtime match saved state to your code.
• Never rename or reorder steps while executions are in flight. AWS warns that this can stop running executions from resuming or make them produce wrong results. More on this in the production section.

Checkpoint + replay: the core idea

This is the concept everything else depends on, so it’s worth going slowly.

Say a workflow has three steps. Steps A and B succeed. Then the environment goes away before Step C finishes. Maybe the invocation hit its timeout, or maybe it suspended on purpose for a wait.

When the execution resumes, AWS runs your handler again from the first line:

Replay → Step A  →  already done, return saved result
       → Step B  →  already done, return saved result
       → Step C  →  not done, execute it now

That’s the whole trick. The function doesn’t jump to line 47. It replays quickly through recorded history until it reaches work that hasn’t happened yet.

This has one big consequence:

Your orchestration code describes the workflow. Durable operations do the work.

Anything outside a step runs again on every replay. So code like this has a hidden problem:

const id = crypto.randomUUID();   // different value on every replay
const now = Date.now();           // different value on every replay

if (now > deadline) { ... }       // the workflow can take a different path

If a replay generates a different ID or reads a different time, the workflow can take a different branch than it did the first time. Then the recorded history no longer matches the code. The fix is to put anything non-deterministic inside a step, so its result gets recorded once:

const id = await context.step("generate-id", async () => crypto.randomUUID());

The same goes for random values, the current time, network calls, and database reads. Logging is a milder case. A console.log outside a step just prints again on every replay. That's harmless but confusing when you're debugging, so use the step context's logger for anything that matters.

If you’ve written Azure Durable Functions orchestrators, this will feel familiar: it’s the same deterministic orchestrator rule, just in a new place.

Waits that don’t cost compute

Long workflows spend most of their time waiting. Durable Functions provide context.wait():

// unnamed
await context.wait({ hours: 2 });

// named (easier to spot in execution history)
await context.wait("cool-down-before-recheck", { minutes: 30 });

When the function reaches a wait, the SDK checkpoints, the invocation ends, and AWS starts a new invocation when the time is up. You don’t pay for Lambda execution time while it waits, so a two-minute wait and a two-week wait cost about the same.

That makes waits a good fit for:

• Polling an external system every N minutes
• Delayed actions like reminders or follow-up emails
• Cool-down periods when a third-party API rate-limits you

Waiting for a human (or another system)

Sometimes you’re waiting for an event, not a time. A manager approves a refund. A partner system confirms a shipment. For this, the SDK has callbacks:

const decision = await context.waitForCallback(
  "wait-for-approval",
  async (callbackId) => {
    await sendApprovalRequest(callbackId, event.requestId);
  }
);

The function suspends until an external system calls the Lambda API (SendDurableExecutionCallbackSuccess or ...Failure) with that callback ID. Before this, you'd build approval flows with a DynamoDB table and a token. Now they take a few lines.

Built-in retries (and why idempotency still matters)

Distributed systems fail in boring, temporary ways: 503 Service Unavailable, timeouts, throttling. Usually you want to retry the one thing that failed, not restart the whole workflow.

Steps accept a retry strategy:

import { createRetryStrategy } from "@aws/durable-execution-sdk-js";

const retryStrategy = createRetryStrategy({
  maxAttempts: 5,
  initialDelay: { seconds: 2 },
  maxDelay: { minutes: 1 },
  backoffRate: 2,
});

const result = await context.step(
  "call-external-api",
  async () => callExternalApi(),
  { retryStrategy }
);

A few details worth knowing:

• Retry delays suspend the function too. The SDK checkpoints the error, exits, and resumes when the next attempt is due. Backoff doesn’t burn execution time.
• There’s a default. With no strategy at all, steps are already retried several times. Set one explicitly so the behavior is clear in code review.
• You can filter what’s retryable with retryableErrors and retryableErrorTypes. A 400 Bad Request should fail fast, not be retried five times.
• The execution itself isn’t retried automatically if it fails completely. For async invocations, configure a dead-letter queue.

Retries bring back a classic distributed-systems problem: idempotency. A step can run more than once. For example, the payment API might succeed, and then the network drops before the checkpoint is saved. If the step creates a payment, a retry could create a second one.

First attempt  → payment created → checkpoint lost
Retry          → payment created AGAIN  ❌

The fix is the same as anywhere else. Send an idempotency key (such as the order ID) so the downstream service sees a repeat request and returns the original result. Durable execution makes reliable workflows much easier to build, but it doesn’t replace good distributed-systems design.

A real-world example: a vehicle-data pipeline

This is where it gets personal. I’ve built scraping pipelines for vehicle marketplace data, and they’re a good stress test for any workflow tool. They involve:

• Thousands of items per run
• Pages that fail in unpredictable ways
• Rate limits that force you to slow down
• Steps that are expensive to repeat, such as rendering a page or calling an enrichment API

The pipeline looks like this:

Start job → Fetch listings → For each vehicle: scrape → normalize → validate → store
          → Calculate derived properties → Publish results

The old version used SQS between every stage, a DynamoDB table of job state, and a fair amount of glue code. As a durable function, it fits in one handler:

export const handler = withDurableExecution(
  async (event, context: DurableContext) => {

    const listingIds = await context.step("fetch-listings", async () =>
      fetchListingIds(event.marketplace)
    );

    const results = await context.map(
      "process-vehicles",
      listingIds,
      async (ctx, listingId) => {
        const raw = await ctx.step(`scrape-${listingId}`, async () =>
          scrapeVehicle(listingId), { retryStrategy }
        );
        const vehicle = await ctx.step(`normalize-${listingId}`, async () =>
          normalizeVehicle(raw)
        );
        return await ctx.step(`store-${listingId}`, async () =>
          storeVehicle(vehicle)   // returns an ID/S3 key, not the full record
        );
      },
      { maxConcurrency: 20 }
    );

    await context.step("calculate-derived", async () =>
      calculateDerivedProperties(results.getResults())
    );

    await context.step("publish", async () => publishResults(event.marketplace));
    return { status: "completed" };
  }
);

What I like about this:

• context.map processes vehicles in parallel with a concurrency cap. That's important when the marketplace rate-limits you.
• Each vehicle has its own checkpointed steps. If vehicle #4,812 fails, the other items keep going, and only that item’s failed step retries.
• If the whole thing is interrupted, it resumes. Vehicles that are already finished aren’t scraped again on replay.
• One file shows the whole pipeline. A new engineer can read it top to bottom.

And the lesson I’d pass on from experience: keep the checkpointed state small. Every step result is stored in the execution history, so don’t return a 2 MB HTML page from a step. Write large payloads to S3 and return the key. AWS calls this out in its own best practices, because large state costs more and slows replay down.

Durable Functions vs AWS Step Functions

This is usually the first question: “AWS already has Step Functions. Why do I need this?”

They overlap, but they’re built for different jobs. AWS’s own framing is roughly: Durable Functions are for application development inside Lambda. Step Functions are for workflow orchestration across AWS.

Choose Durable Functions when:

• Most of your logic already lives in Lambda, and you want it in one codebase.
• Your team prefers TypeScript, Python, or Java over a JSON/ASL state machine.
• You want normal developer tooling: unit tests, debuggers, code review, and refactoring.
• You need fine-grained control over branching, loops, and error handling in code.

Choose Step Functions when:

• You orchestrate many AWS services and want native integrations (200+ services) without writing SDK glue.
• Non-engineers need to see the workflow, and a visual graph helps with cross-team sign-off.
• You want fully managed orchestration with no runtime or dependency updates on your side.

Or use both. A sensible architecture lets Step Functions coordinate the big cross-service picture (ingestion, ECS jobs, Bedrock calls, notifications), while durable Lambdas handle the detailed application logic inside each stage. AWS explicitly recommends this hybrid approach.

Coming from Azure Durable Functions

If you’ve used Azure Durable Functions, you already know most of this. Both are built on the same ideas:

• Orchestration in code, with activities/steps doing the real work
• Event-sourced history with checkpoint and replay
• Durable timers and external events/callbacks
• The deterministic-orchestrator rule

The differences are mostly in packaging. Azure has separate orchestrator and activity functions, built on the Durable Task Framework and a storage provider. AWS keeps everything in one Lambda handler, with steps as inline closures, and AWS manages the state store for you. That means less wiring, but also less visibility into the internals.

My advice to anyone moving between clouds: learn the model, not the API names. Once replay makes sense to you, either platform takes about an afternoon to pick up.

Production considerations

These are the things the getting-started guides tend to rush past.

1. Durability is set at creation time. You configure DurableConfig (with a required ExecutionTimeout, optional RetentionPeriodInDays and an optional KMS key) when you create the function. You can't turn durable execution on for an existing function. So include it in your infrastructure-as-code from the start.

2. Don’t run durable executions on $LATEST. This one catches people. An execution that started last Tuesday might replay against code you deployed this morning. If the new code has renamed a step, added one, or reads stored results differently, the replay no longer matches the history. Always invoke durable functions through a published version or alias, and make sure new code can still handle state from older executions.

With a normal Lambda, a deploy affects the next request. With a durable workflow that lives for weeks, versioning becomes part of correctness.

3. Keep state lean. Return IDs and S3 keys from steps, not large objects.

4. Plan for whole-execution failure. Configure a DLQ for async invocations, and decide what “give up” means for your business process.

5. Monitor executions, not just invocations. One business transaction is now many invocations. Use the execution history APIs and CloudWatch to trace a workflow from start to finish.

When should you (not) use them?

Good fit:

1. Multiple sequential steps where repeating finished work is expensive or unsafe? → Yes.
2. Long pauses (minutes to months) between steps? → Yes.
3. Human approval or waiting on an external system? → Yes, use callbacks.
4. Flaky external dependencies that need retry with backoff? → Yes.
5. Large batches that need parallel processing and partial recovery? → Yes, use map.

Probably not:

• A simple S3 event → resize image → done. A plain Lambda is fine. Don’t add replay semantics you don’t need.
• Orchestrating ten AWS services for a non-technical audience. Step Functions will serve you better.

The architecture should follow the problem, not the latest launch.

Final thoughts

The interesting part of Lambda Durable Functions isn’t that Lambda can now “run for a year”. It’s that durability is now part of the programming model.

Real applications are workflows. They wait. They retry. They call unreliable systems. They need approvals. They fail halfway through and have to carry on tomorrow. We used to model all of that with queues, tables, and glue code. Now we can write it as code that reads like the business process.

The infrastructure hasn’t gone away. AWS now runs it, so we can focus on the workflow itself.

So the useful question isn’t “Should I use Durable Functions?” It’s this:

“Where should the state and orchestration of this workflow live?”

Answer that honestly, and the right tool, whether it’s Lambda, Durable Functions, Step Functions, or a mix, usually becomes clear.

References

• AWS Lambda durable functions
• Getting started with durable functions
• Configure Lambda durable functions
• Durable functions best practices
• Durable functions vs Step Functions
• AWS Durable Execution SDK Developer Guide

AWS is still changing this area quickly, so check the current SDK reference before using these snippets in production.

If this helped, give it a clap and follow for more practical breakdowns on cloud and AI.

AWS Lambda Durable Functions: Steps, Checkpoints, Replay, and Real-World Workflows was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our system uses cameras to understand what’s in front of it: how many objects are in view, how close they are, and whether something is blocking the left side. Each photo can have several labels at once. To do that, we use a convolutional neural network (CNN) called ResNet-152, trained with SageMaker’s built-in image-classification algorithm.

This post explains how that model works, in plain terms first and then with our real settings. You don’t need a deep-learning background to follow it.

What a CNN does

A CNN looks at a photo through lots of small filters. Each filter slides across the image looking for one simple pattern.

• Early layers find edges and color changes.
• Middle layers combine those into shapes and textures.
• Later layers combine shapes into parts of objects.
• The final layer turns all of that into one score per label.

Nobody programs those filters by hand. The network learns them from labeled photos during training.

Why ResNet-152

More layers let a network learn more complex patterns. But very deep networks used to be hard to train: the learning signal got weaker as it passed back through dozens of layers.

ResNet fixed this with shortcut connections. Each block of layers learns a small change and adds it to its input, and the input also skips straight past the block. That shortcut keeps the signal strong, so networks can be very deep. ResNet-152 has 152 layers.

In our settings, that’s one line: num_layers: "152".

We didn’t start from zero: transfer learning.

Training 152 layers from scratch needs millions of photos. We have tens of thousands. So we use transfer learning:

• use_pretrained_model: "1" starts from a ResNet already trained on ImageNet, a huge public photo dataset. It already knows edges, shapes, and textures.
• Our training then teaches it our labels on our photos.

We also often go one step further and start from our own previous model instead of ImageNet. That takes about 10 passes over the data (epochs) instead of 50, because the model already knows most of the job.

One photo, many labels

Most image tutorials pick one label per photo: cat or dog. Ours need several at once, like “few objects," “close range,” and “blocked on the left."

• multi_label: "1" tells the algorithm that each photo can have several labels.
• The model returns a score between 0 and 1 for every label, in the same order as our label map.

Our exact training settings

These are the defaults every model starts from (all values are strings, because SageMaker requires that):

const BASE_HYPERPARAMETERS = {
  num_layers: "152",           // ResNet-152
  image_shape: "3,224,224",    // colour photos resized to 224 × 224
  use_pretrained_model: "1",   // start from ImageNet weights
  multi_label: "1",            // several labels per photo
  epochs: "50",                // 10 when building on our previous model
  mini_batch_size: "32",
  optimizer: "sgd",
  learning_rate: "0.001",
  momentum: "0.9",
  weight_decay: "0.0001",
  augmentation_type: "crop_color_transform",
  use_weighted_loss: "0",
  precision_dtype: "float32",
  top_k: "2",                  // must be 2 or more, or the job fails
  early_stopping: "false",
  checkpoint_frequency: "1",
};

In plain English:

• 224 × 224 color photos. Every photo is resized to the size the network expects.
• 32 photos per step. The model adjusts its filters after each batch of 32.
• Learning rate 0.001 with SGD and momentum. Small, steady steps. Momentum keeps it moving in a consistent direction.
• Weight decay. A small penalty that discourages the model from memorising.
• crop_color_transform. During training, photos are randomly cropped and their colours slightly changed, so the model copes with real-world variety like blur or poor lighting.
• use_weighted_loss: "0". We deliberately don't let the model compensate for rare labels. Instead, we show the imbalance to our team, so they go and label more photos of the rare ones.
• No early stopping. Every run trains the full schedule, so runs are comparable.

Two things are filled in automatically for every run: num_classes (how many labels the model has) and num_training_samples. Nobody can override them, because a wrong value produces a model that gives confident, wrong answers.

Training runs on one ml.g4dn.xlarge, a single NVIDIA T4 GPU.

From scores to answers

The model returns a list of scores. Our app turns them into answers with two simple rules.

Rule 1: Some labels are a group, and only one can win. “Many objects” and “few objects” can’t both be true. So within a group, the highest score wins:

for (const group of groups) {
  const present = group.filter((label) => probMap.has(label));
  if (present.length === 0) continue;
  let maxLabel = present[0], maxProb = -1;
  for (const label of present) {
    const p = probMap.get(label) ?? 0;
    if (p > maxProb) { maxProb = p; maxLabel = label; }
  }
  const hasWinner = maxProb > 0;
  for (const label of present) labels.set(label, hasWinner && label === maxLabel);
}

Rule 2: other labels stand alone, with a 0.5 threshold. “Low lighting” is either true or not, so it’s true when its score is 0.5 or more.

Two edge cases we learned the hard way:

• If every score in a group is 0, nobody wins. That’s the model saying “none of these," not “the first one."
• A label the model doesn’t have shouldn’t read as "no." An older model trained before a label existed returns no score for it. For those labels, the app shows “can’t tell” instead of quietly answering “no” without the model ever looking.

What we’d try next

ResNet-152 is a big network. A smaller depth, such as ResNet-50, might answer faster and cost less to serve, possibly with similar accuracy on our labels. We haven’t tested that yet. With the training pipeline in place, it’s now a one-setting experiment.

Checklist if you’re using the built-in image classifier

1. One label or several per photo? Set multi_label to match.
2. Do you have millions of photos? If not, keep use_pretrained_model on.
3. Is top_k 2 or more? A value of 1 fails the job.
4. Do group labels use “highest score wins," and standalone labels a threshold?
5. Does a label the model doesn’t have to read as “can’t tell," not “no”?

Final thoughts

You don’t need to design a CNN to use one well. The built-in algorithm gave us a proven 152-layer network that already understood photos. Our work was choosing sensible settings and being careful about how its scores turn into answers in the app.

References

• Image classification: MXNet (SageMaker built-in algorithm)
• Image classification hyperparameters
• Deep Residual Learning for Image Recognition (He et al., the ResNet paper)
• Amazon EC2 G4 instances

If this helped, give it a clap and follow for more practical breakdowns on cloud and AI.

The CNN Behind Our Vision Models: ResNet-152, Multi-Label, on SageMaker was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 5 of 5 · SageMaker Without an ML Team. The full series is listed at the end.

Imagine moving house. You don’t carry the furniture out and hope the new place is ready. You set up the new house completely, check the lights and water work, and only then change your address. You keep the old house for a while too, so if something’s wrong you can go back.

That’s how we moved our whole AWS SageMaker setup- every model, training job, and photo- from the Frankfurt region to Stockholm. The actual switch was 16 lines added to one file, nothing removed. Everything that made it safe happened before that.

The trick: same names on both sides

Before the switch, we had already created all 19 models (SageMaker “endpoints”) in the new region, with the same names as the old ones.

That one choice made everything simpler. Our system looks up models by name. Same names meant nothing else had to change. The only question was which region to look in, and that became a single setting. Flip it, and everything points at the new region at once.

The checklist before the switch

The change was tiny. The list of things we checked first was not.

Done in the weeks before:

• All 19 models were running in the new region, and each gave exactly the same answer as the old one for the same photo.
• All 65,967 photos were copied, and the counts matched.
• The training history was copied (18 records), so future training still knew what came before.
• The training setup was deployed in the new region.
• A real training job was run in the new region to prove it worked, instead of trusting a table.

Done in the last hour, because they go out of date:

• A final copy of new photos, since cameras kept uploading to the old region right up to the switch.
• Warming up all 19 models. A “cold” model takes about 18 seconds to answer the first time, against under a second when warm. Switching to 19 cold models at once would have looked like an outage.
• Checking that no training job was running. A job running during the switch would report back to the old region, where nobody was listening anymore, and look stuck forever.

Why such a big switch was safe

We deleted nothing. The old region kept all its models, photos, and setup running.

So if anything went wrong, undoing it was one step: revert the change, and everything pointed back at a setup that had been working ten minutes earlier.

A gradual rollout limits how much breaks at once. Keeping the old setup alive limits how long it stays broken. For a change you can undo instantly, the second matters more. The cost is paying for two of everything for a while, and resisting the urge to tidy up too early.

Copy only what you need first

Copying everything, all the photos, was about 440 GB: hours of work and roughly $9 in transfer costs. The models themselves only needed about 3.3 GB, which copied in minutes for pennies.

The money wasn’t the point. Copying the small part first let us prove the models worked in the new region before moving the big part.

It still broke, a little

After the switch, every finished training job got stuck on a status called “Repacking”, forever, with no error anywhere.

The cause: our code reads the region from a setting, and if the setting is missing, it quietly falls back to the old region. One function in the new region was never given that setting. So it kept calling the old region, where it no longer had permission, and the job just stopped.

The fix was one line per function: tell every function which region it’s in.

The test we added is the clever part. It checks the setup in a region that is not the fallback one. If you test with the fallback region, the bug passes, because the wrong answer and the right answer look the same.

Your checklist for a region move

1. Can the new region run with the same names as the old one? Then the switch can be one setting.
2. Have you proved it works there, with real answers and a real training job?
3. Are the last-minute steps done in the last hour: final copy, warm-up, nothing running?
4. Is the old region still running, so undoing is one step?
5. Does every part of your system know which region it’s in, with no silent fallback?

Wrapping up the series

Across all five posts, the expensive problems had one thing in common: none of them crashed loudly. A score that looked too good. A setting that was quietly ignored. A job stuck with no error. We only found them by checking.

So the most useful question isn’t “does it work?” It’s “if this were wrong, would I be able to tell?” Wherever the answer is no, that’s where to add a check.

Thanks for reading the series.

References

• Amazon SageMaker Serverless Inference
• Docker registry paths for built-in algorithms

If this helped, give it a clap and follow for more practical breakdowns on cloud and AI.

Read the series: SageMaker Without an ML Team

1. 5 Mistakes That Wasted Our Money on AWS SageMaker
2. Your ML Experiment Just Became Everyone’s Default
3. Life After SageMaker Ground Truth
4. Your Train/Test Split Is Probably Leaking
5. How We Moved Our Whole SageMaker Setup to Another AWS Region (you are here)

How We Moved Our Whole SageMaker Setup to Another AWS Region With One Small Change was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Starting a second Amplify Gen 2 project usually takes a few minutes. You clone the same Next.js starter you used for the first one, install the dependencies, and run npx ampx sandbox in the new folder.

That first deploy fails. When you look closer, the command targeted the sandbox of your other project. A quick search suggests giving the new sandbox its own name, so you run npx ampx sandbox --identifier blog, and this time it deploys cleanly.

The next morning you open the first project, start its sandbox as usual, and the deploy stops a few seconds in:

[ERROR] [UnknownFault] ToolkitError: ❌  amplify-awsamplifygen2-alex-sandbox-a8968da97f failed: NotFoundException: API key not found: da2-...

Nothing changed in that project. Its code is the same as yesterday, and its sandbox was healthy when you closed the terminal. The trouble comes from an assumption most of us make without noticing: that a sandbox belongs to the folder you run it from.

In this article, we will look at how two projects end up sharing one sandbox, how to recover from the ampx sandbox api key not found error, and how to prevent it with sandbox identifiers.

Where the Sandbox Name Comes From

The sandbox setup docs say a deploy creates a CloudFormation stack named amplify-<app-name>-<$(whoami)>-sandbox. The ampx source fills in the details. The app name is the name field of the package.json in the directory where you run the command. The second part is your system username, or the value of --identifier when you pass one. A short hash of those two values goes on the end.

The folder path is not one of the inputs. Two folders with the same package.json name, run by the same user, resolve to the same stack.

Starter templates make this easy to hit. The Next.js starter ships with "name": "aws-amplify-gen2" in its package.json, and many projects never change that line. Running the naming logic from @aws-amplify/backend-cli 1.10.0 for a user called alex gives these stack names:

~/code/shop                      amplify-awsamplifygen2-alex-sandbox-a8968da97f
~/code/blog                      amplify-awsamplifygen2-alex-sandbox-a8968da97f
~/code/blog --identifier blog    amplify-awsamplifygen2-blog-sandbox-8b27e47144
~/code/blog, name "blog-app"     amplify-blogapp-alex-sandbox-11c16b5c72

The first two lines are the same stack. The last two are separate stacks, which is what you expected to get from the start.

How a Shared Name Turns Into “API Key Not Found”

This exact sequence is recorded in amplify-backend issue #3044. The reporter started a new project, ran npx ampx sandbox, and got an error. They then realized the command was deploying to the existing project's sandbox, moved the new project to --identifier project2, and that deploy worked. Back in the original project, every redeploy failed with NotFoundException: API key not found.

A maintainer traced the failure to the AppSync API key. According to that investigation, “CloudFormation still has a reference to the old API key ID”, tries to update it, and AppSync replies that the key does not exist. The log in the issue shows UpdateApiKeyCommand as the last AWS call before the deploy failed.

The issue is still open, labeled as a bug at priority p2, and no fix or safe in-place repair has been published. Until then, the practical step is to stop two projects from sharing a stack, which is the part you can control.

Confirm It Before Deleting Anything

Before you remove anything, check that this Amplify sandbox redeploy error really comes from a shared name:

1. Read the stack name in the error line. It starts with amplify-, then the package name with punctuation removed, then the sandbox name.
2. Open package.json in each of your Amplify projects. If two of them have the same name and you ran both without --identifier, they shared a stack.
3. Open Manage sandboxes in the Amplify console, which lists the sandboxes in your account, their status, and last update.

If each project has a different name, the shared stack isn't the cause here. The issue thread is then a better place to compare your logs with the ones already reported.

Recovering the Sandbox

Clearing local build files is the usual first step. In the issue, rm -rf cdk.out .amplify did not help. The mismatch is in the deployed stack, so files on your machine are not where they can be repaired.

Deleting the CloudFormation stack from the AWS console is also tempting. The maintainer withdrew that suggestion in the same thread and advised against it without a full backup, because removing the stack can remove the data in your sandbox tables and buckets.

The route the Amplify team recommends is to delete the sandbox with ampx and create it again. Run this from the project whose sandbox is broken, before you rename anything:

npx ampx sandbox delete
npx ampx sandbox

If the broken sandbox was started with a name, pass the same name to the delete command, for example npx ampx sandbox delete --identifier blog.

This removes the data in that sandbox. The team is direct about it in the thread: sandboxes are meant to be ephemeral, and the setup docs already describe cases where Amplify drops and recreates tables or user pools during normal sandbox work. If you need test data in the new sandbox, the seed feature lets you script it and load it with npx ampx sandbox seed. The reporter resolved the issue the same way, by deleting the sandbox and creating a new one.

Preventing It: One Name per Project, One Identifier per Sandbox

The lasting fix is a unique name in every project's package.json, set before its first sandbox deploy. For the second project in this example, change "name": "aws-amplify-gen2" to "name": "blog-app".

Changing the name changes the stack name, so Amplify treats the project as a new sandbox. The old stack stays in your account until you delete it, which is why the recovery step comes first.

When you want more than one sandbox for the same project, use an ampx sandbox identifier:

npx ampx sandbox --identifier feature-auth

A few details matter once you start using identifiers:

• The CLI accepts letters, numbers, and hyphens, up to 15 characters.
• Secrets are stored per sandbox. According to the sandbox features docs, a named sandbox needs its own values, set with npx ampx sandbox --identifier feature-auth secret set <name>.
• You can switch between named sandboxes, but only one can run at a time.
• Each deploy writes amplify_outputs.json to the project root, so your frontend talks to whichever sandbox deployed last.

The same docs recommend one sandbox per developer per app, since multiple sandboxes make it harder to track which code is deployed where.

Final Thoughts

The API key in the error message is where the deploy failed. The cause happened earlier, when two projects resolved to the same sandbox name. Amplify builds that name from the package.json name and your username, and the folder you run the command in plays no part.

If you only ever work on one Amplify project, the default name is fine. The check matters on the day you start a second project from the same template, and it takes one line in package.json.

FAQ

What does “NotFoundException: API key not found” mean on a sandbox redeploy?

When you see the ampx sandbox api key not found error, CloudFormation is trying to update an AppSync API key that no longer exists. In the reported case, it followed two projects deploying to the same sandbox stack.

Will npx ampx sandbox delete remove my data?

Yes. It deletes the resources in that sandbox, including its tables. Use the seed feature if you need test data again.

What is a sandbox identifier?

A name you pass with --identifier. It replaces your username in the sandbox stack name, so one project can have several separate sandboxes. It accepts letters, numbers, and hyphens, up to 15 characters.

Next in this series: sandbox hot reload failing with an “already exists” error, and how to recover without starting over.

“API Key Not Found” on Sandbox Redeploy: Using Sandbox Identifiers was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 4 of 5 · SageMaker Without an ML Team. The full series is listed at the end.

Imagine a teacher who sets a final exam, but some of the questions are copied straight from the homework. A student who memorized the homework gets a great score. It looks like they learned the subject. They didn’t. They just remembered the answers.

That’s what happened to one of our image models on AWS SageMaker. The score came back better than expected. Nobody investigates good news, so nobody looked. It turned out the model was being tested on photos it had already trained on.

Quick background: train and test photos

When you train a model, you split your photos into two groups:

• Training photos: the model learns from these.
• Test photos (also called validation): kept aside to check how well the model does on photos it has never seen.

If test photos leak into training, the score stops measuring learning and starts measuring memory. It goes up, and it’s wrong.

How we split our photos

We shuffled the list of photos in a fixed, repeatable way (using a “seed”, so the same list always shuffles the same way), then took the first 80% for training and the rest for testing.

Same list in, same split out every time. It looked perfectly reproducible.

What went wrong?

The shuffle is only repeatable if the list is exactly the same. Add even one photo, and the whole shuffle comes out different, because the length of the list affects every step of the shuffle.

When we added ten new photos, we measured what happened: about 80% of the old test photos moved into the training group. Same seed, same code. A completely different split.

Why that was a big problem for us

If you train every model from scratch, a changing split is just untidy. Each model starts fresh, so it hasn’t seen any photos before.

But to save time, we often trained a new model on top of the previous one. The previous model had already learned from most of our photos. When the split reshuffled, many of the new model’s test photos were ones the previous model had trained on.

So the new model was taking an exam with the homework questions on it. Its score looked great. And each new model was then compared against that inflated score, so the problem carried forward.

The 4-line fix

Instead of shuffling a list, we let each photo decide its own group based on its filename.

We turn the filename into a number (a “hash”), then use that number to place the photo:

function isTrainSide(filename: string, ratio: number, seed: number): boolean {
  return fnv1a(filename, seed) % 10000 < ratio * 100;
}

In plain English: every photo gets a fixed number between 0 and 9,999. If you want 80% for training, photos numbered below 8,000 go to training, and the rest go to testing.

Now adding new photos doesn’t move any existing photos. Each photo’s group depends only on its own name.

The small trade-off: you get roughly 80%, not exactly 80% (say 80.05%). So before training starts, we check that neither group came out empty.

A twist we didn’t expect

Because each photo has a fixed number, changing the training percentage only works safely in one direction:

• Raising it (80% → 90%) is safe. The photos still held out for testing were also held out before.
• Lowering it (90% → 80%) brings the leak back. Photos numbered 8,000 to 8,999 were training photos for the previous model, and now they’d be test photos.

So when you build on an earlier model, our system now refuses to lower the training percentage and explains why.

What’s still not fixed

Our cameras take photos a few seconds apart, so two photos can be almost identical. If one lands in training and its near-twin lands in testing, that’s still a small leak. Grouping photos by the moment they were taken, instead of by filename, would fix it. We haven’t done that yet.

Your checklist for a split that doesn’t leak

1. Does adding new photos move existing ones between groups? If so, split by a hash of each photo, not a shuffle.
2. Do you train new models on top of old ones? Then a leaky split will make your scores look too good.
3. Do you lower the training percentage between runs? Don’t, if you’re building on an earlier model.
4. Are near-identical photos kept in the same group?
5. Do you double-check a score that’s better than expected?

Final thoughts

A mistake that makes your score look worse gets noticed the same day. A mistake that makes it look better can hide for weeks, because everyone is happy with the result. When a model suddenly scores higher than it should, that’s the time to check your split.

Next in this series: how we moved our whole SageMaker setup to another AWS region with one small change.

References

• Train and validation data for SageMaker image classification
• Incremental training in Amazon SageMaker

If this helped, give it a clap and follow for more practical breakdowns on cloud and AI.

Read the series: SageMaker Without an ML Team

1. 5 Mistakes That Wasted Our Money on AWS SageMaker
2. Your ML Experiment Just Became Everyone’s Default
3. Life After SageMaker Ground Truth
4. Your Train/Test Split Is Probably Leaking (you are here)
5. How We Moved Our Whole SageMaker Setup to Another AWS Region

Your Train/Test Split Is Probably Leaking (and Why a Good Score Can Be Lying) was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 3 of 5 · SageMaker Without an ML Team. The full series is listed at the end.

Imagine hiring an events company to hand out name tags at a dinner for five friends. They’d do a great job, but you’d be paying for staff, scheduling, and coordination you never needed. You could just write the name tags yourself.

That’s how SageMaker Ground Truth, AWS’s service for labeling training images, felt for our small team. We stopped using it early in 2026. Then AWS closed it to new customers on 30 July 2026 and shut it down completely on 30 September 2026, alongside Mechanical Turk. Here’s why it didn’t fit us and the much simpler setup that replaced it.

What Ground Truth does

To train an image model, someone has to look at each photo and say what’s in it. That’s labeling. Ground Truth helped you manage that work: it found or organised the people doing the labeling, sent them photos, checked their quality, and combined their answers when several people labeled the same photo.

That’s genuinely useful if you need hundreds of thousands of photos labeled once by people you don’t employ.

Why it didn’t fit us

Our situation was the opposite:

• Our labelers were our own colleagues. No outside workforce to manage.
• Nobody labeled the same photo twice, so there were no answers to combine.
• New photos arrived every day in small batches. The real job was “label last night’s 40 photos," a five-minute task, not a big labeling project.

Ground Truth is built around big, one-off labeling jobs. Every batch meant setting up a new job, waiting, and checking it the next day. And you can’t edit a job, only copy it. When someone mistyped a job name, the only fix was to stop it and start a new copy. The typo stayed in the console forever.

Our console ended up with seven jobs that produced two usable datasets. The per-photo fee was small. The real cost was all the setting up, waiting, and redoing.

The surprise: training only needs one file.

When we looked at what Ground Truth actually produced, it was just a text file. One line per photo, saying where the photo is stored and which labels it has:

{"source-ref": "s3://my-bucket/images/img_0417.jpg", "my-labels": [0, 3, 7]}

SageMaker’s training never talks to Ground Truth. It only reads this file. (The real file has a few more housekeeping fields, which the training ignores.) So once you have your own labelers, you don’t need the service. You just need to produce the file.

What we built instead

The file is only one piece. Replacing ground truth took four:

• A labeling screen where our team opens a photo, ticks labels, and saves. The biggest piece.
• A file writer that turns saved labels into that one file. About an afternoon of work.
• A train/test split that decides which photos are for training and which are for testing. Harder than it looks (that’s a later post).
• A missing-photo check before training starts, so one deleted photo doesn’t crash the job (see Part 1).

Our labeling tool, in four screens

The tool follows the life of a photo: Unlabeled → Labeled → Train → Stats. A new photo arrives with no labels, someone labels it; it’s used in training; and the results show up in the stats. Every screen stays clickable, so you can check the stats first and then go label whatever’s missing.

AI suggestions, with two rules

We added AI suggestions that propose labels for a photo. To keep the data trustworthy, two rules are written at the top of that part of the code:

• The AI only runs when you ask it to. Opening the screen costs nothing.
• Nothing is saved without a person clicking Accept.

There is an “Accept All” button, so a tired person could accept 50 suggestions without looking closely. To catch that, every accepted label records one of two values: accepted as suggested or edited before accepting.

That one small detail tells us how good the AI really is. If people edit a lot of suggestions, the AI isn’t good enough yet. If someone accepts hundreds with zero edits, they probably weren’t looking.

What we gave up

Our old, rough labeling tool had keyboard shortcuts: number keys for labels and Enter to save and move on. The new tool is mouse-only. It’s much better in almost every way but slower per photo. We chose trustworthy data over speed per photo, and keyboard shortcuts are on the list.

If you’re replacing a labeling service

1. Do you employ your labelers? If yes, you probably don’t need a workforce service.
2. What file does your training actually read? Produce that file yourself.
3. Does your labeling tool make it easy to label small batches every day?
4. If you use AI suggestions, does a person always approve them?
5. Do you record whether the AI’s suggestion was edited? That’s how you’ll know if it’s any good.

Final thoughts

Ground Truth was a good product for big, one-off labeling projects with outside workers. For a small team labeling a few photos every day, it was mostly overhead. The training only ever needed one file, and producing that file ourselves turned out to be the easy part.

Next in this series: why your train/test split might be quietly leaking, and how that makes a model look better than it is.

References

• Amazon is shutting down Mechanical Turk and SageMaker Ground Truth (Encord).
• Amazon closes Mechanical Turk on September 30 (Let’s Data Science)
• Augmented manifest files for SageMaker training

If this helped, give it a clap and follow for more practical breakdowns on cloud and AI.

Read the series: SageMaker Without an ML Team

1. 5 Mistakes That Wasted Our Money on AWS SageMaker
2. Your ML Experiment Just Became Everyone’s Default
3. Life After SageMaker Ground Truth (you are here)
4. Your Train/Test Split Is Probably Leaking
5. How We Moved Our Whole SageMaker Setup to Another AWS Region

Life After SageMaker Ground Truth: How We Replaced AWS Labeling With One File and Our Own Tool was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.

AWS Lambda environment variables are a convenient way to keep configuration outside your application code. But as a function grows and depends on more services, those values can eventually hit Lambda’s 4 KB total environment-variable limit.

When that happens, the better solution is usually not to keep removing or shortening variables. It is to reconsider where that configuration should live.

One practical option is AWS Systems Manager Parameter Store.

Why the 4 KB Limit Becomes a Configuration Problem

A Lambda function may start with only a few environment variables, but the list often grows as the application becomes more complex. It may eventually need database names, API endpoints, S3 bucket names, queue URLs, service configuration, feature settings, and environment-specific values.

For example:

DATABASE_NAME
API_BASE_URL
BUCKET_NAME
QUEUE_URL
SERVICE_URL
SERVICE_KEY
LOG_LEVEL
OTHER_CONFIGURATION

The important detail is that Lambda does not provide 4 KB for each variable. The combined size of all environment-variable names and values must fit within 4 KB.

This means a function can work normally for a long time and then suddenly fail during deployment after only one or two new values are added. The application code may be completely valid; the problem is simply that the function configuration has grown beyond what Lambda allows.

Removing an unused variable or shortening a value might temporarily make the deployment work again, but that does not really solve the problem. If the application continues to grow, the same limit is likely to appear again.

At that point, the issue is no longer just about fitting another value into Lambda. It becomes a configuration design question: does every configuration value really need to live directly inside the function?

Small values that are closely tied to the function can still remain as environment variables. But larger values, shared configuration, or settings that need to be managed centrally can be moved outside Lambda.

This is where AWS Systems Manager Parameter Store becomes useful. Instead of storing every full value inside the Lambda environment, the function can keep only a small reference and retrieve the actual configuration when it needs it.

Where AWS Systems Manager Fits

AWS Systems Manager is a broader AWS operations service, not just a configuration store. It includes several tools for managing AWS and hybrid environments.

For example, Session Manager can provide controlled access to managed machines without relying on traditional SSH connections. Run Command can remotely execute commands, Patch Manager helps manage patching, State Manager helps keep systems in a desired configuration, and Automation can run operational workflows.

For the Lambda environment-variable problem, the relevant part is Parameter Store. Parameter Store provides a central place to keep configuration values such as service endpoints, resource identifiers, environment settings, and other application parameters. Instead of copying every full value into the Lambda environment, the function can retrieve the values it needs from Parameter Store at runtime.

A Lambda environment could therefore contain only a small configuration reference such as:

CONFIG_PATH=/myapp/prod

while the actual values live in Parameter Store:

/myapp/prod/service-url
/myapp/prod/service-key

This reduces pressure on the Lambda environment-variable limit and also gives the configuration a more organized structure.

Moving Configuration from Lambda to Parameter Store

Suppose an application needs two values. One is a normal service URL, while the other contains sensitive configuration. Parameter Store allows them to be handled differently:

/myapp/prod/service-url
/myapp/prod/service-key

The service URL can be stored as a normal String, while the sensitive value can be stored as a SecureString.

The same path structure can also be used to separate environments:

/myapp/dev/service-url
/myapp/staging/service-url
/myapp/prod/service-url

This makes configuration easier to manage because development, staging, and production values don't need to be mixed together. More importantly, the Lambda no longer needs to hold every full value in its own environment. It only needs enough information to locate the correct parameters and permission to retrieve them.

Understanding Parameter Types and Parameter Tiers

One of the easiest things to confuse in Parameter Store is the difference between a parameter type and a parameter tier.

The parameter type describes how the value is stored. Parameter Store supports String, StringList, and SecureString. A normal String is suitable for ordinary configuration such as an endpoint URL, while SecureString is used when a value needs encryption with AWS KMS.

The parameter tier is a separate decision. The main tiers are Standard and Advanced, and they affect the size limits, available features, quotas, and cost.

This means these combinations are all possible:

• Standard String
• Standard SecureString
• Advanced String
• Advanced SecureString

The key point is that SecureString does not mean Advanced. A sensitive value can still be stored as a Standard SecureString.

For many Lambda configuration use cases, Standard parameters are enough. A Standard parameter supports up to 4 KB per parameter, while Lambda provides 4 KB for the entire collection of environment variables.

Advanced parameters increase the maximum size to 8 KB per parameter and provide additional features such as parameter policies and larger parameter quotas. They also have additional charges.

For a Lambda that only needs to move a few configuration values out of its environment, Standard parameters are often the simpler choice.

How SecureString and AWS KMS Work Together

A normal String parameter suits configuration that doesn't require additional protection. For sensitive values, Parameter Store provides SecureString, which works with AWS Key Management Service (KMS).

The two services have different roles. Parameter Store manages the parameter, while KMS performs the encryption and decryption operations used to protect SecureString values.

When Parameter Store creates a Standard SecureString, it uses a symmetric KMS key to encrypt the value before storing it. Later, when an authorized application requests the parameter with decryption enabled, Parameter Store works with KMS to decrypt it before returning the value.

A customer managed KMS key can also be used when stronger access control is required. This makes it possible to control more precisely which applications, users, or IAM roles are allowed to decrypt values protected by that key.

The choice depends on the security requirements. The AWS managed key is convenient for straightforward use cases, while a customer managed key provides greater control when different applications need clearly separated decryption permissions.

Lambda Still Needs IAM Permission

Moving configuration to Parameter Store does not automatically give a Lambda function access to it. The function uses its IAM execution role when requesting the parameters, so the role needs the required permissions.

For example, the Lambda may need:ssm:GetParameter or ssm:GetParameters . If a SecureString uses a customer managed KMS key, the execution role may also need:kms:Decrypt

Keep these permissions as narrow as possible. A production function, for example, could be allowed to read only parameters under:/myapp/prod/* , while a staging function could be limited to:/myapp/staging/* . This provides a cleaner security boundary than giving every Lambda permission to read every parameter in the account.

Retrieving Parameters Without Adding Unnecessary Latency

Lambda can retrieve Parameter Store values through the AWS SDK. A simplified Node.js example looks like this:

import {
  SSMClient,
  GetParametersCommand,
} from "@aws-sdk/client-ssm";


const ssm = new SSMClient({});
async function loadConfig() {
  const response = await ssm.send(
    new GetParametersCommand({
      Names: [
        "/myapp/prod/service-url",
        "/myapp/prod/service-key",
      ],
      WithDecryption: true,
    })
  );
  const config = {};
  for (const parameter of response.Parameters ?? []) {
    config[parameter.Name] = parameter.Value;
  }
  return config;
}

The WithDecryption: true option tells Parameter Store to return decrypted values for SecureString parameters when the caller has the required permissions.

One important difference exists between environment variables and Parameter Store. Reading an environment variable happens locally inside the Lambda runtime, while retrieving a Parameter Store value normally involves an AWS API request.

For configuration that rarely changes, making that request on every Lambda invocation is unnecessary. The values can often be loaded once and cached in memory while the Lambda execution environment remains warm.

let cachedConfig;

export const handler = async (event) => {
  if (!cachedConfig) {
    cachedConfig = await loadConfig();
  }
  // Use cachedConfig
};

AWS also provides the Parameters and Secrets Lambda Extension, which can retrieve and cache Parameter Store and Secrets Manager values locally. For small functions, simple in-memory caching may be enough, while the extension becomes more useful when configuration access becomes more frequent or complex.

Cost and When Secrets Manager Makes More Sense

For this Lambda configuration problem, Standard Parameter Store is usually the cheapest option. Standard parameters have no additional storage charge, and applications using the default standard throughput can access them without additional Parameter Store API charges.

There is an important difference between Lambda’s 4 KB limit and Parameter Store’s 4 KB limit. In Lambda, the combined size of all environment-variable names and values must fit within 4 KB. With Standard Parameter Store, each individual parameter can contain a value up to 4 KB.

For example, these can all be separate Standard parameters:

/app/service-url      → up to 4 KB
/app/database-name    → up to 4 KB
/app/service-key      → up to 4 KB

These parameters do not share one 4 KB space. Each parameter has its own maximum value size. This is why moving larger configuration values out of Lambda can help: instead of forcing all configuration into Lambda’s single 4 KB environment-variable limit, the values can be stored separately in Parameter Store.

Suppose a Lambda moves two values into Parameter Store: a service URL stored as a String and a sensitive value stored as a SecureString. If both use the Standard tier and the application stays within normal throughput, there is no additional Parameter Store storage charge or standard-throughput API charge. For a small configuration like this, the Parameter Store cost itself can effectively be $0.

Advanced parameters are different. They support values up to 8 KB per parameter, compared with 4 KB for Standard parameters, and also provide features such as parameter policies, higher parameter quotas, and cross-account sharing. Advanced parameters have additional charges, so they are more useful when those extra capabilities are actually required rather than simply because a value needs encryption.

SecureString introduces another part of the cost because it uses AWS KMS for encryption and decryption. When Parameter Store uses the AWS managed AWS/SSM KMS key, there is no monthly storage charge for that AWS managed key. KMS request charges can still apply depending on usage, and AWS currently provides a free tier for eligible KMS requests. A customer managed KMS key, on the other hand, introduces a key charge as well as applicable request charges.

This is also where AWS Secrets Manager becomes worth considering. Parameter Store is well suited to application configuration and relatively static encrypted values, while Secrets Manager is designed more specifically for credentials such as database passwords, API keys, OAuth tokens, and other secrets that may need features such as automatic rotation and stronger secret lifecycle management. AWS charges Secrets Manager based on the number of secrets stored and the API requests made to the service.

For the Lambda 4 KB problem discussed here, Standard Parameter Store is usually enough when the goal is simply to move growing application configuration outside the Lambda environment. Advanced Parameter Store becomes useful when larger values or additional Parameter Store features are required, while Secrets Manager is generally the better choice when the value is a real credential that needs automatic rotation or more dedicated secret management.

The Better Way to Think About the 4 KB Limit

The Lambda 4 KB environment-variable limit may look like a simple quota issue, but it can also signal that application configuration is outgrowing the function itself.

Environment variables are still useful for small, function-specific values. But as configuration grows, AWS Systems Manager Parameter Store provides a cleaner way to manage those values outside Lambda.

For most cases, String works for normal configuration, SecureString KMS can protect sensitive values, and Standard parameters are usually enough unless you need advanced features. IAM controls which functions can retrieve the parameters, while caching helps avoid unnecessary requests.

The goal is not to replace environment variables completely. It is to use them where they make sense and move growing configuration to a service designed to manage it.

Connect with Me

LinkedIn: https://www.linkedin.com/in/pathumscj

How to Fix the AWS Lambda 4 KB Environment Variable Limit was originally published in AWS Fullstack on Medium, where people are continuing the conversation by highlighting and responding to this story.