[{"content":"","date":"1 August 2026","permalink":"/tags/aws/","section":"Tags","summary":"","title":"AWS"},{"content":"","date":"1 August 2026","permalink":"/tags/lambda/","section":"Tags","summary":"","title":"Lambda"},{"content":"","date":"1 August 2026","permalink":"/tags/microvm/","section":"Tags","summary":"","title":"MicroVM"},{"content":"","date":"1 August 2026","permalink":"/posts/","section":"Posts","summary":"","title":"Posts"},{"content":"","date":"1 August 2026","permalink":"/","section":"Pubudu.Dev","summary":"","title":"Pubudu.Dev"},{"content":"Every application that accepts user uploads needs to answer one common question: is the uploaded file safe to process? In order to verify that, you may use an antivirus engine to scan the uploaded file. One of such well known, free and open source antivirus engines is ClamAV.\nRunning ClamAV in a serverless setup can be difficult because the virus signature database is over 200 MB and it is being updated regularly. So, keeping the virus signature database up to date for each scan while reducing latency can be challenging.\nAlso, file scanning should happen in an isolated environment, ideally a single disposable environment per scan. This way, once the file scan is done, those resources can be thrown away safely without impacting the rest of the system.\nAWS Lambda MicroVMs, address both these problems nicely. In this blog post, I discuss why a MicroVM is a good fit for this kind of jobs, walk through a workflow I built that scans user uploaded S3 files with ClamAV running inside a MicroVM, and show how to keep the virus signature database up to date.\nWhy MicroVM is the right fit # Before MicroVMs, there were two most suitable ways to run ClamAV on AWS. You could squeeze it into a Lambda function, and accept the cold start penalty of fetching and loading the database, the packaging effort, and the 15 minute execution cap. Or you could run clamd as an service on ECS or EC2, and accept paying including the idle time, one shared daemon scanning everyone\u0026rsquo;s files, and another piece of infrastructure you need to manage yourself.\nA MicroVM takes the good parts of both the approaches:\nA disposable VM per file. Malware scanning means feeding untrusted input to a software component, so you have to assume the scanner itself can be compromised by a crafted file. Regular Lambda also runs on Firecracker, but execution environments are reused across invocations: a file that exploits the scanner could persist in the warm environment and quietly tamper with the scans that follow it. For example, it can report the infected files as clean. With MicroVM approach, every file gets a fresh VM behind a hardware virtualization boundary, with an execution role that can only read the upload bucket and emit result events, and the VM is terminated right after the scan. The blast radius of the worst case is limited to a single disposable VM that has already done its job.\nThe snapshot removes the latency problem. When the MicroVM image is built, the platform runs the container, waits until the application reports ready, and snapshots its memory and disk. Here, I run freshclam during the image build, so the virus signature database files are baked into the snapshot\u0026rsquo;s disk and nothing is downloaded when a scan runs, and the VM boots with the database already in place. Each scan still spawns clamscan, which loads the database from the snapshot\u0026rsquo;s disk into memory before scanning. But that time duration is trivial comparing to the latency of downloading virus signature database each time per scan.\nPackaging simplicity. A MicroVM image is defined by a very simple plain Dockerfile, and here I use the official clamav/clamav as the base image. There is no runtime interface clients or Lambda layers. The only addition is a small Flask application that runs the scans.\nServerless benefits. A VM is launched per upload and it start fast. Also it terminates itself when the scan is done. So you pay only while it runs. And with the options to configure vCPUs, memory and the running time up to 8 hours, it provides both power and simplicity of serverless.\nArchitecture # Image: Architecture overview This is the high level architecture of the virus scanning flow. There are two main parts of the system: file scanning flow and the functionality to keep the virus signature database up to date.\nFile scanning flow # A user uploads a file to the upload bucket, which has EventBridge notifications enabled. An Object Created rule on the default event bus triggers a small orchestrator Lambda function. This extra step exists because as at now, a MicroVM cannot be an EventBridge rule target directly. Something has to call the RunMicrovm API, and here a Lambda function is the middleman facilitates that. The orchestrator calls RunMicrovm, passing the bucket and key as the run hook payload. As at now, default boto3 in the Lambda runtime doesn\u0026rsquo;t support MicroVM API. So you have to bundle boto3 version 1.43.0 or higher into this function. The platform launches a MicroVM from the ClamAV image snapshot, the application downloads the object, runs clamscan, and publishes a FileScanned event with the result CLEAN, INFECTED or ERROR to a custom event bus. A rule on the custom bus writes the results to CloudWatch Logs. In a real application, you would route them to whatever handles quarantining or further processing of that file. Inside the MicroVM # I use the MicroVM non-interactive way to scan the files. Which means the application logic triggers using the lifecycle hooks. Here are the MicroVM hooks I utilised in this application:\nThe /ready hook is called during the image build. It verifies if the virus signature database is already exists within the MicroVM. It returns 503 and fail if the ClamAV database files not exist, which guarantees the snapshot always contains the virus signature database.\nThe /validate hook runs after the snapshot is taken, on a test MicroVM. I scan the EICAR test string there, which validates if the image actually detects malware before it ever scans actual files.\nThe /run hook is where the actual scan of the file happens. The payload passed to RunMicrovm is delivered as the body of this hook. The application kicks off the scan on a background thread and returns immediately. The scan downloads the object from S3, runs clamscan, and publishes the results to EventBridge. This also includes the matched signature name, the database version and build time read straight from the snapshot. MicroVM execution role provides all the permissions required to talk to the services in use: S3, EventBridge and CloudWatch.\nI have below idle policy for the MicroVM.\n{ \u0026#34;maxIdleDurationSeconds\u0026#34;: 60, \u0026#34;suspendedDurationSeconds\u0026#34;: 0, \u0026#34;autoResumeEnabled\u0026#34;: false } Once the run hook is completed, MicroVM will be suspended after 60 seconds (This is the minimum time allowed). After that, will be terminated immediately.\nKeeping the signatures database fresh # Baking the database into the snapshot is what makes the scan fast, but it comes with an obvious downside. Virus signatures age with the image. An antivirus scanner running week-old signatures is not much of useful. One solution is to download the virus signature database at the file scan time, but it is not the best approach because it adds significant latency and, since each file scan use independent execution environments, there is no way to reuse the downloaded signature database.\nThe better solution is to rebuild the MicroVM image with the latest virus signature database. There is an EventBridge Scheduler invokes an image builder Lambda function every hour. Within that Lambda function it run UpdateMicrovmImage where it starts a new image build using the Dockerfile provided which includes freshclam. freshclam is the official cli tool and daemon used to download and update malware and virus signature databases for the ClamAV antivirus engine. So an hourly Lambda function that updates the scanner image produces a new active version of the image with the latest virus signatures. The next file scan after a rebuild is scanned within a MicroVM that use the latest image so that the latest database will be used.\nRebuilding image with update_microvm_image is quite easy, because we can reuse the configurations from existing image including the S3 code artifacts. And Lambda service will simply take care of the build process and we don\u0026rsquo;t need to handle it.\nRebuilding only when virus database has new updates # A full image build and snapshot is expensive, and ClamAV does not publish new signatures every hour. So before rebuilding the MicroVM image, the function checks whether it needs to.\nClamAV publishes its current signature record as a DNS TXT record at current.cvd.clamav.net, which is the same source freshclam itself uses. The record is a colon-separated string like 1.4.5:63:28072:1785016800:1:90:49192:339, and field index 3 is the UNIX timestamp at which the latest virus database was published.\nIf that timestamp falls inside the last hour, the function starts building the new image. Otherwise it ignores the update. However, if the DNS lookup fails for any reason, function will anyway build the image as a fall back.\nRemoving old image versions # Every rebuild creates an image version, and every version holds a snapshot you pay storage for whether or not anything is running on it. As at now, Lambda does not provide any functionality to automatically expire the old images. So, we have to manage it ourselves.\nWhat I did here was fetching all the image versions available and order them by the created at value and delete one by one using API DeleteMicrovmImageVersion except two most recent image versions.\nPlease note: When there is a image build in progress, the DeleteMicrovmImageVersion API will be rejected for every version of that image, not just the one being built. It is an image level lock.\nBecause of this reason, I run the deletion of old versions before start building the new version.\nTry this yourself # The full code is available at github.com/pubudusj/microvm-file-scan.\nClone the repository and install the dependencies: git clone https://github.com/pubudusj/microvm-file-scan cd microvm-file-scan python3 -m venv .venv \u0026amp;\u0026amp; source .venv/bin/activate pip install -r requirements.txt Deploy both stacks: cdk deploy --all The first deploy takes a few minutes because CloudFormation builds and snapshots the MicroVM image.\nPlease note: As at now, Lambda MicroVMs is available in a limited set of regions. Make sure you deploy the stacks to one of them.\nTesting # Once the project is deployed into your AWS account, there will be several outputs. To test the functionality, first upload a file to the bucket name shown in the stack outputs:\necho \u0026#34;hello\u0026#34; \u0026gt; hello.txt aws s3 cp hello.txt s3://\u0026lt;UploadBucketName\u0026gt;/ Shortly after, a FileScanned event with status: CLEAN appears in the results log group created by the stack. Sample event looks like below:\n{ \u0026#34;version\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;994e766e-cf1b-cba6-2bae-21f9c658620b\u0026#34;, \u0026#34;detail-type\u0026#34;: \u0026#34;FileScanned\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;file.scanner\u0026#34;, \u0026#34;account\u0026#34;: \u0026#34;xxxxxxxxxx\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2026-07-29T12:59:33Z\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;eu-west-1\u0026#34;, \u0026#34;resources\u0026#34;: [], \u0026#34;detail\u0026#34;: { \u0026#34;bucket\u0026#34;: \u0026#34;microvmfilescanstack-uploadbucketxxxxxxxx-xxxxxxxxxxxx\u0026#34;, \u0026#34;key\u0026#34;: \u0026#34;hello.txt\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;CLEAN\u0026#34;, \u0026#34;signature\u0026#34;: null, \u0026#34;dbVersion\u0026#34;: \u0026#34;28076\u0026#34;, \u0026#34;dbTimestamp\u0026#34;: \u0026#34;29 Jul 2026 06:25 +0000\u0026#34;, \u0026#34;scannedAt\u0026#34;: \u0026#34;2026-07-29T12:59:33.861586+00:00\u0026#34; } } To see a detection, upload the harmless, industry standard EICAR test file:\nprintf \u0026#39;X5O!P%%@AP[4\\\\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*\u0026#39; \u0026gt; eicar.com aws s3 cp eicar.com s3://\u0026lt;UploadBucketName\u0026gt;/ The resulting event looks like this:\n{ \u0026#34;version\u0026#34;: \u0026#34;0\u0026#34;, \u0026#34;id\u0026#34;: \u0026#34;b0c38807-a8e7-90e3-031c-20c97ff5e0f5\u0026#34;, \u0026#34;detail-type\u0026#34;: \u0026#34;FileScanned\u0026#34;, \u0026#34;source\u0026#34;: \u0026#34;file.scanner\u0026#34;, \u0026#34;account\u0026#34;: \u0026#34;xxxxxxxxxx\u0026#34;, \u0026#34;time\u0026#34;: \u0026#34;2026-07-29T13:01:40Z\u0026#34;, \u0026#34;region\u0026#34;: \u0026#34;eu-west-1\u0026#34;, \u0026#34;resources\u0026#34;: [], \u0026#34;detail\u0026#34;: { \u0026#34;bucket\u0026#34;: \u0026#34;microvmfilescanstack-uploadbucketxxxxxxxx-xxxxxxxxxxxx\u0026#34;, \u0026#34;key\u0026#34;: \u0026#34;eicar.com\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;INFECTED\u0026#34;, \u0026#34;signature\u0026#34;: \u0026#34;Eicar-Test-Signature\u0026#34;, \u0026#34;dbVersion\u0026#34;: \u0026#34;28076\u0026#34;, \u0026#34;dbTimestamp\u0026#34;: \u0026#34;29 Jul 2026 06:25 +0000\u0026#34;, \u0026#34;scannedAt\u0026#34;: \u0026#34;2026-07-29T13:01:40.882464+00:00\u0026#34; } } Notice the signature, dbVersion and dbTimestamp fields. Every result records exactly which signature set produced it, and dbTimestamp and dbVersion are a result of the hourly build process.\nPlease note: When you create the eicar.com test file in your computer, your anti-virus program will most probably immediately detect and quarantine it (even it is not harmful). Instead of creating this locally, you can open up a cloudshell session in your AWS console and create and upload the test file from there.\nLimitations # The virus database refresh is only as fine grained as the schedule. The interval between attempts to rebuilding the image can be set as per your need, because actual image building will happen only when there is a virus update available.\nThis design launches one MicroVM per upload, which is great for isolation but adds a small launch overhead per file. For high volume workloads, you could reuse a VM for a batch of files but you will loose per-file isolation.\nBe aware that some of the API operations have lower transactions per second ex: RunMicrovm, SuspendMicrovm, TerminateMicrovm. And these rate limits are per account, per region which means if you have other MicroVMs running for other purposes in the same region, you will likely to be throttled. Refer https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#microvms-quotas for more information.\nSummary # For a requirement where you have to scan user uploaded files for malware, it is better to perform it in a disposable isolated environment. It is exactly the kind of job MicroVMs were built for with the vm-level isolation and serverless operational model.\nIn this project I use ClamAV an free and open source antivirus engines, but you can use any similar products suits your the best.\nMicroVM is really fast to start, however, downloading virus signature database every time before a file scan will not be efficient. Baking the virus signature into the MicroVM image is the solution. However, it is necessary to have a workflow that keeps the MicroVM image up to date in regular intervals with the latest virus signature database.\nResources # Lambda MicroVMs developer guide: https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html Lambda MicroVMs API reference: https://docs.aws.amazon.com/lambda/latest/microvm-api/Welcome.html AWS Lambda MicroVMs announcement blog: https://aws.amazon.com/blogs/aws/run-isolated-sandboxes-with-full-lifecycle-control-aws-lambda-introduces-microvms/ ClamAV: https://www.clamav.net ClamAV private mirror / signature distribution notes: https://docs.clamav.net/manual/Usage/SignatureManagement.html EICAR test file: https://www.eicar.org ","date":"1 August 2026","permalink":"/posts/scan-user-uploaded-files-using-microvm/","section":"Posts","summary":"AWS Lambda MicroVMs are isolated serverless compute environments that provide vm-level isolation. In this post, I discuss a scenario where user uploaded files can be scanned for malware using MicroVM and ClamAV an open-source antivirus engine.","title":"Scan user uploaded files using AWS Lambda MicroVM and ClamAV"},{"content":"","date":"1 August 2026","permalink":"/tags/security/","section":"Tags","summary":"","title":"Security"},{"content":"","date":"1 August 2026","permalink":"/tags/serverless/","section":"Tags","summary":"","title":"Serverless"},{"content":"","date":"1 August 2026","permalink":"/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"","date":"3 June 2026","permalink":"/tags/eventbridge-pipes/","section":"Tags","summary":"","title":"EventBridge Pipes"},{"content":"","date":"3 June 2026","permalink":"/tags/sqs/","section":"Tags","summary":"","title":"SQS"},{"content":"Lambda durable functions, introduced in re:Invent 2025, are a great fit for long-running workflows that need to wait for external events, callbacks, or simply pause between steps. A single durable execution can run up to 1 year, which opens up a lot of use cases that previously required Step Functions or custom orchestration.\nEven though the execution can span up to 1 year, it is better to be clear about what \u0026ldquo;1 year\u0026rdquo; actually refers to. A durable execution is not a single Lambda invocation that stays alive for a year. A durable execution can consist of multiple separate invocations. Between those invocations, the execution can be moved to wait state or go to sleep and you will not be charged for that. Every one of those individual invocations still has to finish within the standard 15 minute Lambda function timeout. And, the total duration of those individual invocations and the wait time must be maximum of 1 year. From the caller\u0026rsquo;s perspective the execution can span hours, days, or up to a year, even though a single invocation must run for maximum 15 minutes.\nImage: Durable execution and its individual invocations However, this is not always true. When you start invoking a durable function from an event source mapping (ESM), such as SQS, Kinesis, or DynamoDB Streams, that total duration of the execution is limited to only 15 minutes. This is including intermediate invocations and the waits.\nIn this blog post, I discuss the solution that AWS provides to work around this 15 minute limitation and another (better?) alternative.\nWhy the 15 minute limit exists # Event source mappings poll the source (SQS, Kinesis, etc.) and invoke the target Lambda function synchronously. The poller holds the connection open and waits for the function to return before it marks records as processed.\nFor a standard Lambda function this is fine, because the maximum function timeout is 15 minutes anyway. But for a durable function, the whole durable execution (steps, waits, callbacks, everything from start to finish) must also fit inside that same synchronous call. So even though the underlying durable execution model supports up to 1 year, the synchronous nature of ESM caps the total execution time at 15 minutes.\nPlease note: When using an ESM based trigger, Durable Functions cannot be configured with an execution timeout longer than 15 minutes. Attempting to do so will cause ESM creation to fail.\nThe recommended workaround: using an intermediary function # AWS recommends solving this limitation by adding a standard Lambda function between the event source mapping and the durable function:\nImage: Using an intermediary function to execute durable function with SQS events. The ESM invokes the intermediary Lambda synchronously. The intermediary Lambda invokes the durable function asynchronously (InvocationType=Event) using the Lambda Invoke API. The intermediary Lambda returns success immediately, and the ESM is happy. And, the durable function then runs on its own, independent of the ESM, for as long as it needs (up to 1 year). The intermediary function is an essential piece here, but now you own and maintain a Lambda function whose only job is to forward an event.\nA cleaner approach: EventBridge Pipes # EventBridge Pipes can be the bridge between an event source and a Lambda target, and the target invocation can be configured as fire-and-forget (asynchronous). That is exactly the behavior we need. The source side polling is decoupled from the durable execution, and there is no intermediary Lambda to write or maintain.\nImage: Using EventBridge Pipe to execute durable function with SQS events. The Pipe polls the SQS queue and invokes the durable function asynchronously. The Pipe considers the message processed as soon as the asynchronous invoke succeeds, not when the durable execution completes. The durable function is then free to run for a longer time.\nTry this yourself # The full code is available at github.com/pubudusj/sqs-to-lambda-durable-function-with-pipes.\nDeploy it to your own AWS account using AWS CDK and Python:\nClone the repository and create a virtual environment: python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt Deploy the stack: cdk deploy Send a test message to the QueueUrl output from the stack: aws sqs send-message \\ --queue-url \u0026lt;QueueUrl\u0026gt; \\ --message-body \u0026#39;{\u0026#34;hello\u0026#34;: \u0026#34;durable\u0026#34;}\u0026#39; Open the durable function in the Lambda console using the LambdaFunctionName stack output and watch the execution. You will see the first print_event_before_wait log entry almost immediately, then the wait continues for 30 minutes, then the print_event_after_wait log entry. The execution completes successfully even though it took twice as long as a standard ESM invoked Lambda would allow. Please note: In both the approaches the Pipe or intermediary function succeeds when the async invoke is accepted, not when the durable execution succeeds. If the durable function fails internally (an exception in a step or a callback times out etc), that failure is visible inside the durable execution history, but the SQS source has already considered the message processing is done. It is advised to build your error handling inside the durable function itself for such failures. Source queue level error handling will manage anything that failed to trigger the Pipe or intermediary function.\nSummary # Durable functions are a new way of building long-running, multi-step workflows on Lambda, but the 15 minute ceiling that appears the moment you put an event source mapping in front of them can be a real constraint. The workaround is to use an intermediary Lambda that asynchronously invokes the durable function, but you have to own and manage it.\nEventBridge Pipes does the same job natively. By configuring the Pipe target with FIRE_AND_FORGET, you get the same decoupling between the synchronous poller of ESM and the long-running durable execution, without owning or managing a Lambda function.\nIf you are reaching for durable functions to handle SQS messages with steps, waits, or callbacks that exceed 15 minutes, using Pipes can be a better option than using an intermediary function.\nResources # Lambda durable functions overview: https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html Event source mappings with durable functions: https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking-esm.html Amazon EventBridge Pipes targets: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-event-target.html ","date":"3 June 2026","permalink":"/posts/lambda-durable-functions-triggered-by-sqs/","section":"Posts","summary":"Lambda durable functions can run up to a duration of 1 year. However, this is different when Lambda durable functions are triggered by SQS. Let\u0026rsquo;s look at the reason and the solutions for this.","title":"Triggering Lambda Durable Functions from SQS"},{"content":"","date":"22 February 2026","permalink":"/tags/saas/","section":"Tags","summary":"","title":"SaaS"},{"content":"Lambda tenant isolation is one of the important security features that came out of the 2025 re:Invent season.\nAchieving tenant isolation in SaaS applications is not straightforward, and taking the single-tenant route to solve it introduces its own scaling challenges. This new feature is not a silver bullet, but it does offer much better support for keeping tenants isolated at scale.\nIn this blog post, I discuss what this feature is and the problems it addresses.\nLambda execution environment # When a invoke request reached AWS Lambda service initially, it starts a virtual environment in a EC2 host worker. We call this an execution environment. An execution environment will download the code and required dependencies and process the request. If required, it will return the response.\nOne of the key attributes of this execution environment is that it will not be removed or deleted immediately after processing a single request. It will be kept in \u0026lsquo;warm\u0026rsquo; state to serve another incoming request. When the next request comes in, Lambda service will use the execution environment that is already available to process it, without creating a new environment.\nLikewise, when you invoke a Lambda function, if there are execution environments available for that Lambda function, Lambda service will use them to process the requests else will create new execution environments.\nHowever, this approach can be a concern when it comes to a multi-tenant Lambda function, because execution environments share \u0026rsquo;left over\u0026rsquo; stuff like:\nGlobal variables Objects initialized outside of the handler files saved in /tmp space Multi-tenant Lambda function # Image: Multi-tenant Lambda function In this multi-tenant setup, a Lambda function is shared by more than one tenant. And based on how execution environments behave, irrespective of the tenant that invoke the Lambda function, Lambda service will use execution environments that are already available for that Lambda function. But, as mentioned earlier, execution environments sharing some data across executions can be a security issue.\nHaving data shared across execution environments can be a great optimization if those data are accessible only by the intended tenant. However, when multiple tenants use the same execution environments, tenants will have access to data that they are not intended to.\nFor example, if we take a single execution environment, tenant 1 might fetch some secrets from Secret Manager or save some files in /tmp directory in its execution. If the same execution environments used for an execution of tenant 2, tenant 2 will have access to the secrets fetched for tenant 1 or the contents of the /tmp directory saved by tenant 1.\nSolution # One of the solution for this problem is to reset the execution environment just before processing each request. For example, unsetting any global variables or wipe the /tmp directory etc. However, this approach will not be practical at scale.\nAnother option is to go for tenant specific Lambda functions which is the single-tenant approach. In this case, tenants will have their own dedicated Lambda function each. This will solve the problem of unintended access to temporary data, because different execution environments belongs to different Lambada functions will not share the execution environments.\nImage: Single-tenant Lambda functions However having Lambda function per tenant is not scalable. When there are a lot of tenants available, you end up with a lot of Lambda functions that you need to manage. While this is possible when you use a IAC tool like CDK or Terraform, still in a situation where you need to update source code to introduce a functionality or a bug fix, you will need to update all these tenant specific resources which is not easy. And, most of the time, those resources are not fully utilized too.\nWhat if we can have a single Lambda function shared by all of the tenant (the multi-tenant approach), yet have the isolation we need in a Lambda per tenant (single-tenant approach)?\nLambda tenant isolation # With Lambda tenant isolation, you can have a single Lambda function shared by all of the tenant, yet have the isolation you need in a Lambda per tenant. With this new feature, Lambda service will do the heavy lifting by creating execution environments that are dedicated to a specific tenant. This means that execution environments will not be shared across tenants, and tenants will not have access to data that they are not intended to.\nImage: Tenant isolated Lambda function But, how Lambda determines the incoming request is from which tenant? For that, we need to provide a tenant-id in the request to Lambda. If you use Lambda invoke cli method, you can use the --tenant-id parameter as follows:\naws lambda invoke \\ --function-name tenant-aware-lambda \\ --payload \u0026#39;{ \u0026#34;name\u0026#34;: \u0026#34;Bob\u0026#34; }\u0026#39; \\ --tenant-id t1 \\ response.json If you use Lambda API, you need to provide the value using X-Amz-Tenant-Id as follows:\nPOST /2015-03-31/functions/tenant-aware-lambda/invocations HTTP/1.1 Host: lambda.eu-central-1.amazonaws.com Content-Type: application/json Authorization: AWS4-HMAC-SHA256 Credential=... X-Amz-Tenant-Id: t1 { \u0026#34;name\u0026#34;: \u0026#34;Bob\u0026#34; } Tenant id is case sensitive and it can be any alpha numeric character with maximum length 256 characters. There are few special characters such as hyphens (-), underscores (_), colon (:), equals (=), plus (+), at (@) and periods (.) allowed too.\nOne of the key attributes of tenant id is that we don\u0026rsquo;t need to pre-register those tenant ids. We can pass any dynamic value as the tenant id and Lambda service will take care of creating and maintaining a pool of execution environments for each value passed. Also, we can use any number of unique tenant ids, there is no limit.\nHow to enable this feature in Lambda # If you use the AWS Console, you can go to the Lambda create wizard and enable this in the additional security section.\nImage: Creating Lambda function in AWS console Also, options for CLI, CloudFormation and CDK also available.\nCLI:\naws lambda create-function \\ --function-name tenant-aware-lambda \\ --runtime python3.14 \\ --zip-file fileb://tenant-aware-lambda.zip \\ --handler index.handler \\ --role arn:aws:iam:123456789012:role/execution-role \\ --tenancy-config \u0026#39;{\u0026#34;TenantIsolationMode\u0026#34;: \u0026#34;PER_TENANT\u0026#34;}\u0026#39; CloudFormation:\nMyLambdaFunction: Type: AWS::Lambda::Function Properties: FunctionName: tenant-aware-lambda Runtime: python3.14 Role: !GetAtt LambdaExecutionRole.Arn Handler: index.handler TenancyConfig: TenantIsolationMode: PER_TENANT Code: ZipFile: | ..... ..... Timeout: 10 MemorySize: 128 CDK:\ntenant_aware_lambda = _lambda.Function( self, \u0026#34;TenantAwareFunction\u0026#34;, function_name=\u0026#34;tenant-aware-lambda\u0026#34;, runtime=_lambda.Runtime.PYTHON_3_13, handler=\u0026#34;index.handler\u0026#34;, code=_lambda.Code.from_asset(\u0026#34;src/lambda/tenant_aware\u0026#34;), timeout=Duration.seconds(10), tenancy_config=_lambda.TenancyConfig.PER_TENANT, ) Please note: This feature can be enabled ONLY when the Lambda function is created. You cannot enable this for an existing Lambda function.\nTry this yourself # I have created a sample application for you to see how this feature works. You can deploy it to your AWS environment using CDK with Python.\nClone the repository at github.com/pubudusj/lambda-tenant-isolation-demo and follow the steps below:\nCreate a virtual environment and activate it: python3 -m venv .venv source .venv/bin/activate Install the dependencies: pip install -r requirements.txt Deploy the stack: cdk deploy This will create two Lambda functions:\nA generic Lambda function (without tenant isolation) A Lambda function with tenant isolation enabled Both Lambda functions have a global variable counter which increments on each invocation. This is to simulate the shared state across executions. An API Gateway is also created with two endpoints to trigger these Lambda functions:\n/execute_generic_lambda?tenant_id=\u0026lt;tenant_id\u0026gt; - invokes the generic Lambda function /execute_tenant_aware_lambda?tenant_id=\u0026lt;tenant_id\u0026gt; - invokes the tenant-aware Lambda function Testing # Generic Lambda function # First, let\u0026rsquo;s test the generic Lambda function. Call the /execute_generic_lambda endpoint with a tenant id:\ncurl \u0026#34;\u0026lt;APIGW_BASE_URL\u0026gt;/execute_generic_lambda?tenant_id=tenant1\u0026#34; You will see a response like:\n{\u0026#34;tenant_id\u0026#34;: \u0026#34;tenant1\u0026#34;, \u0026#34;invocation_count\u0026#34;: 1} Now call the same endpoint again but with a different tenant id:\ncurl \u0026#34;\u0026lt;APIGW_BASE_URL\u0026gt;/execute_generic_lambda?tenant_id=tenant2\u0026#34; You can see the invocation count keeps increasing regardless of the tenant id:\n{\u0026#34;tenant_id\u0026#34;: \u0026#34;tenant2\u0026#34;, \u0026#34;invocation_count\u0026#34;: 2} This is because the generic Lambda function shares execution environments across all tenants. The global variable counter retains its value across invocations regardless of which tenant made the request. This is the exact problem we discussed earlier - any global state, cached data or files in /tmp are accessible across tenants.\nTenant-aware Lambda function # Now let\u0026rsquo;s test the tenant-aware Lambda function. Call the /execute_tenant_aware_lambda endpoint with a tenant id:\ncurl \u0026#34;\u0026lt;APIGW_BASE_URL\u0026gt;/execute_tenant_aware_lambda?tenant_id=tenant1\u0026#34; You will see a response like:\n{\u0026#34;tenant\u0026#34;: \u0026#34;tenant1\u0026#34;, \u0026#34;invocation_count\u0026#34;: 1} Now call the same endpoint again with a different tenant id:\ncurl \u0026#34;\u0026lt;APIGW_BASE_URL\u0026gt;/execute_tenant_aware_lambda?tenant_id=tenant2\u0026#34; This time, the invocation count resets:\n{\u0026#34;tenant\u0026#34;: \u0026#34;tenant2\u0026#34;, \u0026#34;invocation_count\u0026#34;: 1} This is tenant isolation in action. Even though both requests are handled by the same Lambda function, Lambda service creates separate execution environments for each tenant. The global variable counter, or any other shared state in the execution environment, is isolated per tenant. Tenant 2 will never see the state left behind by Tenant 1.\nAlso note that in the tenant-aware Lambda function, we can access the tenant id from the Lambda context object (ex: using context.tenant_id in Python), instead of extracting it from the query parameters. Lambda service automatically makes the tenant id available in the context when tenant isolation is enabled. This is useful if you need to do some operations based on the tenant id - for example, fetch some data from other services.\nIn this example, I have used this tenant id available in the context object to publish a custom CloudWatch metric per tenant. This is helpful to monitor per-tenant invocation patterns, which can be useful for billing or auditing purposes.\nEffect on Lambda concurrency and cold starts # One important thing to understand is how tenant isolation affects Lambda concurrency. Since execution environments are not shared across tenants, each tenant will need their own set of execution environments. This means that the overall number of concurrent execution environments can be higher compared to a non-isolated Lambda function where environments are freely shared.\nFor example, if you have 10 tenants each making concurrent requests, instead of reusing a pool of warm execution environments, Lambda needs to maintain separate pools per tenant. This can lead to more cold starts, especially for low-traffic tenants.\nHowever, this is a trade-off worth making when tenant isolation is critical for your application.\nPlease note: Make sure to consider the Lambda concurrency limits in your account when enabling tenant isolation, especially when dealing with a large number of tenants.\nIntegration with API Gateway # As at now, only integration supports Lambda tenant isolation features is API gateway.\nIn the example project, I have mapped the incoming query string tenant_id to the integration request header X-Amz-Tenant-Id for the Lambda service.\ntenant_aware_integration = apigw.LambdaIntegration( tenant_aware_lambda, request_parameters={ \u0026#34;integration.request.header.X-Amz-Tenant-Id\u0026#34;: \u0026#34;method.request.querystring.tenant_id\u0026#34; }, ) When using Lambda integration on API Gateway, it gives us a lot of flexibility in choosing what value to map as the X-Amz-Tenant-Id. It could be the source AWS account id, a value from the request body, or a header value. If authentication and authorization are enabled, it could even be a Cognito user group or a claim from a JWT token. This makes API Gateway a convenient place to resolve and pass the tenant id to Lambda.\nWhat I\u0026rsquo;d like to see next # While Lambda tenant isolation is a solid step forward, there are a few areas where I think it can be even more valuable.\nMore integration support: Currently only API Gateway supports as a integration to Lambda with this feature. It would be great to see this extended to other integrations such as SQS to Lambda via event source mapping. Native per-tenant metrics: Built-in CloudWatch metrics by tenant id would remove the need to publish custom metrics manually. Per-tenant concurrency controls: The ability to set concurrency limits per tenant would help prevent a noisy tenant from consuming all the available concurrency. Conclusion # Tenant isolation is a fundamental requirement in many SaaS applications. While the single-tenant approach provides strong isolation, it comes with significant operational overhead. The multi-tenant approach is operationally simpler but introduces security risks with shared execution environments.\nLambda tenant isolation gives us the best of both worlds. We get a single Lambda function that is easy to manage and deploy, while Lambda service ensures that the execution environments are isolated per tenant. This eliminates the risk of data leakage across tenants without the burden of managing separate Lambda functions for each tenant.\nThis is a great addition to the serverless toolbox. If you are building multi-tenant SaaS applications on AWS Lambda, this feature is worth exploring.\nResources # AWS Lambda tenant isolation documentation: https://docs.aws.amazon.com/lambda/latest/dg/tenant-isolation.html Launch blog post: https://aws.amazon.com/blogs/aws/streamlined-multi-tenant-application-development-with-tenant-isolation-mode-in-aws-lambda ","date":"22 February 2026","permalink":"/posts/understanding-lambda-tenant-isolation/","section":"Posts","summary":"Lambda tenant isolation lets you share a single Lambda function across tenants while keeping their execution environments completely separate. Let\u0026rsquo;s look at why this matters and how it works.","title":"Understanding Lambda Tenant Isolation"},{"content":"Intro # In AWS re:Invent 2025, Lambda introduced Durable Functions with a great set of features. One of the main features is that a single execution can span up to one year. Also with built-in checkpointing it is possible to track the steps that were already completed and when an execution retried from a resume or an interruption, completed steps will be skipped and resumes from the next step.\nWhen it comes to orchestrating multiple AWS services into a workflow, up until now, AWS Step Functions was the best choice. However, now Durable Functions too offer similar functionality to build a workflow within the same familiar Lambda environment, which is great!\nIn this blog post, I explain how Durable Function\u0026rsquo;s callback feature can be used for a human-in-the-loop functionality.\nDurable Function callbacks # When an execution starts, Durable Function will start an invocation. When there are multiple steps within the execution, it is possible to set the execution on hold (or wait) in a certain step for a given time period or until a signal is received from an external process. In case of a external signal, once the signal is received by the Lambda service, based on the success or failure nature of the signal, the execution that was on hold can be resumed or completely terminated.\nThis is a great feature supported natively by Durable Functions that is helpful for an example, in a business process where a human approval is required to continue with the flow.\nSimple leave management with Durable Functions # In this example, an employee can send a leave request and the manager can approve or reject the request. Below are the steps involved and how Durable Function is being used.\nEmployee sends a leave request. Durable function starts an execution. A Leave is created in a DynamoDB table with pending state. The employee receives an email with the request receipt confirmation. The manager receives an email with a callback id to be used to approve or reject the leave. Durable function keeps the execution on hold until manager approval or rejection is received. Once the manager approves or rejects, the execution resumes. If the manager didn’t process the request within a given time period, the request will be expired. The leave status is updated in the DynamoDB table. The employee receives an email with the manager’s decision or expiry. How it works # Image: Architecture There is a proxy function which accepts and validates the incoming request to create a leave. Here I used a Lambda Function URL to submit the request. Once validated, the proxy function will invoke the durable function. Why use the proxy function?\nEven with Durable functions, the synchronous invocations are limited by 15 minutes. Since I need the durable function to run longer, it is required to trigger the durable function asynchronously. Using the proxy function, durable function is invoked with invocation type ‘Event’, where it just fire-and-forget and will not wait for the durable function to complete.\nWhen the durable function starts its execution, first a leave record is created in the DynamoDB table. Then, an email is sent to the employee confirming the receipt of the leave request. In the next step, an email is sent to the manager asking to approve or reject the leave request. This step is a callback step. The execution waits at this step until the manager approves or rejects. Since this is a callback step, there is a callback id generated. This callback id is indicated in the email to the manager. To simulate the manager’s approval or rejection, there is a Function URL exposed to trigger the Process Leave Lambda function. This function accepts callback_id and the decision (approve or reject) as input. Based on the decision, this function will call the boto3 methods of Lambda send_durable_execution_callback_success or send_durable_execution_callback_failure using the given callback id. Please note: These latest SDK methods are not available with the boto3 version provided by Lambda by default as of now. So, it is required to package boto3 version 1.42.1 or newer with your Lambda source code.\nWhen Lambda service receives the decision, the durable execution that was on hold will re-start. If the manager’s decision was not received on time (I set this to 5 minutes at the moment), the wait_for_callback step will throw an CallableRuntimeError exception with an error message - Callback timed out. Based on the decision or expiry of callback, the leave record in DB is updated. Then an email will be sent to the employee with the leave\u0026rsquo;s final status and the execution is completed. Try this yourself # Here is a Github repository of a sample project I created with AWS CDK and Python for you to try out this scenario in your AWS account.\nhttps://github.com/pubudusj/simple-leave-management-with-durable-functions\nClone the repository.\nCopy the .env.example into .env and add the values. The value for SYSTEM_FROM_EMAIL should be already configured in Simple Email Service (SES) to send emails to the manager and employee email addresses.\nDeploy the stack with CDK.\nOnce deployed, there will be two Lambda function urls in the output - one for creating the leave request and the other for process leave request (as manager).\nSend a POST request to the create leaves endpoint with a payload similar to below:\n{ \u0026#34;start_date\u0026#34;:\u0026#34;2026-01-10\u0026#34;, \u0026#34;end_date\u0026#34;: \u0026#34;2026-01-20\u0026#34;, \u0026#34;employee_email\u0026#34;: \u0026#34;employee@email.com\u0026#34; } This will send an email to the employee email address as follows: Image: Leave submitted notification for employee Also, an email to the manager with the callback id as follows: Image: Leave approval request for manager Manager can send a POST request to the leave process endpoint with the callback id in the email with his decision (approve or reject) as follows:\n{ \u0026#34;callback_id\u0026#34;:\u0026#34;[callback id from email]\u0026#34;, \u0026#34;decision\u0026#34;: \u0026#34;approve\u0026#34; } Based on the decision, the employee will receive an email with the status. Image: Leave decision notification for employee If the manager didn\u0026rsquo;t process the decision within given time (for demo purposes, this was set to 5 minutes), the leave will be marked as expired and employee will receive an email as follows: Image: Leave expired notification for employee Please try this and let me know your thoughts!\nResources # AWS Lambda Durable Functions documentation: https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html\nDurable execution SDK: https://docs.aws.amazon.com/lambda/latest/dg/durable-execution-sdk.html\n","date":"13 January 2026","permalink":"/posts/simple-leave-mangement-with-lambda-durable-functions/","section":"Posts","summary":"This blog post explains my experience developing a simple human-in-the-loop functionality for a leave management application using Lambda Durable Functions.","title":"Simple Leave Management with AWS Lambda Durable Functions"},{"content":"","date":"20 October 2025","permalink":"/tags/cloudwatch/","section":"Tags","summary":"","title":"CloudWatch"},{"content":"Intro # When you need to monitor your resources with a CloudWatch alarm, what you normally have to do is to create an alarm with a specific matric of that resource. Although this gives a granular level of monitoring into your resources, you always have to add or remove alarms as and when you have new resources or when you remove a specific resource. Which is an operational overhead although it can be automated if you are using infrastructure as code tool.\nAnother option available is to use aggregated metrics in your alarms such as CPUUtilization for EC2 so you have coverage, but at high level into a group of resources you need to monitor. The downside of this is that it lacks granular visibility into your individual resources. Also, there is only a limited number of resources that support aggregated metrics.\nIn September 2025, Amazon CloudWatch introduced a nice feature of allowing monitor multiple individual metrics via a single alarm using CloudWatch Metric Insights. By using Metric insight SQL queries in the alarm, it automatically updates its query results with each evaluation and adjusts in real time when resources are created and deleted.\nWith the introduction of the multi-metric alarms, you can now get both granular level per resource monitoring as well as less maintenance where you don\u0026rsquo;t need to update alarms when resources are created or removed from your application because the alarm itself will automatically discover and monitor the resources.\nHow it works # You can define a generic SQL statement as the alarm metric.\nFor example, let\u0026rsquo;s assume that you would like to monitor all your SQS queues for available messages and if there are any messages available you need to be notified.\nFor this requirement, you can define a query as follows:\nSELECT MAX(ApproximateNumberOfMessagesVisible) FROM SCHEMA(\u0026#34;AWS/SQS\u0026#34;, QueueName) GROUP BY QueueName ORDER BY COUNT() DESC With this query, the alarm will monitor all your queues for ApproximateNumberOfMessagesVisible count and trigger the alarm if a specific queue has at least one visible message. Also, if you add a new queue after the alarm is created, it will be taken into account when evaluating this query.\nAlso, you can use tags to filter the resources you need to monitor. For example\u0026hellip;\nSELECT MAX(ApproximateNumberOfMessagesVisible) FROM SCHEMA(\u0026#34;AWS/SQS\u0026#34;, QueueName) WHERE tag.Component = \u0026#39;consumer\u0026#39; AND tag.Environment = \u0026#39;production\u0026#39; AND tag.IsDLQ = \u0026#39;true\u0026#39; GROUP BY QueueName ORDER BY COUNT() DESC It would have been a great feature if this supports wildcards or regex based filtering into the resource properties, but as of now, only ways to use filters are using tags or the complete property value as equal or not equal.\nWhen you create the alarm with this Metrics Insights Query, you can see a new section on the alarm called \u0026ldquo;Contributors\u0026rdquo;. Here, you can see all the resources that match the conditions in the alarm query as well as the current state of each contributor.\nImage: CloudWatch multi metric alarm contributors. Alarm states # When a single contributor metric breached the threshold, then its state will be changed to in alarm state. And in the alarm details, it is clearly defined which contributor caused the alarm to change the state. This is super helpful to identify the exact resource that triggered the alarm.\nImage: CloudWatch multi metric alarm notification. The alarm action is based on the contributor transitions. Which means even when the alarm is already in \u0026lsquo;In alarm\u0026rsquo; state, if another contributor breached the threshold and its state becomes \u0026lsquo;In alarm\u0026rsquo;, the alarm action will be triggered.\nImage: CloudWatch multi metric example alarm history. This is how the similar situation shows in the CloudWatch console.\nImage: CloudWatch multi metric example alarm graph. Try this yourself # I have created a Github repository with a SAM template that helps to deploy some AWS resources into your AWS account and try this scenario.\nPre-requisites # You need to enable the CloudWatch setting \u0026ldquo;Resource tags for telemetry\u0026rdquo; if you need to use tag based filters in these queries. Go to CloudWatch \u0026gt; settings \u0026gt; Enable resource tags on telemetry in your region to enable this.\nClone the repository: https://github.com/pubudusj/cloudwatch-multi-metrics-alarm Deploy the stack using AWS SAM CLI. Once the stack is deployed, you can see in the created CloudWatch alarm, the contributors are available. When you add some messages into one of the SQS queues, you can see the alarm state changed and you will receive the alarm notification which includes the contributor. You can uncomment line 64 - 74 and deploy again to create a new resource. This new resource will be evaluated and you can see it is listed in the contributors list. If you add a message to the new queue, the alarm triggers. Likewise, any resource new or old that matches the query will be considered to evaluate the alarm. Limitations # There are some notable limitations in this approach.\nCurrently a single query can return no more than 500 time series and it is a hard limit. Which means as per the above query, maximum 500 resources can be monitored, which might not be not sufficient for a application with a lot of resources. When you have more than 500 resources to monitor, you will have to use multiple alarms.\nThere is no wildcard or regex supported in the query (comparing to some other features in CloudWatch)\nSummary # The new feature of CloudWatch is great for monitoring dynamic resources in your application without the management overhead of adding/removing alarms when the resources are created and deleted. There are some hard limits where this might not be suitable for all the situations. However, I believe there will be more improvements added to this feature in near future - like wild card or regex based resource filtering, that will give better developer experience.\nResources # Release news: Amazon CloudWatch query alarms now support monitoring metrics individually. https://aws.amazon.com/about-aws/whats-new/2025/09/amazon-cloudwatch-alarm-multiple-metrics/\nCreating a Metrics Insights CloudWatch alarm. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-alarm-create.html\nMetrics Insights quotas https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch-metrics-insights-limits.html\n","date":"20 October 2025","permalink":"/posts/cloudwatch-multi-metric-alarms/","section":"Posts","summary":"Amazon CloudWatch now allows you to monitor multiple individual metrics via a single alarm. By dynamically including metrics to monitor via a query, this helps to manage a single alarm for monitoring dynamic resources.","title":"Monitoring multiple dynamic resources using single Amazon CloudWatch alarm"},{"content":"Intro # When delivering messages to different targets using EventBridge, it is important to get notified if there are any delivery failures. EventBridge doesn’t provide this out of the box, but there are several ways to achieve this.\nIn the 1st part of this blog we discussed how we can get notified when a delivery fails to a target using a dead letter queue.\nIn this blog post we will discuss another (better?) option to achieve the same using EventBridge enhanced logging.\nEventBridge Enhanced Logging # On 15th July 2025 AWS introduced enhanced logging for EventBridge. Which means now you can enable logging and EvenBridge will send those logs into a configured log delivery location.\nAvailable log destinations are:\nS3 CloudWatch logs Amazon Data Firehose stream Also you can configure the standard log levels as required: Trace, Info and Errors.\nYou can select more than one log destination and for each destination, you can select the same or a different log level. This is really useful, for example, I can use S3 to log all the traces while using CloudWatch log stream to log only errors.\nHow it works # In this example, I have used EventBridge enhanced logging to log any errors occurring into a CloudWatch log stream.\nImage: Architecture. Simply, EventBridge will log any errors into the CloudWatch log. Once the log record is available in the CloudWatch log stream, there are multiple ways to trigger a CloudWatch alarm. In this example, I use the number of incoming log events as the metric to trigger the alarm.\nIf you use a CloudWatch log stream with log level trace (which includes info and errors as well) and still you want to trigger an alarm based on any error occurring, you may use an option like creating a metric filter in the log group.\nTry this yourself # I have created a Github repository with a AWS SAM template for you to test this scenario in your AWS account.\nClone the Github repository: https://github.com/pubudusj/event-bridge-target-failure-detection-with-enhanced-logging\nDeploy the stack using below command:\nsam deploy \\ --template-file template.yaml \\ --stack-name eb-fail-detection-with-enhanced-logging \\ --capabilities CAPABILITY_IAM \\ --no-confirm-changeset \\ --parameter-overrides NotificationEmail=[YourEmailAddress] Here, add your email address as NotificationEmail, so you will get the notification into your email box when the target fails.\nOnce the stack is deployed, you will get a SNS subscription confirmation email. You need to confirm it in order to receive notifications.\nThen, publish a message into the created event bus with the source as xyzcorp.\nThis way the message will match the rule and try to deliver the message to the target.\nI have blocked the permission for publishing the target intentionally to simulate the failure.\nIn a moment, you should get an email with the alarms status.\nIf you go to the CloudWatch log stream created, you can see the entry with log level ERROR and message type INVOCATION_FAILURE.\nPlease note: As of now, creating the enhanced logging and delivering them to a delivery destination is not a straight forward configuration. You need to create a CloudWatch log group, a delivery source, a delivery destination and a logs delivery.\nRefer: https://github.com/pubudusj/event-bridge-target-failure-detection-with-enhanced-logging/blob/main/template.yaml#L31-L59\nWhich means if you need to send these enhanced logs to multiple destinations, you need to repeat configuring those resources per destination.\nHowever, nice thing about this approach is that you only need to configure this only once on the event bus, and it will log the whole message life cycle within EventBridge systems including ingestion as well as deliveries to all the targets.\nSummary # Overall, enhanced logging is a great improvement to EventBridge because up until now, message delivery was a black box (for customers), specially the consumer side of EventBridge. With this new addition, you can track and debug the flow of your message within EventBridge systems transparently using the logs generated in each and every step that the message is going through, from ingest to delivery (of course depends on the log level that was configured).\nAlso, you can configure more than one log destinations as well as different log levels.\nSince creating single log delivery using CloudFormation requires several AWS resources to be configured, I hope EventBridge team will provide easy to use method to configure this, ideally as properties of EventBridge Bus.\nResources # Monitor and debug event-driven applications with new Amazon EventBridge logging https://aws.amazon.com/blogs/aws/monitor-and-debug-event-driven-applications-with-new-amazon-eventbridge-logging/ ","date":"30 September 2025","permalink":"/posts/detect-eventbridge-target-failure-part-2/","section":"Posts","summary":"When EventBridge cannot deliver a message to a target, there are several ways to get notified. In this blog we discuss how EventBridge enhanced logging can be used in this scenario.","title":"Detect EventBridge target failure: Part 2 - using enhanced monitoring"},{"content":"","date":"30 September 2025","permalink":"/tags/eventbridge/","section":"Tags","summary":"","title":"EventBridge"},{"content":"Intro # When EventBridge delivers messages to its target, there can be many reasons that cause failing to send a message. There can be permission issues, rate limits or the unavailability of the target or even can be a glitch in the AWS itself, just to name a few.\nNo matter what the reason is, it is always ideal to get notified that there is an issue delivering messages and the reason for the failure. In this blog post I will discuss how a dead letter queue can be useful to get notified when the EventBridge fails to deliver messages to its target.\nDead letter queues # Dead letter queues are unsung heroes of the event driven architecture 😀. Those are easy to set up and manage yet greatly improve the resilience of a system. Also it is very cost effective.\nLet’s see how we can capture the target delivery failures in EventBridge using a dead letter queue.\nPlease note that EventBridge supports DLQ in a couple of “levels”. EventBridge bus can have a DLQ itself, or you can set a DLQ per target basis. Let\u0026rsquo;s discuss the differences.\nDLQ on EventBridge bus level # EventBridge bus can have a DLQ of its own. However, this is limited to capture any errors related to the KMS encryption. EventBridge sends events that aren\u0026rsquo;t successfully encrypted to the DLQ.\nYou can only see this DLQ setting in the EventBridge in AWS console only if customer managed KMS is used to encrypt messages. In fact, it is part of the Encryption settings.\nImage: DLQ for Event bus only available when customer managed KMS is in use. However, this DLQ will NOT capture any target related failures, so that we cannot use this for our purpose.\nDLQ on EventBridge target level # When EventBridge cannot deliver a message to a target, we can set up a SQS queue to put that message in, on the target level.\nImage: DLQ on target. Since one rule can have more than one target, each target can have different DLQs as well. You can use the same SQS queue as the DLQ for all the targets, but you have to configure it for each and every target separately. It may sound like repetitive work, but if you use an infrastructure as a code tool like CDK or CloudFormation, this is not complex.\nHow it works # Image: High level architecture. EventBus tries to deliver a message to its target (here, it is a SQS queue) via EventBridge rule.\nLet\u0026rsquo;s assume there is a permission issue, and the message cannot be delivered.\nThen, EventBridge will put the message into the DLQ configured for this specific target.\nIn Cloudwatch, there is an alarm set up to be triggered whenever there is a message in the DLQ.\nWhen the failed message is in the DLQ, Alarm triggers and there is a SNS topic configured as the alarm action.\nAnd when the alarm action publishes a message to SNS topic, it will send the notification to all the subscribers to notify about the failure.\nTry this yourself # I have created an AWS SAM template to try this scenario in your AWS account.\nClone the Github repository: https://github.com/pubudusj/event-bridge-target-failure-detection-with-dlq\nDeploy the stack using below command:\nsam deploy \\ --template-file template.yaml \\ --stack-name event-bridge-target-failure-detection-with-dlq \\ --capabilities CAPABILITY_IAM \\ --no-confirm-changeset \\ --parameter-overrides NotificationEmail=[YourEmailAddress] Here, add your email address as NotificationEmail, so you will get the notification into your email box when the target fails.\nOnce the stack is deployed, you will get a SNS subscription confirmation email. You need to confirm it in order to receive notifications.\nThen, publish a message into the created event bus with the source as xyzcorp.\nThis way the message will match the rule and try to deliver the message to the target.\nI have blocked the permission for publishing the target intentionally to simulate the failure.\nIn a moment, you should get an email with the alarms status.\nFurther, if you check the messages in DLQ, you can see the failed message and in the message attributes, you may see the reason of failure (depends on the reason).\nImage: Message attributes of a failed message in DLQ. You can configure the threshold, period and evaluation periods of the alarm as needed to control the frequency of the notifications in case of a failure. https://github.com/pubudusj/event-bridge-target-failure-detection-with-dlq/blob/main/template.yaml#L61-L63 Summary # EventBridge bus has a DLQ but it is for a different purpose and cannot capture any target failures.\nYou can use this dead letter queue approach, to capture any messages which cannot be delivered to the target. Based on the no of messages in the queue, you can get notified using CloudWatch metric and SNS. However, you will need to configure it for each and every EventBridge target separately. Using an IAC tool to configure this may be convenient.\nI will discuss another solution to achieve the same in part 2 of this blog post.\nResources # Using dead-letter queues to process undelivered events in EventBridge https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-rule-dlq.html ","date":"23 September 2025","permalink":"/posts/detect-eventbridge-target-failure-part-1/","section":"Posts","summary":"When EventBridge cannot deliver a message to a target, there are several ways to get notified. In this blog we discuss an approach of using a dead letter queue","title":"Detect EventBridge target failure: Part 1 - with dead letter queue"},{"content":"","date":"23 September 2025","permalink":"/tags/dlq/","section":"Tags","summary":"","title":"DLQ"},{"content":"Intro # Delivering a message to a target SQS queue from EventBridge is a very common requirement in an event driven application. Sometimes, this SQS queue can be in a different region and may be in a different AWS account. Depending on where the target SQS queue lives, the approach to how you set up the solution differs.\nIn this blog post, we are going to see different scenarios on how EventBridge can deliver messages to a target SQS queue when the target SQS queue is in a different region or different AWS account.\nScenario 1: Event bus, rule and target SQS queue in same AWS account, same region # Image: All resources in the same AWS account, same region This is the most common and simplest scenario. Event bus has a rule which directly configured a SQS queue as a target. All the resources are in the same region in the same AWS account.\nScenario 2: Event bus and rule in one region, target SQS queue in another region in the same AWS account # Image: Event bus and rule in one region, target SQS queue in another region in the same AWS account In this scenario, the Event bus and the rule is in one region. The target SQS queue is in the same AWS account, but in a different region.\nAs of today (September 2025), EventBridge supports SQS as a target only when it is in the same region.\nBecause of that, first you have to create a new Event bus (or use the default Event bus) in the 2nd region. And configure the SQS queue as a target of that Event bus. Then, configure the event bus in the second region to be a target in the EventBridge rule of the first region.\nScenario 3: Event bus, rule in one AWS account, target SQS queue in another AWS account, but all are in the same region # Image: Event bus and rule in one AWS account, target SQS queue in another AWS account, but all are in the same region AWS introduced cross account support for EventBridge targets beginning of 2025. Previously, when delivering messages to cross account resource, you had to use a Event bus in the 2nd account to establish the connection (similar to scenario 2 above).\nWith this new feature, it is possible to have a cross account target which simplifies the message delivery from an event bus to a SQS queue in another account.\nHowever, there is a limitation. Although the cross account delivery is possible, the EventBridge bus and the rule and the target in the 2nd AWS account must be in the same region.\nScenario 4: Event bus, rule in one AWS account, target SQS queue in a different region in another AWS account # Image: Event bus, rule in one AWS account, target SQS queue in a different region in another AWS account As mentioned earlier, cross account delivery is supported only when the target SQS queue is in the same region. If the SQS queue is in a different region, you have to use an Event bus in the 2nd account in between.\nSummary # The pattern is simple. SQS can be a target in an EventBridge rule when the rule and the SQS queue both are in the same region. Target SQS queue can be in another AWS account as long as the region is the same.\nIf the SQS queue is in a different region than the EventBridge rule, you have to use an intermediate EventBus in the 2nd region. This intermediate EventBus and the target SQS can be in another AWS account.\nResources # Event bus targets in Amazon EventBridge - https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-targets.html\nIntroducing cross-account targets for Amazon EventBridge Event Buses - https://aws.amazon.com/blogs/compute/introducing-cross-account-targets-for-amazon-eventbridge-event-buses/\n","date":"17 September 2025","permalink":"/posts/eventbride-to-sqs-cross-region-cross-account-access/","section":"Posts","summary":"When EventBridge needs to deliver messages to SQS, depending on cross account or cross region, the solution is different.","title":"EventBridge to SQS when cross region and cross account"},{"content":"","date":"1 April 2025","permalink":"/tags/amazon-q/","section":"Tags","summary":"","title":"Amazon Q"},{"content":"","date":"1 April 2025","permalink":"/tags/api-gateway/","section":"Tags","summary":"","title":"API Gateway"},{"content":"","date":"1 April 2025","permalink":"/tags/cdk/","section":"Tags","summary":"","title":"CDK"},{"content":"","date":"1 April 2025","permalink":"/tags/dynamodb/","section":"Tags","summary":"","title":"DynamoDB"},{"content":"","date":"1 April 2025","permalink":"/tags/generative-ai/","section":"Tags","summary":"","title":"Generative AI"},{"content":"Intro # Last December 2024, as part of the AWS Game Builder Challenge, I built a simple spelling game. For that, I used AWS Serverless and GenAI services and CDK was used as the IAC tool. This was my first experience building something with the help of GenAI and using Bedrock in an application. This blog post explains the project and my experience building this simple spelling game.\nThis is the final result - the game you can play :)\nhttps://spelling-game.pubudu.dev/\nHow to play the game? # Player can select a language to play the game. As of now, only English (US) and Dutch are the available languages.\nWhen the player selects a language, there will be a maximum 5 words generated. This includes an audio, brief meaning of the word and the number of characters for each word.\nThen the player needs to fill the word in the text box given.\nThere is an indicator next to the text box how many characters have been entered in the text box and how many characters are required for the word. This indicator will be red until the required number of characters are filled, then it turns into green.\nThere is a timer that starts as soon as the words are generated. This is based on the number of words generated.\nWhen the remaining time is less than 30 seconds, the background of the page as well as the background of the timer turns into red.\nPlayer can submit the answers before the timer runs out, else it will be automatically submitted.\nThen, the answers are evaluated and based on the number of correct answers, there is a pop up visible.\nIf all the answers are correct, there will be a \u0026ldquo;Confetti\u0026rdquo; effect appearing on the page.\nClicking on the \u0026lsquo;show results\u0026rsquo; button on the pop up, the player can see the correct/incorrect answers and the correct word (in case of an incorrect answer).\nImplementation # There are 2 main parts of this application.\nBackend - where words are generated and APIs are available. Frontend - Vue.js application for the player to interact. Source code # There are two repositories available with the complete source code.\nBackend - https://github.com/pubudusj/spelling-game-backend\nFrontend - https://github.com/pubudusj/spelling-game-frontend\nDeployment instructions: # Backend is implemented using AWS CDK and can be deployed as a generic CDK application. The URL of the Cloudfront distribution is required for the frontend to work.\nIn the frontend, add the Cloudfront API URL to the VITE_API_BASE_URL in the env file. Install necessary dependencies and then run the application using npm run dev in the dev mode. Or you can build the frontend app using npm run build.\nWhen the backend stack is deployed, it will output the S3 hosting bucket name. You can copy the built frontend app into this bucket which will host the frontend via the Cloudfront distribution created in the backend stack.\nBackend # There are 2 main components of the backend.\nWords generator component - to generate and save words in the database. API component - to serve frontend. Backend - words generator component # Here is the high level overview of the words generator component and the steps within the state machine. Within Step Functions, there will be many AWS services called as explained below.\nImage: Words generator overview Image: Words generator state machine Words Generator Step Functions state machine is responsible for generating words.\nThis Step function execution takes the language code as an input. ex: en-US, nl-NL, etc.\nAs the first step, Bedrock InvokeModel is being called to generate 5 words with descriptions for each word, based on the language.\nHere, the Anthropic Claude 3 Haiku model is being used which gives better balance of accuracy and the pricing in this scenario.\nHere is the prompt I used to generate the words:\n\u0026ldquo;Generate 5 unique words that have a random number of characters more than 4 and less than 10 in Dutch language. For each word, provide a brief description of its meaning in English with more than a couple of words. Produce output only in a minified JSON array with the keys word and description. Word must always be in lowercase.\u0026rdquo;\nHere, the response is a JSON string.\nWithin the step\u0026rsquo;s Result selector, the response will be converted to an array using the intrinsic function - StringToJson.\n{ \u0026#34;words.$\u0026#34;: \u0026#34;States.StringToJson($.Body.content[0].text)\u0026#34; } Then there is a map state where each word is an input to each map.\nWithin the map state, there are branches (currently two) based on the language.\nIn each branch, there is a step to synthesise the word using Polly using StartSpeechSynthesisTask.\nStartSpeechSynthesisTask is an async operation. So, there is an immediate step to check if the synthesis task is completed using Polly\u0026rsquo;s GetSpeechSynthesisTask.\nGreat thing about the StartSpeechSynthesisTask api is that it not only synthesises, but automatically saves the mp3 into the given S3 bucket.\nIf the speech synthesis task is not finished based on the status of the GetSpeechSynthesisTask, it waits and retries the status check.\nOnce synthesis is done, the execution continues to save word to DynamoDB step.\nIn this step, we use DynamoDB\u0026rsquo;s PutItem api to save the generated data to the table. One record consists of below data:\npk - Primary key in the format of Word#{LanguageCode}. ex: Word#en-US sk - MD5 hash of the word - here, intrinsic function States.Hash($.word, \u0026#39;MD5\u0026#39;)is in use. word - The word generated by Bedrock. description - The description generated by Bedrock. s3file - Mp3 file location provided by Polly synthesis task. charcount - Character count of the word. This is retrieved from Polly synthesis task. updated_at - Update timestamp. Since this synthesis and save to db task runs on a map state, after a single execution, there will be a maximum 5 new words available in the ddb for the given language.\nThere are two EventBridge Schedules running every 5 minutes to call this State Machine with the different language codes - English and Dutch.\nBackend - API component # API component is to create resources to interact with the frontend.\nThere are two APIs available in the backend.\nPOST /questions - To generate questions using Step Functions to appear on the frontend. POST /answers - To validate the answers submitted by the player. Image: Frontend and API components Generate questions API # Generate questions API accept one argument. Which is the language code.\nThis /questions api has a proxy integration to a Lambda Function which will start an execution in Questions Generator State Machine synchronously using start_sync_execution SDK call.\nThis Questions Generator State Machine is in type - Express.\nImage: Questions Generator State Machine A sample input is as follows: { \u0026#34;language\u0026#34;:\u0026#34;en-US\u0026#34;, \u0026#34;iterate\u0026#34;:[1, 2, 3, 4, 5] } Here, \u0026ldquo;iterate\u0026rdquo; is a hard coded array to start a map execution within the state machine.\nWithin the state machine, first the map state is executed based on the \u0026ldquo;iterate\u0026rdquo; array from the input.\nInside the map state, first, it fetches max 50 records from DynamoDB. Here, DynamoDB scan is used. However, in order to fetch some random data, a random ExclusiveStartKey is in use with the help of the intrinsic function UUID(). Also, FilterExpression is used to filter the records applicable only for the given language code.\n{ \u0026#34;TableName\u0026#34;: \u0026#34;arn:aws:dynamodb:****\u0026#34;, \u0026#34;Limit\u0026#34;: 50, \u0026#34;ExclusiveStartKey\u0026#34;: { \u0026#34;pk\u0026#34;: { \u0026#34;S.$\u0026#34;: \u0026#34;States.Format(\u0026#39;Word#{}\u0026#39;, $$.Execution.Input.language)\u0026#34; }, \u0026#34;sk\u0026#34;: { \u0026#34;S.$\u0026#34;: \u0026#34;States.UUID()\u0026#34; } }, \u0026#34;FilterExpression\u0026#34;: \u0026#34;pk = :pk\u0026#34;, \u0026#34;ExpressionAttributeValues\u0026#34;: { \u0026#34;:pk\u0026#34;: { \u0026#34;S.$\u0026#34;: \u0026#34;States.Format(\u0026#39;Word#{}\u0026#39;, $$.Execution.Input.language)\u0026#34; } }, \u0026#34;ReturnConsumedCapacity\u0026#34;: \u0026#34;TOTAL\u0026#34; } Next, the number of item counts returned from the previous step is checked.\nIf the count is more than 0, then in the next step, single random record is selected from them. This step is a Pass state with transformation using Parameters, which uses intrinsic functions - ArrayGetItem, MathRandom and ArrayLength.\n{ \u0026#34;item.$\u0026#34;: \u0026#34;States.ArrayGetItem($.items,States.MathRandom(0, States.ArrayLength($.items)))\u0026#34; } Then, the selected record is being sent to the Generate Pre signed URL step. This is a Lambda function, which generates a pre-signed URL for the s3file path of the record. So, from the frontend the mp3 file can be played using this pre-signed url. Also there is a transformation of data within this Lambda function. The expiry of the pre-signed url is set to minimum because it is only required within the session of the game.\nThis is the last step within the map state which outputs the record in below format.\n{ \u0026#34;id\u0026#34;:\u0026#34;XXXX\u0026#34;, \u0026#34;description\u0026#34;:\u0026#34;Description of the word\u0026#34;, \u0026#34;charcount\u0026#34;:8, \u0026#34;language\u0026#34;:\u0026#34;en-US\u0026#34;, \u0026#34;url\u0026#34;:\u0026#34;Presigned-url\u0026#34; } Once all the map steps are completed, there is a final aggregation Lambda function - GetUniqueResultsLambda. Since each map step is independent, there is a possibility of selecting the same record in more than one map state. This Lambda function simply removes such duplications.\nAnd the response is returned to the frontend as the response to the /questions endpoint.\nValidate answers API # POST /answers API is responsible for validating the answers submitted in the frontend.\nThis API endpoint has a proxy Lambda function which accepts the payload in below format:\n{ \u0026#34;language\u0026#34;:\u0026#34;en-US\u0026#34;, \u0026#34;answers\u0026#34;:[ { \u0026#34;id\u0026#34;:\u0026#34;312e8b6583d4b65b\u0026#34;, \u0026#34;word\u0026#34;:\u0026#34;effect\u0026#34; }, { \u0026#34;id\u0026#34;:\u0026#34;99b788c54c1a8265\u0026#34;, \u0026#34;word\u0026#34;:\u0026#34;perilous\u0026#34; }, ... ... ] } Within the Lambda function, it does a DynamoDB\u0026rsquo;s batch_get_item SDK call to fetch words per ids and match with the word provided in the API.\nThen it returns the response in below format:\n[ { \u0026#34;id\u0026#34;: \u0026#34;312e8b6583d4b65b\u0026#34;, \u0026#34;original_word\u0026#34;: \u0026#34;affect\u0026#34;, \u0026#34;correct\u0026#34;: false }, { \u0026#34;id\u0026#34;: \u0026#34;99b788c54c1a8265\u0026#34;, \u0026#34;original_word\u0026#34;: \u0026#34;perilous\u0026#34;, \u0026#34;correct\u0026#34;: true }, ... ... ] Based on the value of \u0026ldquo;correct\u0026rdquo;, frontend will calculate and show the results. Frontend # For the frontend, I have used a simple single page application built with Vue.js. I have very limited knowledge on frontend technologies. Because of that, I used Amazon Q Developer on VSCode to implement the frontend application.\nAlmost 95% of the frontend application was built by Amazon Q Developer. I have asked different questions and in most of the cases, Amazon Q was able to analyse the code and generate the code as per my requirements. This was a step by step process where I asked Amazon Q to generate specific functionality at a time.\nHere are some work in progress \u0026ldquo;versions\u0026rdquo; of the application that was implemented and fine tuned step by step using Amazon Q.\nHere are some examples for the questions I asked from Amazon Q and how it analysed and generated code.\nLessons learnt # Below are some of the lessons I learnt while I was working on this project, including some feedback on some of the services I used here.\nBedrock does not always return JSON. Within the prompt I used, I stated - \u0026ldquo;Produce output only in a minified JSON array with the keys word and description\u0026rdquo;. However, once in a while, Bedrock returns data in different formats. This could have been more accurate if I add the beginning of the response in the prompt, so Bedrock can continue from there. However, this will increase the request token count for each API call. So, to avoid additional cost and also, the error rate is acceptable (since this is anyway a background job), I kept this prompt as it is.\nAmazon Polly\u0026rsquo;s StartSpeechSynthesisTask doesn\u0026rsquo;t support S3 path. We can provide the OutputS3BucketName where the generated audio will be stored. However, we cannot specify a path to save the object. Instead, I have used the OutputS3KeyPrefix parameter to provide the path with the language code so, the audio is saved in s3://bucket_name/language_code/file_name.mp3 However, one minor issue with that is, Polly always adds a dot (.) between the file name and the prefix. So, all the files generated in the sub path start with a dot.\nCost of Polly. Polly has Generative and Neural text-to-speech engines apart from Standard. However, the cost of them is quite high compared to Standard. Also, those are available only for a limited number of languages. https://aws.amazon.com/polly/pricing/\nSelecting random items from the DynamoDB table is hard. There is no straightforward way to achieve this. That\u0026rsquo;s why I had to use a ExclusiveStartKey and fetch maximum 50 items and select one random item. This might introduce some duplications. That\u0026rsquo;s the reason I had to use the GetUniqueResultsLambda which removes any duplicates from the map job.\nI initially used direct integrations to start the Step Functions express workflow to generate questions directly from API Gateway. However, the VTL is complex to build specially to get the response in a specific format. So I stuck to the Lambda proxy option which was more simple.\nInitially I have hosted the frontend using Amplify Hosting. However, I wanted to restrict the API Gateway access only from Amplify project, but there is no option yet. So I switched to Cloudfront, S3 set up. This is described in detail in one of my previous blogs: Enforce CloudFront-Only Access for AWS API Gateway Enforce CloudFront-Only Access for AWS API Gateway 5 March 2025\u0026middot;4 mins\u0026middot; loading AWS Serverless Cloudfront API Gateway EventBridge Scheduler Security In this post, it is discussed how we can secure a AWS API Gateway REST API to accept requests only through CloudFront. For the frontend, I indeed did Vibe coding. So, there is a high chance that any frontend specialist might find unacceptable code in it 😉.\nMost of the time, Amazon Q generated code which includes the expected functionality. However, there were cases it couldn\u0026rsquo;t fix some code snippets. For example, when I asked to center a component, it couldn\u0026rsquo;t fix it even after 20th iteration. May be this is because of the complexity of the single page structure.\nImage: Amazon Q attempts to center a div Possible improvements # Currently there is no \u0026lsquo;history\u0026rsquo; of the games a particular player played. Adding a logged-in mode to record the player status/progress can be a nice feature.\nAdd more languages: Currently only English(US) and Dutch are available to select. Having more languages is nice. However, those need to be supported by Polly.\nFinished game\u0026rsquo;s standing against all other games in the results can be a nice feature. For this, recording the results and comparing against all the results will be required.\nWhat are the other possible improvements you see?\nConclusion # Overall, it was a great experience using Bedrock and Amazon Q. I didn\u0026rsquo;t win the hackathon, but as this was the first time I was using those GenAI services, I learnt a lot from this project and enjoyed a lot. Frontend code was mostly generated by Amazon Q and I am happy with the results of it although I am certain there can be many improvements in the frontend code.\nUseful Links # Getting started with Amazon Q Developer: https://aws.amazon.com/q/developer/getting-started/\nGetting started with Amazon Bedrock: https://docs.aws.amazon.com/bedrock/latest/userguide/getting-started.html\nInvoke and customize Amazon Bedrock models with Step Functions: https://docs.aws.amazon.com/step-functions/latest/dg/connect-bedrock.html\nClaude Prompt engineering overview: https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview\n","date":"1 April 2025","permalink":"/posts/how-i-built-a-simple-spelling-game-with-aws-serverless-and-gen-ai/","section":"Posts","summary":"This blog post explains the process of building a simple spelling game with AWS Serverless and GenAI.","title":"How I built a spelling game with AWS Serverless and GenAI"},{"content":"","date":"1 April 2025","permalink":"/tags/step-functions/","section":"Tags","summary":"","title":"Step Functions"},{"content":"","date":"5 March 2025","permalink":"/tags/cloudfront/","section":"Tags","summary":"","title":"Cloudfront"},{"content":"Recently, I had a requirement to expose a REST API from API Gateway exclusively through CloudFront. Keeping API Gateway behind Cloudfront provides an additional layer of security because Cloudfront comes with automatic protections of AWS Shield Standard, at no additional charge.\nThere are several ways to achieve this, for example, using a signed request using Lambda@Edge in Cloudfront, but I went for a simpler solution, which is using a custom header in the request to API Gateway from the Cloudfront and using Lambda authorizer at the API gateway to validate it. This blog post explains how to implement this solution.\nHow it works # Image: architecture In the Cloudfront distribution, we create an origin for API gateway endpoint.\nHere, we define a custom header on this cloudfront origin.\nWhen a request comes to Cloudfront distribution, as per the behaviour we define, it will call the API gateway endpoint.\nThis request will contain the custom header.\nIn the API Gateway end, there is an Lambda authorizer which validates this incoming header.\nThere is a SSM secure parameter which holds the value of the header.\nWhen it is required to validate the incoming header, Lambda authorizer will fetch this value from the parameter store.\nCommunication between the Cloudfront distribution and the API gateway is secure and there is no way someone who doesn\u0026rsquo;t have access to Cloudfront settings will get to know the header value. However, keeping a static value for a validation seems not correct. So, it is better to rotate the value of the header for added security.\nRotating the header value # There is a EventBridge scheduler which will invoke a Lambda function in a given interval.\nThis Lambda function will perform below actions:\nGenerate a random value for the header. Update the SSM secure parameter. Update the Cloudfront origin\u0026rsquo;s custom header value. Once this Lambda execution is successful, in both sides (Cloudfront and API Gateway) the new header value will be used.\nTry this yourself # Here is a Github repository of the project I created to try out this solution. https://github.com/pubudusj/secure-api-with-cloudfront\nYou can deploy this to your AWS account using CDK.\nPlease note: First, you need to create a secure SSM parameter with any value. Then create an .env file copying the .env.example file in the project root directory and set the name/path of the parameter in the .env file.\nIn the CDK code, you will notice that the initial header value is hardcoded.\norigin=origins.RestApiOrigin( rest_api, origin_path=\u0026#34;/prod\u0026#34;, custom_headers={custom_header_key: \u0026#34;test\u0026#34;}, ), Also, this obviously does not match with the value you have in the SSM parameter.\nThis is fine, because this stack includes a Cloudformation custom resource, which will be executed on create. This custom resource will start the initial token rotation as soon as the stack is created which generates a new header value and syncs both SSM secure parameter and Cloudfront distribution.\nOnce the stack is deployed, you can access the API using API Gateway endpoint and also with Cloudfront endpoint.\nFor example:\nCloudfront endpoint: https://[CloudfrontPrefix].cloudfront.net/prod/hello API Gateway endpoint: https://[APIGWPrefix].execute-api.[region].amazonaws.com/prod/hello You will notice that Cloudfront endpoint works fine while there is a 401 Unauthorized error from the API Gateway endpoint.\nTips/Lesson Learned # Here I have used an SSM Parameter store with a custom built rotation mechanism. However, you can use Secret manager instead with its inbuilt rotation features. Make sure you are aware of the differences. Yan has done this comparison in detail in his blog post https://theburningmonk.com/2023/03/the-old-faithful-why-ssm-parameter-store-still-reigns-over-secrets-manager/\nKeep in mind that when setting a new custom header value on the Cloudfront origin, it will perform a re-deployment. Cloudfront re-deployment takes time. It can be within a couple of minutes to 10-15 minutes too. But for a change like this, it should be faster.\nBecause of this, during a Cloudfront re-deployment, your SSM parameter and incoming header value at API Gateway can be different. One solution I used to address this is to cache the Lambda authorization result. Here I used a ttl of 5 minutes, but you can adjust as required.\nauthorizer = apigateway.RequestAuthorizer( self, \u0026#34;LambdaHeaderAuthorizer\u0026#34;, handler=custom_authorizer, identity_sources=[apigateway.IdentitySource.header(custom_header_key)], results_cache_ttl=Duration.minutes(5), ) Useful Links # Cloudfront custom headers: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/add-origin-custom-headers.html\nAPI Gateway Lambda authorizer: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html\n","date":"5 March 2025","permalink":"/posts/access-api-gw-rest-api-only-from-cloudfront/","section":"Posts","summary":"In this post, it is discussed how we can secure a AWS API Gateway REST API to accept requests only through CloudFront.","title":"Enforce CloudFront-Only Access for AWS API Gateway"},{"content":"","date":"5 March 2025","permalink":"/tags/eventbridge-scheduler/","section":"Tags","summary":"","title":"EventBridge Scheduler"},{"content":"","date":"4 September 2024","permalink":"/tags/encryption/","section":"Tags","summary":"","title":"Encryption"},{"content":"When building distributed applications on AWS, Amazon Simple Queue Service (SQS) often becomes a crucial component in managing message flow between services. Ensuring the security of these messages is important, especially when dealing with sensitive data across multiple AWS accounts. In this post, we’ll explore different encryption options available for SQS and choosing the best option for scenarios like cross-account access.\nIn-Transit (as it travels to and from Amazon SQS) Encryption # You can protect data in-transit using HTTPS (TLS). This ensures that messages are protected as they travel between your application and SQS, preventing attacks such as man-in-the-middle. You can enforce only encrypted connections over HTTPS (TLS) using aws:SecureTransport condition in the queue policy.\n\u0026#34;Condition\u0026#34;: { \u0026#34;Bool\u0026#34;: { \u0026#34;aws:SecureTransport\u0026#34;: \u0026#34;true\u0026#34; } } Another option for in-transit encryption is using client-side encryption where you encrypt data before sending it to SQS. Here you have to manage the encryption-decryption mechanism yourself.\nServer-Side Encryption # Server-side encryption (SSE) adds an extra layer of security by encrypting the contents of your queue at the storage level. SSE protects the contents of messages in queues using SQS-managed encryption keys (SSE-SQS) or keys managed in the AWS Key Management Service (SSE-KMS).\nSSE-SQS (SQS Managed Keys) # This is the simplest option, where Amazon SQS takes care of the encryption keys for you. SQS generates, manages, and uses the encryption key, requiring no additional configuration on your part. Since October 2022, this is by default enabled for any new SQS queue.\nSSE-KMS (AWS Key Management Service Keys) # This option provides more control by allowing you to use AWS Key Management Service (KMS) to manage the encryption keys. With SSE-KMS, you can use an existing KMS key or create a new one specifically for your SQS queue. This method enables finer-grained access control and auditing capabilities compared to SSE-SQS.\nCross-account access with SSE # As one of the most used messaging services between software components, cross account access is a very common scenario for SQS. Still, we need to make sure the messages that are exchanged between SQS queues are secured.\nIn general, you can manage the cross account access for a SQS using its access policy.\nBelow is an IAM policy of my_sqs_queue in account 111111111111. This has granted the account 222222222222 to send messages to my_sqs_queue.\n{ \u0026#34;Sid\u0026#34;: \u0026#34;cross_account_access\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: { \u0026#34;AWS\u0026#34;: \u0026#34;arn:aws:iam::222222222222:root\u0026#34; }, \u0026#34;Action\u0026#34;: [ \u0026#34;sqs:GetQueueAttributes\u0026#34;, \u0026#34;sqs:GetQueueUrl\u0026#34;, \u0026#34;sqs:SendMessage\u0026#34; ], \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:sqs:eu-west-1:111111111111:my_sqs_queue\u0026#34;, } } Depending on the server side encryption used in the queue, there will be additional permission required to grant send messages to my_sqs_queue.\nIf SSE-SQS is used # Good news is if SSE-SQS is used, there are no additional encryption related permissions required by account 222222222222. Which means above IAM permission is sufficient to send messages to my_sqs_queue.\nIf SSE-KMS is used # If this encryption is used, additional permission for the KMS key must be set in order to successfully send messages to my_sqs_queue.\nLet\u0026rsquo;s assume my_sqs_queue is encrypted using a KMS key in the same account 111111111111 which has the alias my_kms_key. In the my_kms_key, you have to grant permission for account 222222222222 as follows:\n{ \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Principal\u0026#34;: { \u0026#34;AWS\u0026#34;: \u0026#34;arn:aws:iam::222222222222:root\u0026#34; }, \u0026#34;Action\u0026#34;: [ \u0026#34;kms:Encrypt\u0026#34;, \u0026#34;kms:Decrypt\u0026#34;, \u0026#34;kms:ReEncrypt*\u0026#34;, \u0026#34;kms:GenerateDataKey*\u0026#34;, \u0026#34;kms:DescribeKey\u0026#34; ], \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:kms:eu-west-1:111111111111:key/[key-id]\u0026#34; } Further, in the account 222222222222, there should be permission kms:GenerateDataKey for the KMS key as follows:\n{ \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;kms:GenerateDataKey\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:kms:eu-west-1:111111111111:key/[key-id]\u0026#34; } Conclusion: What SSE method to choose? # Both SSE-SQS and SSE-KMS support cross-account access for SQS queues. The key difference lies in how much control and responsibility you want over the encryption process.\nSSE-SQS is ideal when you need simple, effective encryption without the additional complexity of managing KMS keys. It\u0026rsquo;s ideal for most general use cases where ease of setup and management is a priority.\nUse SSE-KMS when you require more control over your encryption keys and need to meet strict security and compliance requirements. This option is suited for environments where key management and detailed access control are critical.\nUseful Links # Amazon SQS security best practices: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-security-best-practices.html#implement-server-side-encryption Encryption at rest in Amazon SQS - Developer Guide : https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html Amazon SQS Key management - Developer guide: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-key-management.html ","date":"4 September 2024","permalink":"/posts/sqs-encryption-options/","section":"Posts","summary":"SQS supports both in-transit and server side encryption. For serverside encryption, you can from SSE-SQS or SSE-KMS. What exactly is suitable for you depends on your requirement. In this blog, I focus on how to choose the correct SSE method for cross account access.","title":"SQS encryption options"},{"content":"Intro # When it comes to building a software system, one of the most critical components is error handling, because “everything fails all the time”. While it\u0026rsquo;s impossible to anticipate every possible failure scenario, having a plan in place when an unexpected thing happens can always be helpful for robustness and resilience of the system.\nThis ‘plan’ can be a simple Dead Letter Queue (DLQ)!\nDead Letter Queues can act as the safety net where you can keep the messages when unexpected issues arise. This helps you to isolate problematic messages, debug the issue without disrupting the rest of your workflow, and then retry or reprocess the message as needed.\nMany AWS Serverless services support SQS queues as the dead letter queue natively. However, Step Functions - one of the main Serverless services offered by AWS for workflow orchestration - does not support dead letter queues (yet) natively.\nIn this blog post, I am going to discuss a couple of workarounds to safely capture any messages that have failed to process by a Step Function into a dead letter queue.\nScenario # Let’s consider a very common use case where we have a message in a SQS source queue, which needs to be processed by Step Functions. First the messages are read by a Lambda function that starts a step function execution for each message.\nImage: SQS -\u0026gt; Lambda function -\u0026gt; SF Here, once Step function execution lambda reads the message from Source Queue and successfully starts the Step function execution, the message will be marked as successfully processed and will be deleted from the queue. If there’s any errors in the Step Function execution, the message will be lost.\nSolution 01 # In order to retain the message even when there is a failure in Step Function execution, we can add a 2nd SQS queue which acts as a dead letter queue as follows.\nImage: Solution 1 The state machine will look like this:\nImage: Solution 1 - State machine How it works # Within the State machine, we can create a step namely “Send message to DLQ” which is a SDK integration for SQS SendMessage functionality.\nIn this step, the message to be sent to the DLQ is built based on the execution input retrieved from the context parameters.\nIn the other steps of StateMachine, as required, we can configure the Catch errors in Error handling settings where we use the above “Send message to DLQ” step as the catcher.\nThis way, when an error happens in a state, we can send the message to the DLQ and re-process from there.\nTry this yourself # Here is a Github repository of a sample project I created to try out this solution. https://github.com/pubudusj/dlq-for-stepfunctions\nYou can deploy this to your AWS account using CDK.\nOnce deployed you can test the functionality by sending a message to the source SQS queue in below expected format.\n{ \u0026#34;metadata\u0026#34;: {}, \u0026#34;data\u0026#34;: { \u0026#34;foo\u0026#34;: \u0026#34;bar\u0026#34; } } And you can see the message ends up in the DLQ and the \u0026ldquo;metadata\u0026rdquo; object now includes the \u0026ldquo;attempt_number\u0026rdquo; as well. As an alternative for using the metadata section of the message to set the attempt number, you may use SQS message attribute as well.\nPlease note: In this approach, the DLQ is not a \u0026ldquo;real\u0026rdquo; DLQ and is not configured with the source SQS queue. However, it will help to capture any messages that failed to process by the Step Function execution.\nSolution 02 # In this method, we will use a real DLQ that is configured with the source queue.\nImage: Solution 2 The state machine will look like this:\nImage: Solution 2 - State machine How it works # There is a source SQS queue and a DLQ configured to it.\nIn the DLQ settings, the max receive count is set as \u0026gt; 1 so the message will be available in the DLQ immediately after the first failure.\nThere is a Lambda function which has the trigger set up to process messages from the source queue and initialize SF execution.\nIn the Event Source mapping setting of this lambda function, it is required to set the report_batch_item_failures to True.\nFirst, when the message is processed by the lambda function, we set the visibility timeout of the message to a larger value. This must be larger than the time it takes to complete the Step Function execution.\nThen, the step function execution will be started. Here, we pass the SQS queue url and the message receipt handler values along with the original values from the message from SQS.\nIn the example above, in order to determine if we need to simulate the failure, we use a simple choice state.\nIf it is a successful execution, we will call the step - Delete SQS Message. Here we use the SQS SDK integration to delete the message using the SQS queue url and the receipt handle values received in the input payload.\nIf it is a failure, we will call a step named - “Set message visibility timeout 5s”. Here we will use SQS SDK integration for the action: “changeMessageVisibility” to set the SQS message’s visibility to 5 seconds. For this SDK integration, we use the SQS queue url and the SQS receipt handle values passed in the execution input.\nOnce the message visibility is set to 5 seconds, it will again appear on the source queue after 5 seconds. However, since we have the rule ‘max receive count’ set to more than 1, the message will be immediately put into the DLQ of the source queue.\nTry this yourself # I have another Github repository for you to try this in your own AWS environment. You can set it up using CDK/Python. https://github.com/pubudusj/dlq-for-stepfunctions-2\nTo simulate a failure scenario, send a message into the source queue with a \u0026ldquo;failed\u0026rdquo; value as True.\n{ \u0026#34;foo\u0026#34;:\u0026#34;bar\u0026#34;, \u0026#34;failed\u0026#34;: true } This will make the step function execution fail and the message will be immediately available in the DLQ of the source queue.\nWith this approach, you can use the native DLQ functionality when we cannot process messages in the Step Function execution.\nSummary # Step Functions is one of the widely used AWS Serverless services. However it doesn’t support dead letter queues (DLQs) natively yet. Still there are workarounds to achieve this with few simple steps. This blog post explained two of such workarounds which help to build a better resilient system.\n","date":"26 May 2024","permalink":"/posts/dead-letter-queue-for-aws-step-functions/","section":"Posts","summary":"Step Functions - one of the main Serverless services offered by AWS for workflow orchestration - does not support dead letter queues natively. This is how we can work around to use DLQ with Step Functions.","title":"Dead Letter Queue (DLQ) for AWS Step Functions"},{"content":"In the current business landscape, organisations from startups to enterprises, are increasingly relying on software systems to drive operations, decision-making, and to get advantage over their competitors. Because of that, the reliability and efficiency of their software systems are critical.\nIn this context, the emergence of cloud computing has been a game-changer. To take the advantage of the cloud a step further, serverless technologies have been a choice of many organisations to innovate fast with scalability and cost efficiency.\nResilience of a software system can be defined as the ability to withstand and recover from unexpected failures, faults, or adverse conditions while maintaining essential functionality and performance.\nSo, there are 2 main goals to achieve:\nWithstand and recover from unexpected failures. Maintaining essential functionality and performance. In this blog post I discuss five simple solutions we implemented to improve the resilience of the AWS serverless application I am working on at PostNL — the EBE (EventBroker E-commerce)\nAbout PostNL and the EBE # PostNL is the no. 1 logistic provider in the Benelux region. On average, we deliver 6.9 million letters and 1.1 million parcels a day. This makes PostNL the preferred logistic choice in the region. To cater this demand, there are multiple software systems required to work together. Not only working together, but these systems must maintain a certain amount of performance to ensure a smooth operation. For an organisation like PostNL, it is really important to maintain high resilience as most of our systems are mission critical.\nImage: Components of the EBE application Event Broker E-commerce (EBE) is one of the mission critical applications within PostNL. It is a self service application and built using AWS serverless services.\nAs the name implies it is responsible for receiving messages from different producer applications and delivering them to one or more applications that are subscribed to those events. Producers send messages into EBE using SQS or HTTPS endpoints and EBE delivers the messages to the consumers via SQS or HTTPS end points. EventBridge — the serverless event bus offered by AWS — is the heart of the application that routes messages to consumers according to a given rule. For a given day, we process an average of 10 million messages within the EBE system.\nBelow are five design decisions taken to maintain robust resilience in the EBE system to make sure it recovers from unexpected failures while consistently delivering expected performance.\n1. Minimise the blast radius # We have hundreds of producers and consumers connected our systems. And the EBE system was architected in a way that there are very limited resources shared among individual producers and consumers. When a new producer or consumer registers in the system, we deploy a distinct set of resources (such as SQS queues, DLQs, Lambda functions, CloudWatch alarms, etc.) that are specific to the particular producer or consumer.\nImage: Independant stack for each producer/consumer With this approach of independent CloudFormation stacks, we ensure that any failure occurring in one producer does not impact other producers. Similarly, if one consumer encounters a failure, the remaining consumers remain unaffected by it. This design enhances high resilience within the system, ensuring smooth operation even during unexpected events.\nThis approach comes with its own set of trade-offs, particularly concerning AWS limits, which is suitable for a in-depth analysis in a separate blog post.\n2. Storage first # When a message is received by the EBE, the responsibility of the producer is over. Then, it is EBE application’s responsibility to deliver the message to the consumer application. Also, the communication between a producer and the EBE is one way. If EBE lost a message, there is no way we can request is from the producer application. So, it is crucial for the EBE to ensure that no messages are lost under any circumstances.\nImage: Storage first approach To prevent any message loss, we have implemented a storage-first approach. Every message received is stored within our system until it is successfully delivered to the consumer applications. Primarily, we utilise SQS Queues and DLQs to ensure message retention within the EBE until the deliveries are completed.\nFurther, the system archives all the messages using EventBridge, allowing consumers to replay events within a window of 3 days using EventBridge’s replay option.\nThis combination of the storage-first approach with a replay option ensures the messages are available within EBE until all the deliveries are successful, hence significantly enhancing the system’s reliability and resilience.\n3. Fallback plan — Dead Letter Queues (DLQs) # We use DLQs where ever possible. In the event of a failure occurring within the system, we use a DLQs to capture and manage it effectively. In that way we won’t lose the messages. So, DLQs are the fallback plan or the safety net in our application.\nAnother advantage of having DLQs are that we can set alarm based on the number of messages in the queue. This enables us to identify the specific part of the system encountering errors, including the exact producer or consumer component affected by the failure and quickly react to the failure.\nThere are many AWS services that support DLQ feature such as SQS, SNS, Lambda, EventBridge, StepFunctions, etc. They are very easy to set up, economical and can be very useful to improve the resilience of your system.\n4. Retries # Retrying is one of the fundamental actions we take if we cannot deliver a message to a consumer application in the first attempt. We make up to 100 attempts to deliver the message successfully to the customer within a 24-hour timeframe.\nIt is important that we don’t perform retries in regular intervals. We use Exponential Backoff And Jitter mechanism. This enables efficient retries without overloading our systems, while also safeguarding the consumer application.\nImage: Retry mechanism Buffer Queue is the event source to deliver the messages to the consumer applications. When the egress Lambda function failed to deliver the message to the consumer, it will be send back to the Buffer Queue, and we have max receive count to 1, which results in sending the message to the Buffer DLQ. The failure handler Lambda function consumes the messages in the Buffer DLQ, and calculates the next retry time based on the retry attempt. Here, we use either EventBridge Scheduler or SQS delayed messages feature to send the message back to the Buffer Queue for retrying in the specific time. This cycle continues until the message is delivered successfully or retried for 100 times within 24 hours.\nTypically, this retry mechanism is sufficient for delivering the message to the consumer successfully.\n5. Redrive # In some cases, there are messages that were retried 100 times yet failed to deliver to the consumer application. Then the system concludes that the message cannot be delivered to the consumer. So, those messages will be sent to a final DLQ. And system will automatically send a message to the owners of the consumer application notifying that there are un-deliverable messages available.\nAt this point, the owners of the consumer application can have 2 options in our self service web portal. They can choose to drop those messages and the DLQ will be purged. This is feasible because the DLQ is exclusively for that individual consumer. Else, they can choose to re-drive those failed messages. In re-drive, those messages will be put back to the Buffer Queue for another round of processing.\nImage: Redrive mechanism EBE is a “dumb pipe” which only delivers the messages to the consumers and it doesn’t have any knowledge over the contents and the processing of the messages. Because of this reason, system itself asks the respective owner of the consumer application to take action on the messages that were failed to deliver.\nConclusion # Due to the critical nature of the application it is really important to make sure the EBE application maintains a high resilience, otherwise it will directly affect the business continuity.\nBy implementing a well designed solution using the tools and serverless technologies provided by AWS, we were able to increase the resilience of the system and make sure it recovers itself from any unexpected failures and the effect is minimised so that most part of the application runs without any interruption.\n","date":"21 March 2024","permalink":"/posts/design-a-mission-critical-serverless-application-for-high-resilience/","section":"Posts","summary":"EBE is one of the mission critical Serverless applications at PostNL. We are maintaining the highest resilience possible to ensure business continuity. This blog explains the steps we took to achive high resilience.","title":"Design a mission critical serverless application for high resilience"},{"content":"","date":"21 March 2024","permalink":"/tags/resilience/","section":"Tags","summary":"","title":"Resilience"},{"content":"Hello there 👋 # I am Pubudu Jayawardana. Thanks for visiting my personal website.\nI am based in the Netherlands 🇳🇱, originally from Sri Lanka 🇱🇰\nI work as a senior solutions software engineer (cloud engineer) at PostNL, the Dutch national postal service.\nI am also an AWS Community Builder (since May 2021) in the Serverless category.\nA frequent blogger and public speaker, frequently engaging with the tech community.\nAWS certified - Solutions Architect Professional.\nProficient in a range of technologies, including AWS, Serverless, EDA, AWS CDK, and Python.\nPassionate about sharing my experiences and learnings, and always open to learning from others.\nConnect 🌐 # https://www.linkedin.com/in/pubudusj\nhttps://twitter.com/pubudusj\nhttps://github.com/pubudusj\nhttps://medium.com/@pubudusj\nhttps://dev.to/pubudusj\nPublic Talks🎤 # I love public speaking and below are some of the events where I\u0026rsquo;ve had the privilege of participating as a speaker.\n2026 # AWS UG NL MeetUp - Amsterdam - Better isolation for your SaaS tenants with Lambda tenant isolation AWS Summit Amsterdam - DevChats DEV209 Building modern serverless applications with AWS Lambda capabilities AWS re:Invent re:Cap - AWS Lambda tenant isolation 2025 # AWS Community Day CEE 2025 - Budapest - Rearchitecting a mission-critical serverless application for multi-tenancy AWS Summit Stockholm 2025 - Rearchitecting a mission-critical serverless application for multi-tenancy 2024 # AWS Re:Invent 2024 - From single to multi-tenant: Scaling a mission-critical serverless app (DEV341) AWS Community Day Netherlands - Moving a mission-critical serverless application from single-tenant to multi-tenant JAWS Pankration 2024 - Simplifying AWS Architectures: Exploring Lambda-less Solutions. AWS Summit Berlin 2024 - Design a mission-critical serverless application for high resilience. AWS Community Day Nordics 2024 - Copenhagen - Design a mission-critical serverless application for high resilience. AWS Meetup The Hague - PostNL - Design a mission-critical serverless application for high resilience. 2023 # AWS Community Day Amsterdam 2023 - How to achieve resilience in a high-performance Serverless system. Post-AWS On Tour Meetup Amsterdam - What can AWS Step Functions offer you to make your life easier? AWS Summit Amsterdam 2023 - Large scale parallel data processing with Step Function Distributed Map. AWS Summit Berlin 2023 - Large scale parallel data processing with Step Function Distributed Map. AWS re:Invent 2022 re:Cap Amsterdam - How to achieve resilience in a high-performance Serverless system. 2022 # AWS re:Invent 2022 - DevChats on Building a smart door bell with AWS Serverless. Serverless Saturdays, AWS UG Tirupathi India - Build a simple leave management system with AWS Serverless. AWS User Group Colombo, Sri Lanka - Manage webhooks at scale with AWS Serverless. AWS User Group Coimbatore, India - Create a Smart Doorbell with AWS Serverless. 2021 # JAWSpankration by AWS User Group Japan - How to create a simple OTP system with AWS Serverless services. Previous Nextsads Podcast Appearance 🎙️ # Amazon EventBridge Pipes: A new approach to event driven architecture @ DevOps with Zack ","date":"4 January 2024","permalink":"/about/","section":"Pubudu.Dev","summary":"Hello there 👋 # I am Pubudu Jayawardana.","title":"About Me"},{"content":"Until re:Invent 2023, if you needed to call an external API from Step Functions execution, you had to use a Lambda function. And you needed to manage the responses and make the execution fail or success based on that. Also, most probably due to the fact that the external API is protected, custom code is required to manage the authorisation of the api call.\nAs you see this involves significant custom code which needs to be maintained by the developers.\nGood news is, in the re:Invent 2023, AWS has introduced native integration to HTTPS APIs from Step Functions which allows to call to any 3rd party API endpoint as a step in the execution and, based on the response, you can perform any business logic remaining in the state machine.\nHow it works # In Step Functions, it utilises EventBridge connection to manage authentication credentials for the connection to the 3rd party API.\nCreate EventBridge Connection # In the EventBridge connection, you can select 3 different authentication options.\nBasic (Username/Password) OAuth Client Credentials API Key For this example, we will use OAuth client credentials as the authorisation mechanism.\nTo create a EventBridge connection with OAuth client credentials, you have to provide below information.\nClient id Client secret Auth endpoint HTTP method Once the EventBridge connection creation request is received, first it stores the provided OAuth credentials in AWS Secret Manager.\nNext, it calls the auth endpoint with the given client id and client secret to obtain an auth token.\nIf the auth call is successful, then the auth token will be securely saved in the same secret previously created in the secret manager. And this token will be used in the calls to the api endpoint.\nIf the auth attempt failed, it simply de-register the EventBridge connection and it cannot no longer be usable. Once the EventBridge connection is successfully created, we can use it in a State Machine as the authentication option for the api endpoint.\nCreate State Machine with 3rd party API # In the State Machine, simply add a \u0026ldquo;Call third-party API\u0026rdquo; step and, add the configurations - API endpoint, HTTP method. For authentication, enter the ARN of the EventBridge Connection created previously.\nAlso, you can configure the request payload that sends to the API and you may use the reference paths to build the payload from the runtime data.\nRefreshing OAuth Token # Normally, OAuth uses short-lived tokens. When an API request uses a token that has already expired, it returns 401 Unauthorized error. In such a case, you will need to first renew the token by calling the auth endpoint and use the new token to call the API.\nIn the Step Function HTTP API call, this is taken care of by the EventBridge Connection.\nLet\u0026rsquo;s assume the API returns a 401 error which is the default OAuth response when the token is expired. In that case, EventBridge Connection will automatically call its auth endpoint and retrieve a new token. And update the secret manager entry.\nSo, in the State Machine, you need to set up \u0026ldquo;Retry\u0026rdquo; for the \u0026ldquo;Call third-party API\u0026rdquo; step for the specific error - \u0026ldquo;States.Http.StatusCode.401\u0026rdquo;. This retry will automatically resolve the unthorized error without any additional steps.\nYou can always catch specific error using the HTTP error code as follows: States.Http.StatusCode.[Status_Code]\nIn EventBridge Connection, OAuth tokens are refreshed when a 401 or 407 response is returned.\nSince Step Functions execution uses EventBridge Connections and Secret Manager, necessary permissions must be set in the State Machine role.\nTry this yourself # I have created a sample application that uses Step Function step to call an external API. Here, I have used CDK with Python. Github repository can be found at: https://github.com/pubudusj/step-functions-https-api-integration\nClone the repository. Install the dependencies and deploy the stack using CDK CLI. Once deployed, it will create 2 Lambda functions with Function URLs. One Function URL will be used as the API endpoint while the other will be used as the autorization endpoint. EventBridge connection is also created with autorization endpoint Function URL. And, a State Machine will be created with a single step to call the API endpoint and it has 2 retries set up if API returns status code 401. And authorization via the EventBridge Connection. Images: State Machine with Call 3rd party API state Testing # As soon as the stack is deployed, you can see there is a secret created in Secret Manager with the given client id, client secret and newly generated auth token.\nYou can initialize a Step Functions execution with below input.\n{ \u0026#34;set401\u0026#34;: false } You can see it successfully completes and the output includes the return value from the API Lambda function. Also, in the Lambda logs, you can see the auth token from the secret is being used in the header as the Bearer token.\nTo simulate a 401 error, initialize a Step Functions execution with below input:\n{ \u0026#34;set401\u0026#34;: true } With this input, the API Lambda function, will return a 401 error for the initial attempt. And the state will immediately start retrying. And the step will be successful in the retries.\nIf you check the secret in the Secret Manager, you will be able to see the token has updated with new value. Also, if you check the API Lambda\u0026rsquo;s logs, you can see the request headers include different Bearer token values in the initial and the retry attempts.\nExpress vs Standard workflows # This feature works well in both Express and Standard State Machine types. However, it is wise to keep in mind that (even generally express workflows are cost effective compared to the standard workflows) cost of the express workflows depend on the execution duration too. So, if your APIs are slow, it will cost more. For Standard type, it will be a single state transition, so the time it takes to call the API will not be a concern for pricing.\nSummary # The introduction of HTTP integration support in AWS Step Functions is a significant enhancement that simplifies the process of invoking external APIs. This feature not only reduces the custom Lambda code needed for API calls but also eliminates the necessity for custom code to handle complex retry logic when generating and renewing authentication tokens.\nWith this capability, developers can seamlessly utilise third-party APIs with minimal code, and enhance the overall efficiency of workflow orchestration in AWS Step Functions.\nUseful Links # Step Functions external endpoints AWS Blog: https://aws.amazon.com/blogs/aws/external-endpoints-and-testing-of-task-states-now-available-in-aws-step-functions/ Step Functions external endpoints documentation: https://docs.aws.amazon.com/step-functions/latest/dg/connect-third-party-apis.html EventBridge Connection documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-api-destinations.html#eb-api-destination-connection ","date":"9 December 2023","permalink":"/posts/call-external-apis-with-oauth-within-step-functions/","section":"Posts","summary":"This blog post shows how we can call external HTTPS endpoint within Step Function execution using new native HTTPS integration.","title":"Call external APIs with OAuth within Step Functions"},{"content":"","date":"9 December 2023","permalink":"/tags/eventbridge-connection/","section":"Tags","summary":"","title":"EventBridge Connection"},{"content":"For some requirements, you will need to record all the events that go through your EventBridge Event Bus. CloudWatch can be a suitable target for this. https://repost.aws/knowledge-center/cloudwatch-log-group-eventbridge\nIn this blog post, I am going to discuss how we can implement 3 different rules that can be used to implement catch-all functionality for a EventBridge event bus.\nUsing prefix # You can use the “prefix” pattern matching feature of the EventBridge rule to capture all the events. Keeping the prefix value to an empty string will do the trick.\nEventRuleCatchAllWithPrefix: Type: AWS::Events::Rule Properties: Description: \u0026#34;EventRule to catch all using prefix\u0026#34; EventBusName: !Ref MyEventBus EventPattern: source: - prefix: \u0026#34;\u0026#34; Targets: - Arn: !GetAtt CatchAllWithPrefixLogGroup.Arn Id: \u0026#34;TargetCatchAllWithPrefix\u0026#34; Refers docs of prefix pattern here: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns-content-based-filtering.html#eb-filtering-prefix-matching\nIn this example, I have used the field “source” to apply the prefix filter since each and every event going through an event bus will have a source field. As an alternative, you may use any field that exists in the event.\nUsing version # Similar to the prefix matching, we can exactly match the “version” of the event to capture all the events. For all the events going through Event Bus will include the version field with value 0. As of now, there are no other version values available other than 0, but this might change in future.\nHere is an example how you can define the catch all rule with exactly matching the version.\nEventRuleCatchAllWithVersion: Type: AWS::Events::Rule Properties: Description: \u0026#34;EventRule to catch all using version\u0026#34; EventBusName: !Ref MyEventBus EventPattern: version: [\u0026#34;0\u0026#34;] Targets: - Arn: !GetAtt CatchAllWithVersionLogGroup.Arn Id: \u0026#34;TargetCatchAllWithVersion\u0026#34; Using wildcard # EventBridge recently announced the support for wildcards in their event rules ( https://aws.amazon.com/about-aws/whats-new/2023/10/amazon-eventbridge-wildcard-filters-rules/).\nWe can use this to form the catch all rule as follows. Use any field that exists in the event all the time (here, the “source” field) and apply the wildcard “*”.\nEventRuleCatchAllWithWildcard: Type: AWS::Events::Rule Properties: Description: \u0026#34;EventRule to catch all using wildcard\u0026#34; EventBusName: !Ref MyEventBus EventPattern: source: - wildcard: \u0026#34;*\u0026#34; Targets: - Arn: !GetAtt CatchAllWithWildcardLogGroup.Arn Id: \u0026#34;TargetCatchAllWithWildcard\u0026#34; Refer docs of wildcard pattern here: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-event-patterns-content-based-filtering.html#eb-filtering-wildcard-matching\nTry this yourself # Here is the Github repository I created to show this functionality. You can deploy this into your AWS environment using AWS SAM CLI.\nhttps://github.com/pubudusj/eventbridge-catch-all\nOnce deployed, it will create an Event Bus, 3 different rules as discussed above and 3 different CloudWatch Logs as targets for those rules.\nWhen you send any message into the event bus, you can see them end up in all the CloudWatch Logs.\nSummary # AWS is well known for providing more than one method to achieve the same results. This is one of the examples, where you can implement a catch all functionality for your event bus defining rules in 3 different ways.\n","date":"1 November 2023","permalink":"/posts/3-ways-to-catch-all-the-events-going-through-the-eventbridge-event-bus/","section":"Posts","summary":"This blog post shows how we can implement 3 different rules that can be used to implement catch-all functionality for a EventBridge event bus.","title":"3 ways to catch all the events going through the EventBridge Event Bus"},{"content":"In a previous blog post, it was discussed how we can build a custom retry mechanism for Lambda with SQS. There we used SQS feature- delayed messages - to set the next retry time.\nIf you haven\u0026rsquo;t already read that, you can find it here: https://pubudu.dev/posts/implement-custom-retry-logic-with-sqs-lambda-part-i-using-sqs-delayed-messages\nOne limitation of that approach was we can only delay messages for up to 900 seconds (15 minutes). So, the next retry must be within 15 minutes.\nIn this blog post, I am going to discuss how we can address this limitation by replacing delayed messages with scheduling messages using EventBridge Scheduler.\nArchitecture # Images: Architecture How it works # Let\u0026rsquo;s assume you have a message with this structure. Here, data includes the payload your application needs and metadata is used to identify the message and any information that is used by infrastructure to process the message.\n{ \u0026#34;metadata\u0026#34;:{ \u0026#34;message_id\u0026#34;:\u0026#34;22200000-0749-2113-8e00-000a2b111191\u0026#34; }, \u0026#34;data\u0026#34;:{ // message payload needs to be processed. } } Source Queue is the entry point where this message is added to be processed by Message Processor Lambda Function.\nIn Message Processor Lambda Function, first it validates the message scheme to verify the metadata and data elements are there and the message Id is included in the meta data as an uuid.\nIf this schema validation fails, it sends the message directly into Final DLQ without retrying, because simply retrying will not fix this issue.\nIf the message is valid, let\u0026rsquo;s assume that LambdaX needs to call an external system to successfully process the message, but this external system has some issues so the message processing fails. Then it raises a Runtime exception.\nThen the message is sent back to the source queue.\nOn the Source Queue, there is a Dead Letter Queue (namely Intermediate DLQ) configured. Also, there\u0026rsquo;s a condition set for message max receive count as 1. So, when the failed message is received, it will be sent directly into the DLQ.\nThis Intermediate DLQ has a Lambda function (Failed Message Processor Lambda Function) listening to it.\nSo, this Failed Message Processor Lambda function starts processing the message.\nThe Failed Message Processor Lambda function has several responsibilities.\nFirst it increments the retry attempt count by one if it already exists or if not, set as 1 and add it into the metadata of the message.\nThen, it checks if the number of retry attempts exceeds the pre-defined limit. Here I have used a max retry attempt as 5.\nIf retry attempts exceeded the max amount, then the message is moved into the final DLQ with a Error Type and Error Details as follows:\nImages: Message attributes in Final DLQ If not, it calculates the next retry time based on the logic you provided. Here I used a simple backoff logic. Here, the retry attempt count is multiplied by 60 seconds and then added to the current time to determine the time for the next retry. def _calculate_next_retry_time(message): retry_attempt = message[\u0026#34;metadata\u0026#34;][\u0026#34;retry_attempt\u0026#34;] next_retry_time = datetime.now() + timedelta(seconds=(60 * retry_attempt)) message[\u0026#34;metadata\u0026#34;][\u0026#34;next_retry_time\u0026#34;] = new_datetime.strftime( \u0026#34;%Y-%m-%dT%H:%M:00.0Z\u0026#34; ) return next_retry_time, message Then one time schedule is created in Amazon EventBridge Scheduler given the calculated time.\nPlease Note: With EventBridge scheduler, we can only schedule with minute precision, but not down to the exact second.\nOnce the schedule is created, at the given time, it will call the SQS send message API to queue the message with the given payload.\nMessage that is sent to the Source Queue will look like below:\n{ \u0026#34;metadata\u0026#34;:{ \u0026#34;message_id\u0026#34;:\u0026#34;22200000-0749-2113-8e00-000a2b111191\u0026#34;, \u0026#34;retry_attempt\u0026#34;:1, \u0026#34;next_retry_time\u0026#34;:\u0026#34;2023-10-13T22:52:00.0Z\u0026#34; }, \u0026#34;data\u0026#34;:{ \u0026#34;location_name\u0026#34;:\u0026#34;Amsterdam\u0026#34;, \u0026#34;location_id\u0026#34;:12345 } } Once the message is available to process in the Source Queue, this cycle will continue until the message is processed successfully or sent to the Final DLQ.\nTry this yourself # I have a public GitHub repository created if you need to set up this project in your own AWS environment.\nClone the repository at: https://github.com/pubudusj/retry-with-sqs-and-scheduler\nSet up # This is implemented with CDK and Python. So you need to have Python, AWS and CDK CLIs installed.\nTest # Once stack is deployed, send below message into the Source Queue:\n{ \u0026#34;metadata\u0026#34;:{ \u0026#34;message_id\u0026#34;:\u0026#34;22200000-0749-2113-8e00-000a2b111191\u0026#34; }, \u0026#34;data\u0026#34;:{ \u0026#34;location_name\u0026#34;:\u0026#34;Amsterdam\u0026#34;, \u0026#34;location_id\u0026#34;:12345 } } Result # If you check the log group of Message Processor and Failed Message Processor Lambdas, you can see the records of the message payload with different retry attempts and next retry times for each retry attempt.\nAlso, after a retry attempt, if you check the EventBridge Scheduler, you can see there is a schedule created with the payload to be sent to SQS and with a reschedule time calculated in the Failed Message Processor Lambda.\nAlso, after 5 unsuccessful retries, you can see that message will be available in the Final DLQ with the error attributes.\nAdvantage of this approach # Advantage of this approach is that you can schedule the message for a longer period. Even though practically might not be required a long time, EventBridge schedule trigger time can be set any time into the future.\nAlso, unlike the previous approach, you have some more visibility when the message will be retried, based on the schedule created in the EventBridge Scheduler.\nFurther, AWS has recently announced the feature to automatically delete EventBridge schedules once they are triggered. Which is great, so you don\u0026rsquo;t need to worry about keeping track of the schedules and deleting them manually or programmatically within your application after they are triggered.\n","date":"26 October 2023","permalink":"/posts/implement-custom-retry-logic-with-sqs-lambda-part-ii-using-eventbridge-scheduler/","section":"Posts","summary":"This explains a way to implement custom retry logic with EventBridge Scheduler and Lambda.","title":"Implement custom retry logic with SQS \u0026 Lambda - Part II - using EventBridge Scheduler"},{"content":"It is a very common pattern that you process messages in a SQS queue with a Lambda function. And when there is an error in the Lambda execution, the message will be sent back to the queue if the retries are not set up. If the retries are set up in the Lambda function, it will perform the retries and if still not successful, the message will be sent back to the queue.\nAnd depending on the visibility timeout of the message, this process continues until the message is processed successfully or deleted from the queue.\nBasically, this retry mechanism depends on the visibility timeout and happens in regular intervals.\nSometimes, we might need more control over the retries by retrying in arbitrary intervals or having exponential backoff. With default behaviour of SQS, Lambda and Event Source Mapping, this is not simply possible.\nHere, I am going to discuss a solution where we can retry a failed message with a custom interval and with a condition based on how many times the message should be retried.\nArchitecture # Images: Architecture How it works # Let\u0026rsquo;s assume you have a message with this structure. Here, data includes the payload your application needs and metadata is used to identify the message and any information that is used by infrastructure to process the message.\n{ \u0026#34;metadata\u0026#34;:{ \u0026#34;message_id\u0026#34;:\u0026#34;22200000-0749-2113-8e00-000a2b111191\u0026#34; }, \u0026#34;data\u0026#34;:{ // message payload needs to be processed. } } Source Queue is the entry point where this message is added to be processed by Message Processor Lambda Function.\nIn Message Processor Lambda Function, first it validates the message scheme to verify the metadata and data elements are there and the message Id is included in the meta data as an uuid.\nIf this schema validation fails, it sends the message directly into Final DLQ without retrying, because simply retrying will not fix this issue.\nIf the message is valid, let\u0026rsquo;s assume that LambdaX needs to call an external system to successfully process the message, but this external system has some issues so the message processing fails. Then it raises a Runtime exception.\nThen the message is sent back to the source queue.\nOn the Source Queue, there is a Dead Letter Queue (namely Intermediate DLQ) configured. Also, there\u0026rsquo;s a condition set for message max receive count as 1. So, when the failed message is received, it will be sent directly into the DLQ.\nThis Intermediate DLQ has a Lambda function (Failed Message Processor Lambda Function) listening to it.\nSo, this Failed Message Processor Lambda function starts processing the message.\nThe Failed Message Processor Lambda function has several responsibilities.\nFirst it increments the retry attempt count by one if it already exists or if not, set as 1 and add it into the metadata of the message.\nThen, it checks if the number of retry attempts exceeds the pre-defined limit. Here I have used a max retry attempt as 5.\nIf retry attempts exceeded the max amount, then the message is moved into the final DLQ with a Error Type and Error Details as follows:\nImages: Message attributes in Final DLQ If not, it chooses a random number of seconds between 0 and 900 and sends a message to the Source Queue with the same payload with the delay time as this random seconds. Also, this message contains the next retry attempt time based on the delay.\nMessage that was sent to the Source Queue will look like below:\n{ \u0026#34;metadata\u0026#34;:{ \u0026#34;message_id\u0026#34;:\u0026#34;22200000-0749-2113-8e00-000a2b111191\u0026#34;, \u0026#34;retry_attempt\u0026#34;:1, \u0026#34;next_retry_time\u0026#34;:\u0026#34;2023-10-13T22:52:00.091307Z\u0026#34; }, \u0026#34;data\u0026#34;:{ \u0026#34;location_name\u0026#34;:\u0026#34;Amsterdam\u0026#34;, \u0026#34;location_id\u0026#34;:12345 } } Once the message is sent to the Source Queue, it will not be available to process until it reaches the delayed seconds defined.\nOnce the message is available to process, this cycle will continue until the message is processed successfully or sent to the Final DLQ.\nTry this yourself # I have a public GitHub repository created if you need to set up this project in your own AWS environment.\nClone the repository at: https://github.com/pubudusj/retry-with-sqs-delay-messages\nSet up # This is implemented with CDK and Python. So you need to have Python, AWS and CDK CLIs installed.\n###Test: Once stack is deployed, send below message into the Source Queue:\n{ \u0026#34;metadata\u0026#34;:{ \u0026#34;message_id\u0026#34;:\u0026#34;22200000-0749-2113-8e00-000a2b111191\u0026#34; }, \u0026#34;data\u0026#34;:{ \u0026#34;location_name\u0026#34;:\u0026#34;Amsterdam\u0026#34;, \u0026#34;location_id\u0026#34;:12345 } } Result # If you check the log group of Message Processor and Failed Message Processor Lambdas, you can see the records of the message payload with different retry attempts and next retry times for each retry attempt.\nAlso, after 5 unsuccessful retries, you can see that message will be available in the Final DLQ with the error attributes.\nPlease Note: The max no of seconds can be set as the delay time in SQS is 900 seconds (15 minutes). So, this solution works only if you need to retry within the next 15 minutes.\nSummary # It is a common use case where you have to retry failed messages. However, the default retry mechanism provided by SQS \u0026amp; Lambda might not be sufficient for your business logic.\nMain advantage of this approach discussed here is that you have more control over the retry mechanism and you can fine tune it as per your business needs. For example, here, you can control the max retry attempts and the frequency and the logic of how to determine the next retry attempt time.\n","date":"16 October 2023","permalink":"/posts/implement-custom-retry-logic-with-sqs-lambda-part-i-using-sqs-delayed-messages/","section":"Posts","summary":"This explains a way to implement custom retry logic with SQS and Lambda.","title":"Implement custom retry logic with SQS \u0026 Lambda - Part I - using SQS delayed messages"},{"content":"It is possible for an EventBridge bus to have a rule that sends messages to another EventBridge bus. But, what happens if the second EB bus has a rule that sends any message it receives back to the first EB bus? As expected, there will be a recursive data transfer between two EB buses.\nIn this article, let’s discuss how to address such recursive messages passing between two EB buses.\nFirst, let\u0026rsquo;s look at the architecture.\nImages: Architecture You can find the CDK project that includes the source code to set up this system in your AWS environment using AWS CDK and Python here: https://github.com/pubudusj/eb-to-eb\nHow it works # Once the project is set up, this will create 2 EB buses. Each bus will have two rules.\nOne rule has the other EB bus as the target. The other rule will have a CloudWatch group as a target which will log each message received into the bus. Also each bus will have a Dead Letter Queue setup to preserve any messages that cannot be processed.\nTo test, you can send any message into EB bus A.\nThe expected behaviour is that the message is being sent between the EB buses as a loop, so that there will be a record in CloudWatch Log group for each receipt.\nThe actual behaviour is that, the message is sent from bus A to bus B. CloudWatch groups A has the message logged. Also, CloudWatch group B has logged the message which means the message was received to bus B.\nHowever, those are the only available CloudWatch log records, which means the message was not sent back to bus A.\nInstead, if you check the bus B’s dead letter queue, you can see the message is available there. Also, if you check the message attributes, you can see the error code as “THIRD_ACCOUNT_HOP_DETECTED”. And the error message says — “Event ingestion rejected for event [event id] because an event can be sent to an event bus target only once. This event was previously delivered to an event bus target.”\nImages: Attributes of Failed Message in SQS Which means, at bus B, it failed to process the message when try to send it back to bus A.\nExplanation # Well, this is the default behaviour of Amazon EventBridge Event Buses. EventBridge Bus has a mechanism to not send the same message to another event bus target more than once. In such a scenario, it will simply reject the message, and in this case, since we have DLQ set up, the message was sent there.\nBecause of this feature of Amazon EventBridge, recursion loop is prevented. So, you don’t need to take any additional steps to prevent the recursion here.\n","date":"25 July 2023","permalink":"/posts/manage-amazon-eventbridge-bus-to-bus-recursive-message-passing/","section":"Posts","summary":"Let’s discuss how to address recursive messages passing between two EventBridge buses.","title":"Amazon EventBridge - How to manage Bus to Bus recursive message passing"},{"content":"","date":"18 July 2023","permalink":"/tags/recursion/","section":"Tags","summary":"","title":"Recursion"},{"content":"Recently, AWS Lambda has introduced a new feature to detect and stop recursive loops in Lambda functions most probably due to mis-configuration. This is definitely a great feature to save some thousands of dollars.\nI created a simple Serverless application that you can deploy into your AWS environment and test this new feature. This is built in CDK and Python.\nArchitecture # Images: Recursion between SQS and Lambda Set up # This is a CDK project implemented with Python. So, you need CDK and Python installed in your local environment.\nClone the repository: https://github.com/pubudusj/lambda-recursion-test Go into the cloned directory. To manually create a virtualenv on MacOS and Linux: $ python3 -m venv .venv After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv. $ source .venv/bin/activate If you are a Windows platform, you would activate the virtualenv using: % .venv\\Scripts\\activate.bat Once the virtualenv is activated, you can install the required dependencies. $ pip install -r requirements.txt Then, deploy the application: $ cdk deploy Once the stack is created, note the SQSQueue and LambdaFunction values. How to Test # Add any message into the SQS queue. This will trigger the Lambda function. Within the Lambda execution, it sends the same message into the same SQS queue. This creates an endless loop. Result # Before this new feature, this loop will continue as much as possible and will cost you a lot. Now, this loop will end after 16 Lambda executions. Images: Lambda function invoked only 16 times Then, you can see the message was sent to the DLQ. You also receive the below email from AWS notifying the loop detection. Please note that this email will be sent once per 24 hours per function. Images: Email from AWS notifying the loop detected Important to know # As at now, only SQS and SNS support this loop detection. To support this feature, your SDK must be in a certain version or higher. Refer the documentation here for more information You can turn off this feature if you want (but why?) by contacting AWS support. AWS documentation # https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html\n","date":"18 July 2023","permalink":"/posts/testing-aws-lambda-recursive-loop-detection/","section":"Posts","summary":"This explains the new Lambda recursive loop detection and how to test this new feature.","title":"Testing AWS Lambda Recursive Loop Detection"},{"content":"","date":"14 June 2023","permalink":"/tags/iam/","section":"Tags","summary":"","title":"IAM"},{"content":"It is a best practice and also, one of the AWS Security Hub controls for IAM to have all the IAM user access keys to be rotated every 90 days.\nIt might not be an easy task to check each and every user access key to check if they pass this validation.\nLambda is a great tool not only to be part of a Serverless system but also small house keeping tasks like this. In this blog post, I am going to discuss how we can use a simple AWS Step Function to go through all the user access keys in the AWS account and send a notification if there are keys that were created more than 90 days ago.\nArchitecture # Images: Architecture State machine # Images: State machine As you see, this is a very simple system with very few components.\nThere is an AWS Schedule that runs periodically that initializes a Step Functions execution.\nWithin Step Functions execution, first there is a SDK integration to IAM to get all the users available in the AWS account.\nThen, the result is sent to a Distributed Map state to iterate through each and every user.\nWithin the Distributed Map state, there is another SDK integration to IAM to get all the access keys for a particular user.\nListAccessKeys of IAM will provide the access key details in the below structure:\n{ “AccessKeyId”: “AKIXXXXXXXXX”, “CreateDate”: “2021–11–27T20:15:43Z”, “Status”: “Active”, “UserName”: “pubudu” } Once all the Distributed Map iterations are completed, all the results will be sent to the KeyValidationLambda function as input.\nWithin the KeyValidationLambda function, all the key data are iterated and using the CreatedData, it checks if the particular access key was created 90 days ago.\nIf there are older access keys, it will be sent back to the SF execution.\nIn the next step, there is a Choice state where it checks if there are results returned from the Lambda. If so, it sends this data into a SNS Topic via SDK integration. Then it ends the execution.\nIf there is no result returned from Lambda, SF execution will end successfully.\nYou can subscribe to the SNS topic in many ways (ex: email, Slack, etc) so you will get notified when a message is published to the topic.\nTry this yourself # You can test this solution in your AWS account using AWS SAM with Python. Source code for this project is available at https://github.com/pubudusj/aws-iam-key-checker\nSummary # AWS Step Functions SDK integration is very powerful to build not only full scale Serverless functionality but also this kind of automation around security best practices.\nAt the moment, AWS Step Functions doesn’t support date time related intrinsic functions. When those are available, access key validation checks can be done within the execution which will further reduce the custom code (within the Lambda function).\nUseful Links # Step Functions SDK integrations — https://docs.aws.amazon.com/step-functions/latest/dg/supported-services-awssdk.html\nAWS Security Hub IAM controls — https://docs.aws.amazon.com/securityhub/latest/userguide/iam-controls.html\n","date":"14 June 2023","permalink":"/posts/stay-compliant-with-aws-step-functions-iam-users-access-key-age-notification/","section":"Posts","summary":"This blog post shows to turn off AppConfig Feature Flag based on system behaviour.","title":"Stay compliant with AWS Step Functions — IAM Users’ Access Key Age Notification"},{"content":"It is a very common pattern that you use a Lambda function as the consumer of a SQS FIFO queue. Also SQS FIFO is one of the few services that supports message ordering.\nIn this blog post, I am going to discuss how Lambda concurrency is determined when a SQS FIFO queue is configured as the source of the function.\nMessage Group ID # Message group id is the property of a message that indicates the group within a FIFO queue it belongs to. All the messages that are in the same group will be processed one by one as per the order they are added to the queue.\nWhen processing, sequences of the messages that are in different groups are independent from one another. So, messages with different group ids might be processed irrelevant of the order they are added to the queue.\nLambda concurrency vs Message Group ID # Lambda concurrency is how many Lambda executions can be run at a given time. It is interesting how the lambda concurrency is calculated when it comes to SQS FIFO queue as the source.\nFirst, SQS FIFO queue looks through the first 20,000 messages that are available in the queue.\nThen, it gets the distinct number of message group ids that those 20,000 messages belong to.\nThen, Lambda service initialise a Lambda execution per message group id.\nSo that concurrency is set based on the no of message group ids available in the 20,000 messages.\nFor an example:\nLet\u0026rsquo;s assume there are more than 20,000 messages in the queue that belong to 3 message group ids and messages with all 3 message ids included in the first 20,000 messages.\nThen, Lambda service polls the first 20,000 messages.\nSince there are 3 distinct message ids within this 20,000 messages, at this point, Lambda concurrency will be set to 3.\nLikewise after each poll, Lambda service calculates the concurrency based on the available message group ids.\nImage: Messages in queue with different group ids Test this yourself # I have created a sample application to test this scenario. You can deploy it to your AWS account using CDK with Python.\nClone the repository: https://github.com/pubudusj/sqs-fifo-lambda-concurrency-test\nRun pip install -r requirements.txt to install necessary dependancies.\nRun cdk deploy to deploy the stack.\nThis stack consists of a SQS FIFO queue, a consumer Lambda function and another Lambda function to put some sample messages into the SQS queue for testing.\nWhen deployed, the consumer Lambda function has the SQS FIFO queue as the source but it is not yet enabled.\nOnce deployed, first run the message generator Lambda function. This will add 60,000 messages that belong to 3 message groups into the SQS queue.\nOnce all messages are in the queue, enable the SQS source of consumer Lambda function using AWS console or change the CDK code here.\nThis will start consuming the messages in the SQS queue.\nOnce all the messages are processed, go to the Lambda\u0026rsquo;s CloudWatch metrics and see the \u0026lsquo;Total concurrent executions\u0026rsquo; graph.\nYou can see the maximum concurrency that was set during the processing is only 3.\nImages: Concurrent executions Conclusion # As you see, message group id directly affect the Lambda concurrency when processing messages in a SQS FIFO queue. Try to avoid inserting more messages with the same message group ID as this may lead to large backlog of messages.\nTherefore, as per your business requirement, using a suitable group ID is the key to maximize concurrency when processing data from an SQS FIFO queue. However, if you need a set of messages to be processed sequentially, they must be assigned to the same group. Keeping the number of messages within each group as small as possible ensures that the messages are processed in order while allowing for higher parallelism across different groups.\nUseful Links # SQS FIFO Queue Developer Guide - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html\nHigh throughput for FIFO queues - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/high-throughput-fifo.html\n","date":"3 May 2023","permalink":"/posts/aws-lambda-concurrency-when-sqs-fifo-queue-as-trigger/","section":"Posts","summary":"In this blog, it is discussed how Lambda concurrency is determined when a SQS FIFO queue is configured as the trigger.","title":"AWS Lambda Concurrency when SQS FIFO Queue as Trigger"},{"content":"","date":"3 May 2023","permalink":"/tags/concurrency/","section":"Tags","summary":"","title":"Concurrency"},{"content":"It is a common use case to load data to a database from a large set of files in formats such as CSV, JSON or parquet. This is especially useful in an ETL process where you need to load data into a database in a regular interval.\nThere are many factors that decide how fast and reliable your data is loading. Apart from the limitations the database engine has, capabilities of the service/tool you are using is a major factor in how efficiently the data load can be performed. For example, data can be loaded parallel, sequential or may be as batches. And having a better retry mechanism is also important when there is a failure.\nIn this post, I discuss how we can use AWS Step Function’s latest feature — Distributed Map — to load CSV files located in S3 bucket into a Postgres database. Using Distributed Map, it is easy to parallelise the data loading and effectively retry if there’s any failure. Also you will have better visibility in case of a failure.\nHow it works # Here, we utilise the AWS S3 extension for Postgres to load data from s3 to Postgres.\nWe first load data into a temporary table and once the data load is completed, the existing table is backed up and then the temp table will be renamed to the main table. For that we use two additional schemas along with the default ‘public’ schema of the database.\nUsing Step Functions, we orchestrate these steps in one execution to load data for a single table.\nPrerequisites # Create a RDS Postgres database with public access**\nCreate a parameter in SSM Parameter Store to store the connection string to Postgres database Ex: in the format of postgresql://[username]:[password]@[RDSEndpoint]:[Port]/[DatabaseName]\nCreate two additional schemas in the database as follows:\nCREATE SCHEMA dataimport; CREATE SCHEMA databackup; Install AWS S3 extension for Postgres\nCREATE EXTENSION aws_s3 CASCADE In order to install the application, you need AWS SAM CLI installed in your local machine.\nPlease note: For demonstration purposes, RDS instances need to be created with public access. For a production system, you need to deploy the RDS instance more securely in private subnets and update IAC code mentioned below accordingly for Lambda functions to access RDS instance.\nSet up # Clone the repository https://github.com/pubudusj/s3-to-postgres-with-stepfunctions\nRun below command to install the project: sam build \u0026amp;\u0026amp; sam deploy -g\nThis will prompt you to input DbConnectionStringSSMPath and RDSDatabaseArn. Please note: Input DbConnectionStringSSMPath without leading slash /\nOnce the stack is deployed, there will be 3 values in the output: S3SourceBucket, RDSRole and SNSTopic.\nAttach the RDS role into the database under Manage IAM roles with s3Import as the feature.\nYou can subscribe to the SNS topic created, so you will be notified in case of a failure.\nState Machine # Image: State Machine Testing with Sample Data # To test the functionality, I have used the free data set available at https://registry.opendata.aws/noaa-gsod/. This includes a lot of data, so for the testing purposes, use only the data for 2021 and 2022.\nAWS Cloud shell is best for directly syncing these data into your source s3 bucket. Open a cloud shell in your AWS console and run the command below replacing the S3SourceBucket value from the stack output.\naws s3 sync s3://noaa-gsod-pds/2021 s3://[S3SourceBucket]/year_2021 aws s3 sync s3://noaa-gsod-pds/2022 s3://[S3SourceBucket]/year_2022 This will sync data to your s3 bucket directly from the source.\nIf you have your own dataset, update the get_queries_for_table function at src/share_resources/db.py with the table structure relevant to your CSV data set and re-deploy the stack. Also, the table name must match the path of your data set in S3 bucket. Ex: s3://[S3SourceBucket]/[table_name]/abc.csv\nOnce test data is in your source s3 bucket, initialise a Step Function execution with below input.\n{ \u0026#34;table\u0026#34;: \u0026#34;year_2021\u0026#34; } This will start sync data into your Postgres table: year_2021\nAdjustments/Customisations # This data load functionality depends on the performance of your database. So you might need to adjust MaxConcurrency and MaxItemsPerBatch parameters in your State Machine. You can adjust the ToleratedFailurePercentage value to continue to load data even if the error limit is within this value.\nHere, I use one retry if there is a failure, but by modifying the logic in the ShouldRetry step, you can increase this.\nYou can extend this functionality by adding indexes, constraints, stats, etc. to the table. For that, you can modify the Rename Table Lambda function.\nTips/Lesson Learned # Always use batching and set sensible max concurrency — This will help to control the load into your database and API calls to Parameter Store.\nUse ToleratedFailurePercentage — Depends on your requirement, this helps to load most of the data even if there is a small amount of files failed to import.\nUse express child executions — This helps to reduce cost. If the child execution can run within 5 minutes and doesn’t use standard workflow specific functionality (ex: sync or wait for callback integration patterns). This is a great way to reduce costs.\nConclusion # Using the Step Function Distributed Map you can load data into a database efficiently and in a much controlled manner. Using the in-built retry and error handling along with the tolerated failure settings are helpful to mitigate any errors without any intervention from the outside.\nUseful Links # Distributed Map documentation: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-asl-use-map-state-distributed.html\nBlog Post Step Function Distributed Map: https://aws.amazon.com/blogs/aws/step-functions-distributed-map-a-serverless-solution-for-large-scale-parallel-data-processing/\nStandard vs Experess workflows: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html\n","date":"3 May 2023","permalink":"/posts/load-data-from-s3-to-postgres-using-step-functions/","section":"Posts","summary":"In this post, I discuss how we can use Step Functions Distributed Maps to parallelise the csv data loading to a database.","title":"Load Data From S3 to Postgres using Step Functions"},{"content":"","date":"3 May 2023","permalink":"/tags/postgres/","section":"Tags","summary":"","title":"Postgres"},{"content":"","date":"13 March 2023","permalink":"/tags/eventbridge-pipe/","section":"Tags","summary":"","title":"EventBridge Pipe"},{"content":"AWS CTO, Werner Vogel once said - \u0026ldquo;Everything Fails All the Time\u0026rdquo;. No matter what technology you use, it is impossible to build a system that never fails. Serverless is no exception.\nWhile it is important to reduce the possibility of failing a system, equally it is important to react fast to mitigate the issues when errors occur. Retrying is one of the basic steps towards fixing temporary issues. Further it will be great if the system can heal by itself without any additional input or action.\nIn this blog post, I discuss a scenario where a Lambda based Serverless application reacts to the errors occurring and attempts to re-drive messages to the origin in order to retry in a more controlled manner. For this, I use Lambda destinations, Event Bridge, Event Bridge Pipe and related technologies.\nDead Letter Queue Vs Lambda Destinations # Dead letter queues are great as a way to preserve the messages that were failed to process. For example, assume a scenario where a Lambda function is triggered by a SNS topic. Whenever there is an error in the Lambda execution, we can directly send the original message into a pre configured Dead Letter Queue. However, the downside is, only the original message will be added to the DLQ. No information is available as to WHY the message was unable to be processed.\nIn contrast, if we have a Lambda Destination configured to the same queue for the fail scenario, a message will be added to the DLQ with more useful information along with the original payload.\nBelow is the example message structure for the message added to queue via Lambda Destinations:\n{ \u0026#34;version\u0026#34;:\u0026#34;1.0\u0026#34;, \u0026#34;timestamp\u0026#34;:\u0026#34;2023-03-12T11:20:25.428Z\u0026#34;, \u0026#34;requestContext\u0026#34;:{ \u0026#34;requestId\u0026#34;:\u0026#34;f05e5d90-2b5f-44f2-812a-8ccad64a5795\u0026#34;, \u0026#34;functionArn\u0026#34;:\u0026#34;arn:aws:lambda:...\u0026#34;, \u0026#34;condition\u0026#34;:\u0026#34;RetriesExhausted\u0026#34;, \u0026#34;approximateInvokeCount\u0026#34;:1 }, \u0026#34;requestPayload\u0026#34;:{ \u0026#34;Records\u0026#34;:[ { ... } ] }, \u0026#34;responseContext\u0026#34;:{ \u0026#34;statusCode\u0026#34;:200, \u0026#34;executedVersion\u0026#34;:\u0026#34;$LATEST\u0026#34;, \u0026#34;functionError\u0026#34;:\u0026#34;Unhandled\u0026#34; }, \u0026#34;responsePayload\u0026#34;:{ \u0026#34;errorMessage\u0026#34;:\u0026#34;timeout exceeded\u0026#34;, \u0026#34;errorType\u0026#34;:\u0026#34;RemoteDisconnected\u0026#34;, \u0026#34;requestId\u0026#34;:\u0026#34;f05e5d90-2b5f-44f2-812a-8ccad64a5795\u0026#34;, \u0026#34;stackTrace\u0026#34;:[ \u0026#34;File \\\u0026#34;/var/task/index.py\\\\\u0026#34;, \u0026#34;line 6 in handler\\\\n raise http.client.RemoteDisconnected(\\\\\u0026#39;timeout exceeded\\\\\u0026#39;)\\\\n\u0026#34; ] } } These additional details are useful to determine the next steps of the process if (and how) we are going to retry these messages.\nAbout this project # Image: System architecture In this project, I have 2 SNS topics which are the triggers for 2 Lambda functions. One Lambda function throws an error as it returns from a http client when calling a remote API. The other Lambda function throws an type error which is an error in the Lambda code itself.\nBoth Lambda functions have Lambda Destination configured for the failed scenario into the same SQS queue (Dead Letter Queue).\nEventBridge Pipe is configured to poll the messages from this Dead Letter Queue.\nAn Enrichment Lambda function is configured in the EB Pipe to extract the data from the DLQ message and format the message into the below structure. Also, within this Enrichment Lambda function, the retry count is added or incremented by 1.\n{ \u0026#34;payload\u0026#34;: original_message, \u0026#34;meta\u0026#34;: { \u0026#34;source_topic\u0026#34;: topic_arn, \u0026#34;error_type\u0026#34;: error_type_extracted_from_message, \u0026#34;retry_count\u0026#34;: incremented_retry_count, \u0026#34;response_payload\u0026#34;: response_payload_from_message, } } EventBridge Pipe has Event Bus configured as the target. So these formatted messages are sent to the Event Bus.\nIn the Event Bus, there are 3 rules defined.\nRule #1 # First condition matches the source SNS Topic ARN, the Error Type and the retry count.\nHere, the Error Type condition matches messages that are NOT in a pre-defined error types list.\nAnd it checks if the retry count is less than 3.\nIf these conditions are met, the message will be sent back to the source SNS topic using the rule target.\nRule #2 # Second rule (same as first rule) matches the other source SNS Topic ARN, and the Error Type and the retry count.\nHere, the Error Type condition matches messages that are NOT in the same predefined error types list.\nAnd it checks if the retry count is less than 5.\nIf these conditions are met, the message will be sent back to the second SNS topic using the rule target.\nRule #3 # Third rule matches any messages that do not match with the above 2 rules. There is a SQS queue (retry exhausted queue) defined for this rule as the target and any messages exceeding the retry count will be sent to this queue.\nTry this yourself # I have created a sample project for you to test this scenario in your AWS account. This is created using AWS CDK v2 with Python. So, you need CDK v2 and Python installed in your environment.\nBelow are the deployment details.\nClone the repository: https://github.com/pubudusj/serverless-conditional-error-retry Go into the cloned directory. To manually create a virtualenv on MacOS and Linux: $ python3 -m venv .venv After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv. $ source .venv/bin/activate If you are a Windows platform, you would activate the virtualenv using: % .venv\\Scripts\\activate.bat Once the virtualenv is activated, you can install the required dependencies. $ pip install -r requirements.txt Then, deploy the application: $ cdk deploy Once the application is deployed, in the output you can see values for SNS topics SourceTopicOne,SourceTopicTwo and SQS Queue RetryExhaustedQueue which are required for testing the application. Testing # Publish messages for both SNS topics with below payload structure:\n{ \u0026#34;hello\u0026#34;: \u0026#34;world\u0026#34;, \u0026#34;source_sns\u0026#34;: \u0026#34;one|two\u0026#34; } This will trigger two Lambda functions.\nFirst Lambda Function will throw an http client error and the second Lambda function will trow a key error for this payload.\nIn few moments, if you check the messages in RetryExhaustedQueue , you can see the message that was processed with Lambda one has reached to the queue after 3 retries and the messages processed with Lambda two has reached to the queue without retry (because the error type is \u0026lsquo;KeyError\u0026rsquo; which is defined in the list of errors that should not retry).\nReasoning # The second Lambda function has an error in the code, so no matter how many times you retry, it fails (until you fix the code).\nHowever in a scenario as in the first Lambda function where you received an error from an external API which is most likely a temporary issue, you will need to retry.\nAlso, we have defined the number of retries per each EB rule. This way we can control the retries for each of the components or sources and will not end up in an infinite loop if the error is not fixed.\nConclusion # Retrying on error and re-driving the messages to the sources is one of the simple yet effective way to mitigate any temporary errors. However, if we don\u0026rsquo;t control the retry behaviour based on the error type, we will end up wasting resources.\nWith this proposed solution, we can control the retry behaviour based on no of attempts, error type and per source as well. Which will make a more controlled self-healing application.\nFurther, with the latest addition to AWS Serverless services, the Event Bridge Pipe, it is easy to connect sources and targets with low code and build integration fast between different AWS Serverless services.\nUseful Links # Lambda Destinations: https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/\nEventBridge Pipes documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html\nCloudformation API for Pipes: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-pipes-pipe.html\nPipes Documentation for CDK v2 Python: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_pipes/CfnPipe.html\n","date":"13 March 2023","permalink":"/posts/self-healing-serverless-app-with-lambda-destinations-and-eventbridge/","section":"Posts","summary":"In this blog post, explains how a Lambda based Serverless application reacts to the errors and attempts to re-drive messages to the origin in a controlled manner.","title":"Self healing Serverless App with Lambda Destinations and EventBridge"},{"content":"Last week I composed an article with a sample project on how to manage one time schedules created by Amazon EventBridge Scheduler using StepFunctions.\nhttps://pubudu.dev/posts/manage-eventbridge-schedules-using-step-functions\nThere I use StepFunctions callback pattern to wait for the schedule to run and then delete the schedule.\nIn this post I will explain another way to create the schedule and delete it after the given time without the callback pattern, but using a wait state in the execution.\nStep Functions State Machine # Image: State machine How it works # On the Step Functions execution, it first creates a one time schedule using EventBridge Scheduler SDK integration.\nStep Function execution input must be in the below format:\n{ \u0026#34;scheduleDate\u0026#34;: \u0026#34;YYYY-MM-DD\u0026#34;, \u0026#34;scheduleTime\u0026#34;: \u0026#34;hh:mm:ss\u0026#34;, \u0026#34;flexibleTimeWindow\u0026#34;: 5 } In the next step, the wait time is calculated using a Lambda Function. Here, based on the schedules date and time and the flexibleTimeWindow of the schedule, the wait time is calculated and output as ISO date time format.\nNext step is the \u0026lsquo;Wait\u0026rsquo; state where it waits until the wait time returns for the last step.\nAfter the wait time, the execution continues and in the last step, the schedule will be deleted.\nLimitations # Schedule can only be within 1 year ahead because the maximum duration a standard flow can run is 1 year.\nUnlike the solution that uses callback, Step Function execution will not get to know if the task that was triggered by the schedule is successful or not.\nTechnically for this scenario, you can use Express workflows. However practically it is not suitable, since the maximum execution time is only 5 minutes.\nPossible improvements # Currently Step Functions intrinsic functions doesn\u0026rsquo;t support any date/time operations. When that\u0026rsquo;s available in future, calculating the wait time can be done without the Lambda Function.\nOnce EventBridge team release the feature to auto-delete schedules after they are triggered, this whole scenario will be invalidated.\nTry this yourself # I have created a sample project for you to test this scenario in your AWS account. You need AWS CLI, SAM CLI and GIT installed in your machine.\nBelow are the deployment details.\nClone the repository: https://github.com/pubudusj/manage-eb-schedules-with-stepfunctions-wait\nGo into the directory manage-eb-schedules-with-stepfunctions-wait\nInstall dependencies with sam build\nDeploy the stack with sam deploy -g\nOnce the stack is deployed successfully, you can start a Step Functions execution with below payload format:\n{ \u0026#34;scheduleDate\u0026#34;: \u0026#34;YYYY-MM-DD\u0026#34;, \u0026#34;scheduleTime\u0026#34;: \u0026#34;hh:mm:ss\u0026#34;, \u0026#34;flexibleTimeWindow\u0026#34;: 5 } You can see the schedule is created and the execution waits until the schedule time + the flexible time window and deletes the schedule.\nTo delete the stack, use: sam delete\nSummary # In this use case, you can create one time schedules in EventBridge Scheduler using Step Functions SDK integrations. And wait in the same execution until it runs. Then delete it once it is completed.\nUsefuel Links # Introducing Amazon EventBridge Scheduler Amazon EventBridge Scheduler Docs Amazon EventBridge One Time Schedules ","date":"2 March 2023","permalink":"/posts/manage-eventbridge-schedules-using-step-functions-wait/","section":"Posts","summary":"This is another way of managing EventBridge Schedules withing Step Functions execution.","title":"Manage EventBridge Schedules using Step Functions (Part 2)- with Wait State"},{"content":"","date":"2 March 2023","permalink":"/tags/sam/","section":"Tags","summary":"","title":"SAM"},{"content":"AWS Step functions recently announced SDK support for another 35 services.\nAnnouncement: https://aws.amazon.com/about-aws/whats-new/2023/02/aws-step-functions-integration-35-services-emr-serverless\nThis includes the support for EventBridge Scheduler as well. One time schedule is one of the great features supported by EventBridge Scheduler. As the name implies, it allows you to schedule a task just to run once at a given time.\nAs at now ( February 2023) EventBridge Scheduler doesn\u0026rsquo;t have native functionality to know if the target task is successful or not. Also schedule will not be deleted itself automatically after the execution is completed at the given time.\nIn this blog I explain a simple use case where we can create one time schedules and later remove them after the execution using Step Functions using new SDK integrations.\nStep Functions State Machine # Image: State machine How it works # On the Step Functions execution, it first creates a one time schedule using EventBridge Scheduler SDK integration.\nThis Schedule has a target lambda function.\nThe schedule creation step in the state machine is configured with the call back pattern. So it waits until Step Functions receives a Task Success or Failure signal.\nAt the set time, the Scheduler will trigger the lambda function.\nWithin the Lambda function, based on Success or error in the business logic, task success or task failed Step Functions API can be called.\nOnce Step Functions receives the callback it sends a message to a predefined SNS topic with the result. By subscribing to this SNS topic, you may continue on the business logic as to what to do based on the result.\nIn the next step of Step Functions execution, the schedule will be deleted.\nFew limitations # Since callback pattern is only available for Step Functions standard flows, this solution works only within a standard flow.\nWith this, you can only schedule only up to one year ahead. Because the maximum duration a standard flow can run is one year.\nTry it yourself # I have created a sample project for you to test this scenario in your AWS account. You need AWS CLI, SAM CLI and GIT installed in your machine.\nBelow are the deployment details.\nClone the repository: https://github.com/pubudusj/manage-eb-schedules-with-stepfunctions\nGo into the directory manage-eb-schedules-with-stepfunctions\nInstall dependencies with sam build\nDeploy the stack with sam deploy -g\nOnce the stack is deployed successfully, you can start a Step Functions execution with below payload format:\n{ \u0026#34;scheduleDate\u0026#34;: \u0026#34;YYYY-MM-DD\u0026#34;, \u0026#34;scheduleTime\u0026#34;: \u0026#34;hh:mm:ss\u0026#34;, \u0026#34;error\u0026#34;: 0 } To test the scheduled task failure scenario, set the error value to 1.\nTo delete the stack, use: sam delete\nSummary # In this use case, you can create one time schedules in EventBridge Scheduler and wait until it runs, then delete them once it is completed. All can be handled within a Step Functions execution as a low code solution.\nUseful Links # Introducing Amazon EventBridge Scheduler Amazon EventBridge Scheduler Docs Amazon EventBridge One Time Schedules ","date":"24 February 2023","permalink":"/posts/manage-eventbridge-schedules-using-step-functions/","section":"Posts","summary":"With the support of Step Function SDK integrations, we can manage EventBridge Schedules.","title":"Manage EventBridge Schedules using Step Functions"},{"content":"In this post I discuss how messages in a SQS queue can be split between multiple SQS queues with their original payloads. The idea is to achieve this functionality with a low code solution using EventBridge Pipe, Event Bus and Event Rules.\nAmazon EventBridge Pipes # Amazon EventBridge Pipes was introduced in the last re:Invent 2022. EB Pipes helps to create point to point integration with event producers and consumers with low code.\nThis was a great addition to the Serverless services since this will reduce some custom codes that are required to connect two services together. Also it includes additional capabilities to filter and enrich events as they are passed through the Pipe. As of now, EB Pipe supports streaming sources such as Kinesis, DynamoDB, Self Managed Apache Kafka, Amazon MSK and Amazon MQ and SQS as well.\nAbout this project # Imagine a situation where you need to split messages in a SQS queue based on it\u0026rsquo;s content to separate SQS queues. An example can be where an external producer sends messages to a single SQS queue and you have to process those messages using different consumers. Else, those messages may need to be processed with different priorities. So, first, you need to separate those messages into different SQS queues.\nThis can be achieved using a single Lambda function where Lambda function will first process the messages from the source queue and then send them to different SQS queues.\nHowever, this Lambda function will contain too much business logic and permissions to point correct messages to the respective target queues. Also, it should perform in a way to scale well based on the demand and be reliable to not to be a single point of failure. Further, when there are more targets introduced, this needs to be extended to cater those requirements, which is always a challenge.\nSolution # Image: System architecture How it works # EventBridge Pipe is configured to poll messages from the source SQS queue.\nThere is no filter set up, which means all the messages in the source queue will be processed through the Pipe.\nWhen a message is received from the Pipe, it contains not only the original message in the body parameter, but a lot of metadata (related to SQS) as well. Because of that, in order to send the message to the targets, it is required to extract only the original message from the payload. For this, the enrichment Lambda function is used here.\nThen, this message is sent to the Event Bus.\nThere are event rules defined for the Event Bus with conditions that will match against this original message.\nEach rule has a SQS queue defined as the target.\nIn a scenario where a message content is matched with the rule, that message will be sent to the particular queue.\nTry it yourself # I have created a sample application to test this scenario. This is created using AWS CDK v2 with Python. So, you need CDK v2 and Python installed in your environment.\nHow to set up # Clone the repository: https://github.com/pubudusj/sqs-to-multiple-sqs\nGo into the cloned directory.\nTo manually create a virtualenv on MacOS and Linux:\n$ python3 -m venv .venv After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.\n$ source .venv/bin/activate If you are a Windows platform, you would activate the virtualenv using:\n% .venv\\Scripts\\activate.bat Once the virtualenv is activated, you can install the required dependencies.\n$ pip install -r requirements.txt Then, deploy the application:\n$ cdk deploy Once the application is deployed, in the output you can see 4 values for: SourceQueueUrl, TargetQueueOrderCreated, TargetQueueOrderUpdated and PipeArn. Copy the value of SourceQueueUrl to test the application.\nTest # Here, I have configured 2 Event Bus rules.\nFirst rule will match a message with the field type which has the value \u0026lsquo;OrderCreated\u0026rsquo;. This rule has a target SQS queue named TargetQueueOrderCreated.\nSecond rule will match a message with the field type which has value \u0026lsquo;OrderUpdated\u0026rsquo;. This rule has a target SQS queue named TargetQueueOrderUpdated.\nSend the a message with type \u0026lsquo;OrderCreated\u0026rsquo; into the source queue as follows:\naws sqs send-message \\ --queue-url=SourceQueueUrl \\ --message-body \u0026#39;{\u0026#34;orderId\u0026#34;:\u0026#34;125a2e1e-d420-482e-8008-5a606f4b2076\u0026#34;, \u0026#34;customerId\u0026#34;: \u0026#34;a48516db-66aa-4dbc-bb66-a7f058c5ec24\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;OrderCreated\u0026#34;}\u0026#39; If you check the TargetQueueOrderCreated, you will see the message has arrived into the queue with the original payload.\nSend the a message with type \u0026lsquo;OrderUpdated\u0026rsquo; into the source queue as follows:\naws sqs send-message \\ --queue-url=SourceQueueUrl \\ --message-body \u0026#39;{\u0026#34;orderId\u0026#34;:\u0026#34;125a2e1e-d420-482e-8008-5a606f4b2076\u0026#34;, \u0026#34;customerId\u0026#34;: \u0026#34;a48516db-66aa-4dbc-bb66-a7f058c5ec24\u0026#34;, \u0026#34;type\u0026#34;: \u0026#34;OrderUpdated\u0026#34;}\u0026#39; If you check the TargetQueueOrderUpdated, you will see the message has arrived into the queue with the original payload.\nConclusion # With EventBridge Pipes, it is easy to connect sources, specially streaming sources, with the targets with minimum configurations, where previously custom code is required. It scales well, cost effective and very minimum maintenance is required.\nUseful Links # EventBridge Pipes documentation: https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html\nCloudformation API for Pipes: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-pipes-pipe.html\nPipes Documentation for CDK v2 Python: https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_pipes/CfnPipe.html\nEventBridge Pipes pricing: https://aws.amazon.com/eventbridge/pricing/#Pipes\nPlease feel free to deploy this solution to your own AWS environment and share your experience with me.\nKeep building! Keep sharing!\n","date":"29 January 2023","permalink":"/posts/split-messages-from-single-sqs-queue-to-multiple-sqs-queues/","section":"Posts","summary":"This blog post shows how we can split messages in a SQS queue into mulitple queues usint Amazon EventBride","title":"Split messages from single SQS queue into Multiple SQS queues using EventBridge"},{"content":"At the last AWS reInvent 2022, I was able to speak in Dev Chat under the Community track for the first time. I talked about one of my projects - a Smart doorbell with AWS Serverless. This post explains my experience as a first time Dev Chats speaker.\nThis post is on specifically my own experience and yours and/or other speakers\u0026rsquo; experience may vary.\nI got this opportunity being a member of the AWS Community Builders program. Please read more about the program here: https://aws.amazon.com/developer/community/community-builders/\nBefore the reInvent # Application # In the month of June 2022, Jason Dunn, the AWS Community Builders Program Manager has sent a message to AWS Community Builders\u0026rsquo; Slack, stating that the CFP is open for the talks under community track at reInvent 2022. I decided to submit an application for my project - Smart doorbell with AWS Serverless for a Dev Chat which is a 30 minute talk.\nIn mid August, I received an email from Shantavia Craigg, AWS Developer Event Program Manager with the confirmation of my Dev Chat session approval.\nAfter that, the preparations started!\nSlides deck preparation # AWS has provided a PowerPoint theme, styles and guideline as to how to prepare the slides. This includes the font, font size, colors and any image specific guidelines. Since all the sessions in reInvent are public, all the material that goes into the slides will be verified by the AWS events copy editing team.\nIt was required to prepare the slides deck as soon as possible and submit it for review. Once submitted, the copy editing team will review the slides and if any modifications are required, those need to be adjusted and submitted again for review.\nOnce the slides deck is approved, that will be the version going to be presented and no more modifications are allowed. However, if there is any critical update required, you can request them at the Speaker Ready Room session at the reInvent campus.\nKnow Before You Go Webinar # In order to provide information on what we can expect at reInvent as a speaker, there were a couple of webinars arranged by AWS. Those were named as Know Before You Go webinars, which were conducted by AWS Global Events Team.\nThese webinars were really useful for me, as this was my first time at reInvent.\nSpeaking Coaching Session # Another great opportunity I had in this process is to have access to a speaking coach to rehearse my talk. There were limited speaking coaching sessions available to book online as well as within reInvent campus. I booked an online session which was for 60 minutes.\nI had this session with Blaine Sundrud, Senior Global Public Speaking Coach at AWS. This was my first experience interacting with a speaking coach and I was really amazed by the input he has given. First, I presented my talk to him and he provided me with some great tips on how to improve the talk. Also, he even suggested some modifications to the slides in order to make the whole story much clearer to the audience. Also, since there was a demo I prepared in the talk, he had some great tips to improve user interaction.\nIf you are to present at an AWS event and there is a speaker coaching option, I highly recommend using that opportunity because that will definitely change how you present in public in a positive way!\nAt the reInvent Campus # Speaker Ready Room # Every speaker of reInvent should check-in to a Speaker Ready Room (SRR). This should be done at least 4 hours before the talk. This way, event managers will get to know that the speaker is present in the reInvent campus. It is recommended to book a time slot for SRR even though few walk-ins are also available.\nThere are few things that happen in the SRR. First, speakers can go through their presentation slides deck and verify the contents for one last time before the presentation with a Copy Editor. It is possible to do some minor modifications to the slides with the help of the Copy Editor too.\nNext, you can do a \u0026ldquo;Tech Check\u0026rdquo; in the same SRR. Here, you can check how your slides are visible to the audience. Also, you will be given a quick walk through on how the sound devices, pointing devices work, how switching between slide deck and your computer or any other devices (if applicable) can be done etc. Also, if you need wifi connections to your additional devices, that can be arranged in SRR.\nFor those who prefer in-house speaker training, this is also available with a reservation at the SRR.\nOnce everything is finalised at the SRR, you are ready for the talk.\nImage: Speaker Ready Room The Talk # My talk was scheduled on Wednesday at 4 PM and I had to be at the Dev Lounge in the Expo area half an hour before. This was to make sure everything was set up correctly. This included connecting my own laptop, internet connection to raspberry PI device (for the demo) and a quick test on audio visual units.\nDev Lounge is in the Expo area which is an open space. Since there are a lot of people, the place is always a bit noisy. Therefore the attendees were given headphones to listen into the talks.\nI had a total of 30 minutes for my talk which includes the demo and a few questions and answers as well. Personally I had a great time presenting my project and got nice feedback as well. Thanks a lot to the attendees who decided to be there for my talk.\nImages: Me - during the talk After the reInvent # Feedback # Soon after the reInvent, next Monday I received the feedback about my sessions based on the attendees\u0026rsquo; survey. Which was quite fast and very helpful for me as a speaker.\nSummary # I got this opportunity to be a part of the AWS Community Builders program. So, I would like to thank AWS Community Builders Program Manager Jason Dunn (@jasondunn), and the team (sorry, there\u0026rsquo;s too many people to mention here individually) for the opportunity which definitely helps me to continue what I do!\nOverall, I feel the whole process was well organised with highly attention to details. Even though I am a first time speaker at reInvent, I was comfortable going through this whole process because of all the support I was given from all the people I interacted with. Thanks a lot everyone involved in organising this amazon event.\nThank you AWS for the great experience.\nImage: Jason Dunn and some of the reInvent 2022 Dev Chats speakers ","date":"3 January 2023","permalink":"/posts/my-experience-as-a-dev-chat-speaker-at-aws-reinvent/","section":"Posts","summary":"In re:Invent 2022, I delivered a talk in the Dev Chats. This explains the process of giving a talk at re:Invent","title":"My Experience as a Dev Chat speaker at AWS re:Invent 2022"},{"content":"","date":"3 January 2023","permalink":"/tags/public-speaking/","section":"Tags","summary":"","title":"Public Speaking"},{"content":"","date":"3 January 2023","permalink":"/tags/reinvent/","section":"Tags","summary":"","title":"re:Invent"},{"content":"","date":"29 August 2022","permalink":"/tags/appconfig/","section":"Tags","summary":"","title":"AppConfig"},{"content":"","date":"29 August 2022","permalink":"/tags/feature-flag/","section":"Tags","summary":"","title":"Feature Flag"},{"content":"There can be many different ways to manage feature flags in your applications. AWS AppConfig Feature Flag is a best solution because it is purposely built for this purpose. With the great support of Lambda extensions, Feature Flag is quite easy to use in Serverless applications. In this post, I discuss how we can disable a feature flag that is already enabled based on the activities of the system.\nIn this application, there is a feature flag defined for a discount code. Initially, the feature flag is on and based on that, there is a banner visible in a web site.\nOnce users have consumed the discount code up to a predefined count, the feature flag will be auto-deactivated. So that banner will not be visible to the user.\nAnd below services/functionalities are in use in this project:\nAWS AppConfig Feature Flags Step functions SDK integrations for AppConfig, DynamoDB and SES. Lambda function URLs Amazon DynamoDB Amazon Event Bridge AWS CDK v2 with TypeScript as IAC Architecture # Image: System architecture Image: State machine How it works # To deploy the stack, the discount code needs to be provided as a parameter along with the maximum usage amount possible. Then, there will be a AppConfig feature flag created for this discount code and it will be initially enabled. Also, within the DynamoDB table, a record is created for this discount code. There is Check Discount CodeValidity Lambda Function to check if the feature flag is enabled. Within this Lambda function, using AppConfig Lambda extension it checks the particular feature flag is on. Lambda Function URL is enabled to trigger this Check Discount CodeValidity Lambda Function. To consume the discount code, Consume Discount Code Lambda Function will be used. Using the Lambda Function URL of Consume Discount Code Lambda Function, the user triggers the function. It will publish a message to Event Bridge. There is an Event Bridge rule to trigger when a discount code is consumed, which initializes Step Function execution. Within the Step Function, it first increments the usage count in the DynamoDB table. Next, it checks if the discount code has used up to its maximum usage. If the maximum amount is not reached, it simply goes into a Pass state and ends the execution. If the maximum amount is reached, first an AppConfig Hosted Configuration Version is created with feature flag value to false. Next step, it starts deployment to apply the changes. Here, on stack creation, there is a Custom Deployment Strategy created to make the changes effected immediately. Then, as the last step, an email will be sent to the given admin email address using AWS SES. How to set up # You can deploy this project in your own AWS account. Please follow the steps below:\nPrerequisites # AWS CLI and AWS CDK v2 need to be installed. Verified from-email address in SES Steps # Clone the repository: https://github.com/pubudusj/feature-flag-auto-off Go to the directory feature-flag-auto-off. Update the Makefile with required stack parameters. Run make build to install required dependencies. Run make deploy to deploy the stack. Once the project successfully deployed, there will be 2 urls in the output. FeatureFlagAutoOffStack.checkDiscountCodeValidityApi - This is to check if the feature flag is enabled or not. FeatureFlagAutoOffStack.consumeDiscountCodeApi - This is to consume the discount code once. Testing # First, you can check if the feature flag is enabled using the checkDiscountCodeValidityApi above. You can see it is enabled. Then, you can consume the discount code by calling the consumeDiscountCodeApi. When the discount code reaches the maximum no of usage count, admin email address will receive a notification. When you check the feature flag status, you can see it is disabled. Key points/Lessons learned # AWS AppConfig Extension Layers for Lambda has different ARNs for each region. So, you need to select the correct ARN applicable for your region.\nThere are no official CDK v2 L2 constructs available for AppConfig yet. So, L1 constructs are used in this project.\nIf you would like to learn more about AWS AppConfig, this workshop will be a great resource: https://catalog.us-east-1.prod.workshops.aws/workshops/2ee2fc71-0618-479c-86dd-1d5fb168eb20/en-US\nPlease feel free to deploy this to your own AWS environment and share your experience with me. Keep building! Keep sharing!\n","date":"29 August 2022","permalink":"/posts/turn-off-appconfig-feature-flag-automatically/","section":"Posts","summary":"This blog post shows to turn off AppConfig Feature Flag based on system behaviour.","title":"Turn off AppConfig Feature Flag Automatically"},{"content":"Intro # AWS Step Functions is a Serverless orchestration service that helps to build a workflow with various steps that connect different AWS services. These steps are defined in State Machine, which will be implemented in Amazon States Language (ASL).\nThere are two types of workflows in Step Functions:\nExpress flow Standard flow Among many differences, Express flow can only run up to five minutes and Standard flow can run up to one year.\nSince even five minutes is a long time, there is a possibility of having an active execution when the state machine is updated. In this article, I am going to discuss how such active executions are affected when the State Machine is updated and how to handle those scenarios.\nLet’s assume we have a simple state machine with two Lambda functions as follows. There is a “Wait” state with a three-minute wait time in between Lambda calls.\nImage: Two Lambda Functions with Wait State Scenario 1 # Here, we will run the above state machine, and while there is an active execution, we will update the state machine by adding a third Lambda function after the second Lambda function.\nImage: State Machine updated with 3rd Lamba Function The result is that the active execution is not affected.\nThe reason is when a new execution initialised, the Step Function keeps a ‘snapshot’ of the State Machine and any updates to the State Machine is not affect this ‘snapshot.’\nScenario 2 # Here, we will run the state machine in Fig 1, and this time we will keep the State Machine as it is, but, we will update the second Lambda function code.\nThe result is that the active execution is affected.\nThe reason is although Step Function keeps a ‘snapshot’ of the state machine at the start of the execution, the Lambda functions in the state machine are referenced with their ARNs. This is the latest version of Lambda. So, if the Lambda functions are updated, it affects the running execution since it always refers to the latest version of Lambda.\nHow To Prevent Modifications To Active Executions When Lambda Functions Update? # There is a simple solution for this: use Lambda versions.\nOn deployment, create a Lambda version and in the state machine, refer to the Lambda function with the version in the ARN.\nExample: arn:aws:lambda:[region]:[accountId]:function:[functionName]:[Version]\nAn example of this implementation is this CDK code snippet.\nPreviously (without Lambda versions):\nconst lambdaFunctionOne = new nodejs_lambda.NodejsFunction(this, \u0026#34;LambdaFunctionOne\u0026#34;, { runtime: lambda.Runtime.NODEJS_14_X, entry: path.join(__dirname, `/../lambda/FunctionOne/index.ts`), handler: \u0026#34;handler\u0026#34;, timeout: Duration.minutes(10), }); const waitForXMinutes = new sfn.Wait(this, \u0026#39;Wait\u0026#39;, { time: sfn.WaitTime.duration(Duration.minutes(3)), }); const lambdaFunctionTwo = new nodejs_lambda.NodejsFunction(this, \u0026#34;LambdaFunctionTwo\u0026#34;, { runtime: lambda.Runtime.NODEJS_14_X, entry: path.join(__dirname, `/../lambda/FunctionTwo/index.ts`), handler: \u0026#34;handler\u0026#34;, timeout: Duration.minutes(10), }); const stateMachine = new sfn.StateMachine(this, \u0026#39;SfTestStateMachine\u0026#39;, { definition: new tasks.LambdaInvoke(this, \u0026#39;TriggerLambdaFunctionOne\u0026#39;, { lambdaFunction: lambdaFunctionOne }) .next(waitForXMinutes) .next( new tasks.LambdaInvoke(this, \u0026#39;TriggerLambdaFunctionTwo\u0026#39;, { lambdaFunction: lambdaFunctionTwo }) ) }); Now (with Lambda versions):\nconst lambdaFunctionOne = new nodejs_lambda.NodejsFunction(this, \u0026#34;LambdaFunctionOne\u0026#34;, { runtime: lambda.Runtime.NODEJS_14_X, entry: path.join(__dirname, `/../lambda/FunctionOne/index.ts`), handler: \u0026#34;handler\u0026#34;, timeout: Duration.minutes(10), }); const waitForXMinutes = new sfn.Wait(this, \u0026#39;Wait\u0026#39;, { time: sfn.WaitTime.duration(Duration.minutes(3)), }); const lambdaFunctionTwo = new nodejs_lambda.NodejsFunction(this, \u0026#34;LambdaFunctionTwo\u0026#34;, { runtime: lambda.Runtime.NODEJS_14_X, entry: path.join(__dirname, `/../lambda/FunctionTwo/index.ts`), handler: \u0026#34;handler\u0026#34;, timeout: Duration.minutes(10), }); const stateMachine = new sfn.StateMachine(this, \u0026#39;SfTestStateMachine\u0026#39;, { definition: new tasks.LambdaInvoke(this, \u0026#39;TriggerLambdaFunctionOne\u0026#39;, { lambdaFunction: new lambda.Version(this, \u0026#39;LambdaFunctionOneVersion\u0026#39;, { lambda: lambdaFunctionOne, }) }) .next(waitForXMinutes) .next( new tasks.LambdaInvoke(this, \u0026#39;TriggerLambdaFunctionTwo\u0026#39;, { lambdaFunction: new lambda.Version(this, \u0026#39;LambdaFunctionTwoVersion\u0026#39;, { lambda: lambdaFunctionTwo, }) }) ) }); Conclusion # When using Lambda functions within Step Functions, use the Lambda function versions to make sure any active executions are not affected by Lambda updates.\nIn contrast, if you need any active executions to be updated with the latest Lambda code, do not use Lambda function versions.\nResources # AWS Step Functions Express vs Standard workflows: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-standard-vs-express.html\nDocumentation — Amazon State Language: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-amazon-states-language.html\nDocumentation — State Machine: https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-state-machine-structure.html\nPlease feel free to try this and share your experience with me.\nKeep building! Keep sharing!\n[Header image by davide ragusa on Unsplash]\n","date":"10 June 2022","permalink":"/posts/aws-step-functions-are-active-executions-affected-when-your-state-machine-gets-updated/","section":"Posts","summary":"Looking into different scenario of SF executions vs state machine updates","title":"AWS Step Functions — Are Active Executions Affected When Your State Machine Gets Updated?"},{"content":"","date":"10 May 2022","permalink":"/tags/ses/","section":"Tags","summary":"","title":"SES"},{"content":"Intro # In this project, I will discuss how you can build a simple leave apply/approval system using AWS Serverless services.\nThese are the main functionalities of this implementation.\nApply for a leave Confirmation for leave application. Send notification to the approver regarding new leave request. Send reminder notification to the approver. Send notification upon approval/rejection. And below services/functionalities are in use in this project:\nStep functions SDK integrations for SES, DynamoDB Step functions Parallel processing Step functions wait for callback pattern Lambda function URLs DynamoDB streams with filtering AWS CDK with TypeScript as IAC Architecture # Image: Architecture - Simple Leave System State Machine # Image: State machine How it works # Create a leave request # CreateLeave Lambda function accepts leave requester\u0026rsquo;s email, leave approver\u0026rsquo;s email, leave dates and reason and is responsible to validate and save it to the DyanmoDB table.\nLambda function url is enabled for CreateLeave Lambda function.\nSimple web form is created to simulate creation of leave request which is hosted using Amplify Web Hosting.\nLeave request processing # When a leave request is saved into the DynamoDB table, using DynamoDB steams, Init Lambda function is triggered.\nHere, event source mapping is enabled with filtering, so that only the event type INSERT, record type \u0026rsquo;leaveRequest\u0026rsquo; and record status \u0026lsquo;pending\u0026rsquo; will trigger the lambda function.\nThis Init Lambda function\u0026rsquo;s only responsibility is to format the input and start new Step Function execution.\nWithin step function first step is to send a email notification to the leave applicant to inform the receipt of the request. For this, Step function SDK integration for SES v2 is used.\nNext, there are 2 branches runs parallel. One for leave reject/approval flow, and the other is to remind the approver about leave request until being processed.\nLeave reject/approval flow # In the leave reject/approval flow, first an email will be send to the approver with the leave information and link to mark the leave as approved or rejected. This step has wait for the task token option enabled.\nThis link is generated as the function url of the LeaveApproveOrReject Lambda function.\nAlso, this link consist of the task token generated in the step, so in the LeaveApproveOrReject Lambda function, based on the reject/approval selection, it can call sendTaskSuccess or sendTaskFailure APIs.\nIf the leave is approved with sendTaskSuccess, then the record in the DynamoDB table will be updated using the Step function SDK integration for DynamoDB.\nIf the leave is rejected with sendTaskFailure, then there is an additional step to transform the error object before update the record in the DynamoDB table.\nAs the final step of that branch, an email will be sent to the applicant with the outcome of the processed leave request.\nReminder flow # In the other branch, it starts with a wait status which has a pre-configured wait time before sending the reminder.\nAfter the wait time, it checks the status of the leave request using Step function SDK integration for DynamoDB.\nThen it checks if the leave request status is pending.\nIf the status is pending, it sends an email to the approver reminding the leave request using Step function SDK integration for SES v2.\nAnd the flow goes back to the wait step in order to retry later.\nIf the leave request is not in pending status, it goes into a Pass step to conclude the flow there.\nWith this, until the leave request is being processed, a reminder will be sent to the approver in regular interval defined by the wait time.\nEmail Notifications # Image: Leave request confirmation Image: Leave approver notification Image: Leave approver reminder How to set up # This project is implemented using AWS CDK v2. Source code of this application is available at https://github.com/pubudusj/simple-leave-management\nPrerequisites # AWS CDK CLI (2.22.0+) Verified from-email address in SES Installation # Clone the repository. Update the Makefile with the sending email for SES and AWS CLI profile (if required). Run make build to install required dependancies. Run make deploy to deploy the stacks. Once deployed, using the HostingUrl output value of CreateLeaveFrontendStack, you can access the frontend application to create a leave request. Remove the stack # Run make destroy to remove all the resources of the project. Key points/Lessons learned # SES by default in sandbox mode and only verified email addresses can receive emails. You need to raise a support ticket from your account to take it out of the sandbox mode so you can send emails to any address.\nThis is the first time I used CDK in a project and I am so much impressed how easy it is to deploy Serverless resources with it. This deserves a separate blog post :)\nI have used few CDK modules which are still in alpha version (ex: aws-amplify-alpha). Those functionalities might be changed without prior notice.\nHere, for demo purposes, Lambda function urls are not protected and can be triggered as public url. For a production system, those needs to be protected with AWS_IAM auth type.\nTo deploy the frontend in Amplify Web Hosting, locally built artifacts are used here. However, it is better to build this as part of a CI/CD pipeline where source code is fetched from Github.\nFor demo purposes, retry interval is set to 2 minutes. So, every 2 minutes the approver will get an email until leave request is processed.\nInit Lambda function is only one from many other alternatives to trigger the step function (ex: event bridge rule).\nPlease feel free to deploy this to your own AWS environment and share your experience with me.\nKeep building! Keep sharing!\n","date":"10 May 2022","permalink":"/posts/simple-leave-management-system-with-aws-serverless/","section":"Posts","summary":"This is how I built a simple leave management system using AWS Serverless services","title":"Simple Leave Management System with AWS Serverless"},{"content":"Webhooks are still a preferred choice of many organizations to communicate with 3rd party services. Since webhook calls are event-driven, building a webhook management system with Serverless is a great choice. In this post, I am going to discuss how to build such a webhook management system with AWS Serverless.\nIn an Application Tracking Systems (ATS), whenever a candidate related event occurs (ex: candidate created, applied for a job, candidate state changed, etc), it might need to call to a registered webhook.\nThere are 3 parts to this implementation.\nWebhook registration. Webhook call for the given candidate event. View webhook call history. Architecture # Image: Architecture State Machine: Image: State machine How it works # Webhook Registration # There is an API exposed in the API gateway which proxy to a lambda function (RegisterWebhookFunction). This API accepts companyId, URL, and the eventType (ex: candidate.created). This lambda function generates a hash token which will be used to validate the incoming webhook call. These data will be saved in a DynamoDB table and the generated hash token will be returned as the response. Call webhook # To simulate \u0026lsquo;candidate.create\u0026rsquo; event, I use an API endpoint, which accepts companyId and other candidate\u0026rsquo;s profile data. This data will be received by the CreateCandidate lambda function and data will be saved to the same DyanamoDB table. As you see, it is not required to use the API endpoint to create the candidate, but the only thing matters here is to add the candidate data to the DynamoDB table. Once the candidate data is saved in the DynamoDB table, the TriggerResourceCreated function will be executed using DynamoDB Streams. TriggerResourceCreated function has a filter in place for the DynamoDB stream so it checks the eventName and the type of the resource. In this example, it checks if the eventName is \u0026ldquo;INSERT\u0026rdquo; and the resource type is \u0026ldquo;candidate\u0026rdquo;. If so, the lambda function is executed and an event will be published to the Event Bridge Event Bus. There is a Step Function configured to be executed based on an EventBridge Rule. EventBridge event\u0026rsquo;s detail object will be the input data for the Step Function execution. Within the Step Function # In the input of the step function, only the companyId, eventType and resourceType (ex: candidate) and resourceId (ex: candidateId) included. First, Step Function uses DynamoDB integration to get webhook by companyId and eventType. If the webhook exists, the data goes into a transform state, else, the execution will be skipped. Once the webhook data is transformed, the next step is to fetch resource data given the resourceId and resourceType. For that, the FetchResourceData lambda function is used. If the resource data does not exist, execution is set as failed. If the resource data exists, the CreateWebhookCall lambda function is triggered to prepare the webhook call. This will create a record in the DynamoDB table with the webhook payload and the validation token with the status \u0026lsquo;pending\u0026rsquo;. Then those resource data along with the webhook data sent to an SQS queue. CallWebhook function will consume this SQS queue to call the webhooks with the resource data in the payload. Here, a verification token is also included to verify the webhook call authenticity from the external end. For this, the hash token generated when registering the webhook will be used. In the state machine, this step is set up with retries enabled, so it will retry 3 times if an error occurred. Also, there will be a 1 minute wait time between each retry. If the webhook call was unsuccessful even after all the retries, the state execution will be notified using the sendTaskFailure callback along with some additional data about why this has failed. If the webhook call is successful, the state execution will be notified using the sendTaskSuccess callback. In either case, in the next step, the Step function has a DynamoDB integration to update the webhook call data with its status (as the success/failed) and the additional information from the webhook call. Get webhook call history # An API endpoint is provided to proxy to a lambda function to fetch data from the DynamoDB table for a specific companyId. A list of all the webhook calls made will be returned. How to set up # You can set this up in your own AWS account using AWS SAM. Also, to test/demo the application, I have included a single-page app built with VueJS (in the frontend directory).\nSource code for this application is available at https://github.com/pubudusj/webhook_management To set up backend, run sam build \u0026amp;\u0026amp; sam deploy -g within the backend directory. Once the backend is set up, copy the ApiBaseUrl parameter and copy it to the frontend directory\u0026rsquo;s .env file as the value of VUE_APP_API_BASE_URL env variable. Run npm run serve to test this in your local machine or run run build --production to output the build. Demo # You can see this functionality in action at https://webhooks.pubudu.dev\nYou can create a webhook providing a URL and a companyId. Then create a user for the same companyId. Then if you search the history with the same companyId, you will see a record with \u0026lsquo;pending\u0026rsquo; status. If you search again, you can see the status has changed.\nPlease note: Since the retry is in place for the failed webhook calls, it will take about 2,3 minutes to set the webhook call as failed. Success ones will be updated almost instantly.\nWebhook call can be verified matching the token value in the payload with the output of hmacSHA256(candidateId + createdAt, signingToken). Here singingToken is the one that generated when creating the webhook.\nKey points/Lessons learned # Here I used DynamoDB single table design. So, there will be a lot of events generated into the DynamoDB stream. Luckily, with the recent feature of filtering streaming data including the DynamoDB Streams, now lambda can only use filtered events based on a specific pattern. (Previously, this has to be handled within the Lambda function). In the step function\u0026rsquo;s DynamoDB integration, GetItem integration returns data with all the data type keys. There is no way to filter them out within Step Function unless you know the exact data and object structure. So, here I had to use a lambda function to fetch data from DynamoDB instead of the direct integration. Since this system relies on the external webhook URL availability, the retries are used within the Step Function. Setting up retries and waiting time between the retries are quite easy. Here I used Step Functions Workflow Studio to create and export the state machine. For the SF-DynamoDB integrations, I used native optimized integration instead of the new SDK integration. Both should give the same performance. For demo purposes, I created the event bus within the same stack, however, in a real scenario, this can be a common event bus shared by other services. Please try this and let me know your thoughts.\nKeep building! Keep sharing!\n","date":"15 December 2021","permalink":"/posts/manage-webhooks-at-scale-with-aws-serverless/","section":"Posts","summary":"Webhook calls are event driven. So, building a webhook management system with Serverless is a great choice.","title":"Manage webhooks at scale with AWS Serverless"},{"content":"","date":"27 September 2021","permalink":"/tags/apigateway/","section":"Tags","summary":"","title":"APIGateway"},{"content":"Intro # This post describes how to implement a simple One Time Password (OTP) system with AWS Serverless services which can be used as a part of two step verification.\nBelow tools and technologies used to build this application.\nAWS Lambda API Gateway DynamoDB Simple Email Service - SES Amplify Web Hosting VueJS for frontend Deploy with AWS SAM Architecture # Image: Architecture How it works # In this scenario, I used a login form, which is developed with VueJS and hosted using Amplify static web hosting.\nUser will enter their email and password and once the credentials are validated, an API endpoint is called to execute \u0026ldquo;Generate OTP\u0026rdquo; Lambda function which generates a 6 digit code along with a session id.\nOnce the code and session id is generated, \u0026ldquo;Generate OTP\u0026rdquo; Lambda will save these data into a DynamoDB table.\nThen only the session id will be returned as the response of the API end point.\nDynamoDB streams are enabled in the table. So, once the data are saved, it will trigger the \u0026ldquo;Send Email\u0026rdquo; Lambda function.\nWithin the \u0026ldquo;Send Email\u0026rdquo; Lambda function, it will call the Simple Email Service (SES) to send out an email with the generated code to the email address provided.\nMeanwhile in the frontend side, once the session id is received to from the API, 2nd form is presented to enter the code, which is emailed to the given address.\nOnce user enters the code and submit, it will validate the code along with the session id using another API gateway endpoint which proxy to \u0026ldquo;Verify OTP\u0026rdquo; Lambda function.\nIn \u0026ldquo;Verify OTP\u0026rdquo; Lambda function, it query the DynamoDB table with the given session id and code and returns the success or error responses.\nKey points/Lessons learnt # Here I enabled DynamoDB TTL to delete the entries after specific time to prevent fill out the table very quickly. However DynamoDB will not delete your record immediately when the TTL is expired. It is deleted eventually and AWS only guarantees it to be deleted within 48 hours. Because of this, when verifying the OTP code, it has to consider the same \u0026ldquo;expiredAt\u0026rdquo; field which used to set TTL.\nWhen design the DynamoDB table, I used sessionId + OTP code as a primary key to easily query the required record. So, when verifying the code, I query by primary key with this combination in the DynamoDB table.\nWhen using DynamoDB steam as a trigger for Lambda, it get triggers for all the DynamoDB events (ex: insert, delete, update). So, within the Lambda function, had to filter out only the INSERT events using \u0026ldquo;eventName\u0026rdquo; of the record.\nTo send out emails, I have used AWS\u0026rsquo;s own Simple Email Service (SES). However, you need to first verify your sending email address in order to send emails to any address. This can be done with a support request.\nHere, I used Amplify static web hosting to host the frontend of the application. I have used Amplify features - the auto deployments when Github repository modified and custom domain name set up only with few button clicks.\nI have used AWS SAM to deploy the backend resources. The expiry time of the OTP and no of digits in the OTP code can be configured at the deployment time.\nHow to set up # Prerequisites # AWS cli AWS SAM cli Set up and verified SES send email address Backend # Clone the repository: https://github.com/pubudusj/simple-otp\nRun\nsam init \u0026amp;\u0026amp; sam deploy -g After providing your stack information and AWS environment parameters, this will create the backend stack. Copy the ApiBaseUrl output value.\nFrontend # Copy the .env.example into .env file and add the ApiBaseUrl value as VUE_APP_API_BASE_URL.\nYou may zip the whole frontend directory and use that in Amplify web hosting, or authorise your GitHub repository to automatically deploy the application when a git push is made.\nIf you need to run the frontend in local, navigate to frontend directory and run npm run serve\nDelete the stack # To remove the backend, run sam delete.\nDemo # Demo version of this application is available at : https://simple-otp.pubudu.dev/\nPlease feel free to try this out and let me know your thoughts.\nKeep building, keep sharing!\n","date":"27 September 2021","permalink":"/posts/simple-otp-service-with-aws-serverless/","section":"Posts","summary":"This post describes how to implement a simple One Time Password (OTP) system with AWS Serverless services","title":"How to create a simple OTP service with AWS Serverless services"},{"content":"Intro # This is how I built a photo booth application with AWS Serverless services.\nOnce user capture an image and provide an email, background of the image is removed and new background applied and watermark with timestamp will be added. Then the image will be emailed to the given email address.\nBelow tools and technologies are used to built this.\nBackend:\nAWS Lambda with runtime Node.js Step Function S3 SES remove.bg API Frontend:\nVueJS vue-web-cam plugin Hosted on Amplify web hosting IAC:\nServerless Framework Architecture # Image: Architecture How It Works # In the frontend VueJS app (hosted in Amplify), first a image will be captured with the email address.\nThen the app will request for pre-signed url.\nCaptured image will be uploaded to S3 with the pre-signed url and the email address as a meta data.\nOnce the image is uploaded, Init lambda function will be triggered to initialise step function execution.\nWithin the Step Function execution, first the background of the image will be removed. For this, I use remove.bg external service.\nThen, a new background will be applied to the image. This background must be in the \u0026lsquo;backgrounds\u0026rsquo; path of the S3 bucket.\nAs the next step, watermark will be added with the timestamp.\nFinally, the image will be sent to the given email address using AWS Simple Email Service (SES).\nImage: Photobooth Steps How to set up # Prerequisites # AWS cli Serverless Framework cli yarn \u0026amp; jq installed Create a free account at remove.bg and generate a API token. Set up and verified SES send email address Deployment # Clone the repository: https://github.com/pubudusj/photo-booth\nCopy .env.example to .env and fill in with the appropriate values.\nRun\nmake install \u0026amp;\u0026amp; make deploy This will create the backend stack and make the frontend deployment bundle ready.\nFrontend # Once step 3 above is finished, you can deploy the frontend code in frontend/dist/frontend.zip easily with Amplify web hosting. Of course if you use amplify, you can configure your own custom domain quite easily as well.\nIf you need to run the frontend in local, navigate to frontend directory and run yarn run serve\nTo Delete the stack # To remove the backend, run make down.\n** Make sure the s3 directory created in the stack is empty before run this command.\nDemo # You may check the live demo at https://photobooth.pubudu.dev\nKey points/Lessons learnt # Here I used remove.bg external tool to remove background, because it doesn\u0026rsquo;t need for example green background to extract the foreground objects. I don\u0026rsquo;t have any affiliation with this service and there are lot of other alternatives as well. Please select any service you prefer and you will need to modify the removeBackground lambda accordingly.\nI have used Step Functions general workflows here, but using express workflow also a good option.\nI have NOT implemented any error handling in this project since this is just a POC, but in general, you need to address failure scenarios. Refer error handling with Step Functions here.\nThis is the first time I used Serverless Optimize Plugin and surprised how easy it is to bundle the packages only relevant to each lambda functions.\nAs the email service, I have used AWS\u0026rsquo;s own Simple Email Service (SES). However, you need to first verify your sending email address in order to send emails to any address. This can be done with a support request. Optionally you can use any other email service provider and you have to modify the sendEmail lambda accordingly.\nIn this project, I don\u0026rsquo;t use any database to record the data that goes through the system. I use s3 object meta data to record the email address entered by user to send out the email.\nvue-web-cam is simply awesome plugin for VueJS to get camera input into your application. You have option to select the preferred video source if you have multiple.\nYour valuable feedback on this project is mostly welcome! I would like you to play around with this and if you have any questions or general comments, please reach out to me via LinkedIn, Twitter or Github.\nKeep building, keep sharing!\n[Header image by Kevin Grieve on Unsplash]\n","date":"9 July 2021","permalink":"/posts/create-photo-booth-with-aws-serverless/","section":"Posts","summary":"This explains how I created a simple photo booth with AWS Serverless services","title":"How I created a Photo Booth with AWS Serverless"},{"content":"","date":"9 July 2021","permalink":"/tags/s3/","section":"Tags","summary":"","title":"S3"},{"content":"","date":"28 February 2021","permalink":"/tags/amplify/","section":"Tags","summary":"","title":"Amplify"},{"content":"Intro # This is how I built a simple sprint dashboard using VueJS and used Amplify for backend, frontend, CI/CD, and for custom domain set up as well. In this post, I would like to share my experience using Amplify and some key lessons learned ;)\nThis is the first time I used Amplify as well as GraphQL and I was so impressed with the ease of use of Amplify to rapidly build an application. However, there are some limitations I experienced, which I will discuss further in this post.\nFor the frontend part, I have used the free VueJS theme from Creative Tim and modified as I need . Most of the project setup was done using Amplify CLI, but I have to manually set up a few configurations in AWS console where Amplify doesn\u0026rsquo;t support yet.\nFeatures # Authentication and User Management # For the authentication, I have used amplify auth API. However, I didn\u0026rsquo;t use the UI presets from Amplify but created custom UI components for login, forgot password, etc. Amplify support Cognito out of the box and it was really easy to configure. I used Cognito user pools to create and authenticate users. I created two user groups - Users and Admins. Registration is not open for public, so admin should create an account for user.\nOnce the admin created an account for the user, there will be an email sending out with a temporary password (Cognito invite flow). User will be asked to reset the password in their first login.\nImage: Verify account screen I have created functionality to change the password for logged in user as well as forgot password feature for not loggedin users which use Amplify Auth API under the hood.\nDashboard, Backlog, Tickets, Sprints # Once user logged in, they have access to the dashboard which has summarized data about the current sprint and his/her own tasks.\nImage: Dashboard view They can add, edit, move tickets between sprints. Also, users can move the tickets between different stages of the current sprint, namely - todo, in progress, review, testing, and done.\nImage: Current sprint view To store all these data, I used Amplify GraphQL API. Setting this up is a matter of minutes using Amplify CLI and documentation has good examples of how to model the data structure with different relationships. Once we provide our GraphQL schema, Amplify will automatically create all the DynamoDB tables, indexes, GSIs and it also generates all the code we need to communicate with GraphQL service (queries, mutations and subscriptions). This is really handy and I only had to do a minor modification to the generated code.\nHere is my graphql definition which has 1:many and many:many relationships.\ntype User @model { id: ID! firstname: String lastname: String department: String designation: String verified: Boolean } type Sprint @model { id: ID! title: String! description: String startAt: Int! endAt: Int! isCurrent: Boolean! archived: Boolean! createdBy: User! @connection tickets: [SprintTickets] @connection(keyName: \u0026#34;bySprint\u0026#34;, fields: [\u0026#34;id\u0026#34;]) } type Ticket @model { id: ID! title: String! description: String points: Int priority: String! type: String! status: String! reportBy: User! @connection assignee: User @connection sprints: [SprintTickets] @connection(keyName: \u0026#34;byTicket\u0026#34;, fields: [\u0026#34;id\u0026#34;]) } type SprintTickets @model(queries:null) @key(name: \u0026#34;bySprint\u0026#34;, fields: [\u0026#34;sprintId\u0026#34;, \u0026#34;ticketId\u0026#34;]) @key(name: \u0026#34;byTicket\u0026#34;, fields: [\u0026#34;ticketId\u0026#34;, \u0026#34;sprintId\u0026#34;]) { id: ID! sprintId: ID! ticketId: ID! ticket: Ticket! @connection(fields: [\u0026#34;ticketId\u0026#34;]) sprint: Sprint! @connection(fields: [\u0026#34;sprintId\u0026#34;]) } Admin specific features # Admin users have more functionality than normal users. They can create new users and promote existing users as admins as well.\nImage: User list view To restrict this functionality, I have used Admin Queries API. Amplify provides set of HTTP APIs to interact with Cognito user pools for example to create user, delete user, get all users etc. It is possible to allow access to these APIs only to certain user groups in the Cognito pool easily (in our case, only to \u0026lsquo;Admin\u0026rsquo; users).\nAdmins also can create sprints, close sprints, edit sprints. To restrict normal users perform these activities, I simply use the user group values provided by Auth.currentAuthenticatedUser() method.\nLambda Triggers # Lambda triggers are a great way to set up custom functionality in authentication flow. Amplify provides this with few simple configuration options.\nIn this project, I have set up post-confirmation trigger. When the user confirms his/her account, I add the user to the \u0026lsquo;Users\u0026rsquo; group.\nHub # I found this feature really interesting where we need to communicate with different components of the same application. Hub is a simple pub-sub implementation provided by Amplify out of the box.\nOne of the instances I used Hub in this project, is to fetch data from API when tickets are moved between sprints. It was easy to set up dispatchers and listeners with channels without complex parent-child reactive communications in Vue.\nDispatcher:\nHub.dispatch(\u0026#34;SprintsChannel\u0026#34;, { event: \u0026#34;ticketMoved\u0026#34;, data: {}, message: \u0026#34;\u0026#34;, }); Listener:\nHub.listen(\u0026#34;SprintsChannel\u0026#34;, (data) =\u0026gt; { if (data.payload.event == \u0026#34;ticketMoved\u0026#34;) { this.fetchData(); this.fetchBacklogTickets(); } }); Continuous Deployment # We can easily set up CD functionality in Amplify console. Here, for this project, I have set up CD, where each time I pushed to \u0026lsquo;master\u0026rsquo; branch of the repository, it will build both frontend and backend and deploys automatically.\nThe nice thing is we can set up different branches for different environments easily. This will be quite handy when we have prod, testing, staging environments, so we can deploy to separate frontends/backends with different branches.\nCustom Domains # For this project, I used a custom domain https://dashboard.pubudu.dev. This can be arranged in Amplify console with few clicks. Amplify by default provides a domain but we can always add our own.\nDemo # You may create an account and play around with my project. Login with the below Admin credentials and create your own user, and promote the newly created user as an admin to use the full functionality.\nhttps://dashboard.pubudu.dev username: pubudusj@gmail.com password: admin123#$ You can find the source code here in Github .\nList of functionality # Login, verify account, forgot password View Dashboard, current sprint, backlog, passed sprints Create, edit, close sprints (admin only) Create, edit tickets Move tickets between sprints, backlog Move tickets of the current sprint to different stages. Modify own profile information, password change Create (invite) a user (admin only) Promote normal user to admin (admin only) Some key points to note (Lesson learned) # Although it is a best practice to use a single table in DynamoDB, Amplify/GraphQL api doesn\u0026rsquo;t support it yet. But it creates a __typename column in all the tables which seems to be a sign that this will be supported soon (?)\nInvite user functionality is not yet available with Amplify by default. Here, I had to use a custom method to use Cognito\u0026rsquo;s adminCreateUser functionality in Admin Queries. Also, when user first login, his/her account will be in NEW_PASSWORD_REQUIRED state which is not considered as a confirmed user (yet). So, I had to check for this state and provide the password change form to the user to update the password in the first login.\nSince user invite functionality is not yet available with Amplify, I had to modify the user invite message in Cognito manually.\nAs of now, we can grant permission to a specific user group to access all the admin APIs. We cannot specify/restrict different user groups for different APIs.\nWhen we have many-to-many relationships in our data models, it will be messy to use only two tables. So, best option is to use 3rd table to maintain the relationship. I used SprintTickets DynamoDB table for this purpose and @model(queries:null) flag in the definition as this table is not used to query directly.\nUseful Links/Resources # Amplify \u0026amp; VueJS example application Amplify libraries for javascript Narder Dabit\u0026rsquo;s youtube channel with lot of Amplify resources AWS Amplify blog posts I would like you to play around with my project and provide feedback. I am not so strong in the frontend, so I believe there are definitely better ways to implement these features. 😄\nKeep building\u0026hellip; Keep sharing..!\n","date":"28 February 2021","permalink":"/posts/how-i-built-a-simple-sprint-dashboard-with-aws-amplify/","section":"Posts","summary":"This blog post discuss how to build a simple sprint dashboard with AWS Amplify.","title":"How I built a Simple Sprint Dashboard with AWS Amplify"},{"content":"","date":"27 October 2020","permalink":"/tags/greengrass/","section":"Tags","summary":"","title":"Greengrass"},{"content":"This post contains a weekend project I worked on to create an alarm when my AWS EC2 servers are down. Tools and technologies used: AWS Lambda, EventBridge, Greengrass, Raspberry Pi, Active Buzzer and LED light (as the alarm), GPIO, etc.\nArchitecture # Image: Highlevel architecture How it works # When an EC2 is stopped or terminated, EventBridge will capture the event.\nEventBrige will trigger the Lambda function (PublishToIotCloud) which publish a message to IOT cloud with the topic \u0026lsquo;alarm\u0026rsquo;.\nWithin the Greengrass group, there is a subscription for the topic \u0026lsquo;alarm\u0026rsquo;. Once it receive a message to \u0026lsquo;alarm\u0026rsquo; topic, it sends message to Greengrass core within Raspberry Pi.\nWithin Raspberry Pi, Lambda will be triggered, and connected alarm (active buzzer and led light) will fire.\nImplementation # Installing Greengrass # First it is required to install AWS Greengrass software within the pi. Please refer my other post which will detail all the steps required to install Greengrass within the Pi. Getting started with AWS Greengrass and Raspberry Pi Set up alarm # Here I used active buzzer with a LED light as the alarm. Used a breadboard to connect those to pi. Please make sure you keep the GPIO pin numbers for both active buzzer and LED light as those are required in the next steps to configure the lambda. Learn more about GPIO here.\nCreate services # I have used AWS SAM to create the required Lambdas and EventBridge rules and targets.\nRepository: https://github.com/pubudusj/iot-alarm\nInstall AWS SAM. Clone above repository and run sam build --use-container and sam deploy -g which will take you through the required parameters. (Please make sure you create your stack in the same region as the AWS GreengrassCore created in the Installing Greengrass section.)\nConfigure Greengrass Group # Goto AWS IoT \u0026gt; Greengrass \u0026gt; Groups and select the group already created.\nGoto Lambda under this group and click on Add Lambda button.\nClick on Use existing Lambda and search IotLambda and select the lambda function and click Next.\nIn the next screen, select the Alias: Iotdeployment option and click finish.\nOnce Lambda is created, click on Lambda\u0026rsquo;s Edit Configuration and increase the Timeout value upto 20 seconds. Also add below Environment variables: BuzzerPin: [GPIO pin number for active buzzer] LedPin: [GPIO pin number for LED light]\nNext, goto the Resources section of the group and click on Add a local resource. This is required to configure Raspberry Pi to talk with Greengrass.\nAdd below values: Resource Name: GPIO Device path: /dev/gpiomem Group owner file access permission: Automatically add OS group permissions of the Linux group that owns the resource Lambda function affiliations: select the same lambda function selected in step 3 and select Read and write access and save.\nOnce done, goto Deployments section and click on Deploy which will configure the lambda within the pi.\nIf deployment failed, you can log into the pi and try find the issue within the logs files at /greengrass/ggc/var/log/system/localwatch/localwatch.log or at /greengrass/ggc/var/log/user/[region]/[account_no]/[lambda_function.log]\nConfigure the trigger events # Goto Subscriptions section of the group and click on Add Subscription button.\nSelect IoT Cloud as source and for the target, select the IotLambda function under Lambdas section and click on Next.\nIn the next view, add Topic filter as alarm. and save.\nTesting # To test the functionality within AWS IoT console, goto Test section and publish a message adding the topic alarm. This should trigger the alarm connected to the pi.\nDemo # Create a EC2 instance in the same region as your stack. Once up and running, change the instance state to Stop Instance or Terminate Instance. The alarm should trigger.\nSome points to highlight # Whenever the Lambda is modified, need to re-deploy the Greengrass Group.\nLambda configurations set up in SAM is not valid for IOT, you need to specifically set up env variables, timeout for IOT Lambda within Greengrass group.\nYou can extent the functionality to any event supported by EventBridge (not limited to AWS events).\nIn the IotLambda function, time.sleep(1) used to make the light and sound to be more like an alarm. Having sleep within Lambdas are not really recommended.\nFor the IotLambda function, I have used Python3.7 as the run time, as Python3.7 is already installed in the latest Raspberry Pi OS (previously called Raspbian). But you may use any Python version by installing it in the Raspberry Pi.\nPlease feel free to try this and let me know your thoughts.\nKeep building. Keep sharing!\n","date":"27 October 2020","permalink":"/posts/how-i-created-a-physical-alarm-to-warn-me-when-my-servers-are-down/","section":"Posts","summary":"This explains the process to create an alarm to trigger when AWS EC2 servers are down","title":"How I created a 'physical' alarm to warn me when my servers are down"},{"content":"","date":"27 October 2020","permalink":"/tags/iot/","section":"Tags","summary":"","title":"IoT"},{"content":"This post walks through the steps required to set up AWS Greengrass on your Raspbarry pi.\nInstall OS for Raspberry Pi on SD card # Goto https://www.raspberrypi.org/downloads, download and install the Raspberry Pi Imager software to your local machine which will guide the installation.\nOnce installed, open the Raspberry Pi Imager software and select the relevant OS (Raspberry Pi OS 32bit is recommended), SD Card and it will create the OS image in the SD card provided.\nOnce the OS is installed, you can insert the SD card to Pi and connect external keyboard and screen to it and configure wifi and ssh. Login username is pi and default password is raspberry. If you don\u0026rsquo;t have external keyboard, screen and you only want to connect to Pi via ssh, enable wifi and ssh as below.\nEnable wifi # Create a file at boot directory with the name: wpa_supplicant.conf with below content. Replace country code (ex: NL, UK), wifi SSID and password with your own values.\ncountry=[COUNTRY_CODE] ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev update_config=1 network={ ssid=\u0026#34;[WIFI SSID]\u0026#34; psk=\u0026#34;[WIFI_PASSWORD]\u0026#34; key_mgmt=WPA-PSK } Enable SSH # To enable ssh access, create a empty file at boot directory with the name of ssh.\nOnce this is done, you can safely remove the SD card, insert it into Raspberry Pi and start it.\nOnce started, you need to know the ip of the device in order to ssh into it. For this, you may use one of IP scanner softwares where you can scan all the devices of the network. Else, you can log into your wifi (router) management ui and get the ip of the Raspberry Pi.\nOnce you know the ip, ssh into the Pi as ssh pi@[IP_Address] with the default password raspberry.\nInstall AWS Greengrass # Prepare the system # Before installing AWS IoT Greengrass Core software, there are few configurations need to be done.\nAdd new user and group. Run below command to create user and group.\nsudo adduser --system ggc_user sudo addgroup --system ggc_group Enable hardlink and symlink protection To enable hard and symlik protection, add below 2 line to end of the /etc/sysctl.d/98-rpi.conf file.\nfs.protected_hardlinks = 1 fs.protected_symlinks = 1 Once done, reboot the Pi.\nEnable and mount memory cgroups Open /boot/cmdline.txt file with sudo permission and append below text to the end of the line and save the file.\ncgroup_enable=memory cgroup_memory=1 Once done, reboot the Pi again.\nDependency check To check if all the dependencies required to run AWS Greengrass core, we can download and run the Greengrass dependency checker as below:\nwget https://github.com/aws-samples/aws-greengrass-samples/raw/master/greengrass-dependency-checker-GGCv1.11.x.zip unzip greengrass-dependency-checker-GGCv1.11.x.zip cd greengrass-dependency-checker-GGCv1.11.x/ sudo modprobe configs sudo ./check_ggc_dependencies This will indicate whether all the system requirements are met.\nConfigurations on AWS console # Log into AWS console and goto IoT Greengrass. (Please note, Greengrass is not available in all the regions.)\nClick on Create a Group \u0026gt; Default Group Creation and create a group providing a group name. (ex: MyRaspberryGroup)\nIn the next step, Core name is auto filled based on your group name. Keep it as it is and move to next step.\nIn the Review Group creation step, click on Create Group and Core. This will generate certificate and keys required for the device.\nNext screen, download the provided xxx-setup.tar.gz file which contains the keys and certificates generated.\nSetting up the Pi # Move the xxx-setup.tar.gz file downloaded into Pi by scp.\nLog into Pi\nDownload the Greengrass Core software usign:wget https://d1onfpft10uf5o.cloudfront.net/greengrass-core/downloads/1.11.0/greengrass-linux-armv7l-1.11.0.tar.gz\nExtract downloaded tar.gz files by: sudo tar -xzvf greengrass-linux-armv7l-1.11.0.tar.gz -C /. This will create a new greengrass directory in the root directory.\nExtract the xxx-setup-tar.gz files into the same greengrass directory: sudo tar -xvf xxx-setup.tar.gz -C /greengrass\nGo to /greengrass/config and download the root certificate as: sudo wget -O root.ca.pem https://www.amazontrust.com/repository/AmazonRootCA1.pem\nNext, go to /greengrass/ggc/core/ and run sudo ./greengrassd start. This will start the Greengrass daemon and output the process id.\nGreengrass auto start on boot # In order to start Greengrass daemon when Pi reboot/start, follow the below steps.\nCreate a file using sudo at /etc/systemd/system/greengrass.service with below content:\n[Unit] Description=Greengrass Daemon [Service] Type=forking PIDFile=/var/run/greengrassd.pid Restart=on-failure ExecStart=/greengrass/ggc/core/greengrassd start ExecReload=/greengrass/ggc/core/greengrassd restart ExecStop=/greengrass/ggc/core/greengrassd stop [Install] WantedBy=multi-user.target Change the permission of the file to allow root to execute it.\nsudo chmod u+rwx /etc/systemd/system/greengrass.service Enable and start the greengrass.service:\nsudo systemctl enable greengrass sudo systemctl start greengrass Check if greengrass service is working fine (even after reboot) with ps aux | grep greengrass\nNow, your raspberry pi is ready to use as an IOT device managed by AWS IOT.\n","date":"26 October 2020","permalink":"/posts/getting-started-with-aws-greengrass-and-raspberry-pi/","section":"Posts","summary":"Step by step instructions to install AWS Greengrass in a Raspberry Pi","title":"Getting started with AWS Greengrass and Raspberry Pi"},{"content":"Intro # Recently, there was a hackathon at my work place, Wonderkind, and with one of my colleagues, I created a intelligent door bell with AWS Serverless services + a raspberry pi.\nWhenever someone clicks on the button of the \u0026lsquo;door bell\u0026rsquo;, it will capture a image and check through a Amazon Rekognition faces collection to see if the faces on the image are already indexed. And it will send a message to Slack with the scaled image with a watermark with timestamp indicating the number of people on the image and names of people if they are already in the faces collection.\nThis post describes how we built this project and some of the learning.\nArchitecture # Image: Architecture Image: State Machine How it works # There are two main components - Face indexing and face recognition.\nFace Indexing # We have created a simple frontend with VueJS which is hosted on a S3 bucket. Here, we asked to upload an image with a face and the name of the person.\nOnce uploaded, we proxy a lambda function via API gateway to create a pre-signed url and using this generated pre-signed url, we upload the image to s3 bucket with the person\u0026rsquo;s name as a meta data value.\nOnce the image is uploaded to s3 bucket, there is a lambda function triggers which will detect the face in the image and create a entry in the pre defined AWS Rekognition collection (faces collection) with the external id as the name.\nFace Recognition # Using a Raspberry pi with it\u0026rsquo;s camera module, solderless breadboard, and a button, we created the image capturing part when a button is pressed - \u0026rsquo;the door bell\u0026rsquo;.\nThis captured image is uploaded to AWS S3, a lambda function is triggered to initialize Step function execution.\nWithin the Step function, there are two parallel flows.\nOne flow is to detect the faces of the image and search them in the faces collection. This function will output the total number of faces detected. If there are recognized faces, it will output names of those faces as well.\nThe other flow will resize the image and create a watermark with the timestamp. There are lambda functions used for all these functionality.\nAfter completing both the flows, there is another lambda function triggers to compose and send the message to the Slack channel.\nOutput # In the Slack channel, output will be as follows:\nImage: Example output Here, (my sons) Wanuja and Thenuja are already indexed in the faces collection, not me.\nCode # Complete source code can be found at: https://github.com/pubudusj/serverless-bell\nHow to set up # You may deploy the stack using AWS SAM framework easily.\nPrerequisites:\nAWS SAM cli + AWS profile set up npm (to build frontend) Slack Webhook URL Create a Slack App at https://api.slack.com/apps/. Enable \u0026lsquo;Incoming web hook\u0026rsquo; and add the created webhook to the workspace choosing a channel. This will generate a webhook url in the format of https://hooks.slack.com/services/XXXX/XXXX/XXXXXX\nDeploying backend # First create a AWS Rekognition collection in a region where you are going to deploy the stack:\naws rekognition create-collection \\ --collection-id serverless-bell-faces-collection Clone the github repo. This has several directories for different purposes as described below:\nbackend - source code needs to be deployed using SAM face_index_frontend - source code for face index frontend testing - For local testing without using Raspberry pi, you may use this code to test the face recognition functionality. This will upload the provided image similar as Pi uploads the image to s3. scripts_in_pi - Simple python script to use within Pi which will capture the image from the camera module and upload to s3. In cli, go to /backend directory\nRun command: sam build --use-container This will build the python functions with necessary dependancies.\nThen, to deploy the resources, run: sam deploy -g This will ask you to enter details of the stack to be created in AWS including the stack name, region, Rekognition face collection and slack url.\nPlease make sure you create the stack in the same region as Rekognition faces collection.\nOnce the deployment is done, copy these output values as they are required in the next steps: FaceIndexHostingS3Bucket, FaceIndexWebsiteURL, GeneratePresignedUrl, GeneratePresignedUrlForTesting, FaceDetectUploadBucketName\nNow goto face_index_frontend directory, where face index frontend source code is located.\nCreate new .env file copying .env.example. For the VUE_APP_GENERATE_URL_API variable, use GeneratePresignedUrl output value.\nRun npm install to install required modules and then run npm run build to build the project. This will create dist directory.\nThen, let\u0026rsquo;s upload contents of dist directory to s3 to be used as a s3 hosted web site. Use value of the output FaceIndexHostingS3Bucket as the s3 bucket.\naws s3 cp dist s3://[BucketName] --recursive Now you will be able to access the face index web site using the output value: FaceIndexWebsiteURL.\nUpload a face image with a name and you will see the face is index in the faces collection. aws rekognition list-faces --collection-id \u0026quot;serverless-bell-faces-collection\u0026quot;\nWithin Raspberry pi # Set up Raspberry PI with camera module and AWS profile.\nUse the example script in scripts_in_pi directory to capture and upload image to S3.\nReplace bucket-name with the output value FaceDetectUploadBucketName Use relevant gpiozero Button number as per your set up.\nOnce captured, you can see the message in your Slack channel.\nLocal testing without Raspberry pi # Goto the testing directory.\nCreate new .env file copying .env.example. For the VUE_APP_GENERATE_URL_API variable, use GeneratePresignedUrlForTesting output value.\nRun npm install and npm run serve\nWith the provided url, you can access frontend to upload images to detect faces.\nOnce uploaded, you can see the message in your Slack channel.\nSome lessons learnt # In Rekognition faces collection, ExternalImageId only allow alpha numeric characters. So to store the names with multiple parts spaces in between, we have to replace spaces with underscores and when retrieving the vice versa.\nWhen trigger a lambda function from S3 file upload, lambda will not receive the meta data of the uploaded file. So, to retrieve the meta data of the file, need to read the file again.\nIn SAM, it is not possible to use automatically generated S3 bucket names in the policy objects of the function - [reference] ( https://github.com/aws/serverless-application-model/issues/138). Because of this, we had to build the S3 bucket name instead of SAM generated S3 buckets with random names as described in here.\nPossible improvements # Implement authentication for face index frontends, APIs, Lambda functions. Handle failed scenarios in the Step function execution. Process EXIF orientation data of a uploaded image to get the correct orientation. Please feel free to try this and let me know your thoughts.\n","date":"28 September 2020","permalink":"/posts/how-i-created-a-serverless-door-bell/","section":"Posts","summary":"Explains the steps of me creating a doorbell using Raspberry Pi and AWS Serverless services.","title":"How I created a door bell with AWS Serverless"},{"content":"Without a doubt, security is one of the key aspect to focus in AWS. It is not only authentication is critical, but authorization is a must to consider specially in the production systems where lot of parties (users, resources, services) should be given only the specific permissions in the AWS environment. AWS IAM policies provides great amount of facilities in order to set up granular level of permissions. Tag based permissions are one of the cool feature supported by IAM.\nIn this blog post, I would discuss how below tag based conditions can be used within AWS IAM policies.\nResourceTag RequestTag PrincipalTag TagKeys Please note: Policy snippets describe below are only a part of a complete IAM policy document extracted only to illustrate the \u0026rsquo;tags\u0026rsquo; related conditions, so you will need other necessary permissions to manage the same resources.\nResourceTag # ResourceTag is useful when you need to use the existing tags of a resource to be considered for the permissions. For example, let\u0026rsquo;s assume there are two ec2 instances where both have tag named stage where one has value dev and the other\u0026rsquo;s has value production.\nAnd our requirement is to give permissions to stop or terminate ec2 instances only if the ec2 instances has the stage tag value dev. For this purpose, a IAM policy can be used with ResourceTags condition as below:\n{ \u0026#34;Version\u0026#34;:\u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;:[ { \u0026#34;Sid\u0026#34;:\u0026#34;AllowStopTerminateOnlyWhenTagStageIsDev\u0026#34;, \u0026#34;Effect\u0026#34;:\u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;:[ \u0026#34;ec2:TerminateInstances\u0026#34;, \u0026#34;ec2:StopInstances\u0026#34; ], \u0026#34;Resource\u0026#34;:\u0026#34;arn:aws:ec2:*:*:instance/*\u0026#34;, \u0026#34;Condition\u0026#34;:{ \u0026#34;StringLike\u0026#34;:{ \u0026#34;ec2:ResourceTag/stage\u0026#34;:\u0026#34;dev\u0026#34; } } } ] } As defined in the Condition section, ec2 which the action going to be applied (the Resource) must have the tag named stage which\u0026rsquo;s value is dev. Simple yet powerful, isn\u0026rsquo;t it?\nRequestTag # RequestTag conditions can be used to control/validate which tags to be attached with a given resource. For example, let\u0026rsquo;s assume we need to enforce a user to add his team name TeamX as the team tag when creating a ec2 instance.\nFor this, we can use below IAM policy document, which checks the incoming/requested tag value.\n{ \u0026#34;Sid\u0026#34;:\u0026#34;AllowCreateInstanceOnlyWithTeamTag\u0026#34;, \u0026#34;Effect\u0026#34;:\u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;:[ \u0026#34;ec2:CreateVolume\u0026#34;, \u0026#34;ec2:RunInstances\u0026#34;, \u0026#34;ec2:CreateTags\u0026#34; ], \u0026#34;Resource\u0026#34;:[ \u0026#34;arn:aws:ec2:*:*:volume/*\u0026#34;, \u0026#34;arn:aws:ec2:*:*:instance/*\u0026#34; ], \u0026#34;Condition\u0026#34;:{ \u0026#34;StringEquals\u0026#34;:{ \u0026#34;aws:RequestTag/team\u0026#34;:\u0026#34;TeamX\u0026#34; } } } PrincipalTag # PrincipalTag is useful to check if the Principle has a particular tag attached on their identity. Here the Principle can be a user or a role.\nPlease note: As at now, IAM groups does not support tags. So you cannot apply any tag based permissions for a group directly.\nFor example, assume there are two users in a AWS account and both have a tag team attached to them. User A has team tag value is TeamX and for User B it is TeamY. And there is an ec2 instance, which should only be allowed to be stopped or terminated only if user has tag value TeamX.\nFor this requirement, we can use ResourceTag condition as below:\n{ \u0026#34;Version\u0026#34;:\u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;:[ { \u0026#34;Sid\u0026#34;:\u0026#34;AllowStopTerminateOnlyIfPrincipleFromTeamX\u0026#34;, \u0026#34;Effect\u0026#34;:\u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;:[ \u0026#34;ec2:TerminateInstances\u0026#34;, \u0026#34;ec2:StopInstances\u0026#34; ], \u0026#34;Resource\u0026#34;:\u0026#34;arn:aws:ec2:*:*:instance/*\u0026#34;, \u0026#34;Condition\u0026#34;:{ \u0026#34;StringLike\u0026#34;:{ \u0026#34;aws:PrincipalTag/team\u0026#34;:\u0026#34;TeamX\u0026#34; } } } ] } TagKeys # TagKeys are useful when you need to allow only a set of specified tags to be used on a resource. For example, if you need a ec2 instance to be tagged only with one or both team and department tags you may use below policy snippet. If you provide any tag other than these two, you are not permitted to create the instance.\nPlease note: Here, the value of the tag can be anything, it is not validated. Only the tag key is validated\n{ \u0026#34;Sid\u0026#34;:\u0026#34;OnlyAllowTagsTeamAndDepartment\u0026#34;, \u0026#34;Effect\u0026#34;:\u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;:[ \u0026#34;ec2:CreateTags\u0026#34;, \u0026#34;ec2:RunInstances\u0026#34;, \u0026#34;ec2:CreateVolume\u0026#34; ], \u0026#34;Resource\u0026#34;:[ \u0026#34;arn:aws:ec2:*:*:instance/*\u0026#34;, \u0026#34;arn:aws:ec2:*:*:volume/*\u0026#34; ], \u0026#34;Condition\u0026#34;:{ \u0026#34;ForAllValues:StringEquals\u0026#34;:{ \u0026#34;aws:TagKeys\u0026#34;:[ \u0026#34;team\u0026#34;, \u0026#34;department\u0026#34; ] } } } Please note: When you don\u0026rsquo;t provide a tag when creating the resource, still it can be created with above IAM policy as it only check the provided tags and does not enforce having a specific tag.\nSo, if you need to enforce only a particular tag to be exists, you may do it using a combination of TagKeys and RequestTags as below:\n{ \u0026#34;Sid\u0026#34;:\u0026#34;VisualEditor0\u0026#34;, \u0026#34;Effect\u0026#34;:\u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;:[ \u0026#34;ec2:CreateTags\u0026#34;, \u0026#34;ec2:RunInstances\u0026#34;, \u0026#34;ec2:CreateVolume\u0026#34; ], \u0026#34;Resource\u0026#34;:[ \u0026#34;arn:aws:ec2:*:*:instance/*\u0026#34;, \u0026#34;arn:aws:ec2:*:*:volume/*\u0026#34; ], \u0026#34;Condition\u0026#34;:{ \u0026#34;StringLike\u0026#34;:{ \u0026#34;aws:RequestTag/team\u0026#34;:\u0026#34;*\u0026#34; }, \u0026#34;ForAllValues:StringEquals\u0026#34;:{ \u0026#34;aws:TagKeys\u0026#34;:\u0026#34;team\u0026#34; } } } Here, in the RequestTag condition, we enforce team tag to be exists, still we don\u0026rsquo;t validate the value as we accept any value with wild card (*). And using TagKeys we enforce only team tag can be used as a tag.\n","date":"9 October 2019","permalink":"/posts/aws-iam-tag-based-permissions/","section":"Posts","summary":"AWS IAM policies provides great amount of facilities in order to set up granular level of permissions. Tag based permissions are one of the cool features supported by IAM.","title":"IAM Tag Based Permissions"},{"content":"","date":"9 October 2019","permalink":"/tags/tags/","section":"Tags","summary":"","title":"Tags"},{"content":"Scenario # Let\u0026rsquo;s think about this requirement:\nThere is a lambda function, which calls to a external API and fetch some data. This external API only accepts incoming requests only from pre-configured whitelisted IPs. As per today, AWS doesnt\u0026rsquo; support elastic IPs for Lambda. However, there is a simple work around.\nSolution # Simple solution is, to create the lambda function within a VPC and make the 3rd party API to be accessible via a NAT gateway.\nYou may use a NAT instance as well, but as a best practice, it is recommended to use NAT Gateway wherever possible, unless there is a specific reason to use NAT instance. Find the comparison between NAT instance vs NAT Gateway here.\nThis is the simplest version of the network diagram how you can set up this. As NAT Gateway has Elastic IP assigned, all the traffic from Lambda function to the External API will have the same IP of the NAT Gateway. So, we can whitelist the NAT Gateway\u0026rsquo;s public IP at the external API.\nImage: Network set up for Lambda VPC Below is the VPC configuration settings that is required to set up within the Lambda function. You need to choose the VPC, sub nets you need to use Lambda function and the security groups (for access control) as below.\nImage: VPC settings for Lambda function Please note: As highlighted in the above image, once the Lambda function is inside the VPC, it loses the internet connectivity. You have to explicitly configure internet access within the VPC (in this scenario using a NAT Gateway).\n","date":"27 September 2019","permalink":"/posts/whitelist-lambda-ip/","section":"Posts","summary":"This blog post discuss how we can allow access from Lambda from an external system by whitelisting IP.","title":"Whitelisting Lambda IP for Downstream Services"},{"content":"IAM Policy version is a useful feature in IAM when try to identify the correct access rights for a particular policy and switch between the changes applied to the policies.\nPlease Note: Policy versions are not available for inline policies.\nWhen you update a particular customer manage policy, AWS will not over-write the existing policy. But new version is created and set as the default version.\nUsing AWS Console # Image: Policy versions listing for a single IAM Policy If you navigate to IAM \u0026gt; Policies and select Customer Manage Policy and select a specific policy. Under Policy versions, you may see different versions as per your modifications to the policy document. If this is a fresh policy where you did not do any modifications, still Version 1 can be seen which set as the Default version.\nHere, you can set the default version to a previous versions easily. Or you can delete previous versions.\nPlease Note: Currently, only maximum 5 versions are allowed.\nEven in the AWS Managed Policies, you may see the Policy versions as and when AWS updated them. However you cannot delete the versions or specify the Default version (This is obvious as AWS manage those policies)\nUsing AWS CLI # Using AWS CLI, you can retrieve the same information of policy versions using list-policy-versions command as below:\naws iam list-policy-versions --policy-arn arn:aws:iam::xxxxxxxx:policy/lambda_execute This will output below result which contains created date, version id and default version flag.\n{ \u0026#34;Versions\u0026#34;: [ { \u0026#34;CreateDate\u0026#34;: \u0026#34;2019-03-04T10:48:30Z\u0026#34;, \u0026#34;VersionId\u0026#34;: \u0026#34;v5\u0026#34;, \u0026#34;IsDefaultVersion\u0026#34;: true }, { \u0026#34;CreateDate\u0026#34;: \u0026#34;2019-03-04T10:46:13Z\u0026#34;, \u0026#34;VersionId\u0026#34;: \u0026#34;v4\u0026#34;, \u0026#34;IsDefaultVersion\u0026#34;: false }, { \u0026#34;CreateDate\u0026#34;: \u0026#34;2019-03-04T10:44:14Z\u0026#34;, \u0026#34;VersionId\u0026#34;: \u0026#34;v3\u0026#34;, \u0026#34;IsDefaultVersion\u0026#34;: false }, { \u0026#34;CreateDate\u0026#34;: \u0026#34;2019-03-03T23:59:49Z\u0026#34;, \u0026#34;VersionId\u0026#34;: \u0026#34;v2\u0026#34;, \u0026#34;IsDefaultVersion\u0026#34;: false }, { \u0026#34;CreateDate\u0026#34;: \u0026#34;2019-03-03T23:56:17Z\u0026#34;, \u0026#34;VersionId\u0026#34;: \u0026#34;v1\u0026#34;, \u0026#34;IsDefaultVersion\u0026#34;: false } ] } Also, you can set the default or effective version for a policy using set-default-policy-version command providing the version you need to set as default.\naws iam set-default-policy-version --policy-arn arn:aws:iam::xxxxxxxxxx:policy/lambda_execute --version-id v2 Here, the policy lambda_execute will be revert to v2. There won’t be any output here, but you can verify the result using the list-policy-versions command as described above.\n","date":"26 September 2019","permalink":"/posts/iam-policy-versioning/","section":"Posts","summary":"Explains how you can manage your IAM Policy versions","title":"IAM Policy Versioning"},{"content":"Lambda by nature is highly scalable. However there ares some limitations you need to consider when there are lot of Lambda functions run simultaneously.\nPlease note: This will not applicable for all the scenarios, but for a system with a high throughput.\nAccount Level Concurrent Execution Limit # As at now, Lambda has a soft limit of 1000 concurrent executions per region. Which means, at any given moment, sum of lambda executions running belongs to all of your lambda functions in a single region must be less than 1000.\nExample:\nLet’s say you have 3 Lambda functions in a same region:\nA — SQS poller function which has X number of executions running at this very moment. B — Function to update Dynamodb with Kenisis stream which has Y number of executions running at this very moment. C — Function to process some files in s3 which has Z number of executions running at this very moment.\nSo, X + Y + Z will be equal or less than 1000.\nEven the demand is high, no more new executions will spawn until one execution is completed.\nPlease note: This 1000 limit can be increased by create a case at AWS Support Center under \u0026ldquo;Service Limit Increase\u0026rdquo; with your use case.\nTo understand the rest of calculations easily, will consider this particular region has default 1000 concurrency limit.\nFunction Level Concurrent Execution Limit # You can reserve concurrency limit per function basis which is called reserved concurrency. The rest of the concurrency out of maximum concurrency allowed per region is called unreserved concurrency.\nImage: Setting up lambda reserved concurrency for a function Example:\nA — Function which has 250 reserved concurrent executions. B — Function which has 100 reserved concurrent executions. C — Function which has X number of executions running at this very moment. D — Function which has Y number of executions running at this very moment. So, X + Y will be equal or less than 650 (1000 - (250 + 100))\nPlease note: There must be minimum 100 Unreserved concurrent executions per region. Hence, maximum amount of reserved concurrency across all the functions per region will be 900 (1000–100)\nWhy we need reserved concurrency? # Reserved concurrency is useful to allocate concurrency for a function which need dedicated priority over the others. As we reserve the concurrency, at a given time, this function can run with this maximum amount of concurrency without interrupting or waiting.\nAnother reason is to handle the load of the other related services special VPC based resources. For example, as below diagram, you have a lambda function which is executed by a sqs queue and it does some data processing and save them to a relational database in RDS which sits in a vpc.\nSimply, Elastic Network Interfaces has maximum number of IP addresses depends on the instance type. So it might not be able to save data in the rate of receiving them from lambda, which make a bottle neck at the ENI.\nSo, we need to calculate the maximum rate ENI can manage concurrency and limit the lambda function’s reserved concurrency accordingly in order to not to overload ENI.\nLambda Concurrency Metrics # There are two metrics available in CloudWatch for Lambda Concurrency.\nConcurrentExecutions This is the sum of concurrent executions across all the function in a region at a given point in time.\nUnreservedConcurrentExecutions This is the sum of the concurrency of the functions that do not have a custom concurrency limit specified in a region at a given point in time.\n","date":"26 September 2019","permalink":"/posts/understanding-lambda-concurrency/","section":"Posts","summary":"This blog post explains the behaviour of Lambda when they run simultaneously.","title":"Understanding Lambda Concurrency"},{"content":"","date":"1 January 0001","permalink":"/authors/","section":"Authors","summary":"","title":"Authors"},{"content":"","date":"1 January 0001","permalink":"/categories/","section":"Categories","summary":"","title":"Categories"},{"content":"","date":"1 January 0001","permalink":"/series/","section":"Series","summary":"","title":"Series"}]