Schedule File Transfer from SFTP to S3 with AWS Lambda
Download as PPTX, PDF2 likes11,723 views
This document outlines the steps to set up an AWS Lambda function for scheduled file transfers from SFTP to S3 using Python. It includes creating necessary IAM roles, configuring an EC2 instance, writing the handler code, and scheduling the Lambda function using AWS EventBridge. Additionally, it provides command line instructions for creating the function, invoking it manually, and setting up permissions and event targets.
![Handlerimport paramiko
import boto3
import datetime
import os
LOCALFILE='/tmp/invbatch.txt'
s3 = boto3.client('s3')
def connect_to_sftp():
transport = paramiko.Transport((os.environ['HOST'],int(os.environ['PORT'])))
transport.connect(username=os.environ['USERNAME'],
password=os.environ['PASSWORD'])
sftp = paramiko.SFTPClient.from_transport(transport)
return sftp
def handler(event, context):
sftp = connect_to_sftp()
sftp.get(os.environ['REMOTEFILE'], LOCALFILE)
now = datetime.datetime.today()
S3FILE='test/' + now.strftime('%Y%m%d-%H%M%S') + '_invbatch.txt'
s3.upload_file(Filename=LOCALFILE, Bucket='nufin-test-bucket', Key=S3FILE)](https://image.slidesharecdn.com/awslambdaschedulefiletransfersftp2s3-171021020927/85/Schedule-File-Transfer-from-SFTP-to-S3-with-AWS-Lambda-3-320.jpg)
![Makefile
Schedule the function : Associate the Lambda with the Event
putTargets:
aws events put-targets
--region $(REGION)
--rule $(FUNCTION)-rule
--targets file://targets.json
--profile Administrator
[
{
"Id": "1",
"Arn":"arn:aws:lambda:ap-southeast-1:864962608331:function:sftp2s3"
}
]
targets.json](https://image.slidesharecdn.com/awslambdaschedulefiletransfersftp2s3-171021020927/85/Schedule-File-Transfer-from-SFTP-to-S3-with-AWS-Lambda-9-320.jpg)
Schedule File Transfer from SFTP to S3 with AWS Lambda
- 1. AWS Lambda Scheduled file transfer sftp to s3 python
- 2. Steps 1. Create a new Administrator user in the IAM 2. Create a Role and allow Lambda execution and permissions for S3 operations 3. Take note of the User ARN 4. Deploy 64-bit Amazon Linux EC2 instance 5. SSH in, make project directory/folder 6. Create requirements.txt file, list python packages (except boto3) 7. Write the Handler 8. Zip up code and Dependencies 9. Create Lambda Function 10. Manually invoke the Lambda Function 11. Create the schedule
- 3. Handlerimport paramiko import boto3 import datetime import os LOCALFILE='/tmp/invbatch.txt' s3 = boto3.client('s3') def connect_to_sftp(): transport = paramiko.Transport((os.environ['HOST'],int(os.environ['PORT']))) transport.connect(username=os.environ['USERNAME'], password=os.environ['PASSWORD']) sftp = paramiko.SFTPClient.from_transport(transport) return sftp def handler(event, context): sftp = connect_to_sftp() sftp.get(os.environ['REMOTEFILE'], LOCALFILE) now = datetime.datetime.today() S3FILE='test/' + now.strftime('%Y%m%d-%H%M%S') + '_invbatch.txt' s3.upload_file(Filename=LOCALFILE, Bucket='nufin-test-bucket', Key=S3FILE)
- 4. Makefile Zip up Code and Dependencies packageDependencies: zip -r9 $(FUNCTION).zip sftp_to_s3.py mkdir -p site-packages virtualenv $(FUNCTION) . $(FUNCTION)/bin/activate; pip install -r requirements.txt; cd site-packages; cp -r $$VIRTUAL_ENV/lib/python2.7/site-packages/ ./; cp -r $$VIRTUAL_ENV/lib64/python2.7/site-packages/ ./; cd site-packages; zip -r9 ../../$(FUNCTION).zip . PROJECT = sftp2s3 FUNCTION = $(PROJECT) REGION = ap-southeast-1 ROLE = arn:aws:iam::864962608331:role/service-role/myLambdaRole Makefile Variables
- 5. Makefile Create the Lambda Function createFunction: aws lambda create-function --region $(REGION) --function-name $(PROJECT) --zip-file fileb://$(FUNCTION).zip --role $(ROLE) --environment Variables=“{HOST='12.123.123.12',PORT=22,USERNAME='lhftp',PASSWORD='123456789012',REMOTEFILE='lhftp_uploads/invb --handler sftp_to_s3.handler --runtime python2.7 --profile Administrator --timeout 10 --memory-size 1024
- 6. Makefile manualInvoke: aws lambda invoke --invocation-type RequestResponse --function-name $(FUNCTION) --region $(REGION) --profile Administrator outputfile.txt Manually Invoke the Lambda function to test it
- 7. Makefile # Schedule the Lambda Function # Step 1: createRule: aws events put-rule --region $(REGION) --name $(FUNCTION)-rule --schedule-expression 'cron(0/5 8-17 ? * MON-FRI *)' --profile Administrator Schedule the Lambda function : Create the rule
- 8. Makefile # $(ARNRULE) is taken from the put-rule target output and parsed in # ARNRULE=arn:aws:events:..... # Step 2: addPermission: aws lambda add-permission --region $(REGION) --function-name $(FUNCTION) --statement-id $(FUNCTION)-event --action 'lambda:InvokeFunction' --principal events.amazonaws.com --source-arn $(ARNRULE) --profile Administrator Schedule the function : Give permission to the event
- 9. Makefile Schedule the function : Associate the Lambda with the Event putTargets: aws events put-targets --region $(REGION) --rule $(FUNCTION)-rule --targets file://targets.json --profile Administrator [ { "Id": "1", "Arn":"arn:aws:lambda:ap-southeast-1:864962608331:function:sftp2s3" } ] targets.json