Lab: AWS Lambda, S3 and API Gateway

1.Amazon Simple Storage Service (Amazon S3)

1.1 Overview of AWS S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Visit https://aws.amazon.com/s3

 

Navigate to S3 console at http://s3.console.aws.amazon.com/ .

Click the “Create bucket” button. 

Create a bucket polyuawslab-[your name] (replace [your name] with your name). Note that the name should not contain uppercase characters and you should assign a unique bucket name (over all AWS S3 bucket names)). 

Click next a number of times until the bucket is created.

Click the created bucket

upload files

Verify that the files are uploaded. 

1.2 Create the IAM user with the access permission to S3 

Remark: You can only create IAM user if you are using regular AWS accounts. Use the default access key ID and secret key if you are using AWS starter account. 

Navigate to AWS Identity and Access Management (IAM). Navigate to the Users page. 

go to https://console.aws.amazon.com/iam/home?#/users

Click Add user to create an IAM user s3_user. For access type, choose Programmatic access. Click Next.

Select Attach existing policies directly. Check the box for AmazonS3FullAccess. 

You may expand the AmazonS3FullAccess to view the policy (JSON tab). 

And click the NEXT and create user and download the file

 

 

1.3. Accessing S3 with the AWS Command Line Interface 

Download and install AWS CLI if you have not done so before.

https://aws.amazon.com/cli. 

Open a command prompt, enter:

aws configure

Provide your access Key ID and secret access key. Choose us-east-2 (choose us-east-1 if you are using AWS starter account) as the default region and json as the default output.

Try the following commands 

aws s3 ls 

aws s3 ls s3://[your bucket name]

Refer to the following page for command reference for s3.

https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

 

1.4. Accessing S3 in python

Try the following code which lists the bucket names in s3.

import boto3
from pprint import pprint
s3 = boto3.client('s3',
 aws_access_key_id = '你的access_key',
 aws_secret_access_key='你的secret_key',
 region_name='us-east-2'
 )

def list_buckets():
    # Output the bucket names
    response = s3.list_buckets()
    print('Existing buckets:')

    for bucket in response['Buckets']:
        print(f' {bucket["Name"]}')

list_buckets()

在cmd运行

python s3.py

 

2. AWS Lambda

2.1. Overview 

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume - there is no charge when your code is not running. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically be triggered from other AWS services or call it directly from any web or mobile app. Visit: https://aws.amazon.com/lambda/ 

Examples of serverless applications with lambda

Example 1: Real-time file processing: https://github.com/aws-samples/lambda-refarch-fileprocessing

Example 2: Label detection is triggered when images are uploaded 

 

2.2. Create your first AWS Lambda Function

Step 1: Enter the Lambda Console

Under Services, find Lambda under Compute and click to open the AWS Lambda Console. 

Select US East (Ohio) region. 

https://us-east-2.console.aws.amazon.com/lambda/home?region=us-east-2#/functions

Step 2: Create function

Blueprints provide example code to do some minimal processing. Most blueprints process events from specific event sources, such as Amazon S3, DynamoDB, or a custom application. a. In the AWS Lambda console, select Create a Function.

b. Select Blueprints.

c. In the Filter box, type in hello-world-python and select the hello-world-python blueprint. Click Configure

Step 3: Configure and Create Your Lambda Function

A lambda function consists of code you provide, associated dependencies, and configuration. The configuration information you provide includes the compute resources you want to allocate (for example, memory), execution timeout, and an IAM role that AWS Lambda can assume to execute your Lambda function on your behalf. Enter Basic Information about your Lambda function.

• Name: You can name your Lambda function here. For this tutorial, enter hello-world-python.

• Role: You will create an IAM role (referred as the execution role) with the necessary permissions that AWS Lambda can assume to invoke your Lambda function on your behalf. Select Create new role from AWS policy template(s).

• Role name: type lambda_basic_execution.

Go to the bottom of the page and select Create Function.

then

Modify the lambda function in the Function code section. 

Scroll down the page to view the settings

Click Edit to examine the basic settings. 

Runtime: Currently, you can author your Lambda function code in Java, Node.js, C#, Go or Python. For this tutorial, leave this on Python 3.7 as the runtime.

Handler: You can specify a handler (a method/function in your code) where AWS Lambda can begin executing your code. AWS Lambda provides event data as input to this handler, which processes the event. In this example, Lambda identifies this from the code sample and this should be pre-populated with lambda_function.lambda_handler. We will accept the default settings. Click Cancel. 

Step 4: Invoke Lambda Function and Verify Results The console shows the hello-world-python Lambda function - you can now test the function, verify results, and review the logs. Select Configure Test Event from the drop-down menu called "Select a test event...".

The editor pops up to enter an event to test your function.

• Choose Hello World from the Sample event template list from the Input test event page.

• Type in an event name like HelloWorldEvent.

• You can change the values in the sample JSON, but don’t change the event structure. For this tutorial, replace value1 with hello, world!. 

click test

Upon successful execution, view the results in the console:

• The Execution results section verifies that the execution succeeded.

• The Summary section shows the key information reported in the Log output.

• The Log output section will show the logs generated by the Lambda function execution.

Step 5: Monitor Your Metrics

AWS Lambda automatically monitors Lambda functions and reports metrics through Amazon CloudWatch. To help you monitor your code as it executes, Lambda automatically tracks the number of requests, the latency per request, and the number of requests resulting in an error and publishes the associated metrics.

Invoke the Lambda function a few more times by repeatedly clicking the Test button. This will generate the metrics that can be viewed in the next step. Select Monitoring to view the results. 

Visit https://aws.amazon.com/cloudwatch to understand more about AWS CloudWatch.

Step 6: Delete the Lambda Function

While you will not get charged for keeping your Lambda function, you can easily delete it from the AWS Lambda console. Select the Actions button and click Delete Function.

 

 

3. AWS API Gateway

3.1. Overview

Amazon API Gateway (https://aws.amazon.com/api-gateway/) is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. 

 

3.2. Create a REST API for Lambda Function

Open the AWS Lambda console.

Choose Create function and select Author from scratch.

For Function name, enter my-function. Choose the options as shown below. Click Create function.

By default, the lambda handler code is as follows. 

3.3. Create a REST API in the API Gateway Console 

In this section, you create a simple REST API in the API Gateway console and attach your Lambda function to it as a backend. 

From the Services menu, choose API Gateway to go to the API Gateway console. Under Choose an API type, choose REST API, and click Build. 

1. Under Create new API, choose New API.

2. Under Settings:

• For API name, enter my-api.

• If desired, enter a description in the Description field; otherwise, leave it empty.

• Leave Endpoint Type set to Regional.

3. Choose Create API.

Under Resources, you'll see the endpoint /. This is the root-level resource, which corresponds to the base path URL for your API

Under the resource name (/), you'll see a dropdown menu. Choose GET and then choose the checkmark icon to save your choice. 

In the / – GET – Setup pane:

- Choose Lambda Function as for Integration type

- Check Use Lambda proxy integration.

- For Lambda Region, choose the Region where you created your Lambda function (e.g. us-east-2)

- In the Lambda Function field, input my-function (or whatever name you gave the function that you created in the previous step) from the dropdown menu.

- Leave Use Default Timeout checked. 

Choose Save to save your choice. 

When the Add Permission to Lambda Function popup appears (saying "You are about to give API Gateway permission to invoke your Lambda function…"), choose OK to grant API Gateway that permission.

Now you'll see a / – GET – Method Execution pane:

The Method Execution pane contains these items, in clockwise order:

• Client: This box represents the client (browser or app) that calls your API's GET method

• Method Request: This box represents the client's GET request as it's received by API Gateway.

• Integration Request: This box represents the GET request as it's passed to the backend. 

• Lambda my-function: This box represents the backend Lambda function you created previously.

• If you choose my-function, that opens the my-function Lambda function in the Lambda console

• Integration Response: This box represents the response from the backend, before it's passed to the client as a method response. 

• For Lambda proxy integration, this entire box is grayed out, because a proxy integration returns the Lambda function's output as is. 

•Method Response: This box represents the method response that's returned to the client as an HTTP status code, an optional response header, and an optional response body. 

• By default, the response body that's returned by your Lambda function is passed through as is as a JSON document, so the response body default setting is application/json with an empty model (indicating that the body is passed through as is).  

Click Test.

3.4. Deploy Your REST API in the API Gateway Console 

From the Actions dropdown menu, choose Deploy API. 

From the Deployment stage dropdown menu, choose [New Stage]. For Stage name, enter prod.

Choose Deploy. In the prod Stage Editor, note the Invoke URL at the top. 

If you choose the Invoke URL, it will open a new browser tab with that URL. If you refresh the new browser tab, you'll see the message body ("Hello from Lambda!") returned.

 

3.5. Passing HTTP Query String to Lambda

Amazon API Gateway invokes your function synchronously with an event that contains details about the HTTP request that it received. An Example Amazon API Gateway Message Event is shown below. 

{
 "path": "/test/hello",
 "headers": {
 "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
 "Accept-Encoding": "gzip, deflate, lzma, sdch, br",
 "Accept-Language": "en-US,en;q=0.8",
 "CloudFront-Forwarded-Proto": "https",
 "CloudFront-Is-Desktop-Viewer": "true",
 "CloudFront-Is-Mobile-Viewer": "false",
 "CloudFront-Is-SmartTV-Viewer": "false",
 "CloudFront-Is-Tablet-Viewer": "false",
 "CloudFront-Viewer-Country": "US",
 "Host": "wt6mne2s9k.execute-api.us-west-2.amazonaws.com",
 "Upgrade-Insecure-Requests": "1",
 "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/52.0.2743.82 Safari/537.36 OPR/39.0.2256.48",
 "Via": "1.1 fb7cca60f0ecd82ce07790c9c5eef16c.cloudfront.net (CloudFront)",
 "X-Amz-Cf-Id": "nBsWBOrSHMgnaROZJK1wGCZ9PcRcSpq_oSXZNQwQ10OTZL4cimZo3g==",
 "X-Forwarded-For": "192.168.100.1, 192.168.1.1",
 "X-Forwarded-Port": "443",
 "X-Forwarded-Proto": "https"
 },
 "pathParameters": {
 "proxy": "hello"
 },
 "requestContext": {
 "accountId": "123456789012",
 "resourceId": "us4z18",
 "stage": "test",
 "requestId": "41b45ea3-70b5-11e6-b7bd-69b5aaebc7d9",
 "identity": {
 "cognitoIdentityPoolId": "",
 "accountId": "",
 "cognitoIdentityId": "",
 "caller": "",
 "apiKey": "",
 "sourceIp": "192.168.100.1",
 "cognitoAuthenticationType": "",
 "cognitoAuthenticationProvider": "",
 "userArn": "",
 "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/52.0.2743.82 Safari/537.36 OPR/39.0.2256.48",
 "user": ""
 },
 "resourcePath": "/{proxy+}",
 "httpMethod": "GET",
 "apiId": "wt6mne2s9k"
 },
 "resource": "/{proxy+}",
 "httpMethod": "GET",
 "queryStringParameters": {
 "name": "me"
 },
 "stageVariables": {
 "stageVarName": "stageVarValue"
 }
}

The HTTP query string parameters are passed to the lambda function through the event object (with the “'queryStringParameters” attribute). Change your lambda_handler function as follows.

import json
def lambda_handler(event, context):
 return {
 'statusCode': 200,
 'body': json.dumps(event['queryStringParameters'])
 }

Click Save. 

In the API gateway console, select “Method Request” in API Gateway and add query string parameter “name”. 

Click Method Execution to confirm the change.

change it

Click Test to verify the result. Specify the query string name=John.

3.6. Passing path parameter to Lambda

Select the root resource /.

Define a resource person with {personID} as the path parameter. 

Create a GET method under /{personID}. Configure the method as follows.

Modify your lambda handler code for my-function as follows

import json
def lambda_handler(event, context):
    result = {}
    result["Path"] = event['pathParameters']
    result["QueryString"] = event['queryStringParameters']
    
    return {
    'statusCode': 200,
    'body': json.dumps(result)
    }

Test your API with path parameter mary and query string lang=Eng&mode=1.

Deploy your API. Access the endpoint with the path parameter and query strings (e.g. /peter/?lang=eng&mode=2).

 

4. Integrating AWS Lambda, S3 and DynamoDB

4.1. Creating a role in AWS Identity and Access Management (IAM)

We first create a role inside IAM to allow lambda function to access S3 and Amazon recognition service.

Navigate to IAM console at console.aws.amazon.com/iam. On the left navigation menu click “Roles”. Click blue Create role button. 

On Select Role type page, select “AWS Service” section and choose “Lambda” Click blue “Next: Permissions” button. 

On Select Role type page, select “AWS Service” section and choose “Lambda” Click blue “Next: Permissions” button

On Attach Permissions page, search for and select the checkboxes next to both AWSLambdaFullAccess and AmazonRekognitionFullAccess.

Give your role a name that you like (e.g. LambdaFullAcessWithRekognition). Review the permissions granted by the the policies and Click “Create role”.

4.2. Accessing Dynamo DB in AWS Lambda

In AWS Lambda, create a lambda function (e.g. DemoLambda) and use the created role (e.g. LambdaFullAcessWithRekognition).

In the function code section, input the following code. Click Save.

 

import json
import boto3
import urllib
def insert_item_db():
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('users')
    table.put_item(
        Item={
        'username': 'janedoe',
        'first_name': 'Jane',
        'last_name': 'Doe',
        'age': 25,
        'hobbies':['badminton', 'foodball','singing'],
        'account_type': 'standard_user'
        }
    )

def scan_table(): #scan the entire table
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('users')
    result = table.scan()
    for i in result['Items']:
        print(i)
 
def get_db_item(): #retrieve an item using primary key
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('users')
    response = table.get_item(
        Key={
        'username': 'janedoe'
        }
    )
    item = response['Item']
    print(item)

def lambda_handler(event, context):
    insert_item_db()
    #scan_table()
    #get_db_item()

Click Test to test the lambda function.

Check that the record is created in Dynamo DB

#scan_table()

#get_db_item()

 

4.3. S3 integration in AWS Lambda

Modify your lambda code as follows. Add your S3 bucket name to the code. Save the function

import json
import boto3
import urllib
import pprint
pp = pprint.PrettyPrinter(indent=4)
s3 = boto3.client('s3')
def list_objects(bucket_name): #list objects in bucket
    response = s3.list_objects(Bucket=bucket_name)
    pp.pprint (str(response['Contents']))
def lambda_handler(event, context):
    bucket_name = '<your bucket name>'
    list_objects(bucket_name) #list objects in bucket

Click Test to test the lambda function. Check the execution results

The objects in the bucket should be printed in the log.

4.4. Handling S3 events in AWS Lambda

The Amazon S3 notification feature enables you to receive notifications when certain events happen in your bucket. Visit the following page to know more about the S3 event types 

https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html#supported-notification-event-types

Sample S3 Event Structure:

{
    "Records": [
    {
        "eventVersion": "2.1",
        "eventSource": "aws:s3",
        "awsRegion": "us-east-2",
        "eventTime": "2019-09-03T19:37:27.192Z",
        "eventName": "ObjectCreated:Put",
        "userIdentity": {
            "principalId": "AWS:AIDAINPONIXQXHT3IKHL2"
        },
        "requestParameters": {
            "sourceIPAddress": "205.255.255.255"
        },
        "responseElements": {
            "x-amz-request-id": "D82B88E5F771F645",
            "x-amz-id-2": "vlR7PnpV2Ce81l0PRw6jlUpck7Jo5ZsQjryTjKlc5aLWGVHPZLj5NeC6qMa0emYBDXOo6QBU0Wo="
        },
        "s3": {
            "s3SchemaVersion": "1.0",
            "configurationId": "828aa6fc-f7b5-4305-8584-487c791949c1",
            "bucket": {
                "name": "lambda-artifacts-deafc19498e3f2df",
                "ownerIdentity": {
                    "principalId": "A3I5XTEXAMAI3E"
                },
                "arn": "arn:aws:s3:::lambda-artifacts-deafc19498e3f2df"
            },
            "object": {
                "key": "b21b84d653bb07b05b1e6b33684dc11b",
                "size": 1305107,
                "eTag": "b21b84d653bb07b05b1e6b33684dc11b",
                "sequencer": "0C0F6F405D6ED209E1"
            }
        }
    }
    ]
}

In DynamoDB, create an s3events table with partition key “Event Time”.

In AWS Lambda, add a trigger from S3 and select the bucket that you have created

For Event type, choose “All object create events”. Click Add.