The fat jar approach works fine but if you have multiple functions that use the same dependencies then creating a fat jar of each function is not a good approach and maintenance of such code is also difficult. A change in the version of a jar requires a change in each of the individual lambda functions.

A more elegant solution is to use Lambda layers.

What is Lambda Layers?

A layer is a ZIP archive that contains libraries, a custom runtime, or other dependencies. With layers, you can use libraries in your function without needing to include them in your deployment package. Layers let you keep your deployment package small, which makes development easier.

Layers are extracted to the /opt directory in the function execution environment. Each runtime looks for libraries in a different location under /opt, depending on the language.

You can specify up to 5 layers in your function's configuration, during or after function creation. You choose a specific version of a layer to use. If you want to use a different version later, update your function's configuration.

Let's see it in action

We will write a basic Lambda function that would just print lambda environment variables and event details. We will create 2 maven projects. In the first project, we include amazon and other essential dependencies. This project will serve as our Lambda Layer. In the second project, we include our first project as a dependency and will write the Lambda handler.

Create the first maven project called Lambda-Layer-Base and update pom as below: This project contains all common dependencies required by the functions. In our case, we added lamdba-core and gson dependency. We also added maven-shade-plugin to create fat jar.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-layer-base</artifactId>
    <version>1</version>
    <name>Lambda-Layer-Base</name>
    <description>Lambda base</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

This project will only contain dependencies. Now execute maven clean install and it should generate a jar (lambda-layer-base-1.jar) in the target folder.

Create another maven project and update pom with the below contents. We included our first project as a dependency (lambda-layer-base). Also note, we no longer use maven shade plugin since we don't want dependencies to be included in the jar.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-java-demo</artifactId>
    <version>1</version>
    <packaging>jar</packaging>
    <name>Lambda-Layers-Demo</name>
    <description>HelloWorld Lambda Layers Demo</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencies>
    <dependency>
        <groupId>net.rajanpanchal</groupId>
        <artifactId>lambda-layer-base</artifactId>
        <version>1</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Add a class file in src/main/java that will contain Lambda logic. Here we are just logging some stuff.

package net.rajanpanchal.handlers;

import java.util.Map;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class HelloWorldHandler implements RequestHandler<Map<String,String>, String>{

      public String handleRequest(Map<String,String> event, Context context)
      {
        LambdaLogger logger = context.getLogger();
        String response = new String("200 OK");
        logger.log("ENVIRONMENT:"+System.getenv());
        // log execution details
        Gson gson = new GsonBuilder().setPrettyPrinting().create();
        logger.log("ENVIRONMENT VARIABLES: " + gson.toJson(System.getenv()));
        logger.log("CONTEXT: " + gson.toJson(context));
        // process event
        logger.log("EVENT: " + gson.toJson(event));
        logger.log("EVENT TYPE: " + event.getClass().toString());
        return response;
      }
    }

Execute maven package command to create the jar for the project. A jar (lambda-java-demo-1.jar) should get created in target folder.

Creating Lamdba Layer

Before we head to the console to create the layer, let's complete one important step. Remember that layers gets extracted to a folder in /opt directory. You need to package the layer in folder structure as per your runtime. For java, you need to package jar in java/lib folder. The final folder structure, in the lambda runtime, would be /opt/java/lib/<your jar>. So now create folders java/lib and copy jar from the first project and zip the folder. Now we are ready to upload the jar as lambda layer.

Creating Lambda function

Now create the lambda function and specify the layer that we just created. Also, upload the jar from the second project as our function code.

Add Layers to the function

Upload function code jar and update runtime settings

Create a test event and execute the test.

Source Code:

Conclusion

Lambda layer makes your development easy and error-free. It reduces the size of the function deployment package. You can use the same layer in multiple functions.

If you like the post, feel free to share and follow me on Twitter for updates!

Uses Cases

There are several scenarios where you want to create dynamic content. For example, to create an HTML page response for an API method instead of returning just data in JSON/ XML. The other scenario is to generate a file or message with dynamic content. For instance, create welcome letters for your customers.

Let's Dive In

We are going to create welcome letters for our credit card customers. The letter's template is :

Hello ${name},

Say hi to your new credit card!
Your credit card number is ${creditCardNumber}.

Regards,
Your Credit Company

Lambda would populate name and credit card number. The generated file is uploaded to S3 for further processing like printing and mailing.

STEP 1:

Create an empty maven project and update pom to the following contents:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-java</artifactId>
    <version>1</version>
    <name>Dynamic-content-Lambda</name>
    <description>Generate Dynamic content in Lambda</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-s3 -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-s3</artifactId>
            <version>1.11.865</version>
        </dependency>

        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.velocity</groupId>
            <artifactId>velocity-engine-parent</artifactId>
            <version>2.2</version>
            <type>pom</type>
        </dependency>
        <dependency>
            <groupId>org.apache.velocity</groupId>
            <artifactId>velocity-tools</artifactId>
            <version>2.0</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

We are adding AWS dependencies and apache velocity dependencies. The maven-shade-plugin to create a fat jar.

STEP 2:

We will have our template, discussed earlier, in the resources folder (src/main/resources) with the name welcomeLetter.vm.

STEP 3:

Create a class file with the following content:

package net.rajanpanchal.handlers;

import java.io.File;
import java.io.FileWriter;
import java.util.Map;
import java.util.Properties;

import org.apache.velocity.Template;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.VelocityEngine;
import org.apache.velocity.exception.ParseErrorException;
import org.apache.velocity.exception.ResourceNotFoundException;
import org.apache.velocity.runtime.RuntimeConstants;

import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PutObjectRequest;

public class WelcomeLetterGeneratorHandler implements RequestHandler<Map<String, String>, String> {
    Regions clientRegion = Regions.US_EAST_1;
    String fileObjKeyName = "welcome_letter.txt";
    String bucketName = "welcomelettersbucket";
    static VelocityContext vContext;
    static Template t;
    AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withRegion(clientRegion).build();
    static {
        try {
            // Create a new Velocity Engine
            VelocityEngine velocityEngine = new VelocityEngine();
            // Set properties that allow reading vm file from classpath.
            Properties p = new Properties();
            velocityEngine.setProperty(RuntimeConstants.RESOURCE_LOADER, "file,class");
            velocityEngine.setProperty("class.resource.loader.class",
                    "org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader");
            velocityEngine.init(p);
            t = velocityEngine.getTemplate("welcomeLetter.vm");
            vContext = new VelocityContext();

        } catch (Exception e) {
            throw new RuntimeException(e);

        }
    }

    public String handleRequest(Map<String, String> event, Context context) {
        String response = null;
        LambdaLogger logger = context.getLogger();
        try {
            // Add data to velocity context
            vContext.put("name", "Rajan");
            vContext.put("creditCardNumber", "1234 5678 9012 3456");
            File f = new File("/tmp/"+fileObjKeyName);

            FileWriter writer = new FileWriter(f);
            // merge template and Data
            t.merge(vContext, writer);
            writer.flush();
            writer.close();

            // Upload a file as a new object with ContentType and title specified.
            PutObjectRequest request = new PutObjectRequest(bucketName, fileObjKeyName, f);
            ObjectMetadata metadata = new ObjectMetadata();
            metadata.setContentType("plain/text");
            metadata.addUserMetadata("title", "Welcome Letter");
            request.setMetadata(metadata);
            s3Client.putObject(request);
            response  = new String("Success");
        } catch (Exception ex) {
            response =  new String("Error:"+ex.getMessage());
        }

        return response;
    }
}

In the static block we are creating a velocity engine, setting resource loaders for the engine ('class' and 'file' resource loaders). Then we get the template and create a context to hold data to populate on the template.

In Lambda handler handleRequest, we add data like name and credit card number to the context, and using a FileWriter object we merge the template and the data. Note that we create the file in /tmp/ directory in the Lambda execution environment. Lambda provides some disk space to the function to use. Then we upload this file to the S3 bucket (welcomelettersbucket) with the filename welcome_letter.txt. My bucket exists in US-EAST-1 as indicated by the variable clientRegion.

STEP 4

Now execute maven package command to generate the fat jar with all dependencies. A jar would be generated in the target folder.

STEP 5:

Go to AWS Console, create a Lambda function with runtime Java 8, let it create a default execution role. Upload the jar under Function Code section. Edit Basic Settings and add our handler package and method name

net.rajanpanchal.handlers.WelcomeLetterGeneratorHandler::handleRequest

STEP 6:

Add S3 access policy to the Lambda execution role. Click on Permissions tab - > Execution Role hyperlink. It will take you to the IAM Role. For simplicity, I added AmazonS3FullAccess permission, but you can add fine-grain S3 access.

STEP 7:

Go back to lambda and create Test event. Lambda executed successfully.

Check S3 bucket, the file is saved.

The file is ready for further processing!

Source Code:

Conclusion

It's super easy to create dynamic content in Lambda using Apache Velocity and Java SDK. In addition, using lambda you only pay for what you use. Let me know your thoughts on the article in the comments and how you are going to use this?

If you like the post, feel free to share and follow me on Twitter for updates!

What is IKEA Effect

The IKEA effect is a cognitive bias in which consumers place a disproportionately high value on products they partially created. A 2011 study found that subjects were willing to pay more for furniture they had assembled themselves, than for equivalent pre-assembled items.

Now you might be wondering why we are discussing it on a tech blog? Read on and you would understand how it impacts you as an IT Professional.

When Labor Leads to Love

Normally, when you buy a product from IKEA it requires some level of assembly before the product can be used. The study found that when someone puts labor into something, they start liking it more, even if it is poorly constructed. Hence, the phenomenon,“IKEA effect”, is named in honor of the Swedish manufacturer whose products typically arrive with some assembly required. The study also found that the "IKEA Effect" dissipates when the creations are destroyed. The creation has to stay intact for IKEA effect to be effective.

How you can use IKEA Effect to your advantage?

You might have experienced the IKEA effect even if you didn't know the term before. To take advantage of this, you need to build things. No matter how small it is. If you are learning a new language, a library or a technology, think of a small business problem and try to solve it using the knowledge that you have. For example, I am upskilling myself in AWS. So first I learn the concepts and then I think of a problem that can be solved using different AWS services. Once you solve it you will value your solution more than anything else even if it's not of production grade or perfect. This will keep you motivate to learn more and create more. You can also contribute to open source projects. Open source projects are more popular because IKEA effect is in place. Your name gets connected to projects you make contribution to. This gives a sense of accomplishment and motivates you to contribute more.

Also, don't throw away your solution, instead preserve it using your favorite version control like GitHub, GitLab, etc. Or you can also write blog about it.

Let me know your thoughts in comments !

Now, this post will be incomplete without these memes!

If you like my articles feel free to follow me on Twitter for updates!

What is Serverless Application Model?

What do you do when you want to create a resource on AWS, for example, Lambda function? You log in to the AWS console and manually go to the Lambda service to create a function. Now say you want to create 10 such functions you have to do the same process 10 times. This can be error-prone and time-consuming. With SAM you can just write once and deploy as many times you want flawlessly. It is similar to Cloud Formation. You can say SAM is the superset of Cloudformation.

The AWS Serverless Application Model (SAM) is an open-source framework for building serverless applications. It provides shorthand syntax to express functions, APIs, databases, and event source mappings. With just a few lines of code, you can define the application you want and model it using YAML. During deployment, SAM transforms and expands the SAM syntax into AWS CloudFormation syntax, enabling you to build serverless applications faster.

With the use of Environment variables and parameters, you can dynamically name all your resources and create different environments like DEV, TEST, PROD with the same set of resources.

Getting started with SAM

To use SAM you will need SAM-CLI installed. SAM CLI provides a Lambda-like execution environment that lets you locally build, test, and debug applications defined by SAM templates. You can also use the SAM CLI to deploy your applications to AWS. You can follow the guide here to install SAM CLI. Before we proceed make sure your user has all the permissions to create resources and configured on the machine. To configure AWS account, install AWS CLI and run aws configure command to set up credentials. I am using the admin user that has all the permissions.

Now we are ready to download sample application and examine its contents. Run the following command on terminal

sam init

It will ask you to make several choices. Note: you need to have your runtime already installed.

ubuntu@ubuntu-VirtualBox:~/MyWorkspace/BlogPosts/SAM$ sam init
Which template source would you like to use?
    1 - AWS Quick Start Templates
    2 - Custom Template Location
Choice: 1

Which runtime would you like to use?
    1 - nodejs12.x
    2 - python3.8
    3 - ruby2.7
    4 - go1.x
    5 - java11
    6 - dotnetcore3.1
    7 - nodejs10.x
    8 - python3.7
    9 - python3.6
    10 - python2.7
    11 - ruby2.5
    12 - java8.al2
    13 - java8
    14 - dotnetcore2.1
Runtime: 2

Project name [sam-app]: SAM application

Cloning app templates from https://github.com/awslabs/aws-sam-cli-app-templates.git

AWS quick start application templates:
    1 - Hello World Example
    2 - EventBridge Hello World
    3 - EventBridge App from scratch (100+ Event Schemas)
    4 - Step Functions Sample App (Stock Trader)
    5 - Elastic File System Sample App
Template selection: 1

-----------------------
Generating application:
-----------------------
Name: SAM application
Runtime: python3.8
Dependency Manager: pip
Application Template: hello-world
Output Directory: .

Next steps can be found in the README file at ./SAM application/README.md

Go to the file system and it would have created following file structure:

SAM application/
   ├── README.md
   ├── events/
   │   └── event.json
   ├── hello_world/
   │   ├── __init__.py
   │   ├── app.py            #Contains your AWS Lambda handler logic.
   │   └── requirements.txt  #Contains any Python dependencies the application requires, used for sam build
   ├── template.yaml         #Contains the AWS SAM template defining your application's AWS resources.
   └── tests/
       └── unit/
           ├── __init__.py
           └── test_handler.py

Open template.yml and let's breakdown the contents:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  SAM application

  Sample SAM Template for SAM application

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello_world/
      Handler: app.lambda_handler
      Runtime: python3.8
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hello
            Method: get

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HelloWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
  HelloWorldFunction:
    Description: "Hello World Lambda Function ARN"
    Value: !GetAtt HelloWorldFunction.Arn
  HelloWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hello World function"
    Value: !GetAtt HelloWorldFunctionRole.Arn

AWSTemplateFormatVersion - The first line always stays the same for every template.yml.

Transform - It specifies one or more macros that AWS CloudFormation uses to process your template. This is a required section.

Description is self-explanatory.

Globals - It is unique to the SAM and defines properties shared by the whole template. In this case, the timeout will apply to all functions. Globals don't exist in cloud formation.

Resources - You define all AWS resources under this. This is also a mandatory section. Here we have defined a function with the logical name HelloWorldFunction, it is of type AWS::Serverless::Function with a runtime of python 3.8, CodeUri points to the folder where our handler resides on the filesystem. Handler is the filename and the method. Events would create an API in the API Gateway with method type GET and path /hello.

Outputs- Describes the values that are returned and available for use after the stack is created. For example, in our template, HelloWorldApi output gives the API Url generated by the function. We are also setting Lambda and IAM Role ARN as output. Also note !Sub & GetAtt functions in output, !Sub is used to substitute a variable with a value. Here it will substitute ${AWS::Region} and ${ServerlessRestApi}. GetAtt function returns the value of an attribute from a resource in the template. Here we are asking for ARN of the HelloWorldfunction.

Now examine the app.py file. It is straightforward and returns "hello world" as a response.

import json

# import requests


def lambda_handler(event, context):
    """Sample pure Lambda function

    Parameters
    ----------
    event: dict, required
        API Gateway Lambda Proxy Input Format

        Event doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-input-format

    context: object, required
        Lambda Context runtime methods and attributes

        Context doc: https://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html

    Returns
    ------
    API Gateway Lambda Proxy Output Format: dict

        Return doc: https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html
    """

    # try:
    #     ip = requests.get("http://checkip.amazonaws.com/")
    # except requests.RequestException as e:
    #     # Send some context about this error to Lambda Logs
    #     print(e)

    #     raise e

    return {
        "statusCode": 200,
        "body": json.dumps({
            "message": "hello world",
            # "location": ip.text.replace("\n", "")
        }),
    }

Get back to the terminal and cd into the directory where the template.yml resides, run the command: sam build

ubuntu@ubuntu-VirtualBox:~/MyWorkspace/BlogPosts/SAM/SAM application$ sam build
Building function 'HelloWorldFunction'
Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

The sam build command builds any dependencies that your application has, and copies your application source code to folders under .aws-sam/build to be zipped and uploaded to Lambda. Check the contents of .aws-sam folder in project directory. It would be something similar to shown below. In HelloWorldFunction folder, there would be many other files along with app.py

.aws_sam/
   └── build/
       ├── HelloWorldFunction/
       └── template.yaml

Now its time to run sam deploy -g. It will ask you couple of configuration information like application name, region name where you want to create the stack, etc, and will wait for your confirmation if you asked for confirmation before deploy.

buntu@ubuntu-VirtualBox:~/MyWorkspace/BlogPosts/SAM/SAM application$ sam deploy -g

Configuring SAM deploy
======================

    Looking for samconfig.toml :  Not found

    Setting default arguments for 'sam deploy'
    =========================================
    Stack Name [sam-app]: my-first-sam-app
    AWS Region [us-east-1]: us-east-1
    #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
    Confirm changes before deploy [y/N]: y
    #SAM needs permission to be able to create roles to connect to the resources in your template
    Allow SAM CLI IAM role creation [Y/n]: y
    HelloWorldFunction may not have authorization defined, Is this okay? [y/N]: y
    Save arguments to samconfig.toml [Y/n]: y

    Looking for resources needed for deployment: Found!

        Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1xyg1t2j2ws5k
        A different default S3 bucket can be set in samconfig.toml

    Saved arguments to config file
    Running 'sam deploy' for future deployments will use the parameters saved above.
    The above parameters can be changed by modifying samconfig.toml
    Learn more about samconfig.toml syntax at 
    https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-config.html
Uploading to my-first-sam-app/9601808ead19b558184dcec8285866a3  538937 / 538937.0  (100.00%)

    Deploying with following values
    ===============================
    Stack name                 : my-first-sam-app
    Region                     : us-east-1
    Confirm changeset          : True
    Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-1xyg1t2j2ws5k
    Capabilities               : ["CAPABILITY_IAM"]
    Parameter overrides        : {}

Initiating deployment
=====================
HelloWorldFunction may not have authorization defined.
Uploading to my-first-sam-app/362accae02d25f5921348967d73b9d29.template  1115 / 1115.0  (100.00%)

Waiting for changeset to be created..
CloudFormation stack changeset
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Operation                                                          LogicalResourceId                                                  ResourceType                                                     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+ Add                                                              HelloWorldFunctionHelloWorldPermissionProd                         AWS::Lambda::Permission                                          
+ Add                                                              HelloWorldFunctionRole                                             AWS::IAM::Role                                                   
+ Add                                                              HelloWorldFunction                                                 AWS::Lambda::Function                                            
+ Add                                                              ServerlessRestApiDeployment47fc2d5f9d                              AWS::ApiGateway::Deployment                                      
+ Add                                                              ServerlessRestApiProdStage                                         AWS::ApiGateway::Stage                                           
+ Add                                                              ServerlessRestApi                                                  AWS::ApiGateway::RestApi                                         
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:<account #>:changeSet/samcli-deploy1599948737/03d65ab9-a943-494d-8db6-abf6aad17537


Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]:

Once you confirm, it will start creating stack. Below is the output after the stack creation completes:

2020-09-12 17:13:45 - Waiting for stack create/update to complete

CloudFormation events from changeset
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
ResourceStatus                                    ResourceType                                      LogicalResourceId                                 ResourceStatusReason                            
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS                                AWS::IAM::Role                                    HelloWorldFunctionRole                            -                                               
CREATE_IN_PROGRESS                                AWS::IAM::Role                                    HelloWorldFunctionRole                            Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::IAM::Role                                    HelloWorldFunctionRole                            -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Function                             HelloWorldFunction                                -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Function                             HelloWorldFunction                                Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::Lambda::Function                             HelloWorldFunction                                -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::RestApi                          ServerlessRestApi                                 -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::RestApi                          ServerlessRestApi                                 Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::ApiGateway::RestApi                          ServerlessRestApi                                 -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Permission                           HelloWorldFunctionHelloWorldPermissionProd        Resource creation Initiated                     
CREATE_IN_PROGRESS                                AWS::ApiGateway::Deployment                       ServerlessRestApiDeployment47fc2d5f9d             -                                               
CREATE_IN_PROGRESS                                AWS::Lambda::Permission                           HelloWorldFunctionHelloWorldPermissionProd        -                                               
CREATE_COMPLETE                                   AWS::ApiGateway::Deployment                       ServerlessRestApiDeployment47fc2d5f9d             -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::Deployment                       ServerlessRestApiDeployment47fc2d5f9d             Resource creation Initiated                     
CREATE_IN_PROGRESS                                AWS::ApiGateway::Stage                            ServerlessRestApiProdStage                        -                                               
CREATE_IN_PROGRESS                                AWS::ApiGateway::Stage                            ServerlessRestApiProdStage                        Resource creation Initiated                     
CREATE_COMPLETE                                   AWS::ApiGateway::Stage                            ServerlessRestApiProdStage                        -                                               
CREATE_COMPLETE                                   AWS::Lambda::Permission                           HelloWorldFunctionHelloWorldPermissionProd        -                                               
CREATE_COMPLETE                                   AWS::CloudFormation::Stack                        my-first-sam-app                                  -                                               
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

CloudFormation outputs from deployed stack
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Outputs                                                                                                                                                                                                
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key                 HelloWorldFunctionIamRole                                                                                                                                                          
Description         Implicit IAM Role created for Hello World function                                                                                                                                 
Value               arn:aws:iam::<account #>:role/my-first-sam-app-HelloWorldFunctionRole-27OIM6WD99F0                                                                                                

Key                 HelloWorldApi                                                                                                                                                                      
Description         API Gateway endpoint URL for Prod stage for Hello World function                                                                                                                   
Value               https://3kuuurt63m.execute-api.us-east-1.amazonaws.com/Prod/hello/                                                                                                                 

Key                 HelloWorldFunction                                                                                                                                                                 
Description         Hello World Lambda Function ARN                                                                                                                                                    
Value               arn:aws:lambda:us-east-1:<account #>:function:my-first-sam-app-HelloWorldFunction-1U5YD9NICU5LP                                                                                   
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Successfully created/updated stack - my-first-sam-app in us-east-1

As you can see in the outputs, the HelloWorldApi URL was generated which you can hit in the browser and verify your API is working fine. You can also see the same information in Cloud Formation's output section on AWS console.

Great Job! You have successfully completed your First SAM App!

What else you can do in SAM?

This is not the end, you can also test the function locally before deploy. The AWS SAM CLI provides the sam local command to run your application using Docker containers that simulate the execution environment of Lambda. I am on linux machine, hence I am going to run command : sudo /home/linuxbrew/.linuxbrew/bin/sam local start-api

SAM CLI now collects telemetry to better understand customer needs.

    You can OPT OUT and disable telemetry collection by setting the
    environment variable SAM_CLI_TELEMETRY=0 in your shell.
    Thanks for your help!

    Learn More: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-telemetry.html

Mounting HelloWorldFunction at http://127.0.0.1:3000/hello [GET]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-09-12 22:31:06  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)

When you hit the URL(http://127.0.0.1:3000/hello) in the browser, you should see the same response as before "hello world" and on the terminal, it should show something similar to this

Invoking app.lambda_handler (python3.8)
Failed to download a new amazon/aws-sam-cli-emulation-image-python3.8:rapid-1.1.0 image. Invoking with the already downloaded image.
Mounting /home/ubuntu/MyWorkspace/BlogPosts/SAM/SAM application/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: f4945824-0a1a-1742-2f46-4d0bc2bbe515 Version: $LATEST
END RequestId: f4945824-0a1a-1742-2f46-4d0bc2bbe515
REPORT RequestId: f4945824-0a1a-1742-2f46-4d0bc2bbe515    Init Duration: 216.83 ms    Duration: 2.88 ms    Billed Duration: 100 ms    Memory Size: 128 MB    Max Memory Used: 24 MB    
No Content-Type given. Defaulting to 'application/json'.
2020-09-12 22:38:28 127.0.0.1 - - [12/Sep/2020 22:38:28] "GET /hello HTTP/1.1" 200 -

Where to go from here?

This was a very basic SAM application but I hope you got the gist of it. As an exercise, you can add more resources like CloudFront, Route53, DynamoDB, ACM etc, and see how to model it in YAML. Take a look here and here for more complex SAM Templates.

If you like my articles feel free to follow me on Twitter for updates!

What is Lambda Proxy Integration

When a client submits an API request, API Gateway passes the request to the integrated Lambda function as-is, except that the order of the request parameters is not preserved. This request data includes the request headers, query string parameters, URL path variables, payload, and API configuration data.

The configuration data can include current deployment stage name, stage variables, user identity, or authorization context (if any). The backend Lambda function parses the incoming request data. The API gateway simply acts as a postman to pass requests and responses. One advantage is that you can change the implementation of the Lambda without impacting the API.

Let's implement using AWS Java SDK

Create an empty maven project and update the pom.xml with the following contents.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-api-demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>LambdaApiDemo</name>
    <description>Demonstrates API Request &amp; Response Using Lambda and Java SDK for AWS</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/com.amazonaws/aws-lambda-java-events -->
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-events</artifactId>
            <version>3.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

We are adding 2 main dependencies here:

1. aws-lambda-java-core - This provides RequestHandler
2. aws-lambda-java-events - This provides APIGatewayProxyRequestEvent and APIGatewayProxyResponseEvent

Added 2 plugins:

1. Compiler plugin - determines java version for the compiler.
2. maven-shade-plugin - to generate fat jar aka Uber jar (with all dependencies)

Create a class LambdaHandler, with the following contents.

package net.rajanpanchal.handlers;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyRequestEvent;
import com.amazonaws.services.lambda.runtime.events.APIGatewayProxyResponseEvent;

public class LambdaHandler implements RequestHandler<APIGatewayProxyRequestEvent, APIGatewayProxyResponseEvent> {
    @Override
    public APIGatewayProxyResponseEvent handleRequest(APIGatewayProxyRequestEvent requestEvent, Context context) {
        APIGatewayProxyResponseEvent responseEvent = new APIGatewayProxyResponseEvent();
        try {
            LambdaLogger logger = context.getLogger();
            logger.log("requestEventString:" + requestEvent.toString());
            logger.log("query string param:" + requestEvent.getQueryStringParameters());
            logger.log("headers:" + requestEvent.getHeaders());
            // fetching the value send in the request body
            String message = requestEvent.getBody();
            // Request request = gson.fromJson(message, Request.class);
            logger.log("message body:" + message);

            // setting up the response message
            responseEvent.setBody("Hello from Lambda!");
            responseEvent.setStatusCode(200);

            return responseEvent;

        } catch (Exception ex) {
            responseEvent.setBody("Invalid Response");
            responseEvent.setStatusCode(500);
            return responseEvent;
        }
    }
}

The above code is simple. requestEvent object is of type APIGatewayProxyRequestEvent. You will get headers, body, query-string, etc. from this object. responseEvent object is of type APIGatewayProxyResponseEvent. You can set response headers, body, status, etc in this object.

Now execute maven command mvn package to generate a jar file in the target folder. Login to AWS console, create Lambda function with Java 8 runtime, let it create execution role, and upload the jar file under Function Code section. In Basic Settings, set the handler name in format :

*net.rajanpanchal.handlers.LambdaHandler::handleRequest*

Now go to API Gateway, create a REST API Go to actions and create a new method of type GET. Select the following configuration: To enable proxy integration, make sure to check the checkbox. Save to create a method.

Go to actions and click deploy API and give the following information and click deploy. Use the URL generated to access the API

Testing

Hit the URL in the browser and you should see the below response:

Go to Cloud Watch, access Log Groups from the left-hand navigation, and access the logs corresponding to your lambda function. You should see the log statements from the lambda handler.

Source Code:

Here we complete our REST API with Lambda!

If you like the post, feel free to share and follow me on Twitter for updates!

If you like the post, feel free to share and follow me on Twitter for updates!

Lets look at some of the Lambda Performance Optimizations

Execution context Reuse

When a Lambda function is invoked, AWS Lambda launches an execution context based on the configuration settings you provide like Memory, runtime, etc. The execution context is a temporary runtime environment that initializes any external dependencies of your Lambda function code, such as database connections or HTTP endpoints, etc. After the Lambda function completes the execution it maintains the execution context in hopes that another invocation of same function will happen. If it does happen, then it will reuse the context. So the objects declared outside of the lambda handler (Lambda global variables) can be reused. You can declare any objects there. Following code snippets defines S3 as client that will be reused in subsequent invocation.

import json
import boto3
#DynamoDB defined outside of handler
dynamodb = boto3.resource('dynamodb')
def lambda_handler(event, context):
    table = dynamodb.Table("MyTable")
    response = table.get_item(Key={'userid': username})

    return {
        'statusCode': 200,
        'body': json.dumps(response['Item) 
      }

A word of caution: Its is not guaranteed that subsequent function invocation would go to the same execution context. Lambda may decide to create a new invocation context runtime so don't maintain any state information. Always have logic in Lambda handler to instantiate or reconnect the connections objects if cannot be reused.

Minimize Deployment Package

A deployment package is a ZIP archive that contains your compiled function code and dependencies. In case your Lambda handler is in Java, you need to have a fat-jar aka Uber Jar to upload to Lambda. Uber jar will contain all the dependencies required to execute the handler. Its is important that you only import those packages that are required. For example, if you only need basic Lambda implementation, then import only lambda-java-core dependency - It defines handler method interfaces and the context object that the runtime passes to the handler. If you define your own input types, this is the only library you need. You don't need whole aws-sdk dependency to write simple Lambda handlers. Use Bill of Materials POM (https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bom) in the maven project and provides individual dependencies from BOM POM to import. This will reduce the fat jar size. Keep in mind that in lambda, you are working off a limited disk size and memory as defined in the configuration. In Python and other languages you can only import modules that you need. For example, in TypeScript

// import entire SDK
import AWS from 'aws-sdk';
// import AWS object without services
import AWS from 'aws-sdk/global';
// import individual service
import S3 from 'aws-sdk/clients/s3';

Use Simple Frameworks

Instead of using full blown spring framework try to use lightweight frameworks that accomplish the same tasks. For example: If you need Java dependency injection in your app then use lightweight frameworks like Guice (pronounced 'juice' is a lightweight dependency injection framework for Java 6 and above, brought to you by Google)

Are there any other Lambda optimization techniques? Let us know in comments!

If you like my articles feel free to follow me on Twitter for updates!

To implement handler, we are going to use Maven, Eclipse and AWS SDK for Java. Below are the version:

• Java: 1.8
• Eclipse: Version: 2020-06 (4.16.0)
• AWS SDK: 2.14.11
• Maven: 3.6.3 (Included with eclipse)

To get started, create a new empty maven project in eclipse. Go to File-> New-> Maven Project. In new project dialog box. select "create simple project" and hit next. On next screen. add appropriate information about the project and hit finish. For example:

Now your project is created. Open pom.xml and add following code.

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.rajanpanchal</groupId>
    <artifactId>lambda-java-demo</artifactId>
    <version>1</version>
    <name>Lambda-Java-Demo</name>
    <description>HelloWorld Lambda handler in java</description>
    <properties>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
        <aws.java.sdk.version>2.14.11</aws.java.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>${aws.java.sdk.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.2.0</version>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.8.6</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>${maven.compiler.source}</source>
                    <target>${maven.compiler.target}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.2.4</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

We are setting some properties in <properties> tag like AWS SDK version and Java Version. We are adding AWS SDK dependency and Lambda core dependency with desired version. Google JSON dependency is added to convert between JSON and Java object and vice versa. In plugins, we define maven compiler plugin to compile the code and another important plugin called maven-shade-plugin. This plugin helps to create fat jar a.k.a Uber jar. This jar will contain all the dependencies that are required to successfully run the lambda function.

Now lets create the handler.

package net.rajanpanchal.handlers;

import java.util.Map;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.LambdaLogger;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

public class HelloWorldHandler implements RequestHandler<Map<String,String>, String>{
      Gson gson = new GsonBuilder().setPrettyPrinting().create();
      @Override
      public String handleRequest(Map<String,String> event, Context context)
      {
        LambdaLogger logger = context.getLogger();
        String response = new String("200 OK");
        // log execution details
        logger.log("ENVIRONMENT VARIABLES: " + gson.toJson(System.getenv()));
        logger.log("CONTEXT: " + gson.toJson(context));
        // process event
        logger.log("EVENT: " + gson.toJson(event));
        logger.log("EVENT TYPE: " + event.getClass().toString());
        return response;
      }
    }

Here were are implementing RequestHandler that will accept a Map of String and outputs a String response. The handler is not doing much. Just logging some stuff from context and event and in response it sends 200 OK string. After this right click on project, go to 'Run as' and click 'Maven build'. A configuration window will open. In 'Goals' type package and hit 'Run' . Build should be successful and will create a fat jar in target folder.

Now go to AWS console and create a new Lambda function with Java 8 as runtime and upload this jar from Function Code section. In Basic Settings, you have to specify package.class::methodName in the handler textbox. To test this Lambda, create a test event and execute the test.

The ouput might look similar to this:

START RequestId: c40b4095-27f5-4153-ac37-fd2c103f4476 Version: $LATEST
ENVIRONMENT VARIABLES: {
  "PATH": "/usr/local/bin:/usr/bin/:/bin:/opt/bin",
  "_AWS_XRAY_DAEMON_ADDRESS": "169.254.79.2",
  "LAMBDA_TASK_ROOT": "/var/task",
  "AWS_LAMBDA_FUNCTION_MEMORY_SIZE": "128",
  "TZ": ":UTC",
  "AWS_SECRET_ACCESS_KEY": "<keyid>",
  "AWS_EXECUTION_ENV": "AWS_Lambda_java8",
  "AWS_DEFAULT_REGION": "us-east-1",
  "AWS_LAMBDA_LOG_GROUP_NAME": "/aws/lambda/helloworldjava",
  "_HANDLER": "net.rajanpanchal.handlers.HelloWorldHandler::handleRequest",
  "LANG": "en_US.UTF-8",
  "LAMBDA_RUNTIME_DIR": "/var/runtime",
  "AWS_SESSION_TOKEN": "<session token>",
  "LD_LIBRARY_PATH": "/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib",
  "_X_AMZN_TRACE_ID": "Root\u003d1-5f545c5d-22c8a803badd636859a0f387;Parent\u003d23df427f78bc46e4;Sampled\u003d0",
  "AWS_SECRET_KEY": "<secret key>",
  "AWS_REGION": "us-east-1",
  "AWS_LAMBDA_LOG_STREAM_NAME": "2020/09/06/[$LATEST]57c598b33b164acf8e8151660249e50e",
  "AWS_XRAY_DAEMON_ADDRESS": "169.254.79.2:2000",
  "_AWS_XRAY_DAEMON_PORT": "2000",
  "AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
  "AWS_LAMBDA_FUNCTION_VERSION": "$LATEST",
  "AWS_ACCESS_KEY": "<access key>",
  "AWS_LAMBDA_FUNCTION_NAME": "helloworldjava"
}CONTEXT: {
  "memoryLimit": 128,
  "awsRequestId": "c40b4095-27f5-4153-ac37-fd2c103f4476",
  "logGroupName": "/aws/lambda/helloworldjava",
  "logStreamName": "2020/09/06/[$LATEST]57c598b33b164acf8e8151660249e50e",
  "functionName": "helloworldjava",
  "functionVersion": "$LATEST",
  "invokedFunctionArn": "arn:aws:lambda:us-east-1:<AccountId>:function:helloworldjava",
  "cognitoIdentity": {
    "identityId": "",
    "poolId": ""
  },
  "logger": {}
}EVENT: {
  "key1": "value1",
  "key2": "value2",
  "key3": "value3"
}EVENT TYPE: class java.util.LinkedHashMapEND RequestId: c40b4095-27f5-4153-ac37-fd2c103f4476
REPORT RequestId: c40b4095-27f5-4153-ac37-fd2c103f4476    Duration: 516.04 ms    Billed Duration: 600 ms    Memory Size: 128 MB    Max Memory Used: 79 MB    Init Duration: 393.54 ms

Here we complete our first Lambda handler in Java. In next post we will see how we can implement API using AWS SDK.

How to use GitHub Actions?

Creating a GitHub action is simple. Go to your GitHub repository that you want to automate and click on "Actions"

You will be taken to Actions page where you can create a new Blank workflow or select existing actions from the marketplace. The actions from marketplace are reusable actions that you can use in your workflow. We are going to create a blank action and we will also use some actions from marketplace.

Lets rename the YAML file to workflow.yml. You can name anything you like. We are going to create a Lambda function with API gateway in Serverless Application Model (SAM) template and deploy it using GitHub Actions. Below is our SAM template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  GitHub Actions demonstration App
Resources:
  ApiGatewayApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
    Auth:
     UsagePlan:
      CreateUsagePlan: PER_API
      Description: Usage plan for this API
      Quota:
       Limit: 500
       Period: MONTH
      Throttle:
       BurstLimit: 100
       RateLimit: 50
  LamdbaFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./    
      Handler: lambda.handler
      Runtime: python3.8
      Events:
        getCounter:
          Type: Api
          Properties:
            Path: /hello
            Method: get
            RestApiId: !Ref ApiGatewayApi

lambda.py

def handler(event, context):
            return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body':'Hello from Lambda!'
        ,
        "isBase64Encoded": False
    }

This contains one Lambda function and API with path hello. Lets first deploy manually using SAM CLI and then we will automate it. Create samconfig.toml with below details. create s3_bucket that will be used for SAM deploy and update in samconfig.toml.

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "sam-github-actions-app"
s3_bucket = "aws-sam-cli-managed-default-samclisourcebucket-1xyg1t2j2ws5k"
s3_prefix = "sam-app"
region = "us-east-1"
confirm_changeset = false
capabilities = "CAPABILITY_IAM"

Also create empty requirements.txt along with template.yml. Run SAM build and SAM deploy -g on CLI.

Go to API gateway and hit the url in browser. You should get "hello from Lambda!" response.

Go back to our workflow file on GitHub. We will deploy as soon as we push updates to the repo. Below is our workflow.yml

# This is a basic workflow to help you get started with Actions
name: AWS Lambda & API gateway deployment demonstration
# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the master branch
on:
  push:
    branches: [ master ]
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest
    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2
    # Installs Python
    - name: Set up Python 3.8
      uses: actions/setup-python@v2
      with:
        python-version: 3.8
    # Installs PIP
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
    # Configures AWS credentials from github secrets
    - name: Configure AWS Credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1
    # Build using SAM 
    - name: SAM Build
      uses: youyo/aws-sam-action/python3.8@master
      with:
        sam_command: build
      env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION:  us-east-1
    # Deploy on AWS
    - name: sam deploy
      uses: youyo/aws-sam-action/python3.8@master
      with:
          sam_command: 'deploy --stack-name myApp --no-fail-on-empty-changeset'
      env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          AWS_DEFAULT_REGION: us-east-1

We first indicate that we want to run this action on push to the master. Then we select runner (Ubuntu) on which our steps will execute. In Steps, we first checkout code, then install python and its dependencies. Then we use other actions from marketplace to configure AWS credentials and then we use another action to do SAM build and SAM deploy. Please note that we need ti supply AWS ACCESS KEY and SECRET ACCESS KEY to actions for commands to work. We setup here GitHub Secrets.

As soon as you check-in workflow.yml, the action would trigger.

Deploying stack

Now you can login to the AWS console and confirm the stack is created.

Go to Resources tab and access the API Gateway. Go to stages and access the Prod stage API. Open the URL in browser with the path /hello and you should see below output!

Congratulations! You have successfully automated AWS deployment using GitHub Actions! You can download the code from here:

Here is my badage: AWS Badge

Developer associate is Developer focused and its bit different than the solution architect exam. I found Developer associate exam harder than the Solution Architect, but YMMV.

So here are my tips and preparation topics:

• You success in exam depends upon recognizing keywords in the questions. All answers may look similar but keywords will lead to right answer. Pay attention to words. For example: 'low cost', 'encryption at rest vs encryption in transit'.

• Know how to do Read Capacity Units (RCU) and Write Capacity Units (WCU) calculations. You might get one or two questions on determining RCU/WCU. Don't forget to pay attention to strongly consistent/ Eventual consistent keyword in question.

• Sometimes you may find answer to questions in other questions. While reviewing see if any other question has answer to it.

• Use all the time available for the exam. If finished early, spend time in reviewing all questions.

• Attempt as many mock exams you can. This will prepare you on time management and questions patterns. Questions will be verbose in real exam.

• There could be command level questions like what this command do, when to use etc.

• You need to understand the concepts very well and should have some hands on. It is easy to eliminate two answers.

• Know CI/CD services like CodeDeploy, CodeBuild, CodeCommit. Understand BuildSpec and AppSpec difference. Code Deploy lifecycle hooks. CodeDeploy Deployment Types for Lambda, ECS and EC2.

• Cloudformation, KMS, SQS, SNS, ECS, API gateway, IAM (Inline policies/ Managed Policies) and Lambda (Concurrent execution Limit,

• DynamoDB. Local Secondary Index vs Global. DynamoDB Accelerator. DynamoDB Streams. SCAN vs Query.

• Amazon S3, EC2, Elastic Beanstalk and its deployment types. Knows CORS.

• CloudWatch, CloudTrail, App X-Ray

• You might also get architecture level questions but not much.

Good Luck!

What are Presigned URLs?

A Presigned URL is a URL that provides limited permission and time to make a request. Anyone who receives the presigned URL can then access the object. For example, if you have a file in your bucket and both the bucket and the object are private, you can share the file with others by generating a presigned URL.

Some important points on Presigned URLs

• The creator of Presigned URL should have access to object for URL to work, otherwise URL wont work.
• You can create a presigned URL that's are not usable or doesn’t work.
• It doesn’t cost anything to create Presigned URLs.
• You can set expiration time on the URLs
• If you created a presigned URL using a temporary token, then the URL expires when the token expires, even if the URL was created with a later expiration time
• You can revoke the URL by removing the permissions to access the object from the user created the URL.

How to create Presigned URL?

When you create a presigned URL for your object, you must provide your security credentials, specify a bucket name, an object key, specify method and expiration time.

In Python, using Boto3, you can use generate_presigned_url method to generate the URL

response = s3_client.generate_presigned_url('get_object',
            Params={'Bucket': bucket_name,
            'Key': object_name},
            ExpiresIn=expiration)

The response object will contain the URL which will look similar to this

https://somebucketname-rep.s3.amazonaws.com/someFileName.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAQS3IOUWUZF7YSXGA%2F20200821%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20200821T051228Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Security-Token=FwoGZXIvYXdzEH8aDCfJDxO0y6xQxYmdGCK2AXe71W%2FgZEg%2FSnSWC%2Fw%2FaJHeZ20M7OI7AqMEum5c98Chl6pSNPwE5Awsc3ySwokDF6L8a9wP0ceXWAmxT3WXLSoFeNHDbbEHfUKWnvGL8yFzAxdmf%2Fmi%2B5Tnl62td8Nad%2F0Ct1Sx11Mip1h2qdYxw80OX5bCTq7cAHHjpmupvaDt%2BZ3qVyIA9WZmeS63dCPOlieE9IiBZf%2FjxF4Mcs5w4ZIHtZL%2F3LvqMXAy3XfzCgnlYVZeCNczKLuv%2FfkFMi0mStwkzyO%2BfMIxWJ82GJmyNi7LZuY5r0Hx0mE%2BxLnre8jp9%2FACoV%2FM92GnsR0%3D&X-Amz-Signature=17046b630ad4dede85af1cd57204bba8adc462a1825a35d93e81b656c683ad75

You can use this URL in the browser and access the object! Simple.. isn't it?

Lets see it in Action

We are going to implement this on top of previous two posts, in the first post, we implemented custom identity broker(signup/login), using AWS KMS to encrypt decryt password. In second post, we used AWS STS to AssumeRole to read bucket files names and now we will extend that to use presigned url. Download code for second post from github.com/rajanpanchal/aws-kms-sts and modify it. Here is how overall process looks like

Open file showFiles.py from the Lamdba folder and lets add function to generate presigned URL. Call this function from getFilesList function.

def getSignedUrl(key,s3_client):
    KeyUrl = {}

    response = s3_client.generate_presigned_url('get_object',
                        Params={'Bucket': os.environ['filesBucket'],
                        'Key': key},
                        ExpiresIn=3600)
    KeyUrl[key] = response
    return KeyUrl


# Returns list of files from bucket using STS    
def getFilesList():
    sts_client = boto3.client('sts')

    # Call the assume_role method of the STSConnection object and pass the role
    # ARN and a role session name.
    assumed_role_object=sts_client.assume_role(
        RoleArn=os.environ['s3role'],
        RoleSessionName="AssumeRoleSession1"
    )

    # From the response that contains the assumed role, get the temporary 
    # credentials that can be used to make subsequent API calls
    credentials=assumed_role_object['Credentials']

    # Use the temporary credentials that AssumeRole returns to make a 
    # connection to Amazon S3  
    s3_resource=boto3.resource(
        's3',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
    )
    s3_client = boto3.client('s3',aws_access_key_id=credentials['AccessKeyId'],
aws_secret_access_key=credentials['SecretAccessKey'],
aws_session_token=credentials['SessionToken'])
    bucket = s3_resource.Bucket(os.environ['filesBucket'])
    files=[]
    for obj in bucket.objects.all():
        files.append(getSignedUrl(obj.key,s3_client))
    return files

We are using the temporary credentials obtained from AssumeRole to generate Presigned URL in getSignedUrl function. The getFilesList functions returns list of file names and presigned URL.

Now, modify showFiles.html, to iterate over response and create links for files

<html>
<head>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" integrity="sha512-8bHTC73gkZ7rZ7vpqUQThUDhqcNFyYi2xgDgPDHc+GXVGHXq+xPjynxIopALmOPqzo9JZj0k6OqqewdGO3EsrQ==" crossorigin="anonymous" />
<script
  src="https://code.jquery.com/jquery-3.1.1.min.js"
  integrity="sha256-hVVnYaiADRTO2PzUGmuLJr8BLUSjGIZsDYGmIJLv2b8="
  crossorigin="anonymous"></script>

<script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
</head>
<body>

<div class="ui raised very text container">
<h1 class="ui header">File Access System</h1>
<i class="folder open icon"></i></i><div class="ui label">Files</div>
<div id="files" >Loading..</div>
</div>
</body>
<script>

fetch("    https://g4m3zpzp95.execute-api.us-east-2.amazonaws.com/Prod/showFiles/", {
  credentials: 'include'
})
  .then(response => response.text())
  .then((body) => {
    var files="";

    var obj = JSON.parse(body)
    for (i = 0; i < obj.length; i++) {
            var o = obj[i]

            for(x in o){
                files =  files+ "<i class='file alternate outline icon'><a href='"+o[x]+"' target='_blank'>&nbsp;&nbsp;"+x+"</a>"
            }
    }
    document.getElementById("files").innerHTML= files
  })
  .catch(function(error) {
    console.log(error); 
  });

</script>
</html>

Do SAM build and Deploy.

Modify login.html, signup.html and showFiles.html to update the api urls from cloudformation outputs. Upload these files to the bucket stsexamplebucket or the bucket you created. Keep these files Public

You can find the code here:

Testing

Let me know if you have any questions or comments!

Imagine you are a solution architect in a company with 100s of Sales employees and you are migrating from on premise to AWS Cloud. You want to use existing employee authentication system and you want to store files on S3 that employee uses in their day to day work. You don't want to keep S3 bucket public, that will expose all files to everybody. You have 2 options:

1. Create single role with S3 bucket access and login credentials for all of the employees. With this user have to use 2 different logins. One to access their existing system and other to access S3 files.

2. Use AWS Security Token Service (STS) to assume role with S3 access and use that to give access to the files. User will still authenticate with their existing system.

In this post, we will explore and implement option # 2. Please note that we are building this example on top of previous post.

About Security Token Service (STS)

AWS Security Token Service (AWS STS) is a web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users that you authenticate. You can use AssumeRole action on STS that returns a set of temporary security credentials that you can use to access AWS resources that you might not normally have access to. These temporary credentials consist of an access key ID, a secret access key, and a security token. Typically, you use AssumeRole within your account or for cross-account access. In our case, we are using AssumeRole within same account.

How to setup users and roles?

In our case, we are going to create a role called S3ReadOnlyAccessAssumeRole. As name suggests it has only S3 read access policy. We will also create a Trust Policy and attached to this S3 role. Trust policy will allow this role to be assumed by our lambda execution role. Here is how our SAM will look like for this role.

IamS3Role:
    Type: AWS::IAM::Role
    Properties: 
      AssumeRolePolicyDocument:
       Version: 2012-10-17
       Statement:
          - Effect: Allow
            Principal:
              AWS: !GetAtt ShowFilesFunctionRole.Arn
            Action:
              - 'sts:AssumeRole'
      Description: 'Readonly S3 role for lamdba to Assume at runtime'
      ManagedPolicyArns: 
      - arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess
      RoleName: S3ReadOnlyAccessAssumeRole

In above snippet, AssumeRolePolicyDocument attribute specifies trust policy that allow Lamdba execution role identified by principle AWS: !GetAtt ShowFilesFunctionRole.Arn to AssumeRole to which this policy is attached to i.e S3ReadOnlyAccessAssumeRole . The ManagedPolicyArns attribute species the policy for S3ReadOnlyAccessAssumeRole that allows read only access to S3 buckets.

Lambda Handler

Now, lets write our Lamdba handler that will use this role. Here is the SAM confgiuration

Origin:
    Type: String
    Default: https://stsexamplebucket.s3.us-east-2.amazonaws.com
  FilesBucket:
    Type: String
    Default: s3-sales-rep

ApiGatewayShowFilesApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
    Auth:
     UsagePlan:
      CreateUsagePlan: PER_API
      Description: Usage plan for this API
      Quota:
       Limit: 500
       Period: MONTH
      Throttle:
       BurstLimit: 100
       RateLimit: 50
  ShowFilesFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          userTable: !Ref myDynamoDBTable
          s3role: !GetAtt IamS3Role.Arn
          origin: !Sub ${Origin}
          filesBucket: !Sub ${FilesBucket}
      CodeUri: Lambda/
      Handler: showfiles.lambda_handler
      Runtime: python3.8
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref myDynamoDBTable      
      Events:
        getCounter:
          Type: Api
          Properties:
            Path: /showFiles
            Method: GET
            RestApiId: !Ref ApiGatewayShowFilesApi

We are defining here couple of parameters that will be set as environment variables for Lambda. With Origin we specifying the origin domain for CORS. It is our S3 bucket's Virtual hosted style URL. FilesBucket is the bucket where files are stored. In Serverless Function definition, it uses showfiles.py as lambda handler. Function has permissions to use DB. We also create API for this lamdba with path /showFiles.

Lets see what we do in the Lambda handlers. We modified login.py lambda handler from previous post. We are setting a cookie once the user is authenticated. This is completely optional and really not required for STS to work but you might need some kind of system to identify that user is already authenticated.

 if decryptedPass == pwd : 
      token = secrets.token_hex(16)  
      response = table.update_item(
        Key={
            'userid': uname
        },
        AttributeUpdates={
            'token': {
                'Value': token,
            }
            }
        )

      return {
        'statusCode': 200,
        'headers':{
             'Set-Cookie':'tkn='+uname+'&'+token+';Secure;SameSite=None;HttpOnly;Domain=.amazonaws.com;Path=/',
             'Content-Type': 'text/html'
         },
        'body': '<html><head><script>window.location.href = \''+ os.environ['showFilesUrl']+'\' </script></head><body>Hello</body></html>'
      }
    else:
     response['status'] = 'Login Failed'
     return {
        'statusCode': 200,
        'body': json.dumps(response) 
      }

When user submits the username and password, Login lambda handler will authenticate the user, store the unique id in DB, set the cookie in response with location to showFiles url html page from S3 bucket. On page load, browser will change the location to showfiles url that will trigger the showFiles Lambda handler defined in showFiles.py

showFiles.html

<html>
<head>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.css" integrity="sha512-8bHTC73gkZ7rZ7vpqUQThUDhqcNFyYi2xgDgPDHc+GXVGHXq+xPjynxIopALmOPqzo9JZj0k6OqqewdGO3EsrQ==" crossorigin="anonymous" />
<script
  src="https://code.jquery.com/jquery-3.1.1.min.js"
  integrity="sha256-hVVnYaiADRTO2PzUGmuLJr8BLUSjGIZsDYGmIJLv2b8="
  crossorigin="anonymous"></script>

<script src="https://cdnjs.cloudflare.com/ajax/libs/semantic-ui/2.4.1/semantic.min.js"></script>
</head>
<body>

<div class="ui raised very text container">
<h1 class="ui header">File Access System</h1>
<i class="folder open icon"></i></i><div class="ui label">Files</div>
<div id="files" >Loading..</div>
</div>
</body>
<script>

fetch("https://9nimlkmz74.execute-api.us-east-2.amazonaws.com/Prod/showFiles/", {
  credentials: 'include'
})
  .then(response => response.text())
  .then((body) => {
    var files="";
    var obj =  JSON.parse(body)
    for (i = 0; i < obj.length; i++) {
            files =  files+ "<i class='file alternate outline icon'><a href='#'>&nbsp;&nbsp;"+obj[i]+"</a>"
    }
    document.getElementById("files").innerHTML= files
  })
  .catch(function(error) {
    console.log(error); 
  });

</script>
</html>

We call the showFiles API, that gets the list of Files from S3 bucket and display on the page.

showFiles.py

import json
import logging
import boto3
import os

log = logging.getLogger()
log.setLevel(logging.INFO)

#retuns login cookie information userid and unique token
def getLoginCookie(cookies):
    data ={}
    for x in cookies:
      keyValue = x.split('=')

      if keyValue[0].strip() =='tkn':
        cookieValue = keyValue[1]
        tknvalues = cookieValue.split('&')
        data['uid']=tknvalues[0]
        data['tkn']=tknvalues[1]
      else:
        cookieValue =''
      return data

#verifies unique token that is saved in database vs in request      
def verifyLogin(data):
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table(os.environ['userTable'])
    response = table.get_item(Key={'userid': data['uid']})
    json_str =  json.dumps( response['Item'])

    resp_dict = json.loads(json_str)
    token = resp_dict.get("token")
    return bool(token == data['tkn'])

# Returns list of files from bucket using STS    
def getFilesList():
    sts_client = boto3.client('sts')

    # Call the assume_role method of the STSConnection object and pass the role
    # ARN and a role session name.
    assumed_role_object=sts_client.assume_role(
        RoleArn=os.environ['s3role'],
        RoleSessionName="AssumeRoleSession1"
    )

    # From the response that contains the assumed role, get the temporary 
    # credentials that can be used to make subsequent API calls
    credentials=assumed_role_object['Credentials']

    # Use the temporary credentials that AssumeRole returns to make a 
    # connection to Amazon S3  
    s3_resource=boto3.resource(
        's3',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken'],
    )

    bucket = s3_resource.Bucket(os.environ['filesBucket'])
    files=[]
    for obj in bucket.objects.all():
        files.append(obj.key)
    return files


def lambda_handler(event, context):
    headers = event.get("headers")
    cookies = headers['Cookie'].split(";")
    data = getLoginCookie(cookies)
    isVerified = verifyLogin(data)

    if(isVerified):
        response = getFilesList()

    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin':os.environ['origin'],
            'Access-Control-Allow-Credentials': 'true'
        },
        'body': json.dumps(response)
    }

Focus on the lamdba_handler function here. We first get the cookie and verify the login. If verified we call the getFilesList function where the magic of STS happens. We get the arn of role to be assumed from Lambda environment variables. The assume_role function returns the credentials that contains the access key id, secret access key and session token. You can use this credentials to get S3 resources and access the bucket. We create list of files as array and send it as response.

You can find full SAM template.yml and other code for this here:

Before you run SAM deploy, create a S3 bucket that will store the html files. In our case, its stsexamplebucket. We use urls from this bucket in our SAM Template. On SAM Deploy, It will generate output with 3 URLS for the APIs. Modify the html files to point to those url and upload to S3 bucket. Make sure you make those files public.

Testing

You need to create a bucket with name specified in FilesBucket parameter in SAM template. This bucket will store the files for display.

Summary

In summary, we used a custom identity broker that authenticates the user and then AWS STS that allows access to AWS resources to those users. You might wonder we could have given access to S3 to Lambda Execution role instead of using STS. Of course you can and it will work. But the idea here is to use STS and you can use STS irrespective of Lambda i.e in your other applications like spring boot, java, python application, etc. In next post, we will further extend this to use S3 signed url to give access to files stored in S3.

Feel free to point out any suggestions, errors and give your feedback in comments!

AWS KMS is a Key Management Service that let you create Cryptographic keys that you can use to encrypt and decrypt data and also other keys. You can read more about it here.

Important points about Keys

Please note that the CMK generated can only be used to encrypt small amount of data like passwords, RSA key. You can use AWS KMS customer master keys (CMKs) to generate, encrypt, and decrypt data keys. However, AWS KMS does not store, manage, or track your data keys, or perform cryptographic operations with data keys. You must use and manage data keys outside of AWS KMS. KMS API uses AWS KMS customer master key (CMK) in the encryption operations and they cannot accept more than 4 KB (4096 bytes) of data. To encrypt application data, use the server-side encryption features of an AWS service, or a client-side encryption library, such as the AWS Encryption SDK or the Amazon S3 encryption client.

Scenario

We want to create signup and login forms for a website.Passwords should be encrypted and stored in dynamobDB database.

What do we need?

1. KMS key to encrypt and decrypt data
2. DynamoDB table to store password.
3. Lamdba functions & APIs to process Login and Sign up forms. 4.Sign up/ Login forms in HTML

Lets Implement it as Serverless Appication Model (SAM)!

Lets first create the Key that we will use to encrypt and decrypt password.

KmsKey:
    Type: AWS::KMS::Key
    Properties: 
      Description: CMK for encrypting and decrypting
      KeyPolicy:
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Enable IAM User Permissions
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action: kms:*
          Resource: '*'
        - Sid: Allow administration of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyAdmin}
          Action:
          - kms:Create*
          - kms:Describe*
          - kms:Enable*
          - kms:List*
          - kms:Put*
          - kms:Update*
          - kms:Revoke*
          - kms:Disable*
          - kms:Get*
          - kms:Delete*
          - kms:ScheduleKeyDeletion
          - kms:CancelKeyDeletion
          Resource: '*'
        - Sid: Allow use of the key
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:user/${KeyUser}
          Action:
          - kms:DescribeKey
          - kms:Encrypt
          - kms:Decrypt
          - kms:ReEncrypt*
          - kms:GenerateDataKey
          - kms:GenerateDataKeyWithoutPlaintext
          Resource: '*'

The important thing in above snippet is the KeyPolicy. KMS requires a Key Administrator and Key User. As a best practice your Key Administrator and Key User should be 2 separate user in your Organisation. We are allowing all permissions to the root users. So if your key Administrator leaves the organisation, the root user will be able to delete this key. As you can see KeyAdmin can manage the key but not use it and KeyUser can only use the key. ${KeyAdmin} and ${KeyUser} are parameters in the SAM template. You would be asked to provide values for these parameters during SAM Deploy.

Parameters:
  KeyAdmin:
    Type: String
  KeyUser:
    Type: String

Next, we will create DynamoDB table. The partition key will be user id. This is enough. You can additional attributes as required.

myDynamoDBTable: 
    Type: AWS::DynamoDB::Table
    Properties: 
      BillingMode: PAY_PER_REQUEST 
      AttributeDefinitions: 
        - 
          AttributeName: "userid"
          AttributeType: "S"

      KeySchema: 
        - 
          AttributeName: "userid"
          KeyType: "HASH"

Now, lets create the API and Lamdba handler that will process the Signup and Login requests.

ApiGatewaySignupApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
    Auth:
     UsagePlan:
      CreateUsagePlan: PER_API
      Description: Usage plan for this API
      Quota:
       Limit: 500
       Period: MONTH
      Throttle:
       BurstLimit: 100
       RateLimit: 50
  SignupFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          userTable: !Ref myDynamoDBTable
          keyid: !Ref KmsKey
      CodeUri: Lambda/
      Handler: signup.lambda_handler
      Runtime: python3.8
      Policies:
       - DynamoDBCrudPolicy:
          TableName: !Ref myDynamoDBTable
       - KMSEncryptPolicy:
          KeyId: !Ref KmsKey 
       - KMSDecryptPolicy:
          KeyId: !Ref KmsKey
      Events:
        getCounter:
          Type: Api
          Properties:
            Path: /signup
            Method: POST
            RestApiId: !Ref ApiGatewaySignupApi
  ApiGatewayLoginApi:
    Type: AWS::Serverless::Api
    Properties:
      StageName: Prod
    Auth:
     UsagePlan:
      CreateUsagePlan: PER_API
      Description: Usage plan for this API
      Quota:
       Limit: 500
       Period: MONTH
      Throttle:
       BurstLimit: 100
       RateLimit: 50
  LoginFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          userTable: !Ref myDynamoDBTable
          keyid: !Ref KmsKey
      CodeUri: Lambda/
      Handler: login.lambda_handler
      Runtime: python3.8
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref myDynamoDBTable
        - KMSEncryptPolicy:
            KeyId: !Ref KmsKey 
        - KMSDecryptPolicy:
            KeyId: !Ref KmsKey
      Events:
        getCounter:
          Type: Api
          Properties:
            Path: /login
            Method: POST
            RestApiId: !Ref ApiGatewayLoginApi

Here, we are creating 2 Lamdba handlers, one that handles singup and other handles the login. We are limiting the usage of API under Auth property. This is to prevent abuse of our API. We are setting two environment variables, one for database table name and another for KMS key name. We are not hard-coding the table/key names instead we let cloud formation to name it and we set it as environment variables with auto generated values. This will prevent name conflicts of resources and you can run multiple versions of your whole cloud environment. Then we set policy that allow Lamdba to access dynamoDB and the key. You can find 'out of box' SAM policy templates here.With Events, we create the API Gateway resources. The final part of SAM template is Output section.

Outputs:
  ApiGatewaySignupApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ApiGatewaySignupApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/signup/"
  SignupFunction:
    Description: "Sign Up Lambda Function ARN"
    Value: !GetAtt SignupFunction.Arn
  ApiGatewayLoginApi:
    Description: "API Gateway endpoint URL for Prod stage for Hello World function"
    Value: !Sub "https://${ApiGatewayLoginApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/login/"
  LoginFunction:
    Description: "Login Lambda Function ARN"
    Value: !GetAtt LoginFunction.Arn

We output 2 important urls in above section, ApiGatewayLoginApi and ApiGatewaySignupApi, We use these Urls in the frontend HTML form's action attribute. See below:

<html>
   <head></head>
   <body>
      <h1>Sign Up</h1>
      <form action="https://51v5ifsje1.execute-api.us-east-2.amazonaws.com/Prod/signup/" enctype="application/x-www-form-urlencoded" method="POST">
         <label>Username:</label><input type="text" name="uname" id="uname"> </input><br><br>
         <label>Password:</label><input type="password" name="password" id="password"> </input><br><br>
         <input type="submit"></input>
      </form>
   </body>
</html>

<html>
  <head></head>
  <body>
    <h1>Login</h1>
    <form action="https://4ezqeugqlf.execute-api.us-east-2.amazonaws.com/Prod/login/" enctype="application/x-www-form-urlencoded" method="POST">
      <label>username:</label><input type="text" name="uname" id="uname"> </input><br><br>
      <label>password:</label><input type="password" name="password" id="password"> </input><br><br>
      <input type="submit"></submit>
    </form>
  </body>
</html>

In HTML, we are using encoding type as "enctype="application/x-www-form-urlencoded". With this type of encoding, the form data is sent in below format

username=doryfish&password=nemo

The last piece in the puzzle is the Lamdba handlers. Lets check it out. Python uses Boto3 to access AWS.

import json
from urllib.parse import parse_qs
import urllib.parse
import boto3
import logging
import os
import base64

log = logging.getLogger()
log.setLevel(logging.INFO)

def lambda_handler(event, context):
    log.info(event)
    log.info(event.get("body"))
    qs = parse_qs(event.get("body"))
    log.info(qs)
    uname = qs.get("uname")[0] 
    pwd = qs.get("password")[0]

    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table(os.environ['userTable'])

    log.info('key id:'+os.environ['keyid'])
    key = os.environ['keyid']
    client = boto3.client('kms')
    #Encrypt password
    response = client.encrypt(
    Plaintext=pwd,
    KeyId=key
    )
    log.info(response['CiphertextBlob'])
    b64_pass = str(base64.b64encode(response['CiphertextBlob']),'utf-8')
    log.info(b64_pass)

    response = table.update_item(
        Key={
            'userid': uname
        },
        AttributeUpdates={
            'password': {
                'Value': b64_pass,
            }
            }
        )
    data = {}
    data['status'] = 'Signup Success'
    json_data = json.dumps(data)    
    return {
        'statusCode': 200,
        'body': json_data
    }

The first two statements are logging statements, this is useful in debugging and the log statements will be logged in CloudWatch logs. The parse_qs module is used to read the form data from event object. Data are returned as a dictionary. The dictionary keys are the unique query variable names and the values are lists of values for each name. Hence, in next 2 statements, we get the first value for each of type - uname and password. We retrieve the DynamoDB table using the table name from environment variable - userTable. Environment variables in Lambda can be accessed using os.environ['key']. Now we got the username and password. Its time to encrypt the password and store it in DB. Password to encrypt is passed to Plaintext attribute of the encrypt request and key id is retrieved from environment variable. CiphertextBlob is the encrypted binary value in the response object. For example:

b'\x01\x02\x02\x00x\x9dN"\xa4\xf9\xfe\xb4\xc7&\x01\xdc\xb6J\xdf\xf1\xdc\xf2;)|7\x1b\'{8\xe6(\x80Q\xe5\x11\x8c\x010w"-\x11w\x10b\x9d\xd0w\xa7+\xd1\xa5\xc5\x00\x00\x00e0c\x06\t*\x86H\x86\xf7\r\x01\x07\x06\xa0V0T\x02\x01\x000O\x06\t*\x86H\x86\xf7\r\x01\x07\x010\x1e\x06\t`\x86H\x01e\x03\x04\x01.0\x11\x04\x0ca\xb4\xaa\x00\x10\xc0\xd1\xa6r\x07\xce\xc7\x02\x01\x10\x80"@\nL\xde<\x03s\xc6\xe0g\x80\xd4\x87\x8e\x1e\t\xa2\xac\x10\xfek\xb6\x1d\xf3\x87\x910\xabf\xd1d}x\xdb'

Now we convert this binary value to ASCII text using

str(base64.b64encode(response['CiphertextBlob']),'utf-8')

and the end result we store it in database.

AQICAHidTiKk+f60xyYB3LZK3/Hc8jspfDcbJ3s45iiAUeURjAEwdyItEXcQYp3Qd6cr0aXFAAAAZTBjBgkqhkiG9w0BBwagVjBUAgEAME8GCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMYbSqABDA0aZyB87HAgEQgCJACkzePANzxuBngNSHjh4JoqwQ/mu2HfOHkTCrZtFkfXjb

In Login Lamdba handler, username and password is retrieved from the request like we did in Sign Up.

import json
from urllib.parse import parse_qs
import urllib.parse
import boto3
import secrets
import logging
import os
import base64

log = logging.getLogger()
log.setLevel(logging.INFO)

def lambda_handler(event, context):

    log.info(event.get("body"))
    qs = parse_qs(event.get("body"))

    uname = qs.get("uname")[0] 
    pwd = qs.get("password")[0]

    dynamodb = boto3.resource('dynamodb')

    table = dynamodb.Table(os.environ['userTable'])
    response = table.get_item(Key={'userid': uname})
    json_str =  json.dumps( response['Item'])

    #using json.loads will turn your data into a python dictionary
    resp_dict = json.loads(json_str)
    dbpass = resp_dict.get("password")

    #Decrypt password
    log.info('key id:'+os.environ['keyid'])
    key = os.environ['keyid']
    client = boto3.client('kms')

    response = client.decrypt(
    CiphertextBlob=(base64.b64decode(dbpass)),
    KeyId=key
    )
    log.info("Decrypted value")
    decryptedPass = response['Plaintext'].decode('UTF-8')

    response = {}

    if decryptedPass == pwd : 
      response['status'] = 'Login Success'
      return {
        'statusCode': 200,
        'body': json.dumps(response) 
      }
    else:
     response['status'] = 'Login Failed'
     return {
        'statusCode': 200,
        'body': json.dumps(response) 
      }

In decrypt request, the encrypted password string from database is converted into binary using base64.b64decode(dbpass) and passed to the CiphertextBlob attribute. Decrypted password is returned as bytes in Plaintext attribute of response. The password is encoded in UTF-8 to get the final decrypted password. The decrypted password is then compared with the password from the request and if it matches, 'Login Success' response is sent back.

Phew! Finally, we are done coding.. Lets test it out. Lets do SAM build and Deploy.

Stack creation complete!

Testing

Note that we are not deploying HTML forms as static site on S3 but we will locally open the forms in browser and hit the APIs. Add username/password, hit submit.

Sign Up is Successful!

Item in database. Password is encrypted Now, Login using the same username and password

Login Sucess!

Source Code:

In next post, we will extend this example to use AWS STS!

Few weeks ago I came across Cloud Resume Challenge on Reddit while searching something. I passed my Amazon Solution Architect certification on Jun 26, So I was looking for something to do next on AWS. Initially I was hesitant to take this challenge because it looked too easy to do. However, that was not the case, I realized only later when I progressed on the challenge. But what really caught my attention was to do Infrastructure as Code(IaC). Its easy to do stuff on AWS console but not on SAM/IaC if you are a beginner. So now I was determined to take up this challenge.

If you are not willing to learn, no one can help you. If you are determined to learn no one can stop you. — Zig Ziglar

Challenge Instructions:

Below were the high level requirements of the challenge.

1. Certification Get AWS Cloud Practitioner Certification

2. HTML Your resume needs to be written in HTML.

3. CSS Your resume needs to be styled with CSS.

4. Static S3 Website Your HTML resume should be deployed online as an Amazon S3 static website.

5. HTTPS The S3 website URL should use HTTPS for security.

6. DNS Point a custom DNS domain name to the CloudFront distribution

7. Javascript Your resume webpage should include a visitor counter that displays how many people have accessed the site.

8. Database The visitor counter will need to retrieve and update its count in a database somewhere.

9. API Do not communicate directly with DynamoDB from your Javascript code. Instead, you will need to create an API that accepts requests from your web app and communicates with the database.

10. Python Write Lambda function in Python;

11. Tests You should also include some tests for your Python code.

12. Infrastructure as Code Implement API resources – the DynamoDB table, the API Gateway, the Lambda function as Infrastructure as Code (IaC)

13. Source Control Use Git hub as source control

14. CI/CD (Back end) Set up GitHub Actions to automate build, test and deployment of Code.

15. CI/CD (Front end) Set up git hub actions to automate deployment of frontend code in AWS including cloudfront cache invalidation.

16. Blog post on the learnings.

Implementing the Challenge

The first few steps were easy but things got more messier when I reached Lambda, SAM/IaC portion of the challenge. I got the Resume ready, registered the domain, setup the s3 bucket for static website, setup cloudfront for s3 bucket. At this point I was able to access site using the domain name. I never worked on python before. I spent sometime learning python, Boto3 for interacting with AWS and moto for writing unit test in Python. You can decorate the test to indicate to use mock dynamodb instead of hitting actual services. Python is very particular about indentation. If you are a java developer,you are going to hate Python!

Most time consuming part of the challenge was Serverless Application Model (SAM). There are not enough good articles available on SAM. Only source is AWS Documentation. With some trial and Error I started getting the hang of it. I coded DynamoDB table and Lamdba function in SAM. DynamoDB table would keep track of visitors for each page. I wanted it to be scalable so I added counter per page. PageName being the Primary key.

I got my Lambda function, database and API as IaC in SAM and was working flawlessly using SAM CLI. I can't describe the happiness you feel when you see below line after SAM Deploy.

Although things were working I was not quite satisfied because part of my infrastructure was still configured manually like Route53, Cloudfront, API Gateway. So I decided to make it truly IaC and Implemented All of the above in SAM. I also added custom domain name to the API so that API can be accessed through a URL that doesn't change. For Example:

https://api.rajanpanchal.net/Prod/someMethod

As If Python was not enough to torture me on Indentation , I used YAML for SAM templates instead of JSON. Once in a while I would feel like:

I used only Notepad/GEdit to write the SAM templates. I would recommend you do the same if you are beginner.

Next step was to automate CI/CD for both backend and frontend. I experimented with GitHub Actions to understand how it works and its very powerful. I was ignorant on AWS S3 Sync command and due to this I had to resort to bash scripting to get the changed files in commit for frontend and upload to S3. Later I simply changed to AWS S3 sync command. However, my bash script was not wasted. I used it to invalidate cache on the cloudfront for the files committed. Backend CI/CD

Frontend CI/CD

Final Thoughts

You can check the final outcome here: https://www.rajanpanchal.net Here is the github url:

https://github.com/rajanpanchal Projects: aws-cloud-resume-frontend aws-cloud-resume-backend

I want to thank Forrest Brazeal for bringing up this challenge and the Cloud Resume Challenge Discord community that has been helpful during this challenge. It was a great learning experience. I have sacrificed several nights and weekends for this challenge but it was really worth it.

Where do I go from here?

I want to build more such projects with various other AWS services and grow my skills. I've got a Cloud Fever and the only prescription is more Cloud!