AWS

Migrating an Amplify Backend on Serverless Framework - Part 2

Published May 20, 2022

Updated Feb 13, 2023

8 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

This is Part Two of a three part series on Migrating an Amplify Backend on Serverless Framework. You can find Part One here and Part Three here.

Welcome to the second part of our "Migrating an Amplify Backend to Serverless Framework", where I will give you a step-by-step guide on how to migrate Amplify-based services so they can be deployable using the Serverless Framework. In the first part, we scaffolded our example application and explained how to deploy Cognito user pools. In this blog post, we'll focus on the core of the application, which is the GraphQL API powered by AppSync.

How AppSync Works

Before we go deep into the code, it's worth understanding how AppSync actually works. AppSync is a powerful service by AWS that allows you to deploy GraphQL APIs with ease, as well as connect those APIs with data sources like AWS DynamoDB, lambda, and more. It can work without Amplify, but when you pair Amplify with it, you get some extra benefits. Deployment, for example, is way easier. All you need is a GraphQL schema, and Amplify will generate all the required AppSync components for it to work end-to-end. Some of these components are:

Data sources, which are interfaces to other AWS resources such as lambdas and DynamoDB tables.
Resolvers, which connect your GraphQL types and fields to data sources. A resolver specifies how data is mapped between your GraphQL API and your data sources. This mapping is done using mapping templates, and mapping templates are written using Apache Velocity Template Language (VTL). A mapping template contains one template for request, and one for response.
DynamoDB tables, usually one for each GraphQL type.

Below, you can see the flow of data when a GraphQL request is made that interacts with a DynamoDB table:

A standard GraphQL schema is not enough to generate everything that AppSync needs, though. Amplify needs some additional metadata in the form of Amplify-specific directives which you can place in your GraphQL schema, and Amplify will use the information from the directives to set up everything correctly.

Then, during deployment, Amplify will transform your schema so that all Amplify-specific directives are stripped, and additional GraphQL types are generated if needed. This is especially important for the @connection directive, which allows you to define 1:1, 1:M, or N:M relationships between models (which are basically GraphQL types annotated with @model directive). Having additional GraphQL types is necessary in order for GraphQL API clients to properly read those relationships.

For example, if the Amplify GraphQL schema contains the following type definition:

type List
@model(
  mutations: { create: "createList", update: "updateList" }
  queries: { get: "getList", list: "listLists" }
  subscriptions: null
)
@auth(
  rules: [
    {
      allow: owner
      identityField: "sub"
      ownerField: "owner"
      operations: [create, update, delete, read]
    }
  ]
){
  id: ID!
  cognitoUserId: ID!
  title: String!
  items: [Item] @connection(keyName: "byList", fields: ["id"])
}

Then, this will be transformed to the following in the standard GraphQL schema:

type List {
  id: ID!
  cognitoUserId: ID!
  title: String!
  items(filter: ModelItemFilterInput, sortDirection: ModelSortDirection, limit: Int, nextToken: String): ModelItemConnection
  createdAt: AWSDateTime!
  updatedAt: AWSDateTime!
  owner: String
}

input ModelItemFilterInput {
  id: ModelIDInput
  cognitoUserId: ModelIDInput
  listId: ModelIDInput
  title: ModelStringInput
  notes: ModelStringInput
  completed: ModelBooleanInput
  remindAt: ModelStringInput
  and: [ModelItemFilterInput]
  or: [ModelItemFilterInput]
  not: ModelItemFilterInput
}

enum ModelSortDirection {
  ASC
  DESC
}

type ModelItemConnection @aws_cognito_user_pools {
  items: [Item]!
  nextToken: String
}

It's not quite the same, isn't it? Yet, only the latter form can be recognized by AppSync.

Hence, a simplified sequence of steps executed during deployment is:

Transform your GraphQL schema enriched with Amplify directives to a standard GraphQL schema, which is then uploaded to AppSync.
Create/update/delete DynamoDB tables based on GraphQL types annotated with @model
Create/update/delete data sources based on directives such as @model and @function
Create/update/delete resolvers and connect them to GraphQL schema's fields on one side as well as to data sources on the other side. Resolvers are the "glue" that connects GraphQL fields to data sources.

In our Serverless deployment, we'll need to replicate the above deployment steps. Let's get to it.

Transforming GraphQL Schema

As part of our Amplified ToDo application, we provided a GraphQL schema that will be used for our API. Now, if we tried to deploy the schema to AppSync, with or without Serverless Framework, it would fail because the schema contains Amplify-specific directives. We need to run it through a transformer first - the same one that Amplify runs when the application is deployed using Amplify.

Fortunately, the Amplify CLI is open-source, and the transformer is part of it, meaning that we can freely use it. Guided by an excellent article by Ronny Roeller, we wrote a script that takes an Amplify GraphQL schema and produces an AppSync-compatible schema. Not only that, it also generates VTL templates along with it, just like Amplify!

The script assumes that the Amplify GraphQL schema is located in the amplify-schema.graphql file (we copied it from amplify/backend/api/amplifiedtodo/schema.graphql). In the script, we first create an instance of GraphQLTransform and pass it different transformer instances, each covering one specific Amplify directive. For example, the ModelConnectionTransformer instance will process @connection directives, and the KeyTransformer will process @key directives.

Then, we call the GraphQLTransform.transform() method which will output the transformed schema to appsync-schema.graphql. Finally, we go through the generated types and fields to generate the mapping templates for the resolvers. Mapping templates are stored to serverless.mapping-templates.yml file. There are two types of mapping templates in the file: "UNIT" (which is the default) and "PIPELINE". They correspond to unit and pipeline resolvers, respectively. Unit resolvers are used for mapping to a single data source, while pipeline resolvers allow you to invoke a series of functions in a serial manner. In our case, pipeline resolvers are used for invoking lambda-based data sources.

Invoking the script is a manual step, and it's best to invoke it before deployment. We made an npm task called deploy that runs the transformer, and then runs sls deploy. This task should be used for deploying from now on as it makes sure that no changes to the GraphQL are left out by accident.

Generating GraphQL TypeScript Types

Another thing that Amplify does is that it can generate TypeScript files that correspond to GraphQL types. Unfortunately, this is hard to do without the Amplify CLI, as the code generation part is deep in the Amplify CLI and cannot be extracted like the transformer. If you use TypeScript, you'll need to keep the Amplify CLI to perform code generation using the amplify codegen command. You can add this command to the deploy command as well.

{
  "scripts": {
    "deploy": "node transform-schema.js && amplify codegen && sls deploy"
  },
  "dependencies": {
    // ...
  }
}

Configuring the AppSync Plugin

The final step is configuring the serverless-appsync-plugin plugin so we can deploy our schema to AppSync.

Before we do that, we should create the lambdas and DynamoDB tables that our GraphQL API will use. If you recall from the previous blog post, we've defined one Lambda called processQueue under the Mutation type:

type Mutation {
  processQueue(input: ProcessQueueInput!): ID @function(name: "processQueue-${env}")
}

Let's create a placeholder for that Lambda by copying handlers/insert-user-preference to handlers/process-queue and adding it to our Serverless config:

functions:
  # ...
  processQueue:
    handler: handlers/process-queue/index.handler

We will also need to create one DynamoDB table for each GraphQL type, namely:

List
Item
UserPreference
NotificationQueue

Let's add the table names to the environment so that they can be passed into our Lambda functions:

provider:
  # ...
  environment:
    LIST_TABLE_NAME: List_${self:provider.stage}
    ITEM_TABLE_NAME: Item_${self:provider.stage}
    USER_PREFERENCE_TABLE_NAME: UserPreference_${self:provider.stage}
    NOTIFICATION_QUEUE_TABLE_NAME: NotificationQueue_${self:provider.stage}

Now, let's create the actual tables under the Resources section:

resources:
  Resources:
    # ...
    ListTableResource:
      Type: AWS::DynamoDB::Table
      Properties:
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        BillingMode: PAY_PER_REQUEST
        TableName: ${self:provider.environment.LIST_TABLE_NAME}
    ItemTableResource:
      Type: AWS::DynamoDB::Table
      Properties:
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
          - AttributeName: listId
            AttributeType: S
        GlobalSecondaryIndexes:
          - IndexName: byList
            KeySchema:
              - AttributeName: listId
                KeyType: RANGE
            Projection:
              ProjectionType: ALL
        BillingMode: PAY_PER_REQUEST
        TableName: ${self:provider.environment.ITEM_TABLE_NAME}
    UserPreferenceTableResource:
      Type: AWS::DynamoDB::Table
      Properties:
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
        BillingMode: PAY_PER_REQUEST
        TableName: ${self:provider.environment.USER_PREFERENCE_TABLE_NAME}
    NotificationQueueTableResource:
      Type: AWS::DynamoDB::Table
      Properties:
        KeySchema:
          - AttributeName: id
            KeyType: HASH
        AttributeDefinitions:
          - AttributeName: id
            AttributeType: S
          - AttributeName: isSent
            AttributeType: S
        GlobalSecondaryIndexes:
          - IndexName: bySentStatus
            KeySchema:
              - AttributeName: isSent
                KeyType: HASH
            Projection:
              ProjectionType: ALL
        BillingMode: PAY_PER_REQUEST
        TableName: ${self:provider.environment.NOTIFICATION_QUEUE_TABLE_NAME}

Note how we did not need to define all fields that are defined in the schema. DynamoDB is schemaless, meaning that you can provision arbitrary columns to table rows. We only need to define the columns that are primary key columns as well as those covered by indexes. Indexes are not created automatically by the serverless-appsync-plugin - you need to define manually in the Resources section.

Next, we need to create data sources. Create a file called serverless.data-sources.yml with the following contents:

# Lambda Data Sources
# Naming convention is ${FunctionName}LambdaDataSource, and each should reference a function resource.
# Function resources are created implicitly by Serverless, and are named like ${FunctionName}LambdaFunction.
- type: AWS_LAMBDA
  name: InvokeProcessQueueLambdaDataSource
  config:
    functionName: processQueue
    iamRoleStatements:
      - Effect: Allow
        Action:
          - lambda:InvokeFunction
        Resource: arn:aws:lambda:${aws:region}:${aws:accountId}:*

# Table Data Sources
# Naming convention is ${TableName}DataSource, and each should reference an existing table resource, created explicitly
# in serverless.resources.yml.
- type: AMAZON_DYNAMODB
  name: ListDataSource
  config:
    tableName: { Ref: ListTableResource }
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:BatchGetItem
          - dynamodb:BatchWriteItem
          - dynamodb:PutItem
          - dynamodb:DeleteItem
          - dynamodb:GetItem
          - dynamodb:Scan
          - dynamodb:Query
          - dynamodb:UpdateItem
        Resource: arn:aws:dynamodb:${aws:region}:${aws:accountId}:*
- type: AMAZON_DYNAMODB
  name: ItemDataSource
  config:
    tableName: { Ref: ItemTableResource }
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:BatchGetItem
          - dynamodb:BatchWriteItem
          - dynamodb:PutItem
          - dynamodb:DeleteItem
          - dynamodb:GetItem
          - dynamodb:Scan
          - dynamodb:Query
          - dynamodb:UpdateItem
        Resource: arn:aws:dynamodb:${aws:region}:${aws:accountId}:*
- type: AMAZON_DYNAMODB
  name: UserPreferenceDataSource
  config:
    tableName: { Ref: UserPreferenceTableResource }
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:BatchGetItem
          - dynamodb:BatchWriteItem
          - dynamodb:PutItem
          - dynamodb:DeleteItem
          - dynamodb:GetItem
          - dynamodb:Scan
          - dynamodb:Query
          - dynamodb:UpdateItem
        Resource: arn:aws:dynamodb:${aws:region}:${aws:accountId}:*
- type: AMAZON_DYNAMODB
  name: UserPreferenceDataSource
  config:
    tableName: { Ref: UserPreferenceTableResource }
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:BatchGetItem
          - dynamodb:BatchWriteItem
          - dynamodb:PutItem
          - dynamodb:DeleteItem
          - dynamodb:GetItem
          - dynamodb:Scan
          - dynamodb:Query
          - dynamodb:UpdateItem
        Resource: arn:aws:dynamodb:${aws:region}:${aws:accountId}:*
- type: AMAZON_DYNAMODB
  name: NotificationQueueDataSource
  config:
    tableName: { Ref: NotificationQueueTableResource }
    iamRoleStatements:
      - Effect: Allow
        Action:
          - dynamodb:BatchGetItem
          - dynamodb:BatchWriteItem
          - dynamodb:PutItem
          - dynamodb:DeleteItem
          - dynamodb:GetItem
          - dynamodb:Scan
          - dynamodb:Query
          - dynamodb:UpdateItem
        Resource: arn:aws:dynamodb:${aws:region}:${aws:accountId}:*

The above file defines data sources for our lambdas and DynamoDB tables. The names of data sources are not arbitrary. They need to match the names that were generated by the transformer script in the serverless.mapping-templates.yml file. Also, each data source references a table by using the Ref intrinsic function to find the table name by resource name (defined under Resources section of the main Serverless config).

The final piece are the function configs, which is a configuration part of the serverless-appsync-plugin that is required only for pipeline resolvers. In the serverless.mapping-templates.yml file, each mapping template for a pipeline resolver refers to a function config. For example, the pipeline resolver for the processQueue lambda references a function config named InvokeProcessQueueLambdaDataSource. The function config provides a request/response mapping, also written in VTL templates, for the lambda in question.

Create a file named serverless.appsync-function-configs.yml, with the following contents:

# Naming convention is Invoke${FunctionName}LambdaDataSource, and each should reference a data source named ${FunctionName}LambdaDataSource.
- dataSource: ProcessQueueLambdaDataSource
  name: InvokeProcessQueueLambdaDataSource
  request: default-invoke-lambda.req.vtl
  response: default-invoke-lambda.res.vtl

Create default-invoke-lambda.req.vtl inside the mapping-templates directory, with the following contents:

{
  "version": "2018-05-29",
  "operation": "Invoke",
  "payload": {
    "typeName": "$ctx.stash.get("typeName")",
    "fieldName": "$ctx.stash.get("fieldName")",
    "arguments": $util.toJson($ctx.arguments),
    "identity": $util.toJson($ctx.identity),
    "source": $util.toJson($ctx.source),
    "request": $util.toJson($ctx.request),
    "prev": $util.toJson($ctx.prev)
  }
}

Likewise, create default-invoke-lambda.res.vtl with the following contents:

#if( $ctx.error )
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.result)

As you can see, the VTL request/response templates for pipeline resolvers are not complicated, and they are same for every pipeline resolver if there are more of them. For that reason, all pipeline resolvers can reference the same files: default-invoke-lambda.req.vtl for request, and default-invoke-lambda.res.vtl for response.

Finally, to bring all the pieces together, this is the final configuration of the AppSync plugin:

custom:
  appSync:
    name: AmplifiedToDo_${self:provider.stage}
    authenticationType: AMAZON_COGNITO_USER_POOLS
    schema: appsync-schema.graphql
    userPoolConfig:
      defaultAction: ALLOW
      userPoolId:
        Ref: CognitoUserPoolAmplifiedToDo
    mappingTemplates:
      - ${file(serverless.mapping-templates.yml)}
    functionConfigurations:
      - ${file(serverless.appsync-function-configs.yml)}
    dataSources:
      - ${file(serverless.data-sources.yml)}

Looking above, that's quite a bit of configuration work. It's a smooth ride once you set it up, though.

To help you understand which configuration references which, here's a diagram with all the pieces:

AppSync Configuration Is Complete

With the above configuration in place, the AppSync configuration is complete. You can view the entire code on GitHub. This was by far the most complex part of the migration. In the next and final blog post, we'll be covering some easier topics, like setting up DynamoDB triggers and configuring S3 buckets.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Dario Djuric
Dario is a full-stack engineer who has spent most of his career doing enterprise Java projects. He has always had a hidden passion for frontend, though -- and he is now able to pursue that passion at This Dot. He spends most of his free time with his two sons, but occasionally, he manages to squeeze in some casual sports activities such as jogging, cycling, and soccer.
@dario_djuric @dariodjuric

How to host a full-stack app with AWS CloudFront and Elastic Beanstalk

How to host a full-stack JavaScript app with AWS CloudFront and Elastic Beanstalk Let's imagine that you have finished building your app. You have a Single Page Application (SPA) with a NestJS back-end. You are ready to launch, but what if your app is a hit, and you need to be prepared to serve thousands of users? You might need to scale your API horizontally, which means that to serve traffic, you need to have more instances running behind a load balancer. Serving your front-end using a CDN will also be helpful. In this article, I am going to give you steps on how to set up a scalable distribution in AWS, using S3, CloudFront and Elastic Beanstalk. The NestJS API and the simple front-end are both inside an NX monorepo The sample application For the sake of this tutorial, we have put together a very simple HTML page that tries to reach an API endpoint and a very basic API written in NestJS. The UI The UI code is very simple. There is a "HELLO" button on the UI which when clicked, tries to reach out to the /api/hello endpoint. If there is a response with status code 2xx, it puts an h1 tag with the response contents into the div with the id result. If it errors out, it puts an error message into the same div. ` The API We bootstrap the NestJS app to have the api prefix before every endpoint call. ` We bootstrap it with the AppModule which only has the AppController in it. ` And the AppController sets up two very basic endpoints. We set up a health check on the /api route and our hello endpoint on the /api/hello route. ` Hosting the front-end with S3 and CloudFront To serve the front-end through a CMS, we should first create an S3 bucket. Go to S3 in your AWS account and create a new bucket. Name your new bucket to something meaningful. For example, if this is going to be your production deployment I recommend having -prod in the name so you will be able to see at a glance, that this bucket contains your production front-end and nothing should get deleted accidentally. We go with the defaults for this bucket setting it to the us-east-1 region. Let's set up the bucket to block all public access, because we are going to allow get requests through CloudFront to these files. We don't need bucket versioning enabled, because these files will be deleted every time a new front-end version will be uploaded to this bucket. If we were to enable bucket versioning, old front-end files would be marked as deleted and kept, increasing the storage costs in the long run. Let's use server-side encryption with Amazon S3-managed keys and create the bucket. When the bucket is created, upload the front-end files to the bucket and let's go to the CloudFront service and create a distribution. As the origin domain, choose your S3 bucket. Feel free to change the name for the origin. For Origin access, choose the Origin access control settings (recommended). Create a new Control setting with the defaults. I recommend adding a description to describe this control setting. At the Web Application Firewall (WAF) settings we would recommend enabling security protections, although it has cost implications. For this tutorial, we chose not to enable WAF for this CloudFront distribution. In the Settings section, please choose the Price class that best fits you. If you have a domain and an SSL certificate you can set those up for this distribution, but you can do that later as well. As the Default root object, please provide index.html and create the distribution. When you have created the distribution, you should see a warning at the top of the page. Copy the policy and go to your S3 bucket's Permissions tab. Edit the Bucket policy and paste the policy you just copied, then save it. If you have set up a domain with your CloudFront distribution, you can open that domain and you should be able to see our front-end deployed. If you didn't set up a domain the Details section of your CloudFront distribution contains your distribution domain name. If you click on the "Hello" button on your deployed front-end, it should not be able to reach the /api/hello endpoint and should display an error message on the page. Hosting the API in Elastic Beanstalk Elastic beanstalk prerequisites For our NestJS API to run in Elastic Beanstalk, we need some additional setup. Inside the apps/api/src folder, let's create a Procfile with the contents: web: node main.js. Then open the apps/api/project.json and under the build configuration, extend the production build setup with the following (I only ) ` The above settings will make sure that when we build the API with a production configuration, it will generate a package.json and a package-lock.json near the output file main.js. To have a production-ready API, we set up a script in the package.json file of the repository. Running this will create a dist/apps/api and a dist/apps/frontend folder with the necessary files. ` After running the script, zip the production-ready api folder so we can upload it to Elastic Beanstalk later. ` Creating the Elastic Beanstalk Environment Let's open the Elastic Beanstalk service in the AWS console. And create an application. An application is a logical grouping of several environments. We usually put our development, staging and production environments under the same application name. The first time you are going to create an application you will need to create an environment as well. We are creating a Web server environment. Provide your application's name in the Application information section. You could also provide some unique tags for your convenience. In the Environment information section please provide information on your environment. Leave the Domain field blank for an autogenerated value. When setting up the platform, we are going to use the Managed Node.js platform with version 18 and with the latest platform version. Let's upload our application code, and name the version to indicate that it was built locally. This version label will be displayed on the running environment and when we set up automatic deployments we can validate if the build was successful. As a Preset, let's choose Single instance (free tier eligible) On the next screen configure your service access. For this tutorial, we only create a new service-role. You must select the aws-elasticbeanstalk-ec2-role for the EC2 instance profile. If can't select this role, you should create it in AWS IAM with the AWSElasticBeanstalkWebTier, AWSElasticBeanstalkMulticontainerDocker and the AWSElasticBeanstalkRoleWorkerTier managed permissions. The next step is to set up the VPC. For this tutorial, I chose the default VPC that is already present with my AWS account, but you can create your own VPC and customise it. In the Instance settings section, we want our API to have a public IP address, so it can be reached from the internet, and we can route to it from CloudFront. Select all the instance subnets and availability zones you want to have for your APIs. For now, we are not going to set up a database. We can set it up later in AWS RDS but in this tutorial, we would like to focus on setting up the distribution. Let's move forward Let's configure the instance traffic and scaling. This is where we are going to set up the load balancer. In this tutorial, we are keeping to the defaults, therefore, we add the EC2 instances to the default security group. In the Capacity section we set the Environment type to Load balanced. This will bring up a load balancer for this environment. Let's set it up so that if the traffic is large, AWS can spin up two other instances for us. Please select your preferred tier in the instance types section, We only set this to t3.micro For this tutorial, but you might need to use larger tiers. Configure the Scaling triggers to your needs, we are going to leave them as defaults. Set the load balancer's visibility to the public and use the same subnets that you have used before. At the Load Balancer Type section, choose Application load balancer and select Dedicated for exactly this environment. Let's set up the listeners, to support HTTPS. Add a new listener for the 443 port and connect your SSL certificate that you have set up in CloudFront as well. For the SSL policy choose something that is over TLS 1.2 and connect this port to the default process. Now let's update the default process and set up the health check endpoint. We set up our API to have the health check endpoint at the /api route. Let's modify the default process accordingly and set its port to 8080. For this tutorial, we decided not to enable log file access, but if you need it, please set it up with a separate S3 bucket. At the last step of configuring your Elastic Beanstalk environment, please set up Monitoring, CloudWatch logs and Managed platform updates to your needs. For the sake of this tutorial, we have turned most of these options off. Set up e-mail notifications to your dedicated alert e-mail and select how you would like to do your application deployments. At the end, let's configure the Environment properties. We have set the default process to occupy port 8080, therefore, we need to set up the PORT environment variable to 8080. Review your configuration and then create your environment. It might take a few minutes to set everything up. After the environment's health transitions to OK you can go to AWS EC2 / Load balancers in your web console. If you select the freshly created load balancer, you can copy the DNS name and test if it works by appending /api/hello at the end of it. Connect CloudFront to the API endpoint Let's go back to our CloudFront distribution and select the Origins tab, then create a new origin. Copy your load balancer's URL into the Origin domain field and select HTTPS only protocol if you have set up your SSL certificate previously. If you don't have an SSL certificate set up, you might use HTTP only, but please know that it is not secure and it is especially not recommended in production. We also renamed this origin to API. Leave everything else as default and create a new origin. Under the Behaviors tab, create a new behavior. Set up the path pattern as /api/* and select your newly created API origin. At the Viewer protocol policy select Redirect HTTP to HTTPS and allow all HTTP methods (GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE). For this tutorial, we have left everything else as default, but please select the Cache Policy and Origin request policy that suits you the best. Now if you visit your deployment, when you click on the HELLO button, it should no longer attach an error message to the DOM. --- Now we have a distribution that serves the front-end static files through CloudFront, leveraging caching and CDN, and we have our API behind a load balancer that can scale. But how do we deploy our front-end and back-end automatically when a release is merged to our main branch? For that we are going to leverage AWS CodeBuild and CodePipeline, but in the next blog post. Stay tuned....

Sep 11, 2023

12 mins

NestJSAWSTypeScriptNodeJS

How to set up local cloud environment with LocalStack

How to set up local cloud environment with LocalStack Developers enjoy building applications with AWS due to the richness of their solutions to problems. However, testing an AWS application during development without a dedicated AWS account can be challenging. This can slow the development process and potentially lead to unnecessary costs if the AWS account isn't properly managed. This article will examine LocalStack, a development framework for developing and testing AWS applications, how it works, and how to set it up. Assumptions This article assumes you have a basic understanding of: - AWS: Familiarity with S3, CloudFormation, and SQS. - Command Line Interface (CLI): Comfortable running commands in a terminal or command prompt. - JavaScript and Node.js: Basic knowledge of JavaScript and Node.js, as we will write some code to interact with AWS services. - Docker Concepts: Understanding of Docker basics, such as images and containers, since LocalStack runs within a Docker container. What is LocalStack? LocalStack is a cloud service emulator that runs in a single container on your laptop or in your CI environment. With LocalStack, you can run your AWS applications or Lambdas entirely on your local machine without connecting to a remote cloud provider! Whether you are testing complex CDK applications or Terraform configurations or just beginning to learn about AWS, LocalStack simplifies your testing and development workflow, relieving you from the complexity of testing AWS applications. Prerequisite Before setting up LocalStack, ensure you have the following: 1. Docker Installed: LocalStack runs in a Docker container, so you need Docker installed on your machine. You can download and install Docker from here. 2. Node.js and npm: Ensure you have Node.js and npm installed, as we will use a simple Node.js application to test AWS services. You can download Node.js from here. 3. Python: Python is required for installing certain CLI tools that interact with LocalStack. Ensure you have Python 3 installed on your machine. You can download Python from here. Installation In this article, we will use the LocalStack CLI, which is the quickest way to get started with LocalStack. It allows you to start LocalStack from your command line. Localstack spins up a Docker instance, Alternative methods of managing the LocalStack container exist, and you can find them here. To install LocalStack CLI, you can use homebrew by running the following command: ` If you do not use a macOS or you don’t have Brew installed, you can install the CLI using Python: ` To confirm your installation was successful, run the following: ` That should output the installed version of LocalStack. Now you can start LocalStack by running the following command: ` This command will start LocalStack in docker mode, and since it is your first installation, try to pull the LocalStack image. You should see the below on your terminal After the image is downloaded successfully, the docker instance is spun up, and LocalStack is running on your machine on port 4566. Testing AWS Services with LocalStack LocalStack lets you easily test AWS services during development. It supports many AWS products but has some limitations, and not all features are free. Community Version: Free access to core AWS products like S3, SQS, DynamoDB, and Lambda. Pro Version: Access to more AWS products and enhanced features. Check the supported community and pro version resources for more details. We're using the community edition, and the screenshot below shows its supported products. To see the current products supported in the community edition, visit http://localhost:4566/_localstack/health. This article will test AWS CloudFormation and SQS. Before we can start testing, we need to create a simple Node.js app. On your terminal, navigate to the desired folder and run the following command: ` This command will create a package.json file at the root of the directory. Now we need to install aws-sdk. Run the following command: ` With that installed, we can now start testing various services. AWS CloudFormation AWS CloudFormation is a service that allows users to create, update, and delete resources in an AWS account. This service can also automate the process of provisioning and configuring resources. We are going to be using LocalStack to test creating a CloudFormation template. In the root of the folder, create a file called cloud-formation.js. This file will be used to create a CloudFormation stack that will be used to create an S3 bucket. Add the following code to the file: ` In the above code, we import the aws-sdk package, which provides the necessary tools to interact with AWS services. Then, an instance of the AWS.CloudFormation class is created. This instance is configured with: region: The AWS region where the requests are sent. In this case, us-east-1 is the default region for LocalStack. endpoint: The URI to send requests to is set to http://localhost:4566 for LocalStack. You should configure this with environment variables to switch between LocalStack for development and the actual AWS endpoint for production, ensuring the same code can be used in both environments. credentials: The AWS credentials to sign requests with. We are passing new AWS.Credentials("test", "test") The params object defines the parameters needed to create the CloudFormation stack: StackName: The name of the stack. Here, we are using 'test-local-stack'. TemplateBody: A JSON string representing the CloudFormation template. In this example, it defines a single resource, an S3 bucket named TestBucket. The createStack method is called on the CloudFormation client with the params object. This method attempts to create the stack. If there is an error, we log it to the console else, we log the successful data to the console. Now, let’s test the code by running the following command: ` If we run the above command, we should see the JSON response on the terminal. ` AWS SQS The process for testing SQS with LocalStack using the aws-sdk follows the same pattern as above, except that we will introduce another CLI package, awslocal. awslocal is a thin wrapper and a substitute for the standard aws command, enabling you to run AWS CLI commands within the LocalStack environment without specifying the --endpoint-url parameter or a profile. To install awslocal, run the following command on your terminal: ` Next, let’s create an SQS queue using the following command: ` This will create a queue name test-queue and return queueUrl like below: ` Now, in our directory, let’s create a sqs.js file, and inside of it, let’s paste the following code: ` In the above code, an instance of the AWS.SQS class is created. The instance is configured with the same parameters as when creating the CloudFormation. We also created a params object which had the required properties needed to send a SQS message: QueueUrl: The URL of the Amazon SQS queue to which a message is sent. In our case, it will be the URL we got when we created a local SQS. Make sure to manage this in environment variables to switch between LocalStack for development and the actual AWS queue URL for production, ensuring the same code can be used in both environments. MessageBody: The message to send. We call the sendMessage method, passing the params object and a callback that handles error and data, respectively. Let’s run the code using the following command: ` We should get a JSON object in the terminal like the following: ` To test if we can receive the SQS message sent, let’s create a sqs-receive.js. Inside the file, we can copy over the AWS.SQS instance that was created earlier into the file and add the following code: ` Run the code using the following command: ` We should receive a JSON object and should be able to see the previous message we sent ` When you are done with testing, you can shut down LocalStack by running the following command: ` Conclusion In this article, we looked at how to set up a local cloud environment using LocalStack, a powerful tool for developing and testing AWS applications locally. We walked through the installation process of LocalStack and demonstrated how to test AWS services using the AWS SDK, including CloudFormation and SQS. Setting up LocalStack allows you to simulate various AWS services on your local machine, which helps streamline development workflows and improve productivity. Whether you are testing simple configurations or complex deployments, LocalStack provides the environment to ensure your applications work as expected before moving to a production environment. Using LocalStack, you can confidently develop and test your AWS applications without an active AWS account, making it an invaluable tool for developers looking to optimize their development process....

Jun 19, 2024

6 mins

AWSDevOps

A Look at Playwright Parallelism

A Look at Playwright Parallelism Playwright is an open-source automation library developed by Microsoft, designed for testing web applications across multiple browsers. It enables developers and testers to write scripts that simulate real user actions when interacting with web pages. Playwright supports major browsers like Chrome, Firefox, and Safari, including their headless modes. Recently, it has gained a lot of popularity for its robustness, ease of use, and cross-browser compatibility. In this blog post, we will take a look at one very useful aspect of Playwright: running tests in parallel. Parallelism in Playwright Parallelism in Playwright refers to running multiple test spec files or even test cases within a spec file simultaneously, greatly improving test execution speed. This is achieved by running the tests in worker processes, where each worker process is an OS process, running independently, orchestrated by the test runner. All workers have identical environments, and each starts its own browser instance. Parallel execution is particularly beneficial in continuous integration and continuous deployment (CI/CD) pipelines, reducing overall build and testing times. Playwright's architecture inherently supports parallelism, and most modern test runners integrating with Playwright can leverage this feature. The parallel execution feature makes Playwright a highly efficient tool for large-scale web application testing. Enabling Parallelism By default, if you scaffold the project using npm init playwright@latest, parallelism is already enabled. Assuming that Playwright's configuration includes three browsers and there is a single spec file with two test cases, the total number of tests that Playwright needs to execute is 3x2 = 6. ` Playwright will decide on how many workers to use based on several factors: - Hardware support: Depending on the number of CPU cores and other system resources, the operating system will decide how many processes Playwright can spin up. - The workers property in the playwright.config.ts file. - The --workers argument to the playwright test command. For example, npx playwright test --workers 4. The --workers argument overrides the workers property in the configuration. However, in both cases, the number of workers can go up to the number of processes that Playwright can spin up, as decided by the operating system. Once it has determined the number of workers, and if the number of workers is larger than 1, Playwright will then decide how to split the work between workers. This decision also depends on several factors, such as the number of browsers involved and the granularity of the parallelism, which can be controlled in several ways: - In the test spec file, you can specify whether to run the test cases in parallel. To run tests in parallel, you can invoke test.describe.configure({ mode: 'parallel' }); before your test cases. - Alternatively, you can configure it per project by setting the fullyParallel: true property. - And finally, you can set it globally in the config, using the same property: fullyParallel: true. Therefore, if there is more than one worker and the parallel mode is enabled, Playwright will assign test cases to each worker as they become available. This scenario is ideal because it means that each test is stateless, and the resources are used most efficiently. ` Disabling Parallelism What if, however, the tests are not stateless? Imagine one test changes the global configuration of the app via some sort of administration page, and the configuration affects different parts of the app, like enabling or disabling features. Other tests, which may be testing those features, would be impacted and would report incorrect results. In such cases, you might want to disable parallelism. You can disable any parallelism globally by allowing just a single worker at any time. Following the instructions from the previous sections to configure the number of workers, you can either set the workers: 1 option in the configuration file or pass --workers=1 to the command line. ` Let's have a look at our test output in this case: ` Now, compare the time it took with one worker versus the time it took with four workers. It took 8.2 seconds with one worker compared to only 3.9 seconds with multiple workers. That might be an inefficient usage of resources, of course, especially if you have a large test suite and some of the tests are stateless and can be run without impacting other tests. Tweaking Parallelism If you want some tests not to run in parallel, but still want to utilize your workers, you can do that. Again, following the configuration options from the previous sections, you can annotate the entire spec file with test.describe.configure({ mode: 'serial' }); to have the tests run sequentially in that spec file, or use the fullyParallel: false property to run tests sequentially on the project level, or using the same property to run tests sequentially on the global level. This means you can still split the tests between workers, but the tests would be run sequentially within a worker depending on the configuration. For example, let's set the number of workers to 4, but set fullyParallel: false globally. ` The tests need to be run sequentially, but since each browser instance effectively provides an isolated environment, this means tests cannot impact each other between environments, and they are safe to be executed sequentially within an environment. This means setting fullyParallel: false on the global level is not the same as having workers: 1, since the browsers themselves offer an isolated environment for the tests to run sequentially. However, since we only have 3 environments (3 browsers), we cannot fully utilize 4 workers as we wanted, so the number of workers is 3. Conclusion In conclusion, Playwright's workers are the core of its parallelism capabilities, enabling tests to run concurrently across different environments and browsers with efficiency. Through its many configuration properties, Playwright allows you to configure parallelism at multiple levels, offering a granular approach to optimizing your test runs. Beyond just executing tests in parallel on a single machine, Playwright's parallelism can even extend to splitting work across multiple machines through sharding, significantly enhancing the scalability of your testing. We hope this blog post was useful. For those interested in delving deeper into the world of Playwright and its powerful features, we've also recently released a JS Drop titled Awesome Web Testing with Playwright, and we also hosted Debbie O'Brien from Microsoft in a Modern Web episode....

Mar 20, 2024

5 mins

JavaScriptPlaywrightTesting

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Migrating an Amplify Backend on Serverless Framework - Part 2

How AppSync Works

Transforming GraphQL Schema

Generating GraphQL TypeScript Types

Configuring the AppSync Plugin

AppSync Configuration Is Complete

Dario Djuric

You might also like

How to host a full-stack app with AWS CloudFront and Elastic Beanstalk

How to set up local cloud environment with LocalStack

A Look at Playwright Parallelism

Introduction to Zod for Data Validation

Let's innovate together!

You might also like

How to host a full-stack app with AWS CloudFront and Elastic Beanstalk

How to set up local cloud environment with LocalStack

A Look at Playwright Parallelism

Introduction to Zod for Data Validation