AWS

Utilizing API Environment Variables on Next.js Apps Deployed to AWS Amplify

Published Jun 16, 2023

Updated Jun 16, 2023

3 min read

Although Next.js is a Vercel product, you may choose not to deploy to Vercel due to their pricing model or concerns with vendor lock-in. Fortunately, several other platforms fully support deployment of Next.js including AWS Amplify. Whether you’re using the Next.js app directory or not, you still have API routes that get deployed as serverless functions to whatever cloud provider you choose. This is no different on AWS Amplify. However, Amplify may require an extra step for the serverless functions if you’re using environment variables. Let’s explore how AWS Amplify is deploying your API routes, and how you can properly utilize environment variables in this context.

How AWS Amplify manages Next.js API Routes

When you deploy Next.js apps via Amplify, it takes the standard build outputs, stores them in S3, and serves them from behind a Cloudfront distribution. However, when you start introducing server side rendering, Amplify utilizes Lambda Edge functions. These edge functions execute the functionality required to properly render the server rendered page. This same flow works for API routes in a Next.js app. They’re deployed to individual lambdas.

In Next.js apps, you have two (2) types of environment variables. There are the variables prefixed with NEXT_PUBLIC_ that indicate to Next.js that the variable is available on the frontend of your application and can be exposed to the general public. At build time, Amplify injects these variables, and values that are stored in the Amplify Console UI, into your frontend application. You also have other environment variables that represent secrets that should not be exposed to users. These will not be included in your build. However, neither set of these variables will be injected into your API routes.

If you need any environment variable in your API routes, you will need to explicitly inject these values into your application at build time so they can be referenced by the Next.js systems, and stored alongside your lambdas.

Injecting Environment Variables into the Amplify Build

By default, Amplify generates the following amplify.yml file that controls your application’s continuous delivery (CD). The following is that default file for Next.js applications:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*

To inject variables into our build, we need to write them to a .env.production file before the application build runs in the build phase. We can do that using the following bash command:

env | grep -e <VARIABLE NAME> >> .env.production

env pulls all environment variables accessible. We use the pipe operator (|) to pass the result of that command to the grep -e which searches the output for the matching pattern. In this case, that’s our environment variable which will output the line that it is on. We then use the >> operator to append to the .env.production file, or create it if it does not exist. Be careful not to use a single > operator as that will overwrite your file’s full content.

Our amplify.yml should now look like this:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - env | grep -e <VARIABLE NAME> >> .env.production
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*

It is important to note that you have to do this for all environment variables you wish to use in an API route whether they have the NEXT_PUBLIC_ prefix or not.

Now, you can use process.env.[VARIABLE NAME] in your API routes to access your functions without any problems. If you want to learn more about environment variables in Next.js, check out their docs.

Conclusion

In short, AWS Amplify deploys your Next.js API routes as Lambda Edge functions that can’t access your console set environment variables by default. As a result, you’ll need to use the method described above to get environment variables in your function as needed. If you want to get started with Next.js on Amplify today, check out our starter.dev kit to get started, and deploy it to your AWS Amplify account. It’ll auto-connect to your git repository and auto-deploy on push, and collaborating with others won’t cost you extra per seat.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Dustin Goodman
Engineering Manager with a passion for web and application development. Speaker and writer on work experiences and software development. Dog dad and musician fueled by coffee.
@dustinsgoodman @dustinsgoodman

Integrating Next.js with New Relic

Integrating Next.js with New Relic When it comes to application monitoring, New Relic is genuinely one of the veterans in the market. Founded in 2008, it offers a large set of tools designed for developers to help them analyze application performance, troubleshoot problems, and gain insight into user behavior through real-time analytics. We needed to integrate New Relic's application monitoring with a modern Next.js app built on top of the app router. This blog post will explain your options if you want to do the same in your project. Please keep in mind that depending on when you read this blog post, there may be some advances in how New Relic works with Next.js. You can always refer to the official New Relic website for up-to-date documentation. The approach is also different, depending on whether you use Vercel's managed Next.js hosting (or any other provider that is lambda-based) or self-host Next.js on an alternative provider such as Fly.io. We'll start with the typical steps you need to do for both ways of deploying; then we'll continue with the self-hosted option, which uses a complete set of features from New Relic, and finish with Vercel where we recommend the OpenTelemetry integration for now. Common Prerequisite Steps First, let's install the necessary packages. These were built by New Relic and are needed whether you self-host your Next.js app or host it on Vercel. ` The next thing you'll want to do is define your app name. The name will be different depending on the environment the app is running in, such as a local environment, staging, or production. This will ensure that the data you send to New Relic is always tagged with a proper app name and will not be mixed between environments. An environment-changing parameter, such as the app name, should always be included in an environment variable. In your local environment, that would be in .env.local, while staging/production or any other hosted environments would be configured in the provider. ` To get full coverage with New Relic, in Next.js, we need to cover the parts running on the client (in the browser) and on the server. The server-running part is different on Vercel's managed hosting, where all of the logic is executed within lambdas, and self-hosting variants, where the server logic is executed on a Node.js server. The client part is configured by going to the New Relic dashboard and clicking "Add Data" / "Browser Monitoring". Choose "place a JavaScript snippet in frontend code," name it the same name you used in the environment variable above, click "Continue" to the last step, and copy the part between the script tags: ` Then paste it to a variable inside any module of your code. For example, this is how we did it: ` Now, create a component that will use this script and put it to Next's Script component. Then you can use the component in your root layout: ` There are several things to note here. First, we need to set a unique ID for the Script component for Next.js to track and optimize the script. We also need to set the strategy to beforeInteractive so that the New Relic script is added to the document’s head element. Finally, we use some environment helper variables. IS_PRODUCTION and IS_STAGING are just helper variables that read the environment variables to determine if we are in a local environment, staging, or production. Now you can create an app router error page and report any uncaught errors to New Relic: ` With this in place, the client part is covered, and you should start seeing data in the New Relic dashboard under "All Entities" / "Browser Applications." Let's now focus on the server part, with both a self-hosted option and Vercel. Self-Hosting Next.js If you self-host Next.js, It will typically run through a Node server (containerized or not), meaning you can fully utilize New Relic's APM agent. Start by adding this to the Next.js config (next.config.js) to mark the newrelic package as external, otherwise you would not be able to import it. ` The next step that New Relic recommends is to copy the newrelic.js file from the newrelic library to the root of the folder. However, we want to keep our project root clean, and we'll store the file in o11y/newrelic.js (“o11y” is an abbreviation for observability). This will also demonstrate the configuration needed to make this possible. This file, by default, will look something like this. ` As you can see, some environment variables are involved, such as NEW_RELIC_APP_NAME and NEW_RELIC_LICENSE_KEY. These environment variables will be needed when the New Relic agent starts up. The trick here is that the agent doesn't start up with your app but *earlier* by preloading the library through another environment variable, NODE_OPTIONS. On the server, that would be done by modifying the start script to this: ` This ensures the agent is preloaded before starting the app, and the rest of the environment variables (such as NEW_RELIC_APP_NAME and NEW_RELIC_LICENSE_KEY) should already exist. Making sure this works in local development is not that easy due to different stages of loading environment variables. The environment variables from .env.local are loaded by the framework, which is too late for the New Relic agent (which is preloaded by the Node process). However, it is possible with a trick. First, add the following new environment variables to .env.local: ` (We have not mentioned NEW_RELIC_HOME so far, but the New Relic agent implicitly uses it to locate the newrelic.js file. We need to specify that since we moved it to the o11y folder.) Next, install the nodenv package. ` The nodenv package provides a cross-platform way of preloading environment variables before we start the Node process. If we provide it with the .env.local file, this means all of the environment variables from that file will be available to the New Relic agent! Change the scripts in package.json: ` And that's it. If you now start the local server using the dev command, nodenv will make environment variables available to the Node process. The Node process will preload the New Relic agent (because we have NODE_OPTIONS in .local.env), the agent will read all the environment variables needed by newrelic.js, and everything will work as expected. The first thing you might notice is the creation of the newrelic_agent.log. This is just a log file by the agent which you can use to troubleshoot if the agent is not working correctly. You can safely add this file to .gitignore. From this point on, the server-side part of the app can utilize New Relic's API (such as noticeError and recordLogEvent) and the agent will automatically be recording transactions made through the server. The data should be visible in the New Relic dashboard, under "All Entities" / "Services - APM". This is how you would log an error in your server action, for example: ` The full source of this type of integration is available on StackBlitz. Using Vercel's Managed Next.js Hosting Using the New Relic APM agent on Vercel is currently not possible, as the server part of Next.js runs within lambdas, and not within a typical Node server as in the self-hosted option. In the future, this may be possible. At the moment, the only way to integrate Vercel and New Relic is by using the Vercel OpenTelemetry collector, which is only available for some plans. The first step is to enable the integration in Vercel's dashboard and then install the following packages: ` Create an instrumentation.ts file in the root of your project and add the following code: ` In the the Next.js config (next.config.js), add: ` The OpenTelemetry integration does not work locally, hence the above conditional check. With the above configuration, the collected OpenTelemetry data should be visible in the New Relic dashboard under "All Entities" / "Services - OpenTelemetry". Optionally, Logging via New Relic API When using the Vercel OpenTelemetry integration, there is no easy way to submit custom logs to New Relic. New Relic offers a logging API, however, so to send custom logs from your backend, you could use a logger such as Winston, which has good library support for custom transports, such as the one for New Relic. Install the following packages: ` Then create a logger.ts file with the following code: ` NewRelicTransport utilizes the New Relic logging API to send application logs. Now you can use the serverLogger in your backend code, from server components to server actions. Conclusion As you can see, various options exist for integrating New Relic with the modern, app-router-based Next.js. Sadly, at the time of writing, no one-size-fits-all solution works equally for all hosting options, as Vercel's managed Next.js platform is still relatively young, and so is the app router in Next.js. If you liked this blog post, feel free to browse our other Next.js blog posts for more tips and tricks in other areas of the Next.js ecosystem....

Jul 19, 2024

7 mins

NextJS

How to build an AI assistant with OpenAI, Vercel AI SDK, and Ollama with Next.js

In today’s blog post, we’ll build an AI Assistant using three different AI models: Whisper and TTS from OpenAI and Llama 3.1 from Meta. While exploring AI, I wanted to try different things and create an AI assistant that works by voice. This curiosity led me to combine OpenAI’s Whisper and TTS models with Meta’s Llama 3.1 to build a voice-activated assistant. Here’s how these models will work together: * First, we’ll send our audio to the Whisper model, which will convert it from speech to text. * Next, we’ll pass that text to the Llama 3.1 model. Llama will understand the text and generate a response. * Finally, we’ll take Llama’s response and send it to the TTS model, turning the text back into speech. We’ll then stream that audio back to the client. Let’s dive in and start building this excellent AI Assistant! Getting started We will use different tools to build our assistant. To build our client side, we will use Next.js. However, you could choose whichever framework you prefer. To use our OpenAI models, we will use their TypeScript / JavaScript SDK. To use this API, we require the following environmental variable: OPENAI_API_KEY— To get this key, we need to log in to the OpenAI dashboard and find the API keys section. Here, we can generate a new key. Awesome. Now, to use our Llama 3.1 model, we will use Ollama and the Vercel AI SDK, utilizing a provider called ollama-ai-provider. Ollama will allow us to download our preferred model (we could even use a different one, like Phi) and run it locally. The Vercel SDK will facilitate its use in our Next.js project. To use Ollama, we just need to download it and choose our preferred model. For this blog post, we are going to select Llama 3.1. After installing Ollama, we can verify if it is working by opening our terminal and writing the following command: Notice that I wrote “llama3.1” because that’s my chosen model, but you should use the one you downloaded. Kicking things off It's time to kick things off by setting up our Next.js app. Let's start with this command: ` After running the command, you’ll see a few prompts to set the app's details. Let's go step by step: * Name your app. * Enable app router. The other steps are optional and entirely up to you. In my case, I also chose to use TypeScript and Tailwind CSS. Now that’s done, let’s go into our project and install the dependencies that we need to run our models: ` Building our client logic Now, our goal is to record our voice, send it to the backend, and then receive a voice response from it. To record our audio, we need to use client-side functions, which means we need to use client components. In our case, we don’t want to transform our whole page to use client capabilities and have the whole tree in the client bundle; instead, we would prefer to use Server components and import our client components to progressively enhance our application. So, let’s create a separate component that will handle the client-side logic. Inside our app folder, let's create a components folder, and here, we will be creating our component: ` Let’s go ahead and initialize our component. I went ahead and added a button with some styles in it: ` And then import it into our Page Server component: ` Now, if we run our app, we should see the following: Awesome! Now, our button doesn’t do anything, but our goal is to record our audio and send it to someplace; for that, let us create a hook that will contain our logic: ` We will use two APIs to record our voice: navigator and MediaRecorder. The navigator API will give us information about the user’s media devices like the user media audio, and the MediaRecorder will help us record the audio from it. This is how they’re going to play out together: ` Let’s explain this code step by step. First, we create two new states. The first one is for keeping track of when we are recording, and the second one stores the instance of our MediaRecorder. ` Then, we’ll create our first method, startRecording. Here, we are going to have the logic to start recording our audio. We first check if the user has media devices available thanks to the navigator API that gives us information about the browser environment of our user: If we don’t have media devices to record our audio, we just return. If they do, then let us create a stream using their audio media device. ` Finally, we go ahead and create an instance of a MediaRecorder to record this audio: ` Then we need a method to stop our recording, which will be our stopRecording. Here, we will just stop our recording in case a media recorder exists. ` We are recording our audio, but we are not storing it anywhere. Let’s add a new useEffect and ref to accomplish this. We would need a new ref, and this is where our chunks of audio data will be stored. ` In our useEffect we are going to do two main things: store those chunks in our ref, and when it stops, we are going to create a new Blob of type audio/mp3: ` It is time to wire this hook with our AudioRecorder component: ` Let’s go to the other side of the coin, the backend! Setting up our Server side We want to use our models on the server to keep things safe and run faster. Let’s create a new route and add a handler for it using route handlers from Next.js. In our App folder, let’s make an “Api” folder with the following route in it: We want to use our models on the server to keep things safe and run faster. Let’s create a new route and add a handler for it using route handlers from Next.js. In our App folder, let’s make an “Api” folder with the following route in it: ` Our route is called ‘chat’. In the route.ts file, we’ll set up our handler. Let’s start by setting up our OpenAI SDK. ` In this route, we’ll send the audio from the front end as a base64 string. Then, we’ll receive it and turn it into a Buffer object. ` It’s time to use our first model. We want to turn this audio into text and use OpenAI’s Whisper Speech-To-Text model. Whisper needs an audio file to create the text. Since we have a Buffer instead of a file, we’ll use their ‘toFile’ method to convert our audio Buffer into an audio file like this: ` Notice that we specified “mp3”. This is one of the many extensions that the Whisper model can use. You can see the full list of supported extensions here: https://platform.openai.com/docs/api-reference/audio/createTranscription#audio-createtranscription-file Now that our file is ready, let’s pass it to Whisper! Using our OpenAI instance, this is how we will invoke our model: ` That’s it! Now, we can move on to the next step: using Llama 3.1 to interpret this text and give us an answer. We’ll use two methods for this. First, we’ll use ‘ollama’ from the ‘ollama-ai-provider’ package, which lets us use this model with our locally running Ollama. Then, we’ll use ‘generateText’ from the Vercel AI SDK to generate the text. Side note: To make our Ollama run locally, we need to write the following command in the terminal: ` ` Finally, we have our last model: TTS from OpenAI. We want to reply to our user with audio, so this model will be really helpful. It will turn our text into speech: ` The TTS model will turn our response into an audio file. We want to stream this audio back to the user like this: ` And that’s all the whole backend code! Now, back to the frontend to finish wiring everything up. Putting It All Together In our useRecordVoice.tsx hook, let's create a new method that will call our API endpoint. This method will also take the response back and play to the user the audio that we are streaming from the backend. ` Great! Now that we’re getting our streamed response, we need to handle it and play the audio back to the user. We’ll use the AudioContext API for this. This API allows us to store the audio, decode it and play it to the user once it’s ready: ` And that's it! Now the user should hear the audio response on their device. To wrap things up, let's make our app a bit nicer by adding a little loading indicator: ` Conclusion In this blog post, we saw how combining multiple AI models can help us achieve our goals. We learned to run AI models like Llama 3.1 locally and use them in our Next.js app. We also discovered how to send audio to these models and stream back a response, playing the audio back to the user. This is just one of many ways you can use AI—the possibilities are endless. AI models are amazing tools that let us create things that were once hard to achieve with such quality. Thanks for reading; now, it’s your turn to build something amazing with AI! You can find the complete demo on GitHub: AI Assistant with Whisper TTS and Ollama using Next.js...

Sep 27, 2024

8 mins

AINextJSJavaScript

The Dangers of ORMs and How to Avoid Them

Background We recently engaged with a client reporting performance issues with their product. It was a legacy ASP .NET application using Entity Framework to connect to a Microsoft SQL Server database. As the title of this post and this information suggest, the misuse of this Object-Relational Mapping (ORM) tool led to these performance issues. Since our blog primarily focuses on JavaScript/TypeScript content, I will leverage TypeORM to share examples that communicate the issues we explored and their fixes, as they are common in most ORM tools. Our Example Data If you're unfamiliar with TypeORM I recommend reading their docs or our other posts on the subject. To give us a variety of data situations, we will leverage a classic e-commerce example using products, product variants, customers, orders, and order line items. Our data will be related as follows: The relationship between customers and products is one we care about, but it is optional as it could be derived from customer->order->order line item->product. The TypeORM entity code would look as follows: ` With this example set, let's explore some common misuse patterns and how to fix them. N + 1 Queries One of ORMs' superpowers is quick access to associated records. For example, we want to get a product and its variants. We have 2 options for writing this operation. The first is to join the table at query time, such as: ` Which resolves to a SQL statement like (mileage varies with the ORM you use): ` The other option is to query the product variants separately, such as: ` This example executes 2 queries. The join operation performs 1 round trip to the database at 200ms, whereas the two operations option performs 2 round trips at 100ms. Depending on the location of your database to your server, the round trip time will impact your decision here on which to use. In this example, the decision to join or not is relatively unimportant and you can implement caching to make this even faster. But let's imagine a different situation/query that we want to run. Let's say we want to fetch a set of orders for a customer and all their order items and the products associated with them. With our ORM, we may want to write the following: ` When written out like this and with our new knowledge of query times, we can see that we'll have the following performance: 100ms * 1 query for orders + 100ms * order items count + 100ms * order items' products count. In the best case, this only takes 100ms, but in the worst case, we're talking about seconds to process all the data. That's an O(1 + N + M) operation! We can eagerly fetch with joins or collection queries based on our entity keys, and our query performance becomes closer to 100ms for orders + 100ms for order line items join + 100ms for product join. In the worst case, we're looking at 300ms or O(1)! Normally, N+1 queries like this aren't so obvious as they're split into multiple helper functions. In other languages' ORMs, some accessors look like property lookups, e.g. order.orderItems. This can be achieved with TypeORM using their lazy loading feature (more below), but we don't recommend this behavior by default. Also, you need to be wary of whether your ORM can be utilized through a template/view that may be looping over entities and fetching related records. In general, if you see a record being looped over and are experiencing slowness, you should verify if you've prefetched the data to avoid N+1 operations, as they can be a major bottleneck for performance. Our above example can be optimized by doing the following: ` Here, we prefetch all the needed order items and products and then index them in a hash table/dictionary to look them up during our loops. This keeps our code with the same readability but improves the performance, as in-memory lookups are constant time and nearly instantaneous. For performance comparison, we'd need to compare it to doing the full operation in the database, but this removes the egregious N+1 operations. Eager v Lazy Loading Another ORM feature to be aware of is eager loading and lazy loading. This implementation varies greatly in different languages and ORMs, so please reference your tool's documentation to confirm its behavior. TypeORM's eager v lazy loading works as follows: - If eager loading is enabled for a field when you fetch a record, the relationship will automatically preload in memory. - If lazy loading is enabled, the relationship is not available by default, and you need to request it via a key accessor that executes a promise. - If neither is enabled, it defaults to the behavior ascribed above when we explained handling N+1 queries. Sometimes, you don't want these relationships to be preloaded as you don't use the data. This behavior should not be used unless you have a set of relations that are always loaded together. In our example, products likely will always need product variants loaded, so this is a safe eager load, but eager loading order items on orders wouldn't always be used and can be expensive. You also need to be aware of the nuances of your ORM. With our original problem, we had an operation that looked like product.productVariants.insert({ … }). If you read more on Entity Framework's handling of eager v lazy loading, you'll learn that in this example, the product variants for the product are loaded into memory first and then the insert into the database happens. Loading the product variants into memory is unnecessary. A product with 100s (if not 1000s) of variants can get especially expensive. This was the biggest offender in our client's code base, so flipping the query to include the ID in the insert operation and bypassing the relationship saved us _seconds_ in performance. Database Field Performance Issues Another issue in the project was loading records with certain data types in fields. Specifically, the text type. The text type can be used to store arbitrarily long strings like blog post content or JSON blobs in the absence of a JSON type. Most databases use a technique that stores text fields off rows, which requires a special file system lookup operation to fetch that data. This can make a typical database lookup that would take 100ms under normal conditions to take 200ms. If you combine this problem with some of the N+1 and eager loading problems we've mentioned, this can lead to seconds, if not minutes, of query slowdown. For these, you should consider not including the column by default as part of your ORM. TypeORM allows for this via their hidden columns feature. In this example, for our product description, we could change the definition to be: ` This would allow us to query products without descriptions quickly. If we needed to include the product description in an operation, we'd have to use the addSelect function to our query to include that data in our result like: ` This is an optimization you should be wary of making in existing systems but one worth considering to improve performance, especially for data reports. Alternatively, you could optimize a query using the select method to limit the fields returned to those you need. Database v. In-Memory Fetching Going back to one of our earlier examples, we wrote the following: ` This involves loading our data in memory and then using system memory to perform a group-by operation. Our database could have also returned this result grouped. We opted to perform this operation like this because it made fetching the order item IDs easier. This takes some performance challenges away from our database and puts the performance effort on our servers. Depending on your database and other system constraints, this is a trade-off, so do some benchmarking to confirm your best options here. This example is not too bad, but let's say we wanted to get all the orders for a customer with an item that cost more than $20. Your first inclination might be to use your ORM to fetch all the orders and their items and then filter that operation in memory using JavaScript's filter method. In this case, you're loading data you don't need into memory. This is a time to leverage the database more. We could write this query as follows in SQL: ` This just loads the orders that had the data that matched our condition. We could write this as: ` We constrained this to a single customer, but if it were for a set of customers, it could be significantly faster than loading all the data into memory. If our server has memory limitations, this is a good concern to be aware of when optimizing for performance. We noticed a few instances on our client's implementation where the filter operations were applied in functions that appeared to run the operation in the database but were running the operation in memory, so this was preloading more data into memory than needed on a memory-constrained server. Refer to your ORM manual to avoid this type of performance hit. Lack of Indexes The final issue we encountered was a lack of indexes on key lookups. Some ORMs do not support defining indexes in code and are manually applied to databases. These tend not to be documented, so an out-of-sync issue can happen in different environments. To avoid this challenge, we prefer ORMs that support indexes in code like TypeORM. In our last example, we filtered on the cost of an order item, but the cost field does not contain an index. This leads to a full table scan of our data collection filtered by the customer. The query cost can be very expensive if a customer has thousands of orders. Adding the index can make our query super fast, but it comes at a cost. Each new index makes writing to the database slower and can exponentially increase the size of our database needs. You should only add indexes to fields that you are querying against regularly. Again, be sure you can notate these indexes in code so they can be replicated across environments easily. In our client's system, the previous developer did not include indexes in the code, so we retroactively added the database indexes to the codebase. We recommend using your database's recommended tool for inspection to determine what indexes are in place and keep these systems in sync at all times. Conclusion ORMs can be an amazing tool to help you and your teams build applications quickly. However, they have gotchas for performance that you should be aware of and can identify while developing or during code reviews. These are some of the most common examples I could think of for best practices. When you hear the horror stories about ORMs, these are some of the challenges typically discussed. I'm sure there are more, though. What are some that you know? Let us know!...

Jul 3, 2024

8 mins

Architectural DesignDesign PatternsArchitecture

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Utilizing API Environment Variables on Next.js Apps Deployed to AWS Amplify

How AWS Amplify manages Next.js API Routes

Injecting Environment Variables into the Amplify Build

Conclusion

Dustin Goodman

You might also like

Integrating Next.js with New Relic

How to build an AI assistant with OpenAI, Vercel AI SDK, and Ollama with Next.js

The Dangers of ORMs and How to Avoid Them

Introduction to Zod for Data Validation

Let's innovate together!

You might also like

Integrating Next.js with New Relic

How to build an AI assistant with OpenAI, Vercel AI SDK, and Ollama with Next.js

The Dangers of ORMs and How to Avoid Them

Introduction to Zod for Data Validation