General

Quality of Life With Static Site Generator

Published Mar 25, 2021

Updated Jan 31, 2023

6 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

Intro

We're building a new marketing site here at Fancy Pants Inc. A totally new greenfield project within an existing organization is fun and exciting, but offers a unique set of challenges. Let's outline some of the most common scenarios and see how to tackle them head on.

For this, we're going to start with the idea of sort of a team meeting, a kind of round table discussion. We have a new site that has static content and maybe some forms.

Some of the requirements from our collective bosses: everyone needs to be able contribute, but only the director or those bestowed with the power can publish content. The other big need is to be able to see changes, updates, new material on the site and pass around an internal url to gather feedback before it ever goes live. There are a few more catches that will come up as we scaffold out this exciting new project.

Like all good sized projects, we've made some choices up front and want to tease out if they will meet our needs before writing up the work and getting started. Based on a few of the requirements outlined earlier, we'll be using Sanity content platform. Everything will hopefully go there from importing existing content, rich media, and even copy for the landing page. For our development choice and deployment, we've chosen Eleventy paired with Netlify for deploy. The idea is to make the developer experience vastly superior to our current system: to make deployment and preview of the project as seamless and transparent as possible. Let's get started.

Where do we start

When we talk about everyone in the company being able to 'contribute', what exactly do we mean? Everyone should be able to make a blog post about the latest and greatest? That sounds fine. What about changing the copy on the homepage or updating the Terms and Service fine print buried in the footer? That's a little less clear now, right? These questions raise important technical decisions about were to apply friction. We often talk about 'reducing friction' as a positive thing. We want to reduce the friction of custom checkout so that more people will buy our product or goods. There are cases, however, were we want to increase friction. We want to ensure that updating legal copy or replacing the hero image on the home page is not just a one click away. There needs to be some amount of process or intentional friction to be a sort of checks and balance.

With that in mind we'll create some simple roles within Sanity 'creator' and 'moderator'. Sanity has a great sample project and the CMS studio itself is React components all the way down, so we can customize it to great extent. We're not entirely sure what the fine grain controls will look like, so let's start broad and then we can narrow down those controls overtime.

For now, creators can create any new content that includes page updates and posts, but only moderators can actually publish that content from its draft version to production. With the content up and running, the actual site will be built leveraging Eleventy Static Site Generator, and the main reason is for its simplicity and flexibility.

In Eleventy, data is merged from multiple different sources before the template is rendered. The data is merged in what Eleventy calls the Data Cascade.

While we are only getting data from Sanity to start with, we have multiple other data sources that will need to be added. Even if that list were zero, a growing business and product will need to evolve and take on new responsibilities in the future. You'll thank your past self when you need to add Shopify or some other datasource in addition to the CMS.

We don't needs gates

Just when we are starting to get some momentum in the project, a hiccup. As we onboard different departments into adding content, there are several requests for a more fine grain approach to content permissions. Our "anyone can create" approach isn't going to fly. After some research, it's decided that Contentful has the sort of finer permissions built in that teams are looking for. Great, but what do we do with our implementation now?

A suggestion is to run git init a new project and just start over. We've spend a nontrivial mount of time working on templates and layouts in Eleventy, is it salvagable? The short answer is, yeah absolutely. Remember: our site doesn't care where the data comes from or how many places it comes from. We can wire-up Contentful as a data source, adjust the templates to any changes in the structure of the content (post.body may now be something like post.content) and then run a new build. Having this deployed on Netlify, a couple of us can work on a feature branch, deploy, and share it without slowing down the group working on the page layout, footers, or contact page!

Cache is still hard

Cache continues to be awful. Humans are not built for the level complexity Goldberg machines that deploy software. We want to keep things simple, and more importantly deterministic. We don't want deployment should not be a constant consideration when creating, improving, or trying new and inventive things within the company.

Our goal is whatever is merged into the main branch will be deployed to the production url. So let's follow those steps. A pull request is merged into main. This kicks off a build of the app on Netlify. It runs npx @11ty/eleventy, gathers up all the data sources from across a pleura of sources, and starts to build, on success Netlify points, all the incoming http requests from Fancy Pants dot com to the newly built site that our build step outputted to _site.

What if it fails? That might happen. Sooner or later it's going to happen. Will the customers of our fine establishment be forever disappointed with the HTTP 500 ERROR they witnessed? Nope. They won't even know. When a build fails, not if, the site continues to run the last successful build it has been. We can setup a web hook to send us a slack message, 'ops, looks like the build failed'. This is a very different message then: 'the software team broke production and we're losing money every minute it's down, everything is on fire!!!'.

Those messages invoke very different feelings, don't they? When you've broken a build, gone back and fixed it, then deployed again, you likely forget about it the next day. If you've ever broken production and spent endless hours trying to put it back together again, you'll never forget it.

Quality of Life

This is very much a quality of life that a system like the JamStack provides us: it's secret sauce. Things are going to go wrong, and when they do, you have the time, space, and mental where-with-all to fix it. When it fails, and it will at some point fail, it doesn't affect the product in production. Other teams internally may not even notice. This gives everyone explicit permission to ship. If that's publishing content with some cool new media, or the latest and greatest GraphQL data collection: if it fails, it's fine. The risk is so low as to be trivial to ship and to try new things. If this all sounds too good to be true, it's not, I promise. Come join us on the moon.

'What a waste of time. It's physically draining to do release engineering. There are other things we want to be doing. And it's not FE release engineering. ' ~ Jason Lengstorf

'It's not something we worry about on our team' ~ Zach Leatherman

I want to thank my guests on Build In Better, Jason Lengstorf with Netlify and Zach Leatherman, Open Source Author of Eleventy, for an amazing conversation and sharing their knowledge with me. This article would not be possible without their time and insight. Thank you. 👋

Special Thanks to the fine folks at Fancy Pants wearables and underoos 😉

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Jesse Tomchak
I work at as a Senior Engineer building dev tools to help everyone be super productive. Teaching JavaScript fundamentals from beginning to excited is a personal passion. Always up for a conversation about types and the paradigm of functional programming. I have dad jokes for days.
@jtomchak @jtomchak

Leveling Up Your Work Through Architecture Design and Time Estimation

> This post can be useful for developers at any level! However, it is written mostly for entry-level developers who are starting to transition into a more intermediate role. You’re rocking your first development job. You’re completing tasks, the team loves you, you still have tons of questions but you can get answers and build what you’re asked to build. But you want to keep improving your skills and giving yourself more options. Some of that skill can only come from actually building things over time. But surely there’s something you can do to level up besides that, right? There is! One way to help deepen your understanding of the software you’re writing and make yourself look more impressive is by improving your ability to estimate your time, and how you talk about your work. That’s what we’re going to dig into today! Some terminology You might have heard of, or have read terms like “software architecture” and “quality attributes” and, upon trying to look into them, been met with a LOT of jargon and dense terminology. So before we get too far into this, let’s go over a few terms we’ll use. (This is by no means meant to be a complete description of these terms. The aim is to be clear enough to continue our discussion here, and let you get started.) - Software architecture: This term means how your system is organized. It’s like the blueprint for the application or website you’re building. How do all the pieces we build fit together and interact with each other? What’s the main goal we’re trying to achieve with this codebase? That’s what this term is talking about. - Software design: Once we have the blueprint, we need to figure out how to build the pieces to make that structure come to life. This is where the design comes in. It’s more code-specific and focuses more on the specifics of what we need to build and how. What languages are we using? How do we get this component to be interactive or usable like we want? This is all the design. - Software quality attributes: Sometimes referred to as “ilities” because a lot of them end with those letters, these are words we use to describe the software we write. Common ones would be accessibility, reliability, or performance. These are words used to describe some of the key qualities we want our software to have. You can start with this list of quality attributes or do a search for “software quality attributes”, and find all sorts of articles covering a lot of the common ones. - Time estimation: This is the skill of being able to consider a task and estimate how long it would take you to complete the task. Time can be a funny thing, and it’s pretty common to think something will be easy and have it take way longer, or think something will be complex, and it turns out to be straightforward. - Trade-offs: This is the idea that two things can’t be equally balanced and/or important. There’s a joke about how people want things fast, cheap, and good and you can’t have all three. This is the same idea. Very often, by increasing one quality, you have to decrease another. The balance of those, and the choice to focus on one quality more than another, is called a trade-off. You’re trading the strength of one quality for another. When we’re using the phrase “architecture design” in this article, we’re combining all these concepts together. How do we get started? Ok, so we’ve got some terms now. But what do we do with them? How do we get better at designing software, and estimating our time? I’ve got two templates to share with you that can help you start to develop these skills. Both will involve writing, but I’ve tried to make them as straightforward to fill out as possible so it’s not a chore to use them. I’ll share links for the templates later on. But first, we’ll go over the actual contents of both of them and talk about how to use them. These templates can be useful for any task, no matter how complex. However, super simple tasks like fixing a broken link on a page, or adjusting a color value to improve its contrast may not be the best choice for it. It is useful to use this template on a task that you fully understand, so you get some practice with thinking about the tradeoffs you’re making, and get used to some of the terminology and how to talk about the software you build. But this can really help when you start getting larger tasks or features- ones that might contain multiple parts working together or some complex logic to get the task working properly. The architecture design template For this template, the goal is to complete this first before you do any work on your task. Use this as a tool to help you think through your work before you get started on it so you can have a better understanding of what the task is asking of you, and let you start thinking about the attributes you’re building for. There are 7 sections for this document, each with a title and description. Let’s go over each piece to see how they work. Big picture: What’s the ask? *What’s the ticket/task trying to solve? What’s the end goal of this body of work? Describe the problem in the most ELI5 way you can.* Use this section to describe the task you’re working on. Act like you’re talking to someone with very little understanding of the situation, like a project manager on another team or to a friend of yours that doesn’t work with you. This section helps you make sure you have a strong handle on the work you’re being asked to do, and what the end result should be. Requirements *Is there acceptance criteria? A specific way it needs to work? What makes this ticket / task count as complete? Steps to reproduce and/or screenshots are helpful here too, if available.* Some of this can often be copied over from the ticket you’re working on. Make note of how you can tell that this task is complete, and any specifics that you need to make sure work as expected when you’re ready to have someone review it. Constraints *Are there specific things you can / can’t use? Something you have to make sure doesn’t break? Tools you have to use because of where you have to work in the codebase?* Making note of constraints can be super helpful. Maybe you have to use a specific UI library to complete this because the rest of the project uses it. Or perhaps this task has been attempted before, and you don’t want to repeat a version that didn’t work. Getting used to thinking about the constraints you’re working within can help prep your brain to think around these concepts. Architecture and Tradeoffs *Where does this code live? (login, ui, db, etc) What software quality attributes are we focused on for this task? List out the primary and secondary attributes (maybe a third if it feels necessary).* In this section, we’re focusing on developing our architecture and design skills in detail. Talk a little about how the codebase is structured, and where the work for this task will be located. Also, pick one or two software quality attributes that you think fit this task. Does the work you’re doing help improve the site’s overall accessibility? Does it make the application more reliable for end users? Does it help to keep our codebase’s maintainability within a reasonable level? Use one of the lists above or your own searches to build out a list of the attributes you might build for regularly, and pick out one or two of them that relate the most to this task. Then, talk a little about why you picked those traits and your thoughts about why they’re applicable. Being able to explain your reasoning here will both help you gain more understanding of the work you’re doing, and the words you’re using to describe it, and give whoever reads this document a better glimpse into the work you do and how you think about it! Initial Gameplan *Considering all the above, what seems like the path forward? What are the steps for how you think it can be solved/completed? Where are you starting?* Now that you’ve spent a little time thinking through the task, and some of the things you should remember about it (like how it’s structured, the tools you have to use, and what’s important to keep in mind while you’re solving it) - write out the steps you think you should take to accomplish the task. It’s perfectly okay to not fully follow this plan as you start to build! We can never fully predict everything that might happen as we start to work in a codebase. The goal here is to give yourself a solid place to start, with all the details we’ve listed fresh in your mind. Process Notes *As you solve this - keep this area updated with rough notes. What did you try that didn’t work? Why not? If you have to pivot, what did you switch to and why? Maybe something worked but you still pivoted - what tradeoffs were made, and what’s the reasoning behind it?* As the description implies, this space is all for you. Try to keep notes as you go. If you do have to change course from your game plan, why and what did you change? Did you find another problem you didn’t even know existed? Or maybe you were able to try a new concept out and it worked! Celebrate your process here. None of this needs to be in complete sentences or easy for anyone but you to understand. This space is all yours to help you keep thinking through the work as you’re doing it! Final Solution *You did it! Form a narrative. Now that you’ve got the issue solved, what does it look like? How did you do it? What did you learn along the way? Is the final solution the same as what you thought, or did the ask pivot along the way? Share screenshots if available.* The final wrap-up! Try to write this similarly to how you started with the big picture. Did your initial plan end up working, and if not why? Were there any interesting plot twists throughout the process? Share the wins, the struggles, and the end result here. Screenshots can be perfect here to see a visual of your work! I also typically leave a little space at the bottom to wrap up any lessons I learned for myself, and leave a little room to talk about my time estimate and reasonings behind how it turned out the way it did. But those are completely optional. --- Because I use a Notion database to help me keep track of these documents, I also have a few properties I can fill out related to the project this document is for, what my time estimate and result were, and the quality attributes I selected. Having those visible at a glance is super helpful for me. The biggest thing I’d recommend to track with this is the date you filled it out. Having that date to help you organize and find your documents is super helpful as your collection grows. I typically title mine based on the project it’s for and the name of the task, but you can name yours whatever you’d like! Now let’s talk about how to track our time for this task. The time estimate template This one is a lot less detailed, but just as important! Being able to improve your knowledge of how long different tasks take you will be a super handy skill as your career continues. The main idea with this template is to keep track of two sections: how long you think something will take you (the estimate), and how long it truly takes you. Our goal is to get those two sections to be as close to the same number as possible. Remember that our goal here is NOT perfection. That’s impossible to reach. Our goal is simply to get them to be close to each other. Most people (even managers) understand that an estimate is not a guarantee. But the closer you can get your estimates to the true time it takes you, the more reliable you seem and the more accurately your team can make progress. There are three main sections to this template. The actual numbers Most importantly, we want an estimated total time and an actual total time, as well as a calculation of the difference between those two numbers. Is our estimate higher, or is the actual total time higher? That difference is what we’re trying to keep as close to zero as possible. I like to break my time tracking down into categories, so I can also see if particular parts of my work take me longer (or shorter) than I think. For each of these, I do the same thing: make an estimate, and record the actual value. The categories I like to track are: Investigation: time to fill out my architecture design, do a little looking into the task, make sure I understand fully what I’m being asked to do, and that I have all the information I need to complete it. Coding: time to do the actual work. Most of my time calculation goes into this category. Testing: I use this for either writing actual unit or end-to-end tests, or for manual testing. This is where I double-check to make sure I didn’t break anything or track time spent on that one last piece of functionality that doesn’t quite work right. Review: any feedback I get on my Pull Requests or code reviews that require me to make changes go into this category. Space to record the numbers as I go The Pomodoro technique works really well for me with time tracking, though I change up my “working time” numbers to make them easier for me to calculate. I’ve found the most straight forward way for me to do this is to have a title for each section of time that I’m tracking, with space underneath each one. Then, I have a legend of colored dot emojis related to a different amount of time: 15 minutes, 30 minutes, and 60 minutes. Then, as I do my rounds of working time, each time I’m done with a round, I select the right colored dot for the amount of time I did, copy it, and paste it under the section the work I did belongs to. Keep repeating this until the work is complete! Once your task is done, all you need to do is count your dots, and record the total number they represent in your actual time section from above. Here’s a screenshot of what one of my completed sections look like, so you can better see what I mean. From this tracking section, I can quickly see that I spent 15 minutes on both testing and review, and an hour and 45 minutes on coding. If I don’t happen to spend any time on a section (in this instance, I’d done some investigating in a separate ticket, so I already knew the work that needed to be done), I just leave it blank. I have a section at the top of my page for tracking the total number for each category. So I use this area to keep track as I’m doing small rounds of work. Then, when I’m done, I add up each number and put it in the section at the top of the page for that category. Notes I also have a section for notes at the bottom of my document. I keep this area for anything relating to my time tracking. Did something break unexpectedly, causing my estimate to be off? Did I not need a section for some reason? Or did something work way better than I thought it would? Those are the kinds of things worth keeping notes of here. Can you tell why your estimate and actual times were off? Sometimes we simply don’t know, but being able to keep notes on the things we can realize as we’re doing them helps us get better the next time! Tying these together and talking about it with others You can use both of these templates together or on their own, and you’ll gain a great amount of knowledge about your skill level and your growth over time from them! But they can also really shine using them together. While these are great tools for your own personal knowledge and growth, I also highly recommend sharing them with someone else. It’s a great idea to set a goal to be better with estimating your time, and sharing that goal with your manager. Then, you can use the time estimation template to build up some estimates, and share those with your manager. Maybe you have a senior developer on your team that you really respect, and you’d like to get their opinion on the task you’re working on. If you’ve filled out the architecture design for that task, you can share it with them and have a conversation about it. They can potentially provide you with things to consider for your next task, or get you thinking more deeply about the quality attributes you selected, and why they may or may not have been the best fit for your work. The design documents are also great to share with your manager! It’s a great way to show that you’re starting to think about your work on a deeper level and starting to consider the quality and complexity of your work. It can also be helpful reference material for them when promotions and new work becomes available. They’re more likely to think you might be a good pick for the next big thing if you’re already showing them you’re starting to think about things at a higher level! The Notion template links The links for both of these documents as Notion templates are below. Please feel free to duplicate a copy for yourself if you like using Notion, or just take a peek at them to see the actual layout of them and adapt that for whatever tool works best for you. These documents are set up to be used within a Notion database. We won’t cover that in detail here, but the Notion documentation should be able to take care of those details for you (and we have a link to get you started below as well). In general, you can create a database within Notion, and then set a template to use for each new entry. That way, when you go to create a new item, it will automatically pull up these templates for you, so you don’t have to copy and paste them every time! And the fields at the top will be visible when you go to look at your database, which makes it a little nicer to get a quick overview of your documents and the progress you’re making. Now go forth and deepen your knowledge! - Time estimation template - Architecture design template - Notion database templates documentation...

Jan 6, 2023

14 mins

Career developmentArchitectureNotion

It's Impossible For This Code to Fail - with Loren Sands-Ramshaw

Loren Sands-Ramshaw, Developer Relations Engineer at Temporal joins Rob Ocel to talk about reliable application development. They introduce the topic of durable execution and talk about reliability in systems, unraveling common issues developers face and showcase the benefits that durable execution can bring to software development. They also talk about the challenges of traditional programming and the complexities of event-driven architecture. Listen to the full podcast here: https://modernweb.podbean.com/e/modern-web-podcast-s11e19-its-impossible-for-this-code-to-fail/...

Jan 24, 2024

1 min

ArchitectureJavaScript

Build A Static Site With Dynamic Flair

Building a new marketing site, a totally new greenfield project within an existing organization, is fun and exciting, but offers a unique set of challenges. Let's outline some of the most common scenarios and see how to tackle them head on. The idea: we have a new site that has static content and maybe some forms. Some of the requirements from our collective bosses: everyone needs to be able contribute, but only the director or those bestowed with the power can publish content. The other big need is to be able to see changes, updates, new material on the site, pass around an internal url to gather feedback, before it ever goes live. There are a few more catches that will come up as we scaffold out this exciting new project. Like all good sized projects, we've made some choices up front and want to make sure they will meet our needs before writing up the work and getting started. Based on a few of the requirements outlined earlier, we'll be using Sanity content platform. Everything will hopefully go there from importing existing content, rich media, and copy for the landing page. For our development choice and deployment, we've chosen Next.js paired with Vercel for deploy. The idea is to make the developer experience vastly superior to our current system: to make deployment and preview of the project as seamless and transparent as possible. Let's dive in. Sanity Content Platform Headless CMS offers several advantages over the traditional method of content. For a platform like Wordpress, the user-generated content, structure, persistence, layout, and style are all combined into a single application. This has benefits such as ease of use, a rich ecosystem of plugins and tools, and many more. It also tightly couples our process and curation of content with the site itself. What Sanity offers is a rich content creation experience, called Studio, separate from our end result, meaning we can distribute it quickly and easily across a global CDN network. That's where Vercel comes in, but we'll get to that. With the Studio comes dedicated SDKs and a generated GraphQL endpoint to get that structured content in just the right way, at just the right time. While GROQ is similar to GraphQL queries, it offers a bit more specificity when required. We can give specific access to everyone in the company using roles that allow them to create, edit, or update curtain content. We'll reserve a role 'owner' for those with final sign off to hit publish and send changes out into the world. The real win with Sanity and other headless CMS's is we can have the development team build out a structured scheme for blog posts, marketing updates, product releases, landing pages, or anything else using Sanity's document based objects and then (and here's the key) hand it over the the rest of the team to build it using their expertise. They can then add titles, descriptions, the right copy, images with specific crops, highlights and metadata for a new project rollout. All that stuff developers are not involved in, the rest of the team can take the scheme and the studio and build just what they need, on their own terms. Next.js Now, to get all that hard work from the content backend and out in to world, we're goning to use a React framework, Next.js. The biggest selling point for this project is two forms of pre-rendering: Static Generation and Server-side Rendering that Next.js offers. This gives a couple features right away. We can statically generate all that content on build time once, send it out to the edges and users will have a super fast, high quality experience. Once that snazzy page has loaded, we can selectively hydrate any dynamic content we might want to keep users engages, maybe load inventory of our latest project from Shopify. That's the other big win with Next.js: it's data hungry. We can load, fetch, and hydrate data from any source in a multitude of ways. First and foremost, we can get content from Sanity on build, and then add sources as the project grows: maybe it's Shopify, or some legacy inventory service over a REST endpoint. The powerful idea that makes Next.js a good fit for this project is that it has opinions on when to fetch data, but the how and where are completely open to mix and match. As a bonus, we also get Server-side Rendering for a great preview experience that we'll look into shortly. How to Query We've chosen our respective platform, deployment method, framework, and content management system. As we begin to wire up the data on the backend to the client, there's a decision to make. Sanity offers a unique query language called groq and also a generated GraphQL endpoint based on the scheme that we create in our studio. So, which do we use? Here's the great part: Next.js is unopinionated about the _way_ we fetch data, only really _where_ it's fetched, be that server side, at build, or client side. GROQ isn't totally foreign as a query language: `` *[_type == 'movie' && releaseYear >= 1979] | order(releaseYear) { _id, title, releaseYear } `` Using the provided Sanity SDK to build and fetch queries or mutations is similar enough to GraphQL, so it won't be a huge lift in learning. That being said, the team is already comfortable with GraphQL, so that makes for a logically choice to start with. However, if or when the need arises for multiple queries or some gnarly nested query, GROQ is in our tool belt and can be additive to the project without backtracking on the work already accomplished. That being said, we also have the flexibility to add additional datasources: maybe there's a DAM in the company that years of assets we need to use or an aggregate sales service that the backend team built for us with using SalesForce. We can selectively fetch this data when the user expects it. By just relying on the components of Next.js, we even get prefetching on pages without having to make any configurations. These are a couple of the small wins we get when choosing a framework with some opinions baked in for us. Datasets, Preview, and Production Things are going great. We've got the studio up with structured schemes for the other teams in the company to add content. An early version of the site deployed on Vercel and tied to the main branch in our git repository. Teams are moving quickly and the site is automatically built and deployed on PR merges. So how do teams starting sharing content, work, ideas, before it goes out for the whole world to see? Preview Mode. When we enable preview mode on our Next.js site, it's just a boolean flag in the cookie, but what it does when a request comes in with preview=true is rather than serve static data or props that we've fetched on build, we can have it fetch draft or unpublished data from our studio. ` In practice, anyone with access to the studio can go in, update the layout and copy of a landing page, maybe give it a bit more flair and click the 'Live Preview' tab that we put right there in the studio! It's going to make an api request to our site at /api/preview passing the draft id and the super secret token that we generate and keep in our ENV list on Vercel. Our endpoint will verify the token, maybe check that the draft id exists to be safe, sets that preview cookie to true, and finally redirects to the site page where getServerSideProps picks up on preview mode. Now we're looking at that super rad landing page rendered in the site with all the CSS styles, headers, footers, side nav, exactly what it will look like when it's published. ` Wrap up > "Service, [and] amplify, and give new skills to non technical people and users." > Guillermo Rauch > "The power, that in the past, only developers had." > ~Kapehe When it comes down to it, what we want in a successful project is to empower users and creators to make without limits. What we've gone over together and outlined above is to that service. As developers, we want to enable those around us, in our companies and projects, to bring their skillset and expertise to the table and contribute in a meaningful way. Do that with as little friction as possible, and you're off to a great start. I want to thank my guests on Build IT Better, Kapehe, Devrel with Sanity CMS, and Guillermo Rauch, CEO of Vercel, for a fantastic conversation and sharing their knowledge with me. This article would not be possible without their time and insight. Thank you. 👋 * Sanity Community...

Mar 11, 2021

6 mins

ArchitectureNextJS

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Quality of Life With Static Site Generator

Intro

Where do we start

We don't needs gates

Cache is still hard

Quality of Life

Jesse Tomchak

You might also like

Leveling Up Your Work Through Architecture Design and Time Estimation

It's Impossible For This Code to Fail - with Loren Sands-Ramshaw

Build A Static Site With Dynamic Flair

Introduction to Zod for Data Validation

Let's innovate together!

You might also like

Leveling Up Your Work Through Architecture Design and Time Estimation

It's Impossible For This Code to Fail - with Loren Sands-Ramshaw

Build A Static Site With Dynamic Flair

Introduction to Zod for Data Validation