General

Intro to EdgeDB - The 10x ORM

Published Apr 26, 2024

Updated Apr 26, 2024

11 min read

Intro to EdgeDB - The 10x ORM

I’ve written a couple of posts recently covering different TypeScript ORMs. One about Prisma, and another about Drizzle. ORM’s are a controversial topic in their own right - some people think they are evil, and others think they are great. I enjoy them quite a bit. They make it easy to interact with your databases. What is more important and magical for an application than data? SQL without an ORM is amazing as well, but there are some pain points with that approach. Today I’m excited to write about EdgeDB, which isn’t exactly an ORM or a database from my perspective (although they call themselves one). It is, however an incredibly impressive piece of technology that solves these common pain points in a pretty novel way.

So if it’s not an ORM or a database, what exactly is it?

I don’t think I can answer that in one or two sentences, so we will explore the various pieces that make up EdgeDB in this article. From a high-level standpoint, though, an interface/query language that sits in front of PostgreSQL. This may seem like some less important implementation detail, but in my eyes, it’s a feature and one of the most compelling selling points.

Data Modeling

EdgeDB advertises itself as a “graph-relational database”. The core components of EdgeDB data modeling are the schema, type system, and relationship definitions. A schema will consist of objects that contain various typed attributes and links that connect the objects. In SQL, a table is analogous to an Object, and a foreign-key is associated with a link.

Here’s what a simple schema in EdgeDB looks like

type User {
  required email: str {
    constraint exclusive;
  };
}

type Post {
  required content: str;
  required author: User;
}

There’s a few things to highlight here

We defined two different objects (tables) User and Post
Each object contains properties with their types defined
str is one of several scalar types (bool, int, float, json, datetime, etc)
the author property is a required link to the User object

Defining relations / associations

In our example above we defined a one-to-many relationship between a user and posts. All the relation types that you can define in traditional SQL are available. One interesting feature though is called backward links. These can be defined in your schema and it allows you to access related data from both sides of a relationship.

type User {
  multi likes: Tweet;
}

type Tweet {
  text: str;
  multi link likers := Tweet.<likes[is User];
}

likes are a many-to-many relationship between Tweet and User. With a backlink defined multi link likers := Tweet.<likes[is User]; - we can access likes from a User and likers from a Tweet.

select User {
    name,
    likes: {
        text
    }
};

select Tweet {
    text,
    likers: {
        name
    }
};

That's how we can access these relations in our queries. You might be looking at these queries and thinking it looks a lot like GraphQL. This is why they call it a ‘Graph-relational’ database.

We’ve only scratched the surface of EdgeDB schema’s. Hopefully, I’ve at least managed to pique your interest.

Computed properties

Computed properties are a super powerful feature that can be added to your schema or queries. This example user schema creates a computed discriminator and username property. The discriminator uses an EdgeDB standard library function to generate a discriminator number and the username property is a combination between the name and discriminator properties.

type User {
  required name: str;
	discriminator := std::random(1000, 9999);
	username := .name ++ "#" ++ <str>.discriminator;
}

Globals and Access Policies

EdgeDB allows you to define global variables as part of your schema. The most common use case I’ve seen for this is to power the access policy feature.

You can define a global variable as part of your schema: global current_user: uuid;

With a global variable defined, you can provide the value as a sort of context from your application by providing it into your EdgeDB driver/client.

const client = createClient().withGlobals({
  current_user: '2141a5b4-5634-4ccc-b835-437863534c51',
});

You can then add access policies directly to your schema, for example, to provide fine-grained access control to blog posts in your blogging application.

type BlogPost {
    required title: str;
    required author: User;

    access policy author_has_full_access
      allow all
      using (global current_user    ?= .author.id
        and  global current_country ?= Country.Full) {
       errmessage := "User does not have full access";
      }
     access policy author_has_read_access
       allow select
       using (global current_user    ?= .author.id
         and  global current_country ?= Country.ReadOnly);
  }

Aside from access policies, you can use your global variables in your queries as well. For example, if you wanted a query to select the current user.

select User filter .id = global current_user;

Types and Sets

EdgeDB is very type-centric. All of your data is strongly typed. We’ve touched on some of the types already.

Scalars - There are a lot of scalar types available out of the box
Custom scalars - Custom scalar types are user-defined extensions of existing types
Enums - Are supported out of the box - enum<Admin, Moderator, Member>
Arrays - Defined by passing the singular value type - array<str>;
Tuples - In EdgeD, tuples can contain more than 2 elements and come in named and unnamed varieties - <str, number>; tuple<name: str, jersey_number: float64, active: bool>;

All queries return a Set which is a collection of values of a given type. In the query language, all values are Sets. Sets are a collection of values of a given type. A comma-separated list of values inside a set of {curly braces}. A query with no results will return an empty or singleton set. If we have no User values stored yet - select User returns {}.

Paired with the query language types and set provides an incredibly powerful and expressive system for interacting with your data. If you thought TypeScript was cool, wait until you start writing EdgeQL! 🙂

EdgeQL

Now to the fun stuff: the query language. We’ll use a schema from the docs and start by looking at some of those queries and build on those.

The example schema has an abstract type Person with two sub-types based on it Hero and Villian. This is known as a polymorphic type in EdgeDB. The Movie type includes a 1:m association with Person

module default {
  abstract type Person {
    required name: str { constraint exclusive };
  }

  type Hero extending Person {
    secret_identity: str;
    multi villains := .<nemesis[is Villain];
  }

  type Villain extending Person {
    nemesis: Hero;
  }

  type Movie {
    required title: str { constraint exclusive };
    required release_year: int64;
    multi characters: Person;
  }
}

Selecting properties / data

Before we dig into some real queries we should just touch on how we select actual data from a query. It’s pretty obvious and GraphQL-like but worth mentioning. To specify properties to select, you attach a shape. This works for getting nested link / association data as well.

Based on our schema, here’s how we could select fields from Movie, including data from the collection of related characters.

select Movie {
	title,
	release_year,
	characters: {
		name
	}
};

There is also a feature called a splats that allows you to select all fields and/or all linked fields without specifying them individually.

# select all properties
select Movie {*}; 

# select all properties including linked properties
select Movie {**};

If you don’t specify any properties or splats, only id’s get returned select Movie; .

Adding some objects with insert

To get started, we can use insert to add objects to our database.

We’ll start big by looking at the nested insert example. This example is interesting because it shows the creation of two objects in a single query. You’ll notice the simplicity of the syntax. Even though this is the first EdgeQL query we’re looking at, in my experience, it’s like this across the board. I’ve found EdgeQL queries to be simple and intuitive to the point where I’ve been able to intuit how to accomplish things in my head without having to reference the docs or ask the AI.

This example adds a new Villian and a new Hero which gets assigned as a link or association to the nemesis field on our Villian. To accomplish this we see that we can nest queries by wrapping them in ().

insert Villain {
  name := "The Mandarin",
  nemesis := (insert Hero {
    name := "Shang-Chi",
    secret_identity := "Shaun"
  })
};

The next example is pretty similar, but instead of creating the linked property, we are select ing and adding several potential objects to the characters list of Movie since it is a multi link. This is a pretty complex query that is doing a lot of different things. It’s deceivingly succinct. To accomplish the same thing with SQL this would probably be about 3 different queries. This query finds the objects to add to the characters multi link by filtering on a collection of different strings to match against the name property.

insert Movie {
  title := "Spider-Man: No Way Home",
  release_year := 2021,
  characters := (
    select Person
    filter .name in {
      'Spider-Man',
      'Doctor Strange',
      'Doc Ock',
      'Green Goblin'
    }
  )
};

The last thing we’ll cover for insert is bulk inserts. This is particularly useful for things like seed scripts.

In this example, you can just imagine that you have a JSON array of objects with hero names that gets passed in as an argument to your query

with
  raw_data := <json>$data,
for item in json_array_unpack(raw_data) union (
  insert Hero { name := <str>item['name'] }
);

Querying data with select

We’ve already seen subqueries and a select in the last section where we found a collection of Person records with a filter. We’ll build on that and see what tools are available to us when it comes to querying data.

This one covers a lot of ground. Very similar to SQL we have order and limit, and offset operators to support sorting and pagination. Also there is a whole standard library of functions and operators like count that can be used in our queries. This example returns a collection of villian names, excluding the first and last result.

select Villain {name}
order by .name
offset 1
limit count(Villain) - 1;

Most commonly, you will want to filter by an id

select Villian {*} filter .id = <uuid>"6c22c502-5c03-11ee-99ff-cbacc3918129";

Here’s another common example filtering by datetime. Since we’re using a string value here we need to cast it to the EdgeDB datetime type.

select Movie {*}
filter
    Movie.release_date > <cal::local_datetime>'2020-01-01T00:00:00';

You get a pretty similar toolbox to SQL when it comes to filtering with your common operators and things. Combined with all the tools in the standard library, you can get pretty creative with it.

Changing values and links with update

The update..filter..set statement is how we can update existing data with EdgeQL. set is followed by a shape with assignments of properties to be updated.

update Hero
filter .name = "Hawkeye"
set { name := "Ronin" };

You can replace links for an object

update movie
filter .title = "Black Widow"
set {
 characters := (
  select Person
  filter .name in { "Black Widow", "Yelena", "Dreykov" }
 )
};

or add additional ones

update Movie
filter .title = "Black Widow"
set {
 characters += (insert Villain {name := "Taskmaster"})
};

An even more interesting example is removing links matched on a type. Since Villian is a sub-type of Person , this query will remove all characters linked of the Villian type.

update Movie
filter .title = "Black Widow"
set {
 characters -= Villain # remove all villains
};

Deleting objects with delete

Deleting is pretty straight forward. Using the delete command you can just filter for the objects that you would like to remove.

delete Hero
filter .name = 'Iron Man';

When the EdgeQL pieces fall into place

As you become more familiar with the EdgeQL query language chances are you’ll start writing very complex queries fluently because everything just makes sense once you’ve learned the building blocks.

Domain and business concerns

I don’t think they explicitly mention this as a goal anywhere but it’s something that I picked up on pretty quickly. EdgeDB nudges you to move more of what might have traditionally been application logic into your database layer. This is a topic that can bring a lot of division since even things like foreign keys and constraints in SQL are frowned upon in some circles. EdgeDB goes as far as providing constraints, global variables, contexts, and authorization support built into the database. I think that the ability to bake some of these concerns into your EdgeDB Schema is great. The way you model your schema and database in EdgeDB map to your domain in a much more intuitive way where domain concerns don’t really feel out of place there.

Database Clients and Query Builders and Generators

We’ve covered a lot so far to highlight what EdgeDB is and how to handle common use cases with the query language. To use it in your project though, you will need a client/driver library. There are clients available in several different languages. The one that they clearly have put the most investment into is the TypeScript query builder. We’ll briefly look at both options: simple driver/client and query builder. Whichever you end up choosing you will need to instantiate a driver and make sure you have a connection to your database instance configured.

Basic client

Although the TS query builder is very popular and pretty amazing, I couldn’t get away from just writing EdgeQL queries. In my application, I composed queries using template strings, and it worked great. The clients all have a set of methods available for passing in EdgeQL queries and parameters.

querySingle is a method for queries where you are only expecting a single result. If your query will have multiple results you would use query instead. There is also a queryRequiredSingle which will throw an error if no results are found. There are some other methods available as well including one for running queries in a transaction

import * as edgedb from "edgedb";

const client = edgedb.createClient();

async function main() {
  const result = await client.querySingle(`
    select Movie {
      title,
      actors: {
        name,
      }
    } filter .title = <str>$title
  `, { title: "Iron Man 2" });

  console.log(JSON.stringify(result, null, 2));
}

The first argument is the query, and the second is a map of parameters. In this example we include the title parameter and it is accessed in our query via $title.

TypeScript query builder

If you have a TypeScript app and type-safety is important, you might prefer using the query builder. It is a pretty incredible feat of TypeScript magic initially developed by the same developer behind the popular library Zod. We can’t cover it in very much depth here but we’ll look at an example just to have an idea of what the query builder looks like in an application.

import * as edgedb from "edgedb";
import e from "./dbschema/edgeql-js";

const client = edgedb.createClient();

async function main() {
  // result will be inferred based on the query
  const result = await e
    .select(e.Movie, () => ({
      title: true,
      actors: () => ({ name: true }),
      filter_single: { title: "Iron Man 2" },
    }))
    .run(client);

  console.log(JSON.stringify(result, null, 2));
}

The query builder is able to infer the result type automatically. It knows which fields you’ve selected, it knows that the result will be a single item.

Query generator

There are generators for queries and types. So even if you opt out of using the query builder you can still have queries that are strongly typed. It’s nice to have this option if you want to just write your queries as EdgeQL in .edgeql files.

└── queries
    └── getUser.edgeql
    └── getUser.query.ts    <-- generated file

We end up with an exported function named getUser that is strongly typed.

import { getUser } from "./queries/getUser.query";

const user = await getUser(client, newUser); // GetUserReturns

Tools and Utilities

The team at EdgeDB puts a big emphasis on developer experience. It shows up all over the place. We’ve already seen some utilities with the generators that are available. There are some other tools available as well that help complete the entire experience.

EdgeDB CLI

The first and most important tool to mention is the CLI. If you’ve started using EdgeDB then you’ve most likely already installed and used it. The CLI is pretty extensive. It includes commands for things like migrations, managing EdgeDB versions and installations, managing projects and local/cloud database instances, dumps and restores, a repl, and more. The CLI makes managing EdgeDB a breeze.

Admin UI

The CLI includes a command to launch an admin UI for any project or database. The Admin UI includes a awesome interactive diagram of your database schema, a repl for running queries, and a table to inspect and make changes to the data stored in your database.

Summary

Adopting newer database technology is a tough sales pitch. Replacing your application’s database technology at any point in its lifecycle is not a problem that anyone wants to have. This is one of the reasons why EdgeDB being a superset of PostgreSQL is a huge feature in my opinion. The underlying database technology is tried and true, and EdgeDB is open-source. Based on this, I would feel confident using EdgeDB if it aligned well from a technical and business perspective.

We’ve covered a lot of ground in this post. EdgeDB is feature-packed and powerful. Databases is a tough nut to crack, and I commend the team for all their hard work to help continue pushing forward one of the most important components of almost any application. I’m typically pretty conservative when it comes to databases, but EdgeDB took a great approach, in my opinion. I recommend at least giving it a try. You might catch the EdgeDB bug like I did!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Dane Grant
JavaScript developer building things on web, native, and server environments. Free time filled with reading, cooking, dog, sports, comedy, and music.
@danecando @danecando

Making Seamless Page Transitions with the View Transitions API

Make Seamless Page Transitions using the View Transitions API Traditionally web applications have always had a less polished experience, both functionally and visually speaking, compared to native applications on mobile and other platforms. This has been improving over the years with the introduction of new web APIs and standards. One such new API that bridges that gap is the View Transitions API. What is the View Transitions API? The View Transitions API is a new browser feature that allows developers to more easily create animated transitions between pages and views, similar to how mobile apps have transitions when navigating between pages. Adding view transitions to your application can reduce the cognitive load on your users and make the experience feel less inconsistent. One great thing about view transitions is that they can be used by both SPAs (single-page applications) and MPAs (multi-page applications) alike! Using the View Transitions API for SPAs Let’s see how we can use view transitions ourselves. We will use examples adapted from demos created by the wonderful Jake Archibald and Bramus. There are some alterations to a few of the examples to make them work with the newest version of the View Transitions API, but otherwise, they’re mostly the same. The process of getting view transitions working for SPAs is as simple as calling document.startViewTransition right before starting navigation and replacing your page’s content immediately after that. You need to make this call in your router or in an event listener that triggers your page navigation. The following is an example of how you can hook into page navigation in a vanilla JavaScript application. This example uses the Navigation API if it’s available. As of the time of writing this article only Chromium-based browsers support the Navigation API, but there is a fallback to using a click listener on link elements so that this works with Firefox and Safari. ` If you’re using a framework, then there’s probably a better way to do this specific to that framework than by using this contrived example. For example, if you are implementing usage of this API with a Next application, you may opt for something like next-view-transitions. For this article, we’re going to focus on the way this is done in vanilla JavaScript so that the fundamentals of using the API are understood. Knowing how it’s implemented in vanilla JavaScript should give you enough understanding to implement this in the framework of your choice if there isn’t already a library that does it for you. Now we have a way to know when a navigation is being requested. Using the above utility function we can start the animation and replace our content as follows: ` getMyPageContentSomehow can be any function that returns HTML. How that is implemented will depend on your application and framework of choice. If you want to run these examples on your machine, the demo repository has instructions in its README. In essence, it’s as simple as hosting an HTTP server at the root of the project and visiting localhost. No fancy build system is required! By default, a fade is done that only lasts for a moment. The animation that happens can be configured using CSS with the ::view-transition-group pseudo-element on the element you want to animate, which in our case is the entire page, and we pass in root. ` Given that the animation properties are used, you have much control over what kind of animation to do. You can change the duration and the interpolation, add keyframes and gradients, and even add image masks. You can learn more about the animation properties in CSS on MDN. Using the View Transitions API for MPAs As mentioned, view transitions also work for MPAs with no client-side routing. Setting up view transitions for MPAs is much easier than for SPAs. All you have to do is include the CSS rule on all pages where you want the animations. ` Of course, like how it works with SPAs, you can customize the animations to your heart’s content using the view transition pseudo-elements. Conclusion I hope this article helped you understand the fundamentals of using the View Transitions API. Now that most browsers support it and it’s not too difficult to add it to existing applications, it is a great time to learn how to use it. You can view a full list of examples here (made by Jake Archibald and Bramus) if you want to see how advanced the animations can get. I like the Star Wars ones especially. The README explains how to get all the examples up and running (a total of 43!)....

Aug 2, 2024

4 mins

JavaScript

Exploring Angular Forms: A New Alternative with Signals

Exploring Angular Forms: A New Alternative with Signals In the world of Angular, forms are essential for user interaction, whether you're crafting a simple login page or a more complex user profile interface. Angular traditionally offers two primary approaches: template-driven forms and reactive forms. In my previous series on Angular Reactive Forms, I explored how to harness reactive forms' power to manage complex logic, create dynamic forms, and build custom form controls. A new tool for managing reactivity - signals - has been introduced in version 16 of Angular and has been the focus of Angular maintainers ever since, becoming stable with version 17. Signals allow you to handle state changes declaratively, offering an exciting alternative that combines the simplicity of template-driven forms with the robust reactivity of reactive forms. This article will examine how signals can add reactivity to both simple and complex forms in Angular. Recap: Angular Forms Approaches Before diving into the topic of enhancing template-driven forms with signals, let’s quickly recap Angular's traditional forms approaches: 1. Template-Driven Forms: Defined directly in the HTML template using directives like ngModel, these forms are easy to set up and are ideal for simple forms. However, they may not provide the fine-grained control required for more complex scenarios. Here's a minimal example of a template-driven form: ` ` 2. Reactive Forms: Managed programmatically in the component class using Angular's FormGroup, FormControl, and FormArray classes; reactive forms offer granular control over form state and validation. This approach is well-suited for complex forms, as my previous articles on Angular Reactive Forms discussed. And here's a minimal example of a reactive form: ` ` Introducing Signals as a New Way to Handle Form Reactivity With the release of Angular 16, signals have emerged as a new way to manage reactivity. Signals provide a declarative approach to state management, making your code more predictable and easier to understand. When applied to forms, signals can enhance the simplicity of template-driven forms while offering the reactivity and control typically associated with reactive forms. Let’s explore how signals can be used in both simple and complex form scenarios. Example 1: A Simple Template-Driven Form with Signals Consider a basic login form. Typically, this would be implemented using template-driven forms like this: ` ` This approach works well for simple forms, but by introducing signals, we can keep the simplicity while adding reactive capabilities: ` ` In this example, the form fields are defined as signals, allowing for reactive updates whenever the form state changes. The formValue signal provides a computed value that reflects the current state of the form. This approach offers a more declarative way to manage form state and reactivity, combining the simplicity of template-driven forms with the power of signals. You may be tempted to define the form directly as an object inside a signal. While such an approach may seem more concise, typing into the individual fields does not dispatch reactivity updates, which is usually a deal breaker. Here’s an example StackBlitz with a component suffering from such an issue: Therefore, if you'd like to react to changes in the form fields, it's better to define each field as a separate signal. By defining each form field as a separate signal, you ensure that changes to individual fields trigger reactivity updates correctly. Example 2: A Complex Form with Signals You may see little benefit in using signals for simple forms like the login form above, but they truly shine when handling more complex forms. Let's explore a more intricate scenario - a user profile form that includes fields like firstName, lastName, email, phoneNumbers, and address. The phoneNumbers field is dynamic, allowing users to add or remove phone numbers as needed. Here's how this form might be defined using signals: ` > Notice that the phoneNumbers field is defined as a signal of an array of signals. This structure allows us to track changes to individual phone numbers and update the form state reactively. The addPhoneNumber and removePhoneNumber methods update the phoneNumbers signal array, triggering reactivity updates in the form. ` > In the template, we use the phoneNumbers signal array to dynamically render the phone number input fields. The addPhoneNumber and removePhoneNumber methods allow users to reactively add or remove phone numbers, updating the form state. Notice the usage of the track function, which is necessary to ensure that the ngFor directive tracks changes to the phoneNumbers array correctly. Here's a StackBlitz demo of the complex form example for you to play around with: Validating Forms with Signals Validation is critical to any form, ensuring that user input meets the required criteria before submission. With signals, validation can be handled in a reactive and declarative manner. In the complex form example above, we've implemented a computed signal called formValid, which checks whether all fields meet specific validation criteria. The validation logic can easily be customized to accommodate different rules, such as checking for valid email formats or ensuring that all required fields are filled out. Using signals for validation allows you to create more maintainable and testable code, as the validation rules are clearly defined and react automatically to changes in form fields. It can even be abstracted into a separate utility to make it reusable across different forms. In the complex form example, the formValid signal ensures that all required fields are filled and validates the email and phone numbers format. This approach to validation is a bit simple and needs to be better connected to the actual form fields. While it will work for many use cases, in some cases, you might want to wait until explicit "signal forms" support is added to Angular. Tim Deschryver started implementing some abstractions around signal forms, including validation and wrote an article about it. Let's see if something like this will be added to Angular in the future. Why Use Signals in Angular Forms? The adoption of signals in Angular provides a powerful new way to manage form state and reactivity. Signals offer a flexible, declarative approach that can simplify complex form handling by combining the strengths of template-driven forms and reactive forms. Here are some key benefits of using signals in Angular forms: 1. Declarative State Management: Signals allow you to define form fields and computed values declaratively, making your code more predictable and easier to understand. 2. Reactivity: Signals provide reactive updates to form fields, ensuring that changes to the form state trigger reactivity updates automatically. 3. Granular Control: Signals allow you to define form fields at a granular level, enabling fine-grained control over form state and validation. 4. Dynamic Forms: Signals can be used to create dynamic forms with fields that can be added or removed dynamically, providing a flexible way to handle complex form scenarios. 5. Simplicity: Signals can offer a simpler, more concise way to manage form states than traditional reactive forms, making building and maintaining complex forms easier. Conclusion In my previous articles, we explored the powerful features of Angular reactive forms, from dynamic form construction to custom form controls. With the introduction of signals, Angular developers have a new tool that merges the simplicity of template-driven forms with the reactivity of reactive forms. While many use cases warrant Reactive Forms, signals provide a fresh, powerful alternative for managing form state in Angular applications requiring a more straightforward, declarative approach. As Angular continues to evolve, experimenting with these new features will help you build more maintainable, performant applications. Happy coding!...

Oct 4, 2024

5 mins

AngularJavaScript

D1 SQLite: Schema, migrations and seeds

I’ve written posts about some of the popular ORM’s in TypeScript and covered their pros and cons. Prisma is probably the most well known and Drizzle is a really popular up and comer. I like and use ORM’s in most of my projects but there’s also a camp of folks who believe they shouldn’t be used. I started a small Cloudflare Workers project recently and decided to try using their D1 SQLite database without adding any ORM. This is the first post in a 2 part series where we’ll explore what this experience is like using only the driver and utilities made available in the Wrangler CLI. Introduction If you’re unfamiliar with Cloudflare D1 - it’s a distributed SQLite database for the Cloudflare Workers platform. Workers are lightweight serverless functions/compute distributed across a global network. The platform includes services and API’s like D1 that provide extended capabilities to your Workers. At the time of this writing, there are only two ways to interact with a D1 database that I’m aware of. * From a Worker using the D1 Client API * D1 HTTP API In this 2 part series, we will create a simple Workers project and use the D1 Client API to build out our queries. Getting Started For this tutorial, we’ll create a simple Cloudflare Worker project and treat it like a simple node script/http server to do our experiments. The first step is initializing a new Cloudflare Worker and D1 database: ` ` We need to take our binding and add it to our project wrangler.toml configuration. Once our binding is added we can re-generate the types for our Worker project. ` We now have our DB binding added to our project Env types. Let’s add a simple query to our worker script to make sure our database is setup and working: ` Start the development server with npm run dev and access the server at http://localhost:8787 . When the page loads we should see a successful result {"1 + 1":2} . We now have a working SQLite database available. Creating a schema Since we’re not using an ORM with some kind of DSL to define a schema for our database, we’re going to do things the old fashioned way. “Schema” will just refer to the data model that we create for our database. We’ll define it using SQL and the D1 migrations utility. Let’s create a migration to define our initial schema: ` For our demo purposes we will build out a simple blog application database. It includes posts, authors, and tags to include some relational data. We need to write the SQL in our migration to create all the tables and columns that we need: ` The SQL above defines our tables, columns, and their relations with foreign keys. It also includes a join table for posts and tags to represent a many-to-many relationship. If you don’t have a lot of experience writing SQL queries it might look a little bit intimidating at first. Once you take some time to learn it it’s actually pretty nice. DataLemur is a pretty great resource for learning SQL. If you need help with a specific query, Copilot and GPT are quite good at generating SQL queries. Just make sure you take some time to try to understand them and check for any potential issues. After completing a migration script it needs to be applied: ` I added the --local flag so that we’re working against a local database for now. Typing our Schema One of the downsides of our ORMless approach is we don’t get TypeScript types out of the box. In a smaller project, I think the easiest approach is just to manage your own types. It’s not hard, you can even have GPT help if you want. If managing type definitions for your schema is not acceptable for your project or use case you can look for a code generation tool or switch to an ORM / toolset that provides types. For this example I created some basic types to map to our schema so that we can get help from the lsp when working with our queries. ` Seeding development data Outside of our migrations, we can write SQL scripts and execute them against our D1 SQLite database using wrangler. To start we can create a simple seeds/dev.sql script to load some initial development seed data into our local database. Another example might be a reset.sql that drops all of our tables so we can easily reset our database during development as we rework the schema or run other experiments. Since our database is using auto incrementing integer ID’s, we can know up front what the ID’s for the rows we are creating are since our database is initially empty. This can be a bit tricky if you’re using randomly generated ID’s. In that case you would probably want to write a script that can collect ID’s of newly created records and use them for creating related records. Here we are just passing the integer ID directly in our SQL script. As an example, we know up front that the author Alice Smith will have the id 1, Bob Johnson 2, and so on. Post_tags looks a little bit crazy since it’s just a join table. Each row is just a post_id and a tag_id. (1, 1), (1 2), etc. Here’s the code for a dev seed script: ` Here’s the code for a reset script - it’s important to remember to drop the migrations table in your reset so you can apply your migrations. ` Using the wrangler CLI we can execute our script files against our local development and remote d1 database instances. Since we have already applied our migrations to our local database, we can use our dev.sql seed script to load some data into our db. ` The Wrangler output is pretty helpful - it lets us know to add the --remote flag to run against our remote instance. We can also use execute to run commands against our database. Lets run a select to look at the data added to our posts table. ` This command should output a table showing the columns of our db and the 7 rows we added from the dev seed script. Summary Using wrangler and the Cloudflare D1 platform we’ve already gotten pretty far without an ORM or any other additional tooling. We have a simple but effective migrations system in place and some initial scripts for easily seeding and resetting our databases. There are also some other really great things built-in to the D1 platform like time travel and backups. I definitely recommend at least skimming through the documentation at some point. In the next post we will start interacting with our database and sample data using the D1 Client API....

Aug 30, 2024

5 mins

Cloudflare

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Intro to EdgeDB - The 10x ORM

Intro to EdgeDB - The 10x ORM

Data Modeling

Defining relations / associations

Computed properties

Globals and Access Policies

Types and Sets

EdgeQL

Selecting properties / data

Adding some objects with insert

Querying data with select

Changing values and links with update

Deleting objects with delete

When the EdgeQL pieces fall into place

Domain and business concerns

Database Clients and Query Builders and Generators

Basic client

TypeScript query builder

Query generator

Tools and Utilities

EdgeDB CLI

Admin UI

Summary

Dane Grant

You might also like

Making Seamless Page Transitions with the View Transitions API

Exploring Angular Forms: A New Alternative with Signals

D1 SQLite: Schema, migrations and seeds

Introduction to Zod for Data Validation

Let's innovate together!

You might also like

Making Seamless Page Transitions with the View Transitions API

Exploring Angular Forms: A New Alternative with Signals

D1 SQLite: Schema, migrations and seeds

Introduction to Zod for Data Validation