General

The Dangers of ORMs and How to Avoid Them

Published Jul 3, 2024

Updated Jul 3, 2024

8 min read

Background

We recently engaged with a client reporting performance issues with their product. It was a legacy ASP .NET application using Entity Framework to connect to a Microsoft SQL Server database. As the title of this post and this information suggest, the misuse of this Object-Relational Mapping (ORM) tool led to these performance issues. Since our blog primarily focuses on JavaScript/TypeScript content, I will leverage TypeORM to share examples that communicate the issues we explored and their fixes, as they are common in most ORM tools.

Our Example Data

If you're unfamiliar with TypeORM I recommend reading their docs or our other posts on the subject. To give us a variety of data situations, we will leverage a classic e-commerce example using products, product variants, customers, orders, and order line items. Our data will be related as follows:

The relationship between customers and products is one we care about, but it is optional as it could be derived from customer->order->order line item->product. The TypeORM entity code would look as follows:

import { Entity, PrimaryGeneratedColumn, Column, OneToMany, ManyToOne, JoinColumn } from 'typeorm';

@Entity()
export class Product {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  name: string;

  @Column({ type: 'text' }) // Use 'text' type for potentially long descriptions 
  description: string;

  @OneToMany(() => ProductVariant, (variant) => variant.product)
  variants: ProductVariant[];

  @OneToMany(() => OrderLineItem, (item) => item.product)
  lineItems: OrderLineItem[];
}

@Entity()
export class ProductVariant {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  name: string;

  @Column()
  price: number;

  @ManyToOne(() => Product, (product) => product.variants)
  @JoinColumn() // Specifies the foreign key column
  product: Product;
}

@Entity()
export class Customer {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  name: string;

  @OneToMany(() => Order, (order) => order.customer)
  orders: Order[];
}

@Entity()
export class Order {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  orderDate: Date;

  @Column()
  amount: number;

  @ManyToOne(() => Customer, (customer) => customer.orders)
  @JoinColumn()
  customer: Customer;

  @OneToMany(() => OrderLineItem, (item) => item.order)
  items: OrderLineItem[];
}

@Entity()
export class OrderLineItem {
  @PrimaryGeneratedColumn()
  id: number;

  @Column()
  quantity: number;

  @ManyToOne(() => Product, (product) => product.lineItems)
  @JoinColumn()
  product: Product;

  @ManyToOne(() => Order, (order) => order.items)
  @JoinColumn()
  order: Order;
}

With this example set, let's explore some common misuse patterns and how to fix them.

N + 1 Queries

One of ORMs' superpowers is quick access to associated records. For example, we want to get a product and its variants. We have 2 options for writing this operation. The first is to join the table at query time, such as:

import { getRepository } from 'typeorm';

async function getProductWithVariants(productId: number) {
  const productRepository = getRepository(Product);
  const product = await productRepository.findOne({ 
    where: { id: productId },
    relations: ['variants']  // Eagerly load variants via join
  });

  if (product) {
    return product; // The product object now includes its variants
  } else {
    throw new Error('Product not found');
  }
}

Which resolves to a SQL statement like (mileage varies with the ORM you use):

SELECT *
FROM Product as p, ProductVariant as pv
WHERE p.id = productId
AND p.id = pv.productId;

The other option is to query the product variants separately, such as:

import { getRepository } from 'typeorm';

async function getProductWithVariants(productId: number) {
  const productRepository = getRepository(Product);
  const productVariantRepository = getRepository(ProductVariant);

  const product = await productRepository.findOne({ where: { id: productId } });

  if (product) {
    const variants = await productVariantRepository.find({
      where: { product: { id: productId } } 
    });

    return { …product, variants };
  } else {
    throw new Error('Product not found');
  }
}

This example executes 2 queries. The join operation performs 1 round trip to the database at ~200ms, whereas the two operations option performs 2 round trips at ~100ms. Depending on the location of your database to your server, the round trip time will impact your decision here on which to use. In this example, the decision to join or not is relatively unimportant and you can implement caching to make this even faster. But let's imagine a different situation/query that we want to run. Let's say we want to fetch a set of orders for a customer and all their order items and the products associated with them. With our ORM, we may want to write the following:

import { getRepository } from 'typeorm';

async function fetchCustomerOrdersWithDetails(customerId: number) {
  const orderRepository = getRepository(Order);
  const orderItemRepository = getRepository(OrderItem);
  const productRepository = getRepository(Product);

  const orders = await orderRepository.find({ 
    where: { customer: { id: customerId } } 
  });

  for (const order of orders) {
    const orderItem = await orderItemRepository.findOne(order);
    order.items.push(orderItem);

    for (const item of order.items) {
      const product = await productRepository.findOne(item.product);
      item.product = product;
    }
  }

  return orders; // All orders with items and products populated
}

When written out like this and with our new knowledge of query times, we can see that we'll have the following performance: 100ms * 1 query for orders + 100ms * order items count + 100ms * order items' products count. In the best case, this only takes 100ms, but in the worst case, we're talking about seconds to process all the data. That's an O(1 + N + M) operation! We can eagerly fetch with joins or collection queries based on our entity keys, and our query performance becomes closer to 100ms for orders + 100ms for order line items join + 100ms for product join. In the worst case, we're looking at 300ms or O(1)!

Normally, N+1 queries like this aren't so obvious as they're split into multiple helper functions. In other languages' ORMs, some accessors look like property lookups, e.g. order.orderItems. This can be achieved with TypeORM using their lazy loading feature (more below), but we don't recommend this behavior by default. Also, you need to be wary of whether your ORM can be utilized through a template/view that may be looping over entities and fetching related records. In general, if you see a record being looped over and are experiencing slowness, you should verify if you've prefetched the data to avoid N+1 operations, as they can be a major bottleneck for performance. Our above example can be optimized by doing the following:

async function fetchCustomerOrdersWithDetails(customerId: number) {
  const orderRepository = getRepository(Order);
  const orderItemRepository = getRepository(OrderItem);
  const productRepository = getRepository(Product);

  const orders = await orderRepository.find({ 
    where: { customer: { id: customerId } } 
  });
  const orderItems = await orderItemRepository.find({
    where: orders.map(order => ({ order: { id: order.id } }))
  });
  const products = await productRepository.find({
    where: orderItems.map(orderItem => ({ id: orderItem.product.id }))
  });

  const orderItemsByOrder = groupBy(orderItems, 'orderId');
  const productsById = indexBy(products, 'id');

  for (const order of orders) {
    order.items = orderItemsByOrder[order.id];

    for (const item of order.items) {
      item.product = productsById[item.productId];
    }
  }

  return orders;
}

function indexBy(arr, key) {
  return arr.reduce((acc, item) => {
    acc[item[key]] = item;
    return acc;
  }, {});
}

function groupBy(arr, key) {
  return arr.reduce((acc, item) => {
    if (!acc[item[key]]) {
      acc[item[key]] = [];
    }
    acc[item[key]].push(item);
    return acc;
  }, {});
}

Here, we prefetch all the needed order items and products and then index them in a hash table/dictionary to look them up during our loops. This keeps our code with the same readability but improves the performance, as in-memory lookups are constant time and nearly instantaneous. For performance comparison, we'd need to compare it to doing the full operation in the database, but this removes the egregious N+1 operations.

Eager v Lazy Loading

Another ORM feature to be aware of is eager loading and lazy loading. This implementation varies greatly in different languages and ORMs, so please reference your tool's documentation to confirm its behavior. TypeORM's eager v lazy loading works as follows:

If eager loading is enabled for a field when you fetch a record, the relationship will automatically preload in memory.
If lazy loading is enabled, the relationship is not available by default, and you need to request it via a key accessor that executes a promise.
If neither is enabled, it defaults to the behavior ascribed above when we explained handling N+1 queries.

Sometimes, you don't want these relationships to be preloaded as you don't use the data. This behavior should not be used unless you have a set of relations that are always loaded together. In our example, products likely will always need product variants loaded, so this is a safe eager load, but eager loading order items on orders wouldn't always be used and can be expensive.

You also need to be aware of the nuances of your ORM. With our original problem, we had an operation that looked like product.productVariants.insert({ … }). If you read more on Entity Framework's handling of eager v lazy loading, you'll learn that in this example, the product variants for the product are loaded into memory first and then the insert into the database happens. Loading the product variants into memory is unnecessary. A product with 100s (if not 1000s) of variants can get especially expensive. This was the biggest offender in our client's code base, so flipping the query to include the ID in the insert operation and bypassing the relationship saved us seconds in performance.

Database Field Performance Issues

Another issue in the project was loading records with certain data types in fields. Specifically, the text type. The text type can be used to store arbitrarily long strings like blog post content or JSON blobs in the absence of a JSON type. Most databases use a technique that stores text fields off rows, which requires a special file system lookup operation to fetch that data. This can make a typical database lookup that would take ~100ms under normal conditions to take ~200ms. If you combine this problem with some of the N+1 and eager loading problems we've mentioned, this can lead to seconds, if not minutes, of query slowdown. For these, you should consider not including the column by default as part of your ORM.

TypeORM allows for this via their hidden columns feature. In this example, for our product description, we could change the definition to be:

@Entity()
export class Product {

  // … omitted for brevity

  @Column({ type: 'text', select: false })
  description: string;

  // … omitted for brevity
}

This would allow us to query products without descriptions quickly. If we needed to include the product description in an operation, we'd have to use the addSelect function to our query to include that data in our result like:

const product = await productRepository
  .addSelect('product.description')
  .findOne({ 
    where: { id: productId },
  });

This is an optimization you should be wary of making in existing systems but one worth considering to improve performance, especially for data reports. Alternatively, you could optimize a query using the select method to limit the fields returned to those you need.

Database v. In-Memory Fetching

Going back to one of our earlier examples, we wrote the following:

const orderItems = await orderItemRepository.find({
  where: orders.map(order => ({ order: { id: order.id } }))
});
const orderItemsByOrder = groupBy(orderItems, 'orderId');

This involves loading our data in memory and then using system memory to perform a group-by operation. Our database could have also returned this result grouped. We opted to perform this operation like this because it made fetching the order item IDs easier. This takes some performance challenges away from our database and puts the performance effort on our servers. Depending on your database and other system constraints, this is a trade-off, so do some benchmarking to confirm your best options here.

This example is not too bad, but let's say we wanted to get all the orders for a customer with an item that cost more than $20. Your first inclination might be to use your ORM to fetch all the orders and their items and then filter that operation in memory using JavaScript's filter method. In this case, you're loading data you don't need into memory. This is a time to leverage the database more. We could write this query as follows in SQL:

SELECT o.*
FROM Order as o, OrderLineItem li
WHERE o.customerId = customerId
AND o.id = li.orderId
AND li.amount > 20

This just loads the orders that had the data that matched our condition. We could write this as:

const orderRepository = getRepository(Order); 
const orders = await orderRepository 
  .createQueryBuilder('o') // Alias for Order 
  .innerJoinAndSelect('o.items', 'li') // Join with OrderLineItem, alias 'li' 
  .where('o.customerId = :customerId', { customerId }) // Filter by customer ID 
  .andWhere('li.amount > 20') // Filter by item cost 
  .getMany();

We constrained this to a single customer, but if it were for a set of customers, it could be significantly faster than loading all the data into memory. If our server has memory limitations, this is a good concern to be aware of when optimizing for performance. We noticed a few instances on our client's implementation where the filter operations were applied in functions that appeared to run the operation in the database but were running the operation in memory, so this was preloading more data into memory than needed on a memory-constrained server. Refer to your ORM manual to avoid this type of performance hit.

Lack of Indexes

The final issue we encountered was a lack of indexes on key lookups. Some ORMs do not support defining indexes in code and are manually applied to databases. These tend not to be documented, so an out-of-sync issue can happen in different environments. To avoid this challenge, we prefer ORMs that support indexes in code like TypeORM. In our last example, we filtered on the cost of an order item, but the cost field does not contain an index. This leads to a full table scan of our data collection filtered by the customer. The query cost can be very expensive if a customer has thousands of orders. Adding the index can make our query super fast, but it comes at a cost. Each new index makes writing to the database slower and can exponentially increase the size of our database needs. You should only add indexes to fields that you are querying against regularly. Again, be sure you can notate these indexes in code so they can be replicated across environments easily.

In our client's system, the previous developer did not include indexes in the code, so we retroactively added the database indexes to the codebase. We recommend using your database's recommended tool for inspection to determine what indexes are in place and keep these systems in sync at all times.

Conclusion

ORMs can be an amazing tool to help you and your teams build applications quickly. However, they have gotchas for performance that you should be aware of and can identify while developing or during code reviews. These are some of the most common examples I could think of for best practices. When you hear the horror stories about ORMs, these are some of the challenges typically discussed. I'm sure there are more, though. What are some that you know? Let us know!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

Dustin Goodman

Engineering Manager with a passion for web and application development. Speaker and writer on work experiences and software development. Dog dad and musician fueled by coffee.

@dustinsgoodman @dustinsgoodman

Remix Deployment with Architecture

Intro Today’s article, will give a brief overview of the Architect framework and how to deploy a Remix app. I’ll cover a few different topics, such as what Architect is, why it’s good to use, and the issues I ran into while using it. It is a straightforward process, and I recommend using it with the Grunge Stack offered by Remix. So let’s jump on in and start talking about Architect. Prerequisites There are a few prerequisites and also some basic understandings that are expected going into this. The first is to have a GitHub account, then an AWS account, and finally some basic understanding of how to deploy. I also recommend checking out the *Grunge Stack* here if you run into any issues when we progress further. What is Architect? First off, Architect is a simple framework for Functional Web Apps (FWAs) on AWS. Now you might be wondering, "why Architect?" It offers the best developer experience, works locally, has infrastructure as code, is secured to the least privilege by default, and is open-source. Architect prioritizes speed, smart configurable defaults, and flexible infrastructure. It also allows users to test things and debug locally. It defines a high-level manifest, and turns a complex cloud infrastructure into a build artifact. It complies the manifest into AWS CloudFormation and deploys it. Since it’s open-source, Architect prioritizes a regular release schedule, and backwards compatibility. Remix deployment Deploying Remix with Architect is rather straightforward, and I recommend using the Grunge Stack to do it. First, we’re going to head on over to the terminal and run *npx create-remix --template remix-run/grunge-stack*. That will get you a Remix template with almost everything you need. *Generating remix template* For the next couple of steps, you need to add your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to your repo’s secrets. You’ll need to create the secrets on your AWS account, and then add them to your repo. *AWS secrets* *GitHub secrets* The last steps before deploying include giving CloudFormation a SESSION_SECRET of its own for both staging and production environments. Then you need to add an ARC_APP_SECRET. *Adding env variables* ` With all of that out of the way, you need to run *npx arc deploy*, which will deploy your build to the staging environment. You could also do *npx arc deploy —-dry-run* beforehand to verify everything is fine. Issues I Had in My Own Project Now let's cover some issues I had, and my solution for my own project. So, I had a project far into development, and while looking at the Grunge Stack, I was scratching my head, trying to figure out what was necessary to bring over to my existing project. I had several instances of trial and error as I plugged in the Architect related items. For instance: the arc file, which has the app name, HTTP routes, static assets for S3 buckets, and the AWS configuration. I also had to change things like the *remix.config* file, adding a *server.ts* file, and adding Architect related dependencies in the package.json. During this process, I would get errors about missing files or missing certain function dirs(), and I dumped a good chunk of time into it. So while continuing to get those errors, I concluded that I would follow the Grunge Stack, and their instructions for a new project. Then I would merge my old project with the new project; and that resolved my issues. While not the best way, it did get me going, and did not waste any more time. That resolved my immediate issues of trying to get my Remix app deployed to AWS, but then I had to figure out what got pushed. That was easy, since it created a lambda, API gateway, and bucket based on the items in my arc file. Then I realized, while testing it out live, that my environmental variables hadn’t carried over, so I had to tweak those on AWS. My project also used a GitHub OAuth app. So that needed to be tweaked with the new URL. Then it was up and running. Conclusion Today’s article covered a brief overview of what Architect is, why it’s good to use, how to deploy a Remix app and the issues you might have. I hope it was useful and will give others a good starting point....

Aug 30, 2022

4 mins

RemixOpen SourceArchitectureAWS

Leveling Up Your Work Through Architecture Design and Time Estimation

> This post can be useful for developers at any level! However, it is written mostly for entry-level developers who are starting to transition into a more intermediate role. You’re rocking your first development job. You’re completing tasks, the team loves you, you still have tons of questions but you can get answers and build what you’re asked to build. But you want to keep improving your skills and giving yourself more options. Some of that skill can only come from actually building things over time. But surely there’s something you can do to level up besides that, right? There is! One way to help deepen your understanding of the software you’re writing and make yourself look more impressive is by improving your ability to estimate your time, and how you talk about your work. That’s what we’re going to dig into today! Some terminology You might have heard of, or have read terms like “software architecture” and “quality attributes” and, upon trying to look into them, been met with a LOT of jargon and dense terminology. So before we get too far into this, let’s go over a few terms we’ll use. (This is by no means meant to be a complete description of these terms. The aim is to be clear enough to continue our discussion here, and let you get started.) - Software architecture: This term means how your system is organized. It’s like the blueprint for the application or website you’re building. How do all the pieces we build fit together and interact with each other? What’s the main goal we’re trying to achieve with this codebase? That’s what this term is talking about. - Software design: Once we have the blueprint, we need to figure out how to build the pieces to make that structure come to life. This is where the design comes in. It’s more code-specific and focuses more on the specifics of what we need to build and how. What languages are we using? How do we get this component to be interactive or usable like we want? This is all the design. - Software quality attributes: Sometimes referred to as “ilities” because a lot of them end with those letters, these are words we use to describe the software we write. Common ones would be accessibility, reliability, or performance. These are words used to describe some of the key qualities we want our software to have. You can start with this list of quality attributes or do a search for “software quality attributes”, and find all sorts of articles covering a lot of the common ones. - Time estimation: This is the skill of being able to consider a task and estimate how long it would take you to complete the task. Time can be a funny thing, and it’s pretty common to think something will be easy and have it take way longer, or think something will be complex, and it turns out to be straightforward. - Trade-offs: This is the idea that two things can’t be equally balanced and/or important. There’s a joke about how people want things fast, cheap, and good and you can’t have all three. This is the same idea. Very often, by increasing one quality, you have to decrease another. The balance of those, and the choice to focus on one quality more than another, is called a trade-off. You’re trading the strength of one quality for another. When we’re using the phrase “architecture design” in this article, we’re combining all these concepts together. How do we get started? Ok, so we’ve got some terms now. But what do we do with them? How do we get better at designing software, and estimating our time? I’ve got two templates to share with you that can help you start to develop these skills. Both will involve writing, but I’ve tried to make them as straightforward to fill out as possible so it’s not a chore to use them. I’ll share links for the templates later on. But first, we’ll go over the actual contents of both of them and talk about how to use them. These templates can be useful for any task, no matter how complex. However, super simple tasks like fixing a broken link on a page, or adjusting a color value to improve its contrast may not be the best choice for it. It is useful to use this template on a task that you fully understand, so you get some practice with thinking about the tradeoffs you’re making, and get used to some of the terminology and how to talk about the software you build. But this can really help when you start getting larger tasks or features- ones that might contain multiple parts working together or some complex logic to get the task working properly. The architecture design template For this template, the goal is to complete this first before you do any work on your task. Use this as a tool to help you think through your work before you get started on it so you can have a better understanding of what the task is asking of you, and let you start thinking about the attributes you’re building for. There are 7 sections for this document, each with a title and description. Let’s go over each piece to see how they work. Big picture: What’s the ask? *What’s the ticket/task trying to solve? What’s the end goal of this body of work? Describe the problem in the most ELI5 way you can.* Use this section to describe the task you’re working on. Act like you’re talking to someone with very little understanding of the situation, like a project manager on another team or to a friend of yours that doesn’t work with you. This section helps you make sure you have a strong handle on the work you’re being asked to do, and what the end result should be. Requirements *Is there acceptance criteria? A specific way it needs to work? What makes this ticket / task count as complete? Steps to reproduce and/or screenshots are helpful here too, if available.* Some of this can often be copied over from the ticket you’re working on. Make note of how you can tell that this task is complete, and any specifics that you need to make sure work as expected when you’re ready to have someone review it. Constraints *Are there specific things you can / can’t use? Something you have to make sure doesn’t break? Tools you have to use because of where you have to work in the codebase?* Making note of constraints can be super helpful. Maybe you have to use a specific UI library to complete this because the rest of the project uses it. Or perhaps this task has been attempted before, and you don’t want to repeat a version that didn’t work. Getting used to thinking about the constraints you’re working within can help prep your brain to think around these concepts. Architecture and Tradeoffs *Where does this code live? (login, ui, db, etc) What software quality attributes are we focused on for this task? List out the primary and secondary attributes (maybe a third if it feels necessary).* In this section, we’re focusing on developing our architecture and design skills in detail. Talk a little about how the codebase is structured, and where the work for this task will be located. Also, pick one or two software quality attributes that you think fit this task. Does the work you’re doing help improve the site’s overall accessibility? Does it make the application more reliable for end users? Does it help to keep our codebase’s maintainability within a reasonable level? Use one of the lists above or your own searches to build out a list of the attributes you might build for regularly, and pick out one or two of them that relate the most to this task. Then, talk a little about why you picked those traits and your thoughts about why they’re applicable. Being able to explain your reasoning here will both help you gain more understanding of the work you’re doing, and the words you’re using to describe it, and give whoever reads this document a better glimpse into the work you do and how you think about it! Initial Gameplan *Considering all the above, what seems like the path forward? What are the steps for how you think it can be solved/completed? Where are you starting?* Now that you’ve spent a little time thinking through the task, and some of the things you should remember about it (like how it’s structured, the tools you have to use, and what’s important to keep in mind while you’re solving it) - write out the steps you think you should take to accomplish the task. It’s perfectly okay to not fully follow this plan as you start to build! We can never fully predict everything that might happen as we start to work in a codebase. The goal here is to give yourself a solid place to start, with all the details we’ve listed fresh in your mind. Process Notes *As you solve this - keep this area updated with rough notes. What did you try that didn’t work? Why not? If you have to pivot, what did you switch to and why? Maybe something worked but you still pivoted - what tradeoffs were made, and what’s the reasoning behind it?* As the description implies, this space is all for you. Try to keep notes as you go. If you do have to change course from your game plan, why and what did you change? Did you find another problem you didn’t even know existed? Or maybe you were able to try a new concept out and it worked! Celebrate your process here. None of this needs to be in complete sentences or easy for anyone but you to understand. This space is all yours to help you keep thinking through the work as you’re doing it! Final Solution *You did it! Form a narrative. Now that you’ve got the issue solved, what does it look like? How did you do it? What did you learn along the way? Is the final solution the same as what you thought, or did the ask pivot along the way? Share screenshots if available.* The final wrap-up! Try to write this similarly to how you started with the big picture. Did your initial plan end up working, and if not why? Were there any interesting plot twists throughout the process? Share the wins, the struggles, and the end result here. Screenshots can be perfect here to see a visual of your work! I also typically leave a little space at the bottom to wrap up any lessons I learned for myself, and leave a little room to talk about my time estimate and reasonings behind how it turned out the way it did. But those are completely optional. --- Because I use a Notion database to help me keep track of these documents, I also have a few properties I can fill out related to the project this document is for, what my time estimate and result were, and the quality attributes I selected. Having those visible at a glance is super helpful for me. The biggest thing I’d recommend to track with this is the date you filled it out. Having that date to help you organize and find your documents is super helpful as your collection grows. I typically title mine based on the project it’s for and the name of the task, but you can name yours whatever you’d like! Now let’s talk about how to track our time for this task. The time estimate template This one is a lot less detailed, but just as important! Being able to improve your knowledge of how long different tasks take you will be a super handy skill as your career continues. The main idea with this template is to keep track of two sections: how long you think something will take you (the estimate), and how long it truly takes you. Our goal is to get those two sections to be as close to the same number as possible. Remember that our goal here is NOT perfection. That’s impossible to reach. Our goal is simply to get them to be close to each other. Most people (even managers) understand that an estimate is not a guarantee. But the closer you can get your estimates to the true time it takes you, the more reliable you seem and the more accurately your team can make progress. There are three main sections to this template. The actual numbers Most importantly, we want an estimated total time and an actual total time, as well as a calculation of the difference between those two numbers. Is our estimate higher, or is the actual total time higher? That difference is what we’re trying to keep as close to zero as possible. I like to break my time tracking down into categories, so I can also see if particular parts of my work take me longer (or shorter) than I think. For each of these, I do the same thing: make an estimate, and record the actual value. The categories I like to track are: Investigation: time to fill out my architecture design, do a little looking into the task, make sure I understand fully what I’m being asked to do, and that I have all the information I need to complete it. Coding: time to do the actual work. Most of my time calculation goes into this category. Testing: I use this for either writing actual unit or end-to-end tests, or for manual testing. This is where I double-check to make sure I didn’t break anything or track time spent on that one last piece of functionality that doesn’t quite work right. Review: any feedback I get on my Pull Requests or code reviews that require me to make changes go into this category. Space to record the numbers as I go The Pomodoro technique works really well for me with time tracking, though I change up my “working time” numbers to make them easier for me to calculate. I’ve found the most straight forward way for me to do this is to have a title for each section of time that I’m tracking, with space underneath each one. Then, I have a legend of colored dot emojis related to a different amount of time: 15 minutes, 30 minutes, and 60 minutes. Then, as I do my rounds of working time, each time I’m done with a round, I select the right colored dot for the amount of time I did, copy it, and paste it under the section the work I did belongs to. Keep repeating this until the work is complete! Once your task is done, all you need to do is count your dots, and record the total number they represent in your actual time section from above. Here’s a screenshot of what one of my completed sections look like, so you can better see what I mean. From this tracking section, I can quickly see that I spent 15 minutes on both testing and review, and an hour and 45 minutes on coding. If I don’t happen to spend any time on a section (in this instance, I’d done some investigating in a separate ticket, so I already knew the work that needed to be done), I just leave it blank. I have a section at the top of my page for tracking the total number for each category. So I use this area to keep track as I’m doing small rounds of work. Then, when I’m done, I add up each number and put it in the section at the top of the page for that category. Notes I also have a section for notes at the bottom of my document. I keep this area for anything relating to my time tracking. Did something break unexpectedly, causing my estimate to be off? Did I not need a section for some reason? Or did something work way better than I thought it would? Those are the kinds of things worth keeping notes of here. Can you tell why your estimate and actual times were off? Sometimes we simply don’t know, but being able to keep notes on the things we can realize as we’re doing them helps us get better the next time! Tying these together and talking about it with others You can use both of these templates together or on their own, and you’ll gain a great amount of knowledge about your skill level and your growth over time from them! But they can also really shine using them together. While these are great tools for your own personal knowledge and growth, I also highly recommend sharing them with someone else. It’s a great idea to set a goal to be better with estimating your time, and sharing that goal with your manager. Then, you can use the time estimation template to build up some estimates, and share those with your manager. Maybe you have a senior developer on your team that you really respect, and you’d like to get their opinion on the task you’re working on. If you’ve filled out the architecture design for that task, you can share it with them and have a conversation about it. They can potentially provide you with things to consider for your next task, or get you thinking more deeply about the quality attributes you selected, and why they may or may not have been the best fit for your work. The design documents are also great to share with your manager! It’s a great way to show that you’re starting to think about your work on a deeper level and starting to consider the quality and complexity of your work. It can also be helpful reference material for them when promotions and new work becomes available. They’re more likely to think you might be a good pick for the next big thing if you’re already showing them you’re starting to think about things at a higher level! The Notion template links The links for both of these documents as Notion templates are below. Please feel free to duplicate a copy for yourself if you like using Notion, or just take a peek at them to see the actual layout of them and adapt that for whatever tool works best for you. These documents are set up to be used within a Notion database. We won’t cover that in detail here, but the Notion documentation should be able to take care of those details for you (and we have a link to get you started below as well). In general, you can create a database within Notion, and then set a template to use for each new entry. That way, when you go to create a new item, it will automatically pull up these templates for you, so you don’t have to copy and paste them every time! And the fields at the top will be visible when you go to look at your database, which makes it a little nicer to get a quick overview of your documents and the progress you’re making. Now go forth and deepen your knowledge! - Time estimation template - Architecture design template - Notion database templates documentation...

Jan 6, 2023

14 mins

Career developmentArchitectureNotion

How to Leverage Apollo Client Fetch Policies Like the Pros

Apollo Client provides a rich ecosystem and cache for interfacing with your GraphQL APIs. You write your query and leverage the useQuery hook to fetch your data. It provides you with some state context and eventually resolves your query. That data is stored in a local, normalized, in-memory cache, which allows Apollo Client to respond to most previously run requests near instantaneously. This has huge benefits for client performance and the feel of your apps. However, sometimes Apollo's default doesn't match the user experience you want to provide. They provide fetch policies to allow you to control this behavior on each query you execute. In this article, we'll explore the different fetch policies and how you should leverage them in your application. cache-first This is the default for Apollo Client. Apollo will execute your query against the cache. If the cache can fully fulfill the request, then that's it, and we return to the client. If it can only partially match your request or cannot find any of the related data, the query will be run against your GraphQL server. The response is cached for the next query and returned to the handler. This method prioritizes minimizing the number of requests sent to your server. However, it has an adverse effect on data that changes regularly. Think of your social media feeds - they typically contain constantly changing information as new posts are generated. Or a real-time dashboard app tracking data as it moves through a system. cache-first is probably not the best policy for these use cases as you won't fetch the latest data from the upstream source. You can lower the cache time of items for the dashboard to avoid the staleness issue and still minimize the requests being made, but this problem will persist for social media feeds. The cache-first policy should be considered for data that does not change often in your system or data that the current user fully controls. Data that doesn't change often is easily cached, and that's a recommended pattern. For data that the user controls, we need to consider how that data changes. If only the current user can change it, we have 2 options: Return the updated data in the response of any mutation which is used to update the cache Use cache invalidation methods like refetchQueries or onQueryUpdated These methods will ensure that our cache stays in sync with our server allowing the policy to work optimally. However, if other users in the system can make changes that impact the current user's view, then we can not invalidate the cache properly using these strategies which makes this policy unideal. network-only This policy skips the cache lookup and goes to the server to fetch the results. The results are stored in the cache for other operations to leverage. Going back to the example I gave in my explanation cache-first of a social media feed, the network-only policy would be a great way to implement the feed itself as it's ever-changing, and we'll likely even want to poll for changes every 10s or so. The following is an example of what this component could look like: ` Whenever this SocialFeed component is rendered, we always fetch the latest results from the GraphQL server ensuring we're looking at the current data. The results are put in the cache which we can leverage in some children components. cache-only cache-only only checks the cache for the requested data and never hits the server. It throws an error if the specified cache items cannot be found. At first glance, this cache policy may seem unhelpful because it's unclear if our cache is seeded with our data. However, in combination with the network-only policy above, this policy becomes helpful. This policy is meant for components down tree from network-only level query. This method is for you if you're a fan of React components' compatibility. We can modify the return of our previous example to be as follows: ` Notice we're not passing the full post object as a prop. This simplifies our Post component types and makes later refactors easier. The Post would like like the following: ` In this query, we're grabbing the data directly from our cache every time because our top-level query should have fetched it. Now, a small bug here makes maintainability a bit harder. Our top-level GetFeed query doesn't guarantee fetching the same fields. Notice how our Post component exports a fragment. Fragments are a feature Apollo supports to share query elements across operations. In our SocialFeed component, we can change our query to be: ` Now, as we change our Post to use new fields and display different data, the refactoring is restricted to just that component, and the upstream components will detect the changes and handle them for us making our codebase more maintainable. Because the upstream component is always fetching from the network, we can trust that the cache will have our data, making this component safe to render. With these examples, though, our users will likely have to see a loading spinner or state on every render unless we add some server rendering. cache-and-network This is where cache-and-network comes to play. With this policy, Apollo Client will run your query against your cache and your GraphQL server. This further simplifies our example above if we want to provide the last fetched results to the user but then update the feed immediately upon gathering the latest data. This is similar to what X/Twitter does when you reload the app. You'll see the last value that was in the cache then it'll render the network values when ready. This can cause a jarring user experience though, if the data is changing a lot over time, so I recommend using this methodology sparsely. However, if you wanted to update our existing example, we'd just change our SocialFeed component to use this policy, and that'll keep our client and server better in sync while still enabling 10s polling. no-cache This policy is very similar to the network-only policy, except it bypasses the local cache entirely. In our previous example, we wrote engagement as a sub-selector on a Post and stored fields there. These metrics can change in real time pretty drastically. Chat features, reactions, viewership numbers, etc., are all types of data that may change in real time. The no-cache policy is good when this type of data is active, such as during a live stream or within the first few hours of a post going out. You may typically want to use the cache-and-network policy eventually but during that active period, you'll probably want to use no-cache so your consumers can trust your data. I'd probably recommend changing your server to split these queries and run different policies for the operations for performance reasons. I haven't mentioned this yet, but you can make the fetch policy on a query dynamic, meaning you combine these different policies' pending states. This could look like the following: ` We pass whether the event is live to the component that then leverages that info to determine if we should cache or not when fetching the chat. That being said, we should consider using subscription operations for this type of feature as well, but that's an exercise for another blog post. standby This is the most uncommon fetch policy, but has a lot of use. This option runs like a cache-first query when it executes. However, by default, this query does not run and is treated like a "skip" until it is manually triggered by a refetch or updateQueries caller. You can achieve similar results by leveraging the useLazyQuery operator, but this maintains the same behavior as other useQuery operators so you'll have more consistency among your components. This method is primarily used for operations pending other queries to finish or when you want to trigger the caller on a mutation. Think about a dashboard with many filters that need to be applied before your query executes. The standby fetch policy can wait until the user hits the Apply or Submit button to execute the operation then calls a await client.refetchQueries({ include: ["DashboardQuery"] }), which will then allow your component to pull in the parameters for your operation and execute it. Again, you could achieve this with useLazyQuery so it's really up to you and your team how you want to approach this problem. To avoid learning 2 ways, though, I recommend picking just one path. Conclusion Apollo Client's fetch policies are a versatile and helpful tool for managing your application data and keeping it in sync with your GraphQL server. In general, you should use the defaults provided by the library, but think about the user experience you want to provide. This will help you determine which policy best meets your needs. Leveraging tools like fragments will enable you to manage your application and use composable patterns more effectively. With the rise of React Server Components and other similar patterns, you'll need to be wary of how that impacts your Apollo Client strategy. However, if you're on a legacy application that leverages traditional SSR patterns, Apollo allows you to pre-render queries on the server and their related cache. When you combine these technologies, you'll find that your apps perform great, and your users will be delighted....

May 17, 2024

7 mins

GraphQL

Migrating From Gatsby to Astro: Migration and Styling Tips with Kathleen McMahon

Kathleen McMahon shares how she migrated her personal site from Gatsby to Astro. She discusses styling Astro Islands and Astro Slots, and their impact on CSS styles on deployment. Viewers also learn about component conversion, and challenges with React Live and display boxes in MDX articles. The transition came with plenty of challenges, but it also highlighted some great benefits of Astro. Here's a look at the hurdles she faced, the specific issues she encountered, and the ultimate wins. Challenges Faced One of the big challenges Kathleen dealt with was styling issues. Astro Islands and Astro Slots are powerful, but they caused unexpected CSS problems on the deployed site. She had to spend time tweaking the stylesheets to make everything look right. Converting components was another tough task. Kathleen's story shows how important it is to pay close attention to detail when switching platforms. Astro's Simplicity and Compatibility Despite the challenges, Kathleen praised Astro for its simplicity and compatibility with her existing setup. The platform's seamless integration with various tools and frameworks made the move smoother. Astro's focus on reducing reliance on JavaScript helped her optimize her website's performance and improve loading times. This simplicity and compatibility make Astro a great option for developers wanting to streamline their workflow. The Importance of Component Conversion Kathleen’s experience highlights how crucial component conversion is. Carefully analyzing and converting each component ensures a smooth transition. The issues with React Live and display boxes in MDX articles show that even minor components can have a big impact on the website’s functionality and look. Attention to detail and thorough testing are key to a successful migration. Conclusion Moving a website from one platform to another is no easy task, as Kathleen McMahon found out during her switch from Gatsby to Astro. The challenges she faced, like unexpected CSS styles and component conversion issues, highlight the need for careful planning and attention to detail. Despite these hurdles, Astro’s simplicity, compatibility, and potential to reduce JavaScript reliance make it a compelling choice for developers. Kathleen’s experience is a valuable lesson for anyone considering a platform migration: meticulous component conversion and a keen eye for detail are essential....

Jul 1, 2024

2 mins

GatsbyAstro

The Dangers of ORMs and How to Avoid Them

Background

Our Example Data

N + 1 Queries

Eager v Lazy Loading

Database Field Performance Issues

Database v. In-Memory Fetching

Lack of Indexes

Conclusion

Dustin Goodman

You might also like

Remix Deployment with Architecture

Leveling Up Your Work Through Architecture Design and Time Estimation

How to Leverage Apollo Client Fetch Policies Like the Pros

Migrating From Gatsby to Astro: Migration and Styling Tips with Kathleen McMahon

You might also like

Remix Deployment with Architecture

Leveling Up Your Work Through Architecture Design and Time Estimation

How to Leverage Apollo Client Fetch Policies Like the Pros

Migrating From Gatsby to Astro: Migration and Styling Tips with Kathleen McMahon