Javascript

Web Scraping with TypeScript and Node.js

Published Apr 21, 2022

Updated Feb 13, 2023

6 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

Sometimes you'll find yourself wanting to use a set of data from a website, but the data won't be available as an API or in a downloadable format like CSV. In these cases, you may have to write a web scraper to download individual web pages and extract the data you want from within their HTML. This guide will teach you the basics of writing a web scraper using TypeScript and Node.js, and will note several of the obstacles you might encounter during web scraping.

If you want to skip straight to the finish code example, check it out on GitHub.

Setup

First things first: we need to initialize our project and install the base dependencies. We'll be writing our web scraper in TypeScript and running it as Node.js scripts using ts-node. For simplicity, we'll create an index.ts file in the project root to work from. From the command line, run the following to get started:

mkdir my-web-scraper && cd my-web-scraper # create project directory
git init # initialize new git repository
echo "node_modules" >> .gitignore # do not track node_modules in git
npm init -y # initialize Node.js project
# install dependencies
npm install typescript ts-node
npm install --save-dev @types/node
touch index.ts # create an empty TypeScript file

Node.js doesn't run TypeScript files natively. Rather than use the TypeScript compiler to output new JavaScript files whenever we want to run the script, we'll use ts-node to run the TypeScript files directly. We'll go ahead and add this to our new package.json file as an npm script.

  "scripts": {
    "scrape": "ts-node ./index.ts"
  }

Now, we'll be able to run our scraper from index.ts with the command npm run scrape.

Fetching Websites

In our examples, we'll be using Axios to make our http requests. If you'd prefer something else, like Node Fetch to match the Fetch API until it's ready in Node.js, that's fine too.

npm install axios

Let's create our first function for fetching a given URL and returning the HTML from that page.

import axios from 'axios';

function fetchPage(url: string): Promise<string | undefined> {
  const HTMLData = axios
    .get(url)
    .then(res => res.data)
    .catch((error: AxiosError) => {
      console.error(`There was an error with ${error.config.url}.`);
      console.error(error.toJSON());
    });

  return HTMLData;
}

This function will use Axios to create a promise to fetch a given URL, and return the HTML it gets back as a string. If there's an error, it will log that error to the console, and return undefined instead. Since you're probably going to be running this scraper from your command line throughout development, a healthy number of console.logs will help you make sure the script is running as expected.

Caching Scraped Pages

In the event that you're trying to scrape many, many static web pages in a single script, you might want to cache the pages locally as you download them. This will save you time and headache as you work on your scraper. You're much less likely to annoy the website you're scraping with high traffic and the bandwidth costs associated with it, and your scripts will probably run faster if they aren't limited by your Internet connection.

Let's go ahead and create a .cache folder in the project root. You probably won't want to keep cached files in your git history, so we'll want to add this folder to your .gitignore file.

mkdir .cache
echo ".cache" >> .gitignore

To cache our results, we'll first check if a cached version of the given page already exists. If so, we'll use that. If not, we'll fetch the page and save it to the .cache folder. For filenames, we're just going to base-64 encode the page's URL. If you prefer some other way to generate a unique filename, that's fine too. I've chosen the base-64 encoded URLs because it's easy and very obviously a temporary sort of file. We also have an optional function argument ignoreCache, in case you've built up your cache but want to scrape fresh data anyway.

import { existsSync, mkdirSync } from 'fs';
import { readFile, writeFile } from 'fs/promises';
import { resolve } from 'path';

async function fetchFromWebOrCache(url: string, ignoreCache = false) {
  // If the cache folder doesn't exist, create it
  if (!existsSync(resolve(__dirname, '.cache'))) {
    mkdirSync('.cache');
  }
  console.log(`Getting data for ${url}...`);
  if (
    !ignoreCache &&
    existsSync(
      resolve(__dirname, `.cache/${Buffer.from(url).toString('base64')}.html`),
    )
  ) {
    console.log(`I read ${url} from cache`);
    const HTMLData = await readFile(
      resolve(__dirname, `.cache/${Buffer.from(url).toString('base64')}.html`),
      { encoding: 'utf8' },
    );
    return HTMLData;
  } else {
    console.log(`I fetched ${url} fresh`);
    const HTMLData = await fetchPage(url);
    if (!ignoreCache && HTMLData) {
      writeFile(
        resolve(
          __dirname,
          `.cache/${Buffer.from(url).toString('base64')}.html`,
        ),
        HTMLData,
        { encoding: 'utf8' },
      );
    }
    return HTMLData;
  }
}

Extracting Data with jsdom

Now that we have HTML to work with, we want to extract the relevant data from it. To do this, we will use jsdom, a JavaScript implementation of the DOM. This will let us interact with the downloaded HTML in the exact same way as if we were working in a browser's console, giving access to methods like querySelector.

(If you prefer a syntax more like jQuery's, Cheerio is also a popular option.)

npm install jsdom
npm install --save-dev @types/jsdom

Now let's import jsdom and use it to return the Document object of our HTML string. Just modify the previous fetchFromWebOrCache to turn HTMLData into a DOM object, and return its window.document.

import { JSDOM } from 'jsdom';

async function fetchFromWebOrCache(url: string, ignoreCache = false) {
  // Get the HTMLData from fetching or from cache
  const HTMLData = '<html>...</html>'
  const dom = new JSDOM(HTMLData);
  return dom.window.document;
}

Now that we're working with a Document instead of a string, we've got access to everything we'd have if we were working in the browser console. This makes it much easier to write code that extracts the pieces of a page that we want! For example, let's scrape whatever is on the front page of Hacker News right now. We'll write a function that accepts the Document of the Hacker News front page, finds all of the links, and gives us back the link text and URL as a JavaScript object.

Using your browser's developer tools, you can easily inspect an element on the page with desired data to figure out a selector path. In our example, we can right-click a link and choose Inspect to view it in DevTools. Then we right-click the DOM element, and choose "Copy > Copy selector" in Chrome or "Copy > CSS Selector" in Firefox, for example.

A copied selector will give you a string of text that selects only the element you copied it from in DevTools. And often that is useful! Just throw your selector into document.querySelector('selector'), and you're good to go. But in our case, we want all of the front page links. So we need a broader selector than copy-pasting from DevTools will give us. This is where you'll have to actually read through the HTML, classes, ids, etc., to figure out how to craft the right selector.

Fortunately for us in this example, all of the links on the Hacker News feed have a unique class: titlelink. So we can use document.querySelectorAll('a.titlelink') to get all of them.

// Pass the scraped Document from news.ycombinator.com to this
// function to extract data about front page links.
function extractData(document: Document) {
  const writingLinks: HTMLAnchorElement[] = Array.from(
    document.querySelectorAll('a.titlelink'),
  );
  return writingLinks.map(link => {
    return {
      title: link.text,
      url: link.href,
    };
  });
}

This function is only a simple example, and would be different depending on what you want to get out of a page. When working with jsdom, remember that you're not working with arrays and objects but with NodeLists and Elements. To get useful data out of your selections, you'll often have to do things like convert a NodeList into an array as shown above.

Sometimes you'll have to get creative with your selections. I recently tried to scrape the information from an HTML table on a page with varying numbers of tables and no classes. Because the number of tables was always different, I couldn't reliably select from a list of tables by which number table it was. I had to select every table present on a page, then filter them by the text in the first cell to get precisely the one table I needed!

// Sometimes, web scraping is just hard...
const table: HTMLTableElement = Array.from(
    data.querySelectorAll('table'),
  ).filter(t =>
    t.children[0].children[0].children[0].innerHTML.match(
      /Unique Text in First Cell which IDs the Table/,
    ),
  )[0];

Extracting Data with Regular Expressions

Unfortunately for us, not all pages on the Internet are well-structured and ready for scraping. Sometimes, they don't even try to use HTML tags properly. In these sad cases, you may need to turn to regular expressions (regex) to extract what you need. We won't need to resort to such extreme measures in our example of scraping Hacker News, but it's worth knowing that you might need to do this.

I'll give you a contrived example where you would need some regex, based on another site I recently scraped. Imagine the following badly-done HTML:

<div class="pokemon">
  Name: Pikachu<br />
  Number: 25<br />
  Type: Electric<br />
  Weakness: Ground
</div>

The various data attributes we care about aren't wrapped by their own HTML elements! Everything is just inside a div with some br tags to create line breaks. If I wanted to extract the data from this, I could use regex to find and match the text and patterns I expect to find. This can require trial and error, and I recommend using a tool like regex101 to test the regular expressions you come up with. In this example, we might write the following code:

const rawPokemonHTML = document.querySelector('.pokemon');
const name = rawPokemonHTML.match(/Name: (\w+)/)[0];
const num = rawPokemonHTML.match(/Number: (\d+)/)[0];
// etc...

Saving Data

Once we've extracted our data from the HTML, we'll want to save it. This is basically the same as when we created a cache for the downloaded HTML files.

import { existsSync, mkdirSync } from 'fs';
import { writeFile } from 'fs/promises';
import { resolve } from 'path';

function saveData(filename: string, data: any) {
  if (!existsSync(resolve(__dirname, 'data'))) {
    mkdirSync('data');
  }
  writeFile(resolve(__dirname, `data/${filename}.json`), JSON.stringify(data), {
    encoding: 'utf8',
  });
}

Putting It All Together

Now that we've got all the necessary pieces, we're ready to build our JSON file of Hacker News front page stories. To see all of our code in one piece, check it out on GitHub.

async function getData() {
  const document = await fetchFromWebOrCache(
    'https://news.ycombinator.com/',
    true, // Hacker News is always changing, so ignore the cache!
  );
  const data = extractData(document);
  saveData('hacker-news-links', data);
}

getData();

When we run our script from the command line, it will execute getData(). That function will fetch the HTML from Hacker News' front page, extract all of the links and their titles, and then save it to data/hacker-news-links.json. And while you probably don't need a list of links from Hacker News, this information should be enough to get you started with collecting some data from the web which you do care about.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Tom VanAntwerp
@tvanantwerp @tvanantwerp

Exploring Angular Forms: A New Alternative with Signals

Exploring Angular Forms: A New Alternative with Signals In the world of Angular, forms are essential for user interaction, whether you're crafting a simple login page or a more complex user profile interface. Angular traditionally offers two primary approaches: template-driven forms and reactive forms. In my previous series on Angular Reactive Forms, I explored how to harness reactive forms' power to manage complex logic, create dynamic forms, and build custom form controls. A new tool for managing reactivity - signals - has been introduced in version 16 of Angular and has been the focus of Angular maintainers ever since, becoming stable with version 17. Signals allow you to handle state changes declaratively, offering an exciting alternative that combines the simplicity of template-driven forms with the robust reactivity of reactive forms. This article will examine how signals can add reactivity to both simple and complex forms in Angular. Recap: Angular Forms Approaches Before diving into the topic of enhancing template-driven forms with signals, let’s quickly recap Angular's traditional forms approaches: 1. Template-Driven Forms: Defined directly in the HTML template using directives like ngModel, these forms are easy to set up and are ideal for simple forms. However, they may not provide the fine-grained control required for more complex scenarios. Here's a minimal example of a template-driven form: ` ` 2. Reactive Forms: Managed programmatically in the component class using Angular's FormGroup, FormControl, and FormArray classes; reactive forms offer granular control over form state and validation. This approach is well-suited for complex forms, as my previous articles on Angular Reactive Forms discussed. And here's a minimal example of a reactive form: ` ` Introducing Signals as a New Way to Handle Form Reactivity With the release of Angular 16, signals have emerged as a new way to manage reactivity. Signals provide a declarative approach to state management, making your code more predictable and easier to understand. When applied to forms, signals can enhance the simplicity of template-driven forms while offering the reactivity and control typically associated with reactive forms. Let’s explore how signals can be used in both simple and complex form scenarios. Example 1: A Simple Template-Driven Form with Signals Consider a basic login form. Typically, this would be implemented using template-driven forms like this: ` ` This approach works well for simple forms, but by introducing signals, we can keep the simplicity while adding reactive capabilities: ` ` In this example, the form fields are defined as signals, allowing for reactive updates whenever the form state changes. The formValue signal provides a computed value that reflects the current state of the form. This approach offers a more declarative way to manage form state and reactivity, combining the simplicity of template-driven forms with the power of signals. You may be tempted to define the form directly as an object inside a signal. While such an approach may seem more concise, typing into the individual fields does not dispatch reactivity updates, which is usually a deal breaker. Here’s an example StackBlitz with a component suffering from such an issue: Therefore, if you'd like to react to changes in the form fields, it's better to define each field as a separate signal. By defining each form field as a separate signal, you ensure that changes to individual fields trigger reactivity updates correctly. Example 2: A Complex Form with Signals You may see little benefit in using signals for simple forms like the login form above, but they truly shine when handling more complex forms. Let's explore a more intricate scenario - a user profile form that includes fields like firstName, lastName, email, phoneNumbers, and address. The phoneNumbers field is dynamic, allowing users to add or remove phone numbers as needed. Here's how this form might be defined using signals: ` > Notice that the phoneNumbers field is defined as a signal of an array of signals. This structure allows us to track changes to individual phone numbers and update the form state reactively. The addPhoneNumber and removePhoneNumber methods update the phoneNumbers signal array, triggering reactivity updates in the form. ` > In the template, we use the phoneNumbers signal array to dynamically render the phone number input fields. The addPhoneNumber and removePhoneNumber methods allow users to reactively add or remove phone numbers, updating the form state. Notice the usage of the track function, which is necessary to ensure that the ngFor directive tracks changes to the phoneNumbers array correctly. Here's a StackBlitz demo of the complex form example for you to play around with: Validating Forms with Signals Validation is critical to any form, ensuring that user input meets the required criteria before submission. With signals, validation can be handled in a reactive and declarative manner. In the complex form example above, we've implemented a computed signal called formValid, which checks whether all fields meet specific validation criteria. The validation logic can easily be customized to accommodate different rules, such as checking for valid email formats or ensuring that all required fields are filled out. Using signals for validation allows you to create more maintainable and testable code, as the validation rules are clearly defined and react automatically to changes in form fields. It can even be abstracted into a separate utility to make it reusable across different forms. In the complex form example, the formValid signal ensures that all required fields are filled and validates the email and phone numbers format. This approach to validation is a bit simple and needs to be better connected to the actual form fields. While it will work for many use cases, in some cases, you might want to wait until explicit "signal forms" support is added to Angular. Tim Deschryver started implementing some abstractions around signal forms, including validation and wrote an article about it. Let's see if something like this will be added to Angular in the future. Why Use Signals in Angular Forms? The adoption of signals in Angular provides a powerful new way to manage form state and reactivity. Signals offer a flexible, declarative approach that can simplify complex form handling by combining the strengths of template-driven forms and reactive forms. Here are some key benefits of using signals in Angular forms: 1. Declarative State Management: Signals allow you to define form fields and computed values declaratively, making your code more predictable and easier to understand. 2. Reactivity: Signals provide reactive updates to form fields, ensuring that changes to the form state trigger reactivity updates automatically. 3. Granular Control: Signals allow you to define form fields at a granular level, enabling fine-grained control over form state and validation. 4. Dynamic Forms: Signals can be used to create dynamic forms with fields that can be added or removed dynamically, providing a flexible way to handle complex form scenarios. 5. Simplicity: Signals can offer a simpler, more concise way to manage form states than traditional reactive forms, making building and maintaining complex forms easier. Conclusion In my previous articles, we explored the powerful features of Angular reactive forms, from dynamic form construction to custom form controls. With the introduction of signals, Angular developers have a new tool that merges the simplicity of template-driven forms with the reactivity of reactive forms. While many use cases warrant Reactive Forms, signals provide a fresh, powerful alternative for managing form state in Angular applications requiring a more straightforward, declarative approach. As Angular continues to evolve, experimenting with these new features will help you build more maintainable, performant applications. Happy coding!...

Oct 4, 2024

5 mins

AngularJavaScript

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

Upgrading from Astro 2 to Astro 4

Upgrading from Astro 2 to Astro 4 Astro is building fast. Right on the heels of their version 3 launch on August 30th, Astro version 4 launched on December 6th, 2023. They've built so fast, that I didn't even have a chance to try 3 before 4 came out! But the short time between versions makes sense, because the two releases are very complementary. Many Astro features introduced in version 3 as experimental are made stable by version 4. If, like me, you're looking at a two-version upgrade, here's what you need to know about Astro 3 and 4 combined. View Transitions Astro makes it easy to include animated transitions between routes and components with the component. You can add it to the of specific pages, or to your site-wide to be enabled across the entire site. No configuration is required, but you'll probably still want to configure it. Adding between pages effectively turns your site into a single-page application, animating in and out content on route change rather than downloading a new statically generated HTML document. You can further customize how specific components animate in and out with the transition:animate property. If you don't want client side routing for a specific link, you can opt out for that link with the data-astro-reload property. Image Optimization The way Astro works with images has changed a lot from version 2. If you were using @astrojs/image, then updating how you handle images is probably going to be the most time-consuming part of your Astro migration. Astro's and components have had API changes, which will require you to make changes in your usage of them. You should definitely check out the full image migration guide in the Astro docs for all of the details. But some of the details you should be aware of: - @astrojs/image is out, astro:assets is in - You get image optimization when you import images inside /src. This can change your entire image src referencing strategy. Optimization only works for images imported from inside /src, so you might want to relocate images you've been keeping inside the /public directory. - Importing an image file no longer returns the path as a string, and an ImageMetadata object with src, width, height, and format properties. If you need the previous behavior, add ?url to the import path. - Markdown documents can reference image paths inside /src for automatic image optimization, no need to use in MDX nor reference the root relative paths of images in the /public directory. - The and components have changes to properties. For example, aspectRatio is no longer a valid property because it is inferred from the width and height of images. A new pictureAttributes property lets you do things like add a CSS style string. - You can use a helper image schema validator in your content collection schemas. Dev Toolbar The Dev Toolbar is a new local development feature to help developers work with their interactive islands and to integrate with other tools. The Inspect option highlights what parts of the page are interactive, and lets you examine them. You can view their props and even open them directly in your editor. Audit checks for accessibility issues, such as missing alt attributes in images. And the Menu lets you use specific integrations. Currently only Storyblok and spotlight are available, but you can expect more integrations in the future. And if you don't want to wait, you can also extend the Dev Toolbar yourself with their API. If you don't like the Dev Toolbar, you can turn it off in the CLI with astro preferences disable devToolbar. Conclusion Astro has added a lot of cool features in 2 major back-to-back releases, and you should absolutely consider upgrading if you're still on version 2. Be prepared to modify how you've handled images, but also get excited to play with view transitions!...

Dec 22, 2023

3 mins

Astro

Learning Paths for Next.JS Developers with Ankita Kulkarni

In this video, Rob Ocel and co-hosts Tracy Lee, Adam Rackis, and Danny Thompson talk with tech educator Ankita Kulkarni about her journey from engineering leader to full-time educator. Ankita shares insights on teaching Next.js, bridging practical knowledge gaps, and helping developers tackle real-world challenges. They discuss Next.js as a React-based framework, its benefits, and the challenges it presents for beginners. Chapters - Introduction to the Podcast and Guests 00:01 - Meet Ankita Kulkarni, Tech Educator 00:26 - Ankita's Transition to Full-Time Education 01:41 - Teaching Practical Knowledge in Next.js 03:19 - Effective Methods for Teaching Next.js 05:27 - Challenges of Being a Full-Time Educator 07:48 - Balancing Broad and Specific Examples 09:54 - Embracing Mistakes as a Teaching Tool 12:13 - Pair Programming and Mentorship 14:00 - Discussion on Next.js and Framework Adoption 16:48 - Advantages and Challenges of Next.js 18:12 - Choosing the Right Framework for Your Needs 20:35 - Impact of Next.js in React Documentation 22:26 - Learning Paths for New Developers 23:24 - The Rise of Full-Stack Web Development 25:09 - Benefits of Frameworks Abstracting Complexity 26:27 - OpenNext and Deployment Flexibility 28:06 - Ankita's Excitement for New Next.js Features 30:35 - The Future of Next.js Without Vercel 32:16 - Final Thoughts and Where to Find Everyone Online 34:21 Follow Ankita Kulkarni on Social Media Twitter: https://x.com/kulkarniankita9 YouTube: https://www.youtube.com/@kulkarniankita Sponsored by This Dot: thisdot.co...

Nov 12, 2024

2 mins

NextJSJavaScript

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Web Scraping with TypeScript and Node.js

Setup

Fetching Websites

Caching Scraped Pages

Extracting Data with jsdom

Extracting Data with Regular Expressions

Saving Data

Putting It All Together

Tom VanAntwerp

You might also like

Exploring Angular Forms: A New Alternative with Signals

Introduction to Zod for Data Validation

Upgrading from Astro 2 to Astro 4

Learning Paths for Next.JS Developers with Ankita Kulkarni

Let's innovate together!

You might also like

Exploring Angular Forms: A New Alternative with Signals

Introduction to Zod for Data Validation

Upgrading from Astro 2 to Astro 4

Learning Paths for Next.JS Developers with Ankita Kulkarni