General

Build a Next-Gen Chat App with AI and WebSockets in Just 3 Hours with Ben Lesh and Tracy Lee

Published May 30, 2024

Updated Dec 11, 2024

1 min read

In this comprehensive 3-hour training session, Tracy Lee and Ben Lesh guided participants through the process of building a chat application using various cutting-edge technologies and methodologies.

The training began with an introduction to using v0.dev, followed by leveraging AI tools like GitHub Copilot and ChatGPT to scaffold the initial version of the application. This segment highlighted the practical use of AI in real-time coding, demonstrating how these tools can significantly streamline development workflows.

Next, the session delved into creating a monorepo using Nx, showcasing how to efficiently manage multiple projects and libraries within a single repository.

Participants then learned how to connect their application to the OpenAI API, incorporating AI functionalities. The training continued with a focus on refactoring the application to enable real-time streaming using WebSockets. This segment was particularly insightful for developers, as it covered essential techniques for implementing WebSocket functionality in real-world scenarios.

Throughout the training, attendees had the opportunity to observe Ben Lesh's debugging process, gaining practical knowledge on troubleshooting and optimizing code.

The session concluded with integrating the application with AstraDB and Cohere to implement the retrieval part of Retrieval-Augmented Generation (RAG). This final step equipped the chat app with the capability to understand and utilize RxJS, bringing the project to fruition.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

Detect Hand Sign Languages with Tensorflow

Interested in learning how to use Tensorflow to detect hand sign languages in your apps? By the end of this read, you will know how to implement Tensorflow in your application with very simple steps. In our example today, we will be using Vue. What is Tensorflow? Tensorflow is an end-to-end platform _(meaning: delivering complex systems or services in functional form after developing it from beginning to end.)_ used for building Machine Learning applications, and it is also open-source. TensorFlow enables you to build dataflow graphs and structures to define how data moves through a graph by taking inputs as a multi-dimensional array called Tensor. You can read more on Tensorflow here. What is a Model? A model is a function with learnable parameters that maps an input to an output. A well-trained model will provide an accurate mapping from the input to the desired output. Tensorflow Models Tensorflow models are pre-trained models, and there are four defined categories of them: - Vision: Analyze features in images and videos. - Body: Detect key points and poses on the face, hands, and body with models from MediPipe - Text: Enable NLP in your web app using the power of BERT and other Transformer encoder architectures. - Audio: Classify audio to detect sounds. If you want to go into more detail, check out Tensorflow Models. All these models are broken down into subs and for our case, we will be making use of the Body Model which has the hand pose detection we need in order to detect the hand signs. Hand Pose Detection This model used a 2D and 3D multi-dimensional array which enables it to predict the keypoints of the hands. Example of a 2D is [[1,2],[3,5],[7,8],[20,44]] and that of a 3D is [[1,2,5],[3,5,8],[7,8,6],[20,44,100]]. This hand pose detection is a model from the MediPipe as we established above, and it provides us with two model types which are lite and full. The accuracy of the prediction increases from lite to full while the inference speed reduces, i.e. the response time will be slower as the accuracy increases. What do we need? There are a few dependencies we need to get things working, and I also will be assuming that you have your project set up as well. You will need to add these dependencies to the project ` Above, in the commands, you will notice we added a fingerpose. Let's talk a little about what we need the figerpose for. Fingerpose Fingerpose is a gesture classifier for hand landmarks detected by Mediapipe hand pose detection. It also allows you to add your own hand gesture, which means that a gesture that signifies the letter Z can signify Hello based on your fingerpose data. We will see an example of how the data looks in a bit. You can check out fingerpose for more details. Get started We are going to use Vue for this illustration. We will start by looking at the HTML first, and then we will cover the JavaScript. Our Template will be a basic HTML that will have a video tag so we can show a video after getting access to our webcam. Template ` The snippet above shows a div and a video tab. The video is used when we gain access to the webcam. We will now be writing the JS required to initialize the webcam. Script ` We imported two methods from vue: onMounted and ref. The onMounted runs when the page is fully mounted while the ref is used to declare a reactive value to reference the video element. If you look at the video tag in the template, you will notice a ref property. You can check out Template ref and onMounted lifecycle hook. In the openCam function, we first try to test if mediaDevices is available on your browser navigation. >The MediaDevices interface provides access to connected media input devices like cameras and microphones, as well as screen sharing. In essence, it lets you obtain access to any hardware source of media data. This MediaDevice has a method getUserMedia which prompts the user for permission to use a media input. You can find all you need to know about getUserMedia here. From the snippet, we can see that getUserMedia returns a promise, and with that, we can get the media stream as a response using then(). We check if the video element has srcObject or not. If it does we assign the media stream to the srcObject and if not, we convert the media stream to a URL and assign it to the src of the video element. With this Snippet and with a few style, you should have your video showing your awesome face! Introducing Tensorflow and Hand Detection Now that we got our webcam working, we will update the Template and the script in order to detect, predict, and display the alphabet based on the hand sign prediction. The updated HTML should now look like this: ` The div with class name alphabet will display the alphabet based on the hand sign prediction. We will be introducing two(2) new functions, createDetectionInstance and handleSignDetection. Firstly, lets begin with the createDetectionInstance which is an integral part of the hand sign detection and then we will introduce handleSignDetection which predicts and displays the hand sign. ` To be able to detect hand poses, we need to create an instance of the handpose detector, and here, we created a function createDetectionInstance which is an asynchronous function. You can check out this Tensorflow blog to see more details. Now that we have created an avenue to detect hand signs, let us start detecting the hand. In that light, we will be adding a handleSignDetection function. ` The handleSignDetection runs after creating the detection instance. We have a setInterval that runs every 2 seconds (PS: _the 2 seconds timing is arbitrary and can be less or more_) to check if there is any hand sign. We also have a conditional statement to ensure the video element exists, and the detection instance was created accordingly. So, the detector calls a method estimateHands, which tries to predict the hand pose by getting keypoints with values that are either in 2D or 3D (Multi-dimensional Array). If you check your console log, you will see an array of data if any hand pose is detected. Now that we can detect hand poses, we will now add fingerpose that will help predict and display the alphabet based on the hand sign. ` Assuming that our detector sensed a hand, it is time to match this value based on the hand signs we created with the fingerpose. The landmark variable is a 3D array pulled from the hand result's keypoint3D key value. There is also a keypoint as well, which is a 2D value, and both will give the same result. Now, using GE.estimate, we can generate a possible gesture that matches the sign, and a score/confidence is assigned to each gesture pending the amount of gesture predicted. So, the gesture with the highest score/confidence is selected since it is estimated to be the closest to the hand sign from the figerpose hand signs we created. We also imported Handsigns and its content looks like this: You can also get the handsigns folder from the 100-ms-vue repository. Looking at the screenshot, there is a GestureDescription instance that takes a string A which will represent what the hand sign will stand for. So, it could be anything you want the handsign to stand for. >onMounted is asynchronous because we need to ensure that our detection instance is created, which is required to detect the hand sign. With the updated code, you should be able to display some letters. Conclusion Don't forget, you can see in detail how this was implemented in one of This Dot Labs' open source projects 100-ms-vue. Please note that what we did is just a basic implementation, and to have a production-ready version, it will need a bigger model, and a more complex detection to be able to identify hand sign language....

Jul 12, 2023

7 mins

Artificial IntelligenceTensorflowMachine Learning

How to build an AI assistant with OpenAI, Vercel AI SDK, and Ollama with Next.js

In today’s blog post, we’ll build an AI Assistant using three different AI models: Whisper and TTS from OpenAI and Llama 3.1 from Meta. While exploring AI, I wanted to try different things and create an AI assistant that works by voice. This curiosity led me to combine OpenAI’s Whisper and TTS models with Meta’s Llama 3.1 to build a voice-activated assistant. Here’s how these models will work together: * First, we’ll send our audio to the Whisper model, which will convert it from speech to text. * Next, we’ll pass that text to the Llama 3.1 model. Llama will understand the text and generate a response. * Finally, we’ll take Llama’s response and send it to the TTS model, turning the text back into speech. We’ll then stream that audio back to the client. Let’s dive in and start building this excellent AI Assistant! Getting started We will use different tools to build our assistant. To build our client side, we will use Next.js. However, you could choose whichever framework you prefer. To use our OpenAI models, we will use their TypeScript / JavaScript SDK. To use this API, we require the following environmental variable: OPENAI_API_KEY— To get this key, we need to log in to the OpenAI dashboard and find the API keys section. Here, we can generate a new key. Awesome. Now, to use our Llama 3.1 model, we will use Ollama and the Vercel AI SDK, utilizing a provider called ollama-ai-provider. Ollama will allow us to download our preferred model (we could even use a different one, like Phi) and run it locally. The Vercel SDK will facilitate its use in our Next.js project. To use Ollama, we just need to download it and choose our preferred model. For this blog post, we are going to select Llama 3.1. After installing Ollama, we can verify if it is working by opening our terminal and writing the following command: Notice that I wrote “llama3.1” because that’s my chosen model, but you should use the one you downloaded. Kicking things off It's time to kick things off by setting up our Next.js app. Let's start with this command: ` After running the command, you’ll see a few prompts to set the app's details. Let's go step by step: * Name your app. * Enable app router. The other steps are optional and entirely up to you. In my case, I also chose to use TypeScript and Tailwind CSS. Now that’s done, let’s go into our project and install the dependencies that we need to run our models: ` Building our client logic Now, our goal is to record our voice, send it to the backend, and then receive a voice response from it. To record our audio, we need to use client-side functions, which means we need to use client components. In our case, we don’t want to transform our whole page to use client capabilities and have the whole tree in the client bundle; instead, we would prefer to use Server components and import our client components to progressively enhance our application. So, let’s create a separate component that will handle the client-side logic. Inside our app folder, let's create a components folder, and here, we will be creating our component: ` Let’s go ahead and initialize our component. I went ahead and added a button with some styles in it: ` And then import it into our Page Server component: ` Now, if we run our app, we should see the following: Awesome! Now, our button doesn’t do anything, but our goal is to record our audio and send it to someplace; for that, let us create a hook that will contain our logic: ` We will use two APIs to record our voice: navigator and MediaRecorder. The navigator API will give us information about the user’s media devices like the user media audio, and the MediaRecorder will help us record the audio from it. This is how they’re going to play out together: ` Let’s explain this code step by step. First, we create two new states. The first one is for keeping track of when we are recording, and the second one stores the instance of our MediaRecorder. ` Then, we’ll create our first method, startRecording. Here, we are going to have the logic to start recording our audio. We first check if the user has media devices available thanks to the navigator API that gives us information about the browser environment of our user: If we don’t have media devices to record our audio, we just return. If they do, then let us create a stream using their audio media device. ` Finally, we go ahead and create an instance of a MediaRecorder to record this audio: ` Then we need a method to stop our recording, which will be our stopRecording. Here, we will just stop our recording in case a media recorder exists. ` We are recording our audio, but we are not storing it anywhere. Let’s add a new useEffect and ref to accomplish this. We would need a new ref, and this is where our chunks of audio data will be stored. ` In our useEffect we are going to do two main things: store those chunks in our ref, and when it stops, we are going to create a new Blob of type audio/mp3: ` It is time to wire this hook with our AudioRecorder component: ` Let’s go to the other side of the coin, the backend! Setting up our Server side We want to use our models on the server to keep things safe and run faster. Let’s create a new route and add a handler for it using route handlers from Next.js. In our App folder, let’s make an “Api” folder with the following route in it: We want to use our models on the server to keep things safe and run faster. Let’s create a new route and add a handler for it using route handlers from Next.js. In our App folder, let’s make an “Api” folder with the following route in it: ` Our route is called ‘chat’. In the route.ts file, we’ll set up our handler. Let’s start by setting up our OpenAI SDK. ` In this route, we’ll send the audio from the front end as a base64 string. Then, we’ll receive it and turn it into a Buffer object. ` It’s time to use our first model. We want to turn this audio into text and use OpenAI’s Whisper Speech-To-Text model. Whisper needs an audio file to create the text. Since we have a Buffer instead of a file, we’ll use their ‘toFile’ method to convert our audio Buffer into an audio file like this: ` Notice that we specified “mp3”. This is one of the many extensions that the Whisper model can use. You can see the full list of supported extensions here: https://platform.openai.com/docs/api-reference/audio/createTranscription#audio-createtranscription-file Now that our file is ready, let’s pass it to Whisper! Using our OpenAI instance, this is how we will invoke our model: ` That’s it! Now, we can move on to the next step: using Llama 3.1 to interpret this text and give us an answer. We’ll use two methods for this. First, we’ll use ‘ollama’ from the ‘ollama-ai-provider’ package, which lets us use this model with our locally running Ollama. Then, we’ll use ‘generateText’ from the Vercel AI SDK to generate the text. Side note: To make our Ollama run locally, we need to write the following command in the terminal: ` ` Finally, we have our last model: TTS from OpenAI. We want to reply to our user with audio, so this model will be really helpful. It will turn our text into speech: ` The TTS model will turn our response into an audio file. We want to stream this audio back to the user like this: ` And that’s all the whole backend code! Now, back to the frontend to finish wiring everything up. Putting It All Together In our useRecordVoice.tsx hook, let's create a new method that will call our API endpoint. This method will also take the response back and play to the user the audio that we are streaming from the backend. ` Great! Now that we’re getting our streamed response, we need to handle it and play the audio back to the user. We’ll use the AudioContext API for this. This API allows us to store the audio, decode it and play it to the user once it’s ready: ` And that's it! Now the user should hear the audio response on their device. To wrap things up, let's make our app a bit nicer by adding a little loading indicator: ` Conclusion In this blog post, we saw how combining multiple AI models can help us achieve our goals. We learned to run AI models like Llama 3.1 locally and use them in our Next.js app. We also discovered how to send audio to these models and stream back a response, playing the audio back to the user. This is just one of many ways you can use AI—the possibilities are endless. AI models are amazing tools that let us create things that were once hard to achieve with such quality. Thanks for reading; now, it’s your turn to build something amazing with AI! You can find the complete demo on GitHub: AI Assistant with Whisper TTS and Ollama using Next.js...

Sep 27, 2024

8 mins

AINextJSJavaScript

Building a Stripe App: A Step-by-Step Guide to QR Code Generation

Building a Stripe App: A Step-by-Step Guide to QR Code Generation Why Build a Stripe App? I recently participated in an audio space with the Stripe team, and something they said really stuck with me: the Stripe app store is a growing area that isn't overly saturated yet. There's a lot of potential for new apps, and companies can use this opportunity to grow. I work at a company called This Dot Labs, and we've created several Stripe apps and even own one. After looking at the data, I can confirm that the Stripe team was right! Creating a QR Code Generator App For this tutorial, we'll build a QR code app that can take a URL and generate a code for it. This is a good use case to help you understand the ins and outs of Stripe's developer tools. Why QR Codes? QR codes are useful tools that have become common in e-commerce, restaurants, and other industries. While Stripe already has a QR code tool, we'll make our own to familiarize ourselves with their syntax and problem-solving approaches. Project Structure Before we dive into the implementation, let's look at the structure of our Stripe App QR Code project: * .vscode: Contains settings for Visual Studio Code * source/views: Holds the main application views * .gitignore: Specifies files to ignore in version control * stripe-app.json: Defines the Stripe app configuration * ui-extensions.d.ts: TypeScript declaration file for UI extensions * .build: this is where the built Stripe app gets placed. Step-by-Step Implementation 1\. Install Stripe Locally First, you need to install Stripe on your local machine. The documentation provides great instructions for this: * For Mac users: Use Brew to install * For Windows users: Download the package and add it to your environment variables You can find the details here in the stripe docs to install the Stripe CLI https://docs.stripe.com/stripe-cli When using Windows, you must do stripe login from Powershell, NOT from Git bash or any other tool. After the server is up, then you can continue using git bash for everything else. After stripe login, you need to enter stripe apps start. Once you do that, the server is up and running and you can go back to using git bash or any other tool. 2\. Install Dependencies We'll be using an extra package for QR code generation. Install it using npm: ` 3\. Set Up the Main Component Let's look at the home.tsx file, where we'll use Stripe's UI components: ` These components are similar to other UI libraries like Bootstrap or Tailwind CSS. 4\. Create the UI Structure Our app will have: * An input field for the URL * Validation using a regex pattern * Error handling for invalid URLs * QR code generation and display Here is the Home.tsx file that is located in the src/views folder ` * ContextView` is at the top level of the app where we see the Title and the link to the Stripe Docs that we placed in our Context View. * Box is how you use Divs. * Banners can be used to show notification errors or any other item you wish to display. * Textfields are input fields. * Everything else is pretty self-explanatory. 5\. Handle Content Security Policy One problem I personally ran into was when I tried to redirect users, the Stripe policies would block it since I did not express that I knew what it was doing. I had to go into the stripe-app.json file and mention the specific security policies. For this particular exercise, I kept these as null. This is my stripe-app.json file. ` 6\. Configure App Views As you can see here, the stripe-app.json file shows the views for each file I have. The Home.tsx file and the Invoice.tsx are also included This is our way of saying that for each view we have, show the app functionality on that page. Our stripe-app.json file will show it but also, the manifest.js file in our .build folder will also show the same. Any view that doesn't have a file will not show the application's functionality. So, if I were to go to transactions, the app would not show the same logic as the home or invoices page. By following these steps, you'll have a fully functional QR code generator app for Stripe. This is just a simple example, but the potential for Stripe apps is massive, especially for businesses serving e-commerce customers. If you need help or get stuck, don't hesitate to reach out, danny.thompson@thisdot.co. The Stripe team is also very active in answering questions, so leverage them as a resource. Happy coding!...

Dec 18, 2024

4 mins

stripe

Svelte 5 is Here!

Svelte 5 was finally released after a long time in development. Fortunately, we've been able to test it for some time, and now it has a stable release. Let's dig into its features and why this is such a significant change, even though Svelte 4 code is almost 100% compatible. Svelte syntax everywhere This is one of my favorite additions. Previously, Svelte syntax was limited to a component element. Refactoring code from a component and moving it to a JavaScript file worked differently. Now you can use the .svelte.js or .svelte.ts extension, which allows you to use the same syntax everywhere. It's important to note that it's a way to express that this is not just JS, and what you write may be compiled to something else, just like .svelte files are not just html even though they look very similar. Runes The introduction of runes is one of the most significant changes in Svelte. Many users felt attracted to variables being instantly reactive in previous versions. ` There was, however, a lot of magic underneath and a considerable amount of work from the compiler to make it behave reactively. $state In Svelte 5, a reactive variable has to be explicitly declared using the $state rune, which brings a clear distinction between what is reactive and what isn’t. ` In the previous code, $state is not an actual function being called; it's a hint for the compiler to do something special with this declaration. Rune names always start with a dollar sign ($) and do not need to be imported. However, there’s much more to the changes than just the way we declare reactive variables. Runes bring a lot of new features that make Svelte 5 a great improvement. In Svelte 5, objects or arrays use Proxies to allow for granular reactivity, meaning that individual properties in an object are reactive, even if they are nested objects or arrays. If you modify a property, it will only update that property and will not trigger an update on the whole object. It also supports triggering updates when methods like array.push are called. In previous versions, an assignment was required to trigger an update: ` To have a similar behavior of svelte 4 (no deep reactivity), use the $state.raw() rune. This syntax of .* is common for related features of a rune: $state.raw() will require an assignment to trigger reactivity. ` Because proxies are used, you may need to extract the underlying state instead of the proxy itself. In that case, $state.snapshot() should be used: ` $derived We will use $derived and $derived.by to declare a derived state. The two runes are essentially the same, except one allows us to use a function instead of an expression, allowing for more complex operations to calculate the derived values. ` $effect Effects are useful for running something triggered by a change in state. One of the things you'll note from the $effect rune is that dependencies don't need to be explicit. Those will be picked from reactive values read synchronously in the body of the effect function. ` Something to bear in mind is that these dependencies are picked each time the effect runs. The values read during the last run become the dependencies of the effect. ` Depending on the result of the random method, foo or bar will stop being a dependency of the effect. You should place them outside the condition so they can trigger reruns of the effect. *Variants* of the effect rune are $effect.pre, which runs before a DOM update, and $effect.tracking(), which checks for the context of the effect (true for an effect inside an effect or in the template). $props This rune replaces the export let keywords used in previous versions to define a component's props. To use the $props syntax, we can consider it to return an object with all the properties a component receives. We can use JavaScript syntax to destructure it or use the rest property if we want to retrieve every property not explicitly destructured.. ` $bindable If you want to mutate a prop so that the change flows back to the parent, you can use the $bindable prop to make it work in both directions. A parent component can pass a value to a child component, and the child component is able to modify it, returning the data back to the parent component. ` $inspect The $inspect rune only works in dev mode and will track dependencies deeply. By default, it will call console.log with the provided argument whenever it detects a change. To change the underlying function, use $inspect.with().with): ` $host The host rune gives access to the host element when compiling as a custom element: ` ` Other changes Another important change is that component events are just props in Svelte 5, so you can destructure them using the $props() rune: ` A special children property can be used to project content into a component instead of using slots. Inside the components, use the [@render[(https://svelte.dev/docs/svelte/@render) tag to place them. ` Optional chaining (?.) prevents from attempting to render it if no children are passed in. If you need to render different components in different places (named slots before Svelte 5, you can pass them as any other prop, and use @render with them. Snippets Snippets allow us to declare markup slices and render them conveniently using the render tag (@render). They can take any number of arguments. ` Snippets can also be passed as props to other components. ` ` Conclusion Some exciting changes to the Svelte syntax were introduced while, at the same time, maximum compatibility efforts were made. This was a massive rewrite with numerous improvements in terms of performance, bundle size, and DX. Besides that, a new CLI has been released, making the whole experience of starting a project or adding features delightful. If you haven't tried Svelte before, it's a great time to try it now....

Nov 27, 2024

4 mins

Svelte

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Build a Next-Gen Chat App with AI and WebSockets in Just 3 Hours with Ben Lesh and Tracy Lee

You might also like

Detect Hand Sign Languages with Tensorflow

How to build an AI assistant with OpenAI, Vercel AI SDK, and Ollama with Next.js

Building a Stripe App: A Step-by-Step Guide to QR Code Generation

Svelte 5 is Here!

Let's innovate together!

You might also like

Detect Hand Sign Languages with Tensorflow

How to build an AI assistant with OpenAI, Vercel AI SDK, and Ollama with Next.js

Building a Stripe App: A Step-by-Step Guide to QR Code Generation

Svelte 5 is Here!