[00:00] In this video, we are going to create a very simple RAG system. If you’re not familiar with RAG, it’s retrieval augmented generation, which means we’re going to grab some data and use that to generate our responses. So I’ve got a little read me here. These are the things we’re going to set up. We’re going to create a database, a table and index, we’re going to set up our connection to a llama, we’re going to do an embedding model and a query model. In this ingest guy, we’re going to insert our book into the database, we’re going to set up this query to query the database and generate the answer. And then index is just going to be our command line tool for running all of this. Jumping into my package, Jason, don’t really have a lot going on here. AI SDK, we’re using lib SQL from torso. And then I’ve got this guy EPUB two, which is going to allow us to read our source, which is going to be the godfather by Mario Puzo. So we’re just going to jump in to my plan here.

[01:00] And we’re going to jump into dbts. It’s going to be really simple. I’m importing create client from lib SQL. And I’m importing SQL from this guy right here where I just wrote up some of the SQL we’re going to need to write. So I’ll come right back to that. We’re going to export a const of db equal to create client. And we’re using, we’re just going to call this ragdb. So that’s all good. Now we are going to net db is going to do these things. So what we’re going to do is we’re going to have this init db function. It’s going to execute this create table and this create index. That’s going to be the extent of creating our database. So I’m going to jump over here really quick. A very simple table called chunks. It’s going to ID, it’s got text, and it’s got this embedding field, which is an F32 blob. But the important part here is that it has 1024 dimensions, or that’s what it’s expecting. When you’re working with different embedding models, they will tell you usually up front how to either set the dimensions or what their fixed dimensions are going to be. I happen to know that the one we’re going to use is 1024. The other thing we did here is we created an index called chunks vec. It’s doing some internal lib SQL stuff. I’m not super familiar with exactly how that works, but I can show you the end result of that. So we’ve got our db that we can pass around and we’ve created our table and our index. So we’ve done these guys. We’re going to jump over to the AI side of things.

[02:40] I do have this query prompt set up here that I’m going to use, but I didn’t want to type that out. So we’re going to import create open AI from the AI SDK. We’re going to import embed and generate text from AI. We are going to set up our connection to a llama with our local host and an API key of a llama because a llama doesn’t really need, it doesn’t need an API key, but the create open AI does. And you know what? There’s a typo there. Okay. So we’ve got our connection. Now we’re going to set up our embed models. So this is the model that’s going to embed pretty close, but we’re not going to use this. We are going to use MX BAI embed large. And that’s that embedding model that I mentioned earlier. That is going to embed our text or chunks of text, uh, in dimensions of 10, 24. Okay. So moving on, we’ve got that guy and we’re going to set up, I’m going to call it our query model. And that is not going to be llama. That is going to be quen three, eight billion.

[04:00] Okay. We’re going to export and const of better. It’s going to be our embedding function. That’s going to be async. It’s going to take in text, which is a string and it’s going to return a promise of number array, which is really what the embedding ends up being. So if you’re not familiar with the embedding, really all it’s doing is it’s taking, uh, a chunk of text in our case and turning it into an array of numbers. Um, and what’s cool about that, and we’ll get more into it is, um, that’s all we end up storing. And then when we do a search later, we can actually embed our search and use that to find other similar embeddings. Uh, so it’s pretty dope. All right, let’s see. This, uh, is close. So embedding, it’s going to equal a weight embed, which we’re bringing in from the AI, uh, uh, library. So we’re going to return that. We’re all good there. And now we’ve got to create our, uh, generate or query. Um, I don’t think, I don’t like any of that. So I’m just going to type this one out. Uh, we’ll call it generate. It’s going to be an async function that takes in context. It’s going to be a string and a question, which is going to be a string. So that context is the data we retrieved from the vector, uh, database. Uh, database. And the question is the question that the user is, um, asking.

[05:30] So from that, we’re going to return a promise of string that looks close, but I’m just going to type it out because I don’t trust all that. So we’re getting text from wait, uh, generate text. Okay. So we’ve got our model. So generate text in case you miss it is coming from the AI, uh, library. It’s got a model. We’re going to set a temperature here. Uh, so the lower the temperature, the more, uh, deterministic the response is going to be, the higher the temperature, the more likely you are to get kind of like, it’s just going to make up some stuff. Uh, and we don’t want to do that. Um, I’m also going to set a system prompt here and that’s going to be down here. Okay. And bring that back. So I call that query prompt and that’s fine. So our system prompt is right there. And that guy just says answer only using the provided context. Uh, if the context doesn’t contain enough information to answer fully, just say so. Uh, so trying to prevent it from making stuff up. Okay. So now we’re going to get into our messages. So that’s an array. It’s going to have one object in it. It’s going to have a key of role, which is going to be user. And then it’s going to have content, which is going to be the context and the question. Yeah. Yeah. That’s perfect. Okay. And then once we get outside of all this, we’re just going to return the text that we got from that guy. So cool. I’m going to clean this up a little bit and our AI, let’s see this embedding model one. So this is just going to be model, embedding model and value is text. Okay. There we go. We’re going to jump back over here. We’ve taken care of the database and the AI. Now we’re going to look at ingest, which this is probably the, I I’d certainly say this is the largest part of our application.

[07:45] So we’ve got EPUB from the EPUB library or database, the embedder that we just created. I’ve got a couple of utilities where I just didn’t want to type out a bunch of stuff. So I’ve got a strip HTML utility and I’ve got this chunk text, uh, uh, utility and, and I, and I’ll come right back to that. Um, and then we’ve got our SQL, which again is just our SQL statement. So here we’re going to be using this insert a SQL statement. Okay. So I’m going to set up my EPUB path. Okay. So it’s our source to the Godfather by Mario Puzo. I’m going to export a const of ingest. Okay. I’m going to log here because this part of the process does take a minute. So ingesting book is fine. Then we’re going to say const EPUB. It’s going to be equal to EPUB dot create async. And then we’re going to pass in our EPUB path. Uh, from there we can do not this. We’re going to say chapters is equal to EPUB dot flow. And that’s just going to give us the chapters of the book, uh, according to how EPUB finds them. Then we’re going to say for const chapter of chapters. So we’re just going to iterate over these guys. Now EPUBs, if you’re not familiar, are really just zipped up HTML files. So what we’re going to do is say let HTML string, and then we’re going to try HTML equals await EPUB dot get chapter. That’s close, but it’s going to be get chapter raw async. And then, yeah, if that fails, we’re just going to continue on. Um, that could be the cover page. That could be all sorts of different things that might get in the way. And we’re really only interested in text. Um, so let’s see where AI has gotten me. So yes, we want to get the text. We’re going to strip that out of the HTML. So that is correct. And then our chunks are going to be going through this chunk text method that I looked at, or I briefly showed you earlier. Um, so what we’re doing here is we have our string, I’ve got a default chunk size, and then I’ve got an overlap. Now the values that you choose for these things are completely up to you. The chunk size of 1000 is actually pretty large, but I do find it works better with this type of content. This is a novel. Um, so larger chunks that it can, it can kind of breeze over. And then the overlap is really simple. Basically for every chunk that goes in, we just back up through the string, the overlap value. So what ends up happening is after the first, uh, chunk of text, the next one contains the last hundred characters of the previous one. And then we just return an array of those. So we’re just pushing that into here based on that. So, so each of these is chunks of text from the book. Um, each one containing the last hundred characters of the previous one at its start. Um, so we’ve got that. And now we’re going to do for chunk of chunks. Yep. And now we’re going to convert that into the embedding. Uh, so what comes back from here is actually just this array of numbers that we’ve gotten back from the embedding and it’s ready to go into that, um, F32 blob field that we set up in our database. Um, so I’m not sure that this is right. We’re not going to do embedding here. We are going to say and we’re going to join that guy. Okay. I think that looks right. If we run into a problem, we’ll check it out. So this one, I did have ingesting book at the top. So when we get done with this guy, yeah, I’m going to console log out done. Okay. If I jump back over here, we’ve got our database, we’ve got our AI set up and our ingest, and now we’re going to create our query.

[12:15] So if I jump over here, this is where we’re going to query the database and generate an answer. We’re bringing our database file, our embedder and our generate from our AI, uh, component and our SQL. Um, and if we look at our SQL, what we’re going to be doing here is querying. So we’ve got this guy here. Um, one thing to mention about this is again, when we create the, um, database, all we’re doing is creating this table and this index called chunks VAC. Um, when we look at this SQL on the query part of it, you’re going to see, you know, we’re getting something from chunks, but we’re also getting some from, from this vector top K what we end up passing into this is that top K is really just how many results do we want to get? And we want the top results. Now there’s vector top K thing. This is something specific to lib SQL that lib SQL gives us to use. Um, but I didn’t actually create that guy. Um, again, it’s, it’s just not here. So you, you can see the two things that I create or the table and the index, this is a function of lib SQL. So if we come in here, uh, first thing I’m going to do is I’m going to set my top K and this is another instance where depending on the type of content or type of data you’re dealing with smaller or larger could have a pretty substantial impact. All right. So we’ve got query async. The questions coming in, we’re turning a promise of a string. That’s all good. Close that out. And we are going to say const vector equals await better. And then we’re just passing in the question. Um, so just so you understand what we’re doing, we have a database of embeddings and those are all just arrays of numbers created by the embedder. Um, the index that we created is what allows it to find those similarities quicker between one embedding and another embedding. So we take our question that the user’s answering and asking, which is a string, which is a string and we convert that into an embedding. And then we can use that to find similar embeddings. And that’s how, uh, embeddings and, and vector databases work. So we’re, we’re able to use this string, convert it to an embedding and compare it to the other embeddings. So certain words are going to create certain, um, embeddings. And so we can go find those embeddings in the database much quicker than we would if we were doing a full database search on strings. So there you go. Okay. Uh, we are going to get our rows and into that we’re going to pass our, uh, SQL query and our, uh, that might work, but I just don’t like it. There we go. So it’s the exact same format that we used, uh, in the previous code. Um, so there’s our rows that we’re getting back from the database. And now this is going to be our context for, uh, the query. And then we’re just going to join all that. So we have a nice string that we can send to our, uh, guy. So a couple of line breaks works for me. And then, yeah, we can just return our generated answer. So to our generate, we are sending the context and the question. The context is really just the, the results of our query to say, take this embedding and compare it to the other embeddings we have and using our top K give us the top 20 that are most similar to our question. And that’s all there is to it.

[16:10] Okay. So where are we at? We got our query. And, uh, one thing I’m going to do here is in this setup, I’m just bringing in our init DB and our ingest and then firing those off since we, we really only want to do that one time. And then in our index, um, we are going to, we are going to import query. We’re not going to import, uh, well, let’s see, say the question is processed at arg2. Otherwise, we’re just going to exit this guy. You didn’t send us a question. So there’s that. And then we’ll log out the question. We’ll get our answer and then we’ll log out the answer. Uh, a little line break here and I think that looks good. So now we’re actually ready to give this guy a try. Uh, so I’m going to come down here. I’m going to say bond run our, uh, source set up TS and we did get an error. Um, so I see it created the database. It doesn’t like the path to this guy. Since I’m calling it from above, I’ll do source slash, see if that fixes it. Um, I can see it created the database. Uh, I’m just going to go ahead and delete that for now. And we’ll try this one more time. See if I’ve got any other issues set up. Okay. Uh, API call errors. So let’s go look here. Uh, you know what this needs forward slash v1 for the, uh, for the, for the open create, open AI, uh, API is going to work like that. Okay. So we’re going to try this one more time. Uh, again, I’m going to delete the database and we’re back. So run the setup. Okay. We’re ingesting the book. Cool. According to this, our book is ingested. If we want to take a look at the database, see, we’ve got all of these blobs in here. Uh, so let’s see if we can run a query. Run, run, run, run source index TS. And I’m going to say, what was Michael’s first wife’s name? See if that works. Um, so what we’re looking for is Apollonia. If you’ve never read the Godfather and let me zoom this in a bit is not Virginia at all. So what’s our problem here? It’s hallucinating. It’s making up stuff. So we need to make this more explicit.

[19:10] So we’ll do that. And then the other thing I’m going to do, so I’m going to make these guys much bigger, 1500 chunk size, which means I’m going to rerun the ingestion. I’m going to have larger chunks that gather, uh, more information all at once. Uh, and I’m hoping that that’s going to fix this up. So I’m going to run this one more time. It’s going to take another two minutes. Cool. We’ve got our book ingested. Uh, I also realized that there may actually be another character named Michael in the book, and maybe I jumped the gun there, but I’m going to say, what is the name of Michael Corleone’s first wife? Cool. So we got, uh, Michael Corleone’s first wife. Her name was Apollonia. Um, so I’m going to add one more. It’s a little more wordy. I just, I got off some trivia website. So when exiled in Sicily, he learns about his roots. What did he discover the word mafia meant? So that’s a super specific detail. Uh, what we’re looking for is, uh, a place of refuge. Yep. Place of refuge. All right. So let’s try, let’s try one that’s, uh, uh, more a bit more abstract and covers a lot more of the book because the story of many of the characters in the book are told in pieces throughout the book. So tell me the story of Luca Bratsi. Let me just clear out what we have. We’ll run that. And this is a good example of where these really large chunks and the overlaps, uh, really help because it’s comparing really large chunks of the text to other really large chunks of the text. And it’s, uh, that makes it more adaptable to like large scale summarization. So it’s going to have to gather this story of this character for many parts of the book. And I’m actually looking for one specific detail that’s not in the movie, but it’s in the book and it involves, uh, Luca Bratsi murdering a baby. Okay. Uh, it’s a feared enforcer. He talks about the olive oil war. Uh, his devotion to the Don was absolute, uh, pivotal. Here it is. Moment in the story involved a midwife named Philomena, who witnessed his dark side arranged to give birth in her home when the newborn was born, arranged by the child’s race. Cause it was Irish ordered the infant killed. Uh, yeah, this all looks, this all looks great. Uh, so there is like a long form summary that we got back from the AI. So I get to sit here and ask it questions about the Godfather all day. I feel very confident about that. At this point we have created a rag system, uh, and taught, uh, AI all about the Godfather.