LlamaForge

[00:00] In this video, we are looking at Llama Forge, which is fucking awesome. So if you’re like me and you use Llama C++ server with the Llama router, you probably have a models any file or you have a whole bunch of aliases with all of your configuration, every little knob tweaked to your heart’s content. That is what Llama Forge takes care of for you. It’s a little rough around the edges, but it’s really cool. Here’s your documentation website. We are going to jump into the setup. We’re going to try some things out and let’s get into it.

[00:36] So I’ve got a directory here called Scratch, where I put a lot of stuff. I’m going to clone that repository and I’m going to tell you right off the bat, once we get in there, if you’re on a Mac, you’re going to want to chmod plus X all of the shell files. So what we’re going to run here is Bootstrap. And right off the bat, it’s going to ask me if I want to clone Llama C++. Now, I can stop here and go into a configuration file and tell it where I have Llama C++ living. But I’m not fully committed to Llama Forge yet, so I’m going to say go ahead and get your own copy. So it’ll be managing a copy within the application.

[01:18] So it’s picking up on some models here from a previous run. But the first thing you really want to do is jump into setup. And what you want to do is, or well, you don’t have to, but this is what I’m going to do. I’m going to say scan for GGUF models. As soon as I do that, it’s actually scanning my entire hard drive or any hard drives connected to my machine for any GGUF models. So I’m going to let that run until it finishes up. So it came back and it said it found all of these models. So I’m going to say add all nine models to my config. Now there’s some additional stuff down here. Do you want to start with a specific one? Do you want to create network access for your Llama C++ server? And do you want to connect an agent? I’m not going to worry about those right now. I’m going to jump over here to models.

[02:04] This is my go-to right here. This Quen 3.6 Q4 KXL. Now it doesn’t have any of my settings here, but I can go in and update all of these. And let me just point out there are way more than I ever knew there were. As I expand all of these sections, I’m going to be honest. My configuration for this model in my models ZINI is not this complex, or I don’t use all of these. And the cool thing is if you go look at the source code, Llama Forge is actually pulling all of these directly from Llama C++. So this should always be in sync with your version of Llama C++, which is really dope. So what I did is I actually went and got my models ZINI, and then I came in here and just pasted all that information in, but now they are all represented here in the UI, which is really, really cool.

[03:08] If I want to go in and just tweak something, I want to change the cache type K, I can just tweak it here. So I go from Q8.0 to Q5.0, I say save and reload. If I load up that models ZINI, you can see it’s been updated there. So it is managing that for me. And I didn’t have to go in and plug each and every one of these in, but now I can go in and just kind of mess with it. And that is a huge time saver because I’ll be honest, I don’t end up messing with these that much. I do on somebody’s recommendation, but they can, you know, once you dial it in just right, as a matter of fact, I got to put that back. Once you dial it in just right, you kind of want to keep that guy where it was. So at this point, I have a fully running Llama server running with router. It’s connected to my model running exactly how I want it to run. And if I want to come in and tweak something, I can just do that.

[03:56] Now there’s a whole bunch of other stuff here. There’s downloading models from a hugging face. You can go through, you can search. There’s a whole bunch of great stuff there. There’s some testing to find out if a particular model will run on your machine. There’s this whole context thing that I haven’t really gotten into where you can set up different profiles that have like a context for them, almost like you’re talking to a specific persona or agent. So this is Llama Forge. It is definitely rough around the edges. There’s a lot of little things. For example, when I come in here, if these guys are closed and I say, I want to search for temp, nothing opens. You have to have them open, but then they will all filter by any that actually have the temp keyword in the label name or the description. So there’s some rough around the edges bits here, but this is like a tool that I didn’t even know I needed until now. And I’m definitely going to be using this going forward, assuming that the progress continues moving forward.

[05:02] This project is by somebody who goes by DadWritesTech. I don’t know who that is, but awesome job, DadWritesTech. I think everyone who’s using Llama C++ with Llama router should absolutely check this out. It’s going to save you a bunch of time and make you less afraid to experiment with your model’s settings. So there you go. That’s Llama Forge.