llmfit

[00:00] If you’re running your AI locally, you probably spend a lot of time testing different models, seeing what works on your hardware or what pushes it too far. This is a tool called LLM Fit that helps with that process. It’s available on GitHub for free. Here in the README, there’s a whole bunch of different ways to install it. I used Brew install. Once you have it installed, you can run LLM Fit. Just take a moment to load up all the models. And here we are, I’m looking at 852 models. We are looking at my hardware. So it’s comparing it to my local hardware to figure out how I can expect it to run.

[00:44] So you can see this fit column here says perfect. But if I want to cycle through that with the F key, I can say these are runnable, perfect. These are good, not great. These are marginal. So you may or may not have good results with this. And then these are too tight, which means you’re going to run out of memory.

[01:03] So if I go back to all, just going through some of these filters really quick, we can filter out certain providers. If we have a bias there, we can choose to only look at certain use cases. So say we, we only want to do embedding. So we don’t need to look at a lot of these larger models. We could just check those off and get embedding. We can do capabilities. So if we’re looking for vision or tool use, we can also sort.

[01:30] So I’m a fan of sorting by tokens per second on my hardware. We already looked at fit. We have available. I don’t really know exactly what that does other than it’s installed or all. We have T, which is if you’re doing tensor parallelism, which is if you want to run it on multiple video cards. And then there’s a little theme guy. So we can also sort, or I’m sorry, search. So let’s just say I only want to look at quen models. So all of these are going to have quen in them. I’m going to go ahead and go back to sorting by, by the date.

[02:13] And so I can see I’ve got a bunch of these installed already. I’m going to come down here and say, you know, let me just look at this guy. So I hit enter on it. I get a whole bunch of information. I get a bunch of different links to various downloads that I can grab. And you can even just, I think it’s D for pull. I can even just say, I want to install it for one of these systems that I’m currently running and download it right here.

[02:40] So if you are spending a lot of time looking for specific models, LLM fit is a really great way to get a headstart on that. To clear out some of the clutter, you don’t have to pull down what’s clearly not going to work. I would say my experience with this is almost universally positive, but I have seen some false positives where it said it wouldn’t run, but it would. But either way, it’s a great starting point to kind of filter out some of the things that you do or don’t want to use and get quick access to them.