omlx

[00:00] If you are running local AI on an Apple machine with an M-Series processor, and you’re not using MLX, you are almost definitely leaving significant performance on the table. We’re going to take a look at a tool called OMLX. But to get us started, what is MLX? It’s a framework from Apple that is specifically designed for native Apple Silicon optimizations when working with LLMs. So here’s their open source Apple website. Here’s the MLX framework website that’s got some more information. At the end of the day, most people are interacting with this Python library, MLX. It could be compared to Llama C++ in terms of what it does and how it works. Now, OMLX is very similar to, say, O Llama, where O Llama is built on Llama C++. OMLX operates on that Python package that I mentioned earlier.

[01:10] If we jump over to their website, they’ve got all your standard stuff, a bunch of wild claims. But at the end of the day, what you want to do if you want to check this out is download this DMG, drag it to your applications, and you’re up and running. And what you get is this little web service running on localhost, 8,000. This is the dashboard. I’m going to jump over to models, and you can see I happen to have two models that I’ve already downloaded. I used their downloader here, where you can just plug in a repository ID. And if you have a hugging face token, that’ll get you some faster download speeds, I believe, but it’s not required. And then you can also just kind of browse some of their recommendations, and you’ll get some notes here. I’m not seeing any right now, but sometimes it’ll say, like, tight fit for your system or whatever. I’m not seeing any right now, but sometimes it’ll say, like, tight fit for your system or whatever. And you can, you know, do search, trending, popular, and so forth. And then, again, I’ve already downloaded these two because I kind of wanted to compare them.

[02:13] And if we jump over to our settings, just something to take a look at. We’ve got this model directory, so this is where it’s downloading those models, too. But if you have some models that you’ve already downloaded, and I do have a few in my hugging face directory where I use the hugging face CLI to download those, I can just add that directory. Hit save settings down here. And now when I jump back to my models, it’s picking up on those models that I have in the hugging face directory. So you don’t have to re-download models that you already have. You just need to let OMLX know where they are. Now, one really cool feature that we get from OMLX is the ability to run some benchmarks. So I’m going to jump in here and say on this 35A3B 4-bit, I’m going to run this benchmark. Now, when this completes, it’s going to give us a bunch of numbers. And I’m not going to pretend that I understand every single one of these tests. But there is a little reference down here that gives you a lot more information as far as what each of these metrics that you’re getting back mean. So the only one I really know for sure is these output tokens. So these are tokens per second. I’m getting 54 and 51 on those two benchmarks. I’m going to want to come up here and say, try it on this NVFP4 version. I can run that. And now I can do a nice little comparison for myself. Cool. So these two seem to be fairly comparable.

[04:08] Okay, we are going to jump over here and we are going to, so I’m on the dashboard here and one feature that we get is this other integration section, which is really cool. Because I can come down here to Pi and say, I want to use Pi with that guy that I just ran the benchmark on. So I can just copy this command. And what I’m going to do is I’m going to jump into this directory that I’ve already got some code and just show you the code really quick. It’s just an index file doing some nifty quadrant llama index stuff that I’ve been playing with. And so Pi is up and running. It’s using OMLX, the model that we specified. And I’m just going to give it a try. I’m going to say, review the code, excluding node modules, and summarize. Blow this up. So Pi just explained the shit out of my one file repository, which is great. And I got to tell you, it was super fast. The output was super fast. So that is a look at OMLX. Again, if you’re running local AI on an Apple machine with an M series processor, I highly suggest you check it out and see what kind of performance improvements you get.