Splash

[00:00] Rather than building an inference engine like all other inference engines that can accommodate all sorts of models, the folks at Inco.ai built Splash, which is an experimental and new type of inference engine built around two models. It’s Quen 3.635 billion A3B and Quen 3.827 billion. Now, depending on which spec you read, their claims are anywhere from two times faster to five times faster than other inferences using the same models. So in this video, we’re going to install Splash, grab their very specific models, and put it to the test with some benchmarks.

[00:20] The models that they use are not like any other models I’ve ever seen. They’re not MLX, they’re not GGUF, and they’re not Hugging Face Transformers. I’m looking at their model card here, and if I jump into their files, there’s some things that are fairly familiar. There’s tokenizers and categorizers and things of that nature. But when you jump into, for example, this target, it’s just all of these bin files layer after layer. And each of these models comes with its own D-Flash 2 draft model. That’s not optional. It comes with it automatically.

[01:10] So right here on their website, the way we can install this is with brew install Inco AI tap splash. So I’ve already got that installed, but I’m going to go ahead and run this anyways. Cool. So it says everything’s installed.

[01:23] Now, when we want to run this guy, we’re going to run splash, serve, double tack, model, Inco AI, and then the model that we want to pull down. Now, the first time you run this, it is going to pull that model down for you. So once that’s completed, we’ve got a local URL running at port 8,000. We’re going to load that up in the browser. And we’ve got what looks like a little open web UI interface. I’m just going to say hello to our model. And I can tell you right now, I’ve never gotten speeds like that. 133.6 tokens per second is easily the fastest I’ve ever seen with this model.

[02:00] So a little while back, large IT data put out this report comparing rapid MLX to OMLX. And along with that, they gave us their open source benchmarking system. And I’ve been using this to benchmark different inference engines for a while. So we’re going to put splash through its paces compared to OMLX with the same model using this benchmark lab.

[02:23] Now, the first one here, I already could see with the naked eye that the decode speed in splash is intense. It’s killing OMLX here. This is basically your tokens per second on average. Our next result is the decode degradation. So as the context gets higher, does that speed slow down? And in fact, OMLX has the advantage here. Splash like really takes a steep dive right from the beginning, even though they end up at about the same place at 32K context. Prefill speed versus context is actually really interesting. OMLX and every benchmark I’ve ever run has dominated this category. But Splash is clearly doing better and taking off at the higher context here. Time to first token, Splash is also killing it. You can see all the way across every single context, Splash is returning that first token much, much faster.

[03:04] So for those of you interested in 3.827B, I did pull that model down with Splash and ran it through the same benchmarks, and it was a significant improvement over MLX with the same model. However, my M3 Pro with 36 gigabytes of RAM and that model measured between 15 and a high about 29 tokens per second, which tends to be insufficient for my personal needs, but it’s definitely something you may want to check out.

[03:16] So really quick to answer the question, some of you might be wondering about, should you give up on OMLX and switch over to Splash? The short answer is no. If you’re primarily using these QN models, there’s definitely a place to work Splash into your local AI workflows. But the great thing about open source AI is everybody’s sharing. And another great thing about OMLX is that they’re very quick or their primary dev is very quick to pick up on interesting projects and incorporate that into OMLX. And if you look at the acknowledgement section of their readme, Splash is already there as an inspiration for some of the specialized kernel work they’ve already done.