headroom

[00:00] So as someone who predominantly runs all of my AI stuff locally, I don’t ever really think that much about token optimization, but we are all very aware of the fact that the entire industry is kind of being taken by surprise right now as context windows get bigger and everything is agentic and automation and these things are long running processes. Everyone’s kind of got sticker shock by the per token pricing or whatever it is, the amount of tokens they’re using and that has given rise to a lot of really interesting token optimization software. Rust Token Killer is very popular and then this one is called Headroom and that’s what we’re going to take a look at here and the reason I’m interested in this is not about saving money, which maybe it would in electricity, but one of the bottlenecks on a Mac running LLMs we all know is pre-fill. So that’s your prompt processing that’s processing the input that’s going in. So what we can do is we can take advantage of the fact that all of this token optimization stuff is available to us and use that to not necessarily improve the speed of our pre-fill, but to reduce the amount of context that the inference engine has to process. So this is headroom. We’re going to install it. But what we’re specifically going to look at is this proxy server.

[01:30] Okay. So this is my terminal. This is the installation of headroom AI proxy and run that and everything’s installed. And we’re going to load that up by saying headroom proxy a port, which is going to be 8787. That’s where headroom is going to live. And then we’re going to pass in our open AI API URL. So this proxy acts as the middleman. So everything that goes to 8787 is going to get forwarded on to 8,000. Now, before I start this, I’m actually going to add one more flag and that’s going to be no optimize. So what I’m doing on this first run is I’m saying proxy, all the information, all the data, all the tokens through this port, but don’t do any optimization, just let it run through. And this is going to give us our baseline. And then we’ll add the optimization and we’ll see what the differences are.

[02:24] So with that up and running, I’m jumping into my PI coding agents models.json and where I have this MTPLX server running. I’m going to change that to 8787. Now, just to be clear, I’m actually running OMLX that’s at 8,000. So in case that makes a difference. So now everything that it tries to request from this model is going to go through that port. So I’m going to save that. And what I’m going to do is I’m going to ask PI coding agent to, this is an index file on a project I have with a bunch of types defined in this TypeScript file. I’m going to tell it to extract all those to their own file and then re-import them into this file and then ensure that tests and linting are still working. And that should technically create a lot of tool outputs and file reads and things of that nature that are going to really increase the size of the data that it needs to pre-fill. So I’m going to drop in my prompt right here. It’s exactly what I said. Extract the types to their own file, re-import them to this file and ensure that the linting and tests still pass. We’re going to run that guy. I’m going to jump over here to 8787 dashboard and here we can get a real-time view or sort of real-time view of the data that’s coming in and how it’s being optimized.

[03:50] Now in our case there’s going to be no optimization so this before and after are always going to be the same and we’re just going to let this process continue until it is done and we’ll see it reading a lot of files, running linting, making mistakes, all that good stuff. So this context should grow pretty significantly. Cool. So that’s done. Let me reload this and see where we landed 153.6 thousand tokens across 13 requests. Now what I’m going to do is come in here. I’m going to undo its work and delete that types and I’m going to clear out any changes. So now we have a clean repository with nothing to commit. We are going to stop headroom. We’re going to restart it without the new optimized flag. So now we’re actually going to do optimization. Load that up. It’s running. We’ll jump back over here. Run pi. Get our dashboard up and it’s all clean. This is on a per session basis. There is a historical tab, but all I’m concerned about right now is the per session. So I’m going to drop in the exact same prompt. And we’ll see what we get. So we’re starting to see small optimizations. And if we come down here to the actual commands that are being sent through, I’m going to zoom this out a bit and we can see that in the beginning, it’s not really doing much optimization, but it will start to do more and more.

[05:37] The way it ends up working is it has like this context router that says, oh, what kind of data is coming in and what are the best ways for me to optimize it? And I think it’s got four different ones. One’s called like summary crusher or JSON crusher. This one is excluding tool calls. Oh yeah. So smart crusher is the one there. So this one did both of those. It’ll show you exactly which optimizations it’s using. And I’m using this pretty bare bones. This is not fully optimized. I’m not enabling all the different flags and really tweaking it. But what we can see. So before and after right now is 133 versus 106. And according to this, that’s 20% savings and our guy is done. So we shaved off 26.9,000 tokens, I guess. And that’s pretty significant. That definitely makes our pre-fill faster. Now it doesn’t actually improve the pre-fill speed. It just, the pre-fill has to do less. Now this was a pretty short running process. What I’m going to do is undo those changes. Once again, I’m going to start a new session with headroom. I’m going to load up PI and I’m going to drop in a much longer, uh, prompt. And you know, it’s actually getting cut off. So I’ll just show you what the prompt is that I pasted it in. It’s the exact same thing that we had before, but then I wanted to go through and extract this function into its own file, ensure that the linting and test still pass, then do the same thing with the next function and the next function. So this build user prompts extract fields, and this function that is actually called some models just won’t fucking listen. Um, I’m not going to make you sit through all of this, uh, but this should be a very long running process that takes a while and has a lot of context. Uh, so I will, uh, run this and then we’ll come back and see what type of savings Headroom says we got.

[08:02] Cool. So our task is done. And if we look at this, we had almost 800,000 drop down to 568. We save 231, which is almost 30%. Now there’s a whole bunch of tweaking I have not done here. Very specifically this learn flag, which will, uh, use some caching techniques to learn how to best optimize on your system. So you, you can definitely get more out of this. Again, I’m just doing a really bare bones look at this thing, but pre-fill being the bottleneck with, uh, running local AI on your Mac, uh, uh, is 100% without a doubt improved because we’re simply sending less to be pre-filled. Uh, so, so even though we don’t care about the cost of tokens, uh, pre-filling less tokens is going to improve our performance.