Transcript
[00:00] One of the things we need to be aware of in our local coding agent is context or what we call tokens. The way that we generally measure tokens is you just take all the messages and divide them by four, meaning every four characters is a token. So what I did is I created this estimated tokens here in this compact file, brought that into our index, and right here at the beginning of our loop, I said, give me a token count. Just to show how this works, I’m gonna try to do something that generates a lot of tokens, so I’m gonna say, give me 50 writing prompts. So just that was 14 tokens. So after this comes back, I’ll ask it something trivial and we’ll see what the current token count is. Okay, so we got our 50 writing prompts and I’ll just say, what is two plus two. And our token count is approaching a thousand. So in memory, we have roughly 4,000 characters in our message history. So what we’re going to do here is we’re going to compact that history. And we’re just going to automate it. So it’s not going to be a tool that the agent needs to use. We’re just going to do it. We’re going to set it to automatically happen. So in my config, I’m going to create a new const called context limit. I’m going to set it to 50,000. We’ll bring all the stuff in that we need. So we’ve got generate text. We’ve got our model, our API, and our context limit. Now we’re going to have a couple internal ones here. So keep recent is going to be three. I’ll explain that in a second. Then we’re going to have const preserve tools. That’s going to be a set. And for right now, we’re just going to be preserving read file. So these are specifically about tool calls. I’m going to go ahead and keep the most recent three tool calls. And we’re always going to preserve read files. So the thinking there is that when we read a file, it’s actually referential information. It’s potentially important.
[02:33] So let me just rearrange this a little bit. Put these guys at the top. Okay. So estimate tokens is going to stay. And then we’re going to have three other functions. We’re going to collect tool results. And we’re going to have micro compact. So this one’s actually going to run on every call through the LLM. Then we’re going to have an auto compact. An auto compact is going to be when our message history has gotten too big. I’m going to drop in a little type here and close this sidebar. Okay. So the type is actually just what our messages look like. And I do want to mention that when the content of a message is just a string, that just means it’s text. There’s nothing there for us to really concern ourselves with. We’ll concern ourselves with it in the auto compact, but not in the tool results or the micro compact. So we’re going to ignore those when it’s just a string. What we’re looking for in the collect tool results in micro compact is tool calls. So that content won’t be a string. It’ll actually be an array of what very much looks like our message history. So it’ll have a type, potentially a tool call ID, a tool name, and a content. So we’ll start with our collect tool results. This guy is going to get our messages, which is an array of model messages. And what it’s going to return is a part with content, which for now will be unknown, and a tool name, which will be a string. So we’re going to gather those names in a new map. And we’re going to return. We’re just going to do all this very functionally. So the first thing we’re going to do is flat map this guy. And we’re going to say, if our content is an array, I’m going to return that as an array of parts. Otherwise, just an empty array. Now we’re going to reduce on that. We have our accumulator and our, we’ll call it P for part. So first, if our P dot type equals tool use, here we can get the actual, or we should be able to get the tool name. So let’s make sure that it has a tool call ID. And if so, we’ll say names dot set, call ID, and the tool name. So now we’ll look for tool results. So let me just copy this. So if a tool result, and we have the tool ID, we’re going to push into our accumulator, a new object with our part, and our tool name, names dot get. And in case that’s not there, we’ll just say unknown. We’ll return our accumulator. And then down here, we will have our array as, not tool. It’s going to be that guy. And that should be an array. Okay, so that is, now we’ve got our names, and we’ve got our tool use and result in our accumulator.
[06:08] So now we get to our microcompact. And we’re going to call collect tool results with our messages, which we’re going to plug in here. This isn’t going to return anything. On that, we’re going to slice 0 to a minus keep recent. So we’re keeping our most recent 3. And we’re just going to for each those guys. Get our part and our tool name. And we’re going to have a big old if condition here. So we’re going to say, if not preserve tools.has tool name. So in our case, if this isn’t read file. And the contents string. And for good measure, we’re going to check to see if the length is greater than 100. And if all of that is the case, we’re going to say part.content equals. And we’re just going to say previous used tool name. So what we’re doing is we’re actually replacing that message in the history with used tool name. So we’re just throwing it away, more or less. So we’re just keeping a reference that it’s there. The rest of the structure for the way that AISDK works with this is still in place. So we’re fine there. We haven’t broken the history. We’re just stripping out that content and saying this was the result of a tool name or whatever. We still keep track of what tool was called. The order in which it was called. All of that remains intact. The message history is there. It’s just really that the results of this guy are stripped out. And we just say, we use this and we don’t really care what the results of it were. Okay. And then for this, what we’re going to do, we’re going to bring all this into our index. And at the very beginning of each of our interactions, we’re just going to drop in micro compact messages. So it’s just going to strip out everything except the last three. And we’re also excluding read files. So everything else, long bash outputs, you know, catting files, all that stuff is going to go away. Okay. So now we have our auto compact. If you notice, I brought in generate text or model, all this good stuff. Oh, you know what this context limit? Oh, we’re not using that yet. Okay. So all those things we haven’t used yet. And what’s groovy about this is we’re actually just going to use the LLM to summarize all this stuff. So we’re going to take in our messages model messages again. And here we are going to, we’re actually going to use the AI. So wait, generate text, our model, which is our API.chat model of our messages. And we’ve done all this before. So I’m not explaining a bunch here, but this is how we interact with our LLAMA service. So we’re passing in the model, which we’ve already got in our config. And now the messages, you know, we just got done iterating through a bunch of those, but this one will just be a user message. So role user. And then the content is going to be this right here. So that’s pretty long. Summarize this conversation for continuity. Include what was accomplished, current state and key decisions made. Be concise. And then we just pass in all our messages and slice it. We slice off the, we slice it to the context limit. That’s it. And then we’re going to return a new message history. That’s it. So let me just grab this. Instead of that content, I’m going to say conversation compressed. Here’s the text. Otherwise, there’s no summary. And that is all there is to that.
[10:35] We are going to jump back over to our index. We’re going to use our estimate tokens. We’re going to say if estimate tokens messages is greater than the context limit. Say const compacted is equal to await auto compact. Pass in our messages. And then we’re going to make sure we clear that out by saying messages length equals zero. And then we’re just going to push all the new ones in there. Clean this up. And that’s everything. So on every call, we are going to do this micro compact. Make sure that our tool calls are getting reduced. We’re always keeping those last three. And we’re always keeping read file. And if we hit the context limit, which in our case is 50,000, we are going to automatically compact those messages. Now, I’m going to pause the video here while I try to figure out a really good way to show that this is occurring. So I did two things here. One, I dropped our context limit down to 3,500. And the other thing I did is I just reworked this a little bit. Get our estimated tokens outside of the check. Console log out what our tokens are. And then we do our compacting. And what I did is I log out. We’re compacting at. And then console log our new tokens. So 3,500 should be pretty easy to hit. So I’m going to say give me 100 writing prompts. So we’re at 14 and a quarter. Okay. So we’ll ask it something simple now and see where we’re at. So we’re at 1648. So give me 200 more writing prompts. Okay, there’s our 200. Let’s see where we’re at. Okay. So we were at 4,443 tokens and it’s compacting now. And again, remember in this case, it’s actually firing off a new request to the LLM in a new session to summarize the content of those messages. And our new tokens are down to 122 and 75. So there you go. We are automatically compacting all of our message history. We went from almost 4,500 tokens down to 122. And you can see from its response here, it clearly hasn’t lost any of the thread. It’s still talking about the writing prompts from our message history. So there you go. We could essentially assume that we could work on a large code base for, if not forever, a very, very long time with this new compaction without concerning ourselves about the context getting too large. Okay.
Stop blowing through your context window with this automated AI memory management system. We build a local coding agent that dynamically compresses message history, preventing context overflow while preserving critical tool calls and reasoning. This approach enables infinite-context workflows for large codebases without hitting token limits.
Measuring Token Usage
To estimate token usage, we can use a simple heuristic that approximates the number of tokens in a message by calculating the length of the JSON string representation of the message. This is a rough estimate, but it’s good enough for our purposes.
import type { ModelMessage } from "ai";
export const estimateTokens = (messages: ModelMessage[]): number => { return JSON.stringify(messages).length / 4;};Using the Token Estimator
const agentLoop = async (messages: ModelMessage[]): Promise<string> => { console. log('token count: ${estimateTokens (messages) }'); let stepsSinceTodo = 0;Try it out
input >> give me 50 writing promptstoken count: 14......input >> what is 2+2?token count: 989.75 # ~4000 charactersLet’s set our agent loop to automatically compact this history
export const compact = (messages: ModelMessage[]): ModelMessage[] => { // TODO: implement};import type { ModelMessage } from "ai";import { generateText } from "ai";import { MODEL, api, CONTEXT_LIMIT } from "./config";
const KEEP_RECENT = 3;const PRESERVE_TOOLS = new Set(["read_file"]);
type Part = { type?: string; toolCallId?: string; toolName?: string; content?: unknown;};
export const estimateTokens = (messages: ModelMessage[]): number => { return JSON.stringify(messages).length / 4;};
const collectToolResults = (messages: ModelMessage[]): { part: { content: unknown; }; toolName: string; }[] => { const names = new Map();
return messages .flatMap(m => (Array.isArray(m.content) ? (m.content as Part[]) : [])) .reduce((acc, p) => { if (p.type === 'tool_use' && p.toolCallId) { names.set(p.toolCallId, p.toolName); } if (p.type === 'tool_result' && p.toolCallId) { acc.push({ part: p as { content: unknown; }, toolName: names.get(p.toolCallId) ?? 'unknown' }); } return acc;
}, [] as { part: { content: unknown; }; toolName: string; }[] );
};
export const microCompact = (messages: ModelMessage[]): void => { collectToolResults(messages) .slice(0, -KEEP_RECENT) .forEach(({ part, toolName }) => { if ( !PRESERVE_TOOLS.has(toolName) && typeof part.content === 'string' && part.content.length > 100 ) { part.content = [`Previous: used ${toolName}`]; } });};
export const autoCompact = async (messages: ModelMessage[]): Promise<ModelMessage[]> => { const { text } = await generateText({ model: api.chat(MODEL), messages: [ { role: "user", content: ` Summarize this conversation for continuity. Include: 1) What was accomplished, 2) Current state, 3) Key decisions made. Be concise.
${JSON.stringify(messages).slice(-CONTEXT_LIMIT)} `, } ] });
return [ { role: "user", content: `[Conversation compressed]\n\n${text || "No summary."}`, } ];};Adding compaction to our Agent Loop
import { generateText, isLoopFinished } from "ai";import type { ModelMessage } from "ai";import * as readline from "readline";import { PARENT_TOOLS, } from "./tools";import { api, MODEL, WORKDIR } from "./config";import { api, CONTEXT_LIMIT, MODEL, WORKDIR } from "./config";import { autoCompact, estimateTokens, microCompact } from "./compact";
/** AGENT LOOP */const agentLoop = async (messages: ModelMessage[]): Promise<string> => { let stepsSinceTodo = 0; microCompact(messages); const estTokens = estimateTokens(messages); console.log (' tokens, ${estTokens}'); if (estTokens > CONTEXT_LIMIT) { console. log ('compacting at $festTokens}'); const compacted = await autoCompact(messages); messages.length = 0; messages.push(...compacted); console.log ('new tokens, ${estimateTokens (messages) }'); }You can try this out by setting a CONTEXT_LIMIT of ~3000.
input >> give me 300 writing promptstokens, 14.25......input >> what is 2+2?tokens, ???