Token budget system #12

New issue

Open

opened 2026-04-07 20:23:10 +00:00 by austin · 0 comments

austin commented

2026-04-07 20:23:10 +00:00

Owner

Implement hard token budget allocation and enforcement for prompt construction.

Define budget slots: system prompt, code context, task description, few-shot examples, generation headroom
Default allocations for an 8K context window: ~256 system, ~1536 code, ~512 task, ~512 few-shot, remainder for generation
Make allocations configurable via localcode.toml
Token counting using the loaded model's actual tokenizer (not approximation)
Enforce budgets: if a section exceeds its allocation, truncate intelligently (prefer dropping the least relevant content, not cutting mid-line)
Report budget utilization so we can tune allocations over time

Prompt construction for 3B models is a zero-sum game. Every token wasted on low-value context is a token stolen from something useful.

Implement hard token budget allocation and enforcement for prompt construction. - Define budget slots: system prompt, code context, task description, few-shot examples, generation headroom - Default allocations for an 8K context window: ~256 system, ~1536 code, ~512 task, ~512 few-shot, remainder for generation - Make allocations configurable via localcode.toml - Token counting using the loaded model's actual tokenizer (not approximation) - Enforce budgets: if a section exceeds its allocation, truncate intelligently (prefer dropping the least relevant content, not cutting mid-line) - Report budget utilization so we can tune allocations over time Prompt construction for 3B models is a zero-sum game. Every token wasted on low-value context is a token stolen from something useful.