In progress
AgentOS
A per-agent KV-cache and context resource-control plane over vLLM V1.
- Role
- Sole author
- Stack
- vLLM V1 · custom scheduler · dynamic KV connector
- Code
- Repository →
84 + 44
CPU + real-torch KV tests
Per-agent
token and KV-block budgets
RTX A4000
vLLM runtime validation
How it works
- 01 Admission agent declares a budget
- 02 Scheduler priority + preemption
- 03 KV accounting blocks charged per agent
- 04 Eviction policy selectable, per-agent scope
- 05 Snapshot / restore resume, do not recompute
An agent is a process. Its KV cache is memory the scheduler owns.
KV block pool, per-agent ownership
Highlighted blocks belong to the priority agent. Under pressure the scheduler evicts inside a low-priority agent's own allocation instead of taking from whoever asked last.
The problem
Serving stacks treat KV cache as one shared pool. Run twenty agents against it and there is no per-agent budget, no policy for whose blocks get evicted, and no way to pause an agent and resume it with its context intact. A background agent can quietly evict the cache of the one a user is waiting on, and nothing in the stack considers that a bug.
The approach
Treat each agent the way an operating system treats a process: live per-agent token and KV-block budgets, a selectable eviction policy, priority and preemption, and KV snapshot/restore so a preempted agent resumes instead of recomputing its history. Built on vLLM's dynamic KV connector plus a custom scheduler.
The hard part
Correctness of injected KV. Restoring blocks into a running engine is easy to make fast and very easy to make subtly wrong — a misaligned block yields plausible tokens drawn from a corrupted past, which no throughput metric catches. The work went into invariants covered by 84 CPU tests and 44 real-torch KV tests.
Where it stands
Scheduler and snapshot paths run against vLLM V1 with the suite green. Nothing else ships per-agent KV as a managed, schedulable resource, which is the reason to keep building it.