--- title: "Harnessing LMCache: Building a Pluggable KV-Cache Management & Testing Framework" author: "Alain Airom (Ayrom)" published: "2026-11-10T08:17:51.000Z" updated: "2026-11-10T08:24:52.889Z" url: "https://zyvop.com/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework-xhv80" canonical: "https://dev.to/aairom/harnessing-lmcache-building-a-pluggable-kv-cache-management-testing-framework-5fej" tags: ["lmcache","Software","kvcache","llmcaching","LLM"] --- # Harnessing LMCache: Building a Pluggable KV-Cache Management & Testing Framework ![Cover image]() LMCache test & implementation. ![ ]() ## Introduction As Large Language Model (LLM) architectures scale and context windows expand to tens of thousands of tokens, prefill latency has become a major bottleneck in AI application performance. Processing long, repeated prompt prefixes—such as system prompts, core reference documents in Retrieval-Augmented Generation (RAG), and extensive agent chat histories—consumes substantial compute cycles and drives up time-to-first-token (TTFT). ## What is LMCache? [LMCache]() is an open-source Key-Value (KV) cache management layer designed to solve this exact bottleneck. Rather than recomputing attention Key and Value tensors for every incoming request, LMCache captures, indexes, and stores these tensors outside the core inference engine, re-injecting them on demand. Key features of LMCache include: - Chunk-Level Prefix Reuse: Token streams are partitioned into fixed-size chunks (defaulting to 256 tokens). Each chunk is assigned a rolling prefix hash (using `blake3`), guaranteeing that matching prefixes immediately hit cached states. - Tiered Storage Architecture: Cache snapshots flow seamlessly across device memory (GPU/CPU RAM), local disk (L2), and remote backends (e.g., Redis, S3, Mooncake, NIXL) with Least Recently Used (LRU) eviction policies. - Engine Independence: Cache snapshots survive engine restarts and slot evictions, allowing cross-instance sharing and disaggregated prefill/decode topologies. ## The LMCache Ecosystem The LMCache ecosystem spans mainstream open-source projects, enterprise hardware vendors, and infrastructure orchestrators: - Inference Engines: Native support targets server-grade Linux/CUDA engines such as vLLM, SGLang, and TensorRT-LLM. - Orchestration Frameworks: Integrates with orchestrators like NVIDIA Dynamo, vLLM Production Stack, `llm-d`, and KServe. - Hardware & Storage Ecosystem: Compatible with storage vendors (Dell, Redis, DDN, Valkey, WEKA, VAST, Samsung, IBM Storage) and accelerator providers (NVIDIA, AMD, Ascend, Intel, Moore Threads, Rebellions, MetaX). - Research Extensions: Incorporates advanced research ideas including TurboQuant, CacheBlend, CacheGen, and Cartridges. > **Excerpt from official Github repository of the project:** ![ ]() LMCache is a **KV cache management layer** for LLM inference. It turns KV cache from a temporary state into reusable _AI-native knowledge_ that can be _stored_ persistently, _reused_ across multiple serving engines, _monitored_ with an observability stack, and _transformed_ for better generation quality. As a result, LMCache **reduces TTFT** (time-to-first-token) and **improves throughput**, especially for long-context agentic, multi-turn conversation, and knowledge-augmented workloads (e.g., RAG). LMCache is **vendor-neutral**. It can be used as a KV cache layer for a range of mainstream open-source serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers. The vendor neutrality allows users to freely switch between serving engines and storage vendors, while reusing the stored KV caches. ![ ]() ### Key features - **Engine-independent deployment**: LMCache, as a standalone daemon process, manages KV cache independently from the inference engine process, so that KV cache will not be lost even if the inference engine crashes (i.e., no fate-sharing with engines). - **Persistent, tiered KV cache offloading and reuse**: Move KV caches out of GPU memory into a tiered storage hierarchy spanning CPU memory, local storage, and remote backends, enabling reuse across requests, sessions, and engine instances to reduce repeated prefill computation and improve TTFT. - **Production-level KV cache observability**: LMCache provides a rich set of KV cache observability metrics, including typical Kubernetes metrics (health monitoring, performance diagnostics), KV-cache-specific metrics (request-level and token-level prefix cache hits, lifecycle, request-level KV cache performance), management metrics (user-specific usage), and more. - **Pluggable storage and transport backends**: Easily integrate remote storage and KV transfer backends through a unified interface, enabling KV cache offloading and sharing across storage providers. Through this interface, LMCache supports storage backends including CPU RAM, local disk (SSD), Redis/Valkey, Mooncake, InfiniStore, S3-compatible object storage, NIXL, and GDS. - **Non-prefix KV reuse**: Extend KV reuse beyond prefix caching by reusing cached KV blocks at any position in the prompt. This leverages CacheBlend to selectively recompute tokens for quality recovery. - **PD disaggregation and KV transfer**: Support KV cache transfer from prefill workers to decode workers over NVLink, RDMA, or TCP through transport layers such as NIXL. - **Pluggable KV transformation**: A simple interface for researchers to write compression, token dropping, and custom serialization through a flexible SERDE interface. LMCache is becoming an integral layer in the LLM inference _ecosystem_, with _community_\-driven integration with serving engines, inference frameworks, hardware vendors, storage systems, and infrastructure providers: ![ ]() * * * ## Implementation > **Disclaimer: _Officially, LMCache targets server-side GPU inference engines (vLLM, SGLang, TRT-LLM) on Linux; official PyPI wheels require CUDA, and feature requests for Ollama/LM Studio support (e.g., issue #923) were closed as "not planned". However, for a mock test and rapid local prototyping, an LMCache-style framework named_ _KVBridge_ \*was implemented to bring these concepts to my local test environment.**\* ![ ]() ## Application overview & purpose KVBridge demonstrates the core value proposition of [LMCache]() — _a KV cache management layer for scalable LLM inference_ — on a developer laptop. LMCache itself targets server-side engines (vLLM, SGLang, TensorRT-LLM) and needs CUDA. KVBridge **re-implements the LMCache concepts** (chunked, content-addressed KV reuse; tiered storage; engine-independent keys) on top of the engines developers actually run: two local ones (llama.cpp, Ollama) and any OpenAI-compatible cloud service, all behind one pluggable engine registry. > **What KVBridge is, and is not** > > KVBridge is an LMCache-_style_ implementation. It is **not** the LMCache package. It optionally reuses LMCache's native `TokenHasher` when the real package can be built (`scripts/install_lmcache.sh`); otherwise a bit-compatible `blake3` rolling hash is used. ## System Architecture & Components KVBridge mirrors LMCache's core mechanisms (rolling prefix hashing, tiered storage, and cache persistence) while adding an extensible engine registry to support local engines (`llama.cpp`, `Ollama`) and any OpenAI-compatible cloud provider. ![ ]() ## Code Layout The local application, uses Gradio for the UI implementation and is configurable to connect to multiple backends. ```python lmcache-test/ ├── kvbridge/ │ ├── __init__.py │ ├── __main__.py # python -m kvbridge │ ├── app.py # Gradio GUI (registry-driven), AppState, main(), SIGTERM handler │ ├── backends.py # engine registry, LlamaCppBackend, OllamaBackend, CloudBackend, make_backend │ ├── engine.py # KVBridgeEngine: lookup, restore, compute, store │ ├── store.py # L1 RAM / L2 disk tiered store │ ├── hashing.py # chunking + chained hashes │ ├── metrics.py # counters, TTFT, hit rates │ ├── charts.py # Plotly figures │ ├── scenarios.py # demo scenario registry │ └── config.py # Settings (GUI/env/.env), presets, update_env_file ├── tests/ # unit + live tests ├── scripts/ # setup, start, stop, test, cleanup, install_lmcache ├── Docs/ # Quickstart, Architecture, Research, this file ├── input/ output/ # tracked via .gitkeep, contents ignored ├── requirements.txt ├── pytest.ini ├── .env.example # every KVB_* variable, local and cloud examples └── README.md # includes the Apache-2.0 licence statement ``` Enter fullscreen mode Exit fullscreen mode ![ ]() ## Evaluation Modes To evaluate performance gains, the architecture routes requests through three distinct operational modes: 1. **cold**: Disables native prompt caching (`cache_prompt=false`) and wipes existing slot state, forcing a full prefill calculation on every request to establish a baseline. 2. **native**: Relies strictly on the engine's built-in, in-slot prefix caching (`cache_prompt=true`). 3. **lmcache**: Drops in-slot state, performs chunk-hash lookups against the L1/L2 store, restores cached tensor state into the engine slot, pre-fills only the un-cached tail suffix, and saves new chunk snapshots upon completion. - **Full KV State Export and Import (**`engine.py`**)**: the primary pipeline manages KV tensor persistence using (only for mock test purpose) `llama.cpp`'s `/slots/{id}?action=save|restore` API: ```bash ""The cache-aware inference pipeline. Request flow (llama.cpp, full integration):: prompt --tokenize--> tokens --chunk+hash--> [h0 h1 h2 ...] | store.lookup (longest prefix) <-+ |hit |miss fetch snapshot (L1 RAM / L2 disk) | restore into engine slot | +---------+----------+ v engine.generate (prefills only the *missing* suffix) v save slot -> store.put (only when new chunks were computed) Three execution modes make the benefit measurable: ``cold`` prefix reuse disabled - every request prefills the whole prompt. ``native`` the engine's own slot/prefix cache only (baseline a user has today). ``lmcache`` native cache is dropped first (modelling eviction by another session, a restart, ...) and the KV layer restores state instead. """ ... def _infer_full(self, prompt: str, rec: RequestRecord, n_predict: int, temperature: float) -> None: """llama.cpp path: real KV tensor save/restore handling.""" b = self.backend tokens = b.tokenize(prompt) if rec.mode == "cold": b.reset_native_cache() self._fill(rec, b.generate(prompt, tokens, n_predict, False, temperature)) return if rec.mode == "native": self._fill(rec, b.generate(prompt, tokens, n_predict, True, temperature)) return # ---- LMCache Mode Pipeline hashes = self.hasher.chunk_hashes(tokens) rec.chunks_total = len(hashes) rec.chunk_status = self.store.chunk_status(hashes) # Drop native slot state to isolate LMCache retrieval benefit b.reset_native_cache() match = self.store.lookup(hashes) rec.lmcache_match_tokens, rec.chunks_hit, rec.tier = match.tokens, match.chunks, match.tier # Restore matching KV snapshot from L1 RAM or L2 Disk if match.entry is not None: try: blob = self.store.fetch(match.entry) n = b.restore_state(blob) rec.restore_ms = (time.perf_counter() - t0) * 1000 except (BackendError, KeyError, OSError) as exc: log.warning("Restore failed (%s); falling back to recompute", exc) rec.chunks_hit, rec.lmcache_match_tokens, rec.tier = 0, 0, "none" b.reset_native_cache() # Compute generation (prefills missing suffix only) res = b.generate(prompt, tokens, n_predict, True, temperature) self._fill(rec, res) # Persist newly computed tail chunks into the Tiered Store if len(hashes) > rec.chunks_hit: blob, n_saved = b.save_state() self.store.put(hashes, len(hashes) * self.s.chunk_size, blob, note=f"{n_saved} tok snapshot") ``` Enter fullscreen mode Exit fullscreen mode - **Extensible Engine Registry (**`backends.py`**)**: to prevent hardcoded conditionals across engine implementations, a centralized registry handles dynamic initialization: ``` """Inference-engine adapters. Three adapters share one small interface (:class:`Backend`) and are looked up through a registry (:func:`register_engine`, :func:`resolve_engine`), so the engine is chosen purely by configuration (``KVB_ENGINE`` or the GUI): ``LlamaCppBackend`` Talks to ``llama-server`` (llama.cpp). llama.cpp exposes the **slot save/restore API** (``POST /slots/{id}?action=save|restore|erase``) which serialises the *real KV tensors* of a sequence. That is the integration point that lets an external cache layer (LMCache semantics) take KV out of the engine and inject it back -> **full integration**. ``OllamaBackend`` Talks to the Ollama REST API. Ollama deliberately hides KV state: there is no KV export/import and no tokenizer endpoint. The adapter therefore works in **shadow mode** - the LMCache layer indexes/accounts for chunks and predicts reuse, while Ollama's own prefix cache does the real work and its ``prompt_eval_cached_count`` is used as ground truth. ``CloudBackend`` Talks to any **OpenAI-compatible** HTTP API (OpenAI, Groq, OpenRouter, Together, Mistral, DeepSeek, vLLM, LM Studio, ...). Hosted engines never expose their KV tensors either, so it also runs in **shadow mode**; the provider's own prompt-cache counter (``cached_tokens``) is the ground truth. Prompts leave the machine - the GUI says so explicitly. """ from __future__ import annotations import jsonimport loggingimport osimport reimport shutilimport signalimport subprocessimport timeimport uuidimport zlibfrom abc import ABC, abstractmethodfrom dataclasses import dataclassfrom pathlib import Pathfrom typing import Optional, Sequencefrom urllib.parse import urlparse import requests from .config import CLOUD_API_STYLES, DEFAULT_LLAMA_SEARCH_PATHS, Settings...@dataclass(frozen=True)class EngineSpec: key: str # Identifier (e.g., KVB_ENGINE) label: str # GUI Label cls: type[Backend] # Adapter implementation class blurb: str = "" aliases: tuple[str, ...] = () _REGISTRY: dict[str, EngineSpec] = {} def register_engine(spec: EngineSpec) -> EngineSpec: _REGISTRY[_norm(spec.key)] = spec return spec def make_backend(kind: str, settings: Settings) -> Backend: return resolve_engine(kind).cls(settings) # Engine Registrationregister_engine(EngineSpec("llamacpp", "llama.cpp (local)", LlamaCppBackend, "Full KV save/restore through llama-server slots.", ("llama",)))register_engine(EngineSpec("ollama", "Ollama (local)", OllamaBackend, "Shadow mode: Ollama hides its KV cache.", ()))register_engine(EngineSpec("cloud", "Cloud / OpenAI-compatible API", CloudBackend, "Shadow mode against hosted engines.", ("openai", "api"))) ``` Enter fullscreen mode Exit fullscreen mode - **Chained Rolling Prefix Hasher (**`hashing.py`**)**: token streams are packed into big-endian `uint32` representations and processed sequentially: ```python """Chunk-level rolling prefix hashing, compatible with LMCache. LMCache splits a token sequence into fixed-size chunks (default 256 tokens) and identifies each chunk by a *rolling prefix hash*:: h_0 = blake3(prefix_seed || pack(chunk_0)) h_i = blake3(h_{i-1} || pack(chunk_i)) where ``pack`` is the big-endian ``uint32`` encoding of the token ids (``lmcache/v1/multiprocess/token_codec.py``). Only *complete* chunks are hashed; a trailing partial chunk is discarded - exactly like ``lmcache.v1.multiprocess.token_hasher.TokenHasher``. This module re-implements that algorithm (so the app runs on macOS where the full LMCache wheel cannot be installed) and can optionally delegate to the real ``TokenHasher`` when the ``lmcache`` package is importable. A unit test verifies both produce identical digests. """ from __future__ import annotations import hashlibimport loggingimport structfrom typing import Protocol, Sequence log = logging.getLogger(__name__) try: # blake3 is the algorithm LMCache uses import blake3 as _blake3 def _digest(data_parts: Sequence[bytes]) -> bytes: h = _blake3.blake3() for part in data_parts: h.update(part) return h.digest() ALGORITHM = "blake3"except ImportError: # pragma: no cover - fallback keeps the app usable def _digest(data_parts: Sequence[bytes]) -> bytes: h = hashlib.sha256() for part in data_parts: h.update(part) return h.digest() ALGORITHM = "sha256 (blake3 missing; NOT LMCache compatible)"def pack_tokens(tokens: Sequence[int]) -> bytes: """Pack token ids as big-endian ``uint32`` (LMCache wire format).""" return struct.pack(f">{len(tokens)}I", *tokens)def namespace_seed(namespace: str) -> bytes: """Derive the initial prefix hash from a model/config namespace. KV tensors are only valid for the exact model, quantisation and context configuration that produced them, so the namespace is mixed into the root of every hash chain (LMCache keys also embed the model name). """ return _digest([namespace.encode("utf-8")])class Hasher(Protocol): chunk_size: int impl: str def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]: ...class ChunkHasher: """Pure-Python implementation of LMCache's rolling chunk hashing.""" impl = "kvbridge (LMCache-compatible blake3)" def __init__(self, chunk_size: int = 256, namespace: str = "") -> None: if chunk_size < 1: raise ValueError("chunk_size must be >= 1") self.chunk_size = chunk_size self.namespace = namespace self.seed = namespace_seed(namespace) def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]: """Return one rolling hash per *complete* chunk of ``tokens``.""" n_complete = len(tokens) - len(tokens) % self.chunk_size prefix = self.seed out: list[bytes] = [] for i in range(0, n_complete, self.chunk_size): prefix = _digest([prefix, pack_tokens(tokens[i : i + self.chunk_size])]) out.append(prefix) return outclass LMCacheHasherAdapter: """Delegates to ``lmcache.v1.multiprocess.token_hasher.TokenHasher``.""" def __init__(self, chunk_size: int = 256, namespace: str = "") -> None: from lmcache.v1.multiprocess.token_hasher import TokenHasher # type: ignore self.chunk_size = chunk_size self.namespace = namespace self.seed = namespace_seed(namespace) self._hasher = TokenHasher(chunk_size=chunk_size, hash_algorithm="blake3") self.impl = "lmcache.TokenHasher (real LMCache)" def chunk_hashes(self, tokens: Sequence[int]) -> list[bytes]: return self._hasher.compute_chunk_hashes(list(tokens), prefix_hash=self.seed)def make_hasher(chunk_size: int, namespace: str, prefer_lmcache: bool = False) -> Hasher: """Create a hasher, preferring the real LMCache one when requested.""" if prefer_lmcache: try: return LMCacheHasherAdapter(chunk_size, namespace) except Exception as exc: # ImportError or LMCache init problems log.warning("LMCache TokenHasher unavailable (%s); using built-in hasher", exc) return ChunkHasher(chunk_size, namespace)def common_prefix_len(a: Sequence[int], b: Sequence[int]) -> int: """Length of the longest common prefix of two token sequences.""" n = min(len(a), len(b)) i = 0 while i < n and a[i] == b[i]: i += 1 return i ``` Enter fullscreen mode Exit fullscreen mode ## Configuration examples & usage scenarios Pick one block for `.env` (or enter the same values in the GUI and press _Connect_). - **.env — local, full integration (llama.cpp):** ```toml KVB_ENGINE=llamacpp KVB_OLLAMA_MODEL=llama3.2:latest # used to locate the GGUF blobKVB_CTX_SIZE=8192 KVB_LLAMA_START_TIMEOUT=300 # slow disk? allow more time ``` Enter fullscreen mode Exit fullscreen mode - **.env — local, shadow mode (Ollama):** ```toml KVB_ENGINE=ollama KVB_OLLAMA_URL=http://127.0.0.1:11434 KVB_OLLAMA_MODEL=llama3.2:latest KVB_OLLAMA_KEEP_ALIVE=30m ``` Enter fullscreen mode Exit fullscreen mode - **.env — hosted provider (key stays in your shell):** ```toml KVB_ENGINE=cloud KVB_CLOUD_PROVIDER=openai KVB_CLOUD_MODEL=gpt-4o-mini KVB_CLOUD_API_KEY_ENV=OPENAI_API_KEY # export OPENAI_API_KEY=... before start.shKVB_CLOUD_TOKEN_BUDGET=200000 # spend guard ``` Enter fullscreen mode Exit fullscreen mode - **.env — self-hosted vLLM / any OpenAI-compatible server:** ```toml KVB_ENGINE=cloud KVB_CLOUD_BASE_URL=https://llm.example.internal/v1 KVB_CLOUD_MODEL=meta-llama/Llama-3.1-8B-Instruct KVB_CLOUD_API_KEY_ENV=INTERNAL_LLM_KEY KVB_CLOUD_EXTRA_HEADERS={"X-Team": "search"} ``` Enter fullscreen mode Exit fullscreen mode - **.env — free smoke test of the cloud path (no account):** ```toml KVB_ENGINE=cloud KVB_CLOUD_PROVIDER=ollama-local-openai # http://127.0.0.1:11434/v1KVB_CLOUD_MODEL=llama3.2:latest ``` Enter fullscreen mode Exit fullscreen mode | Scenario | Engine | What to do | What you see | | --- | --- | --- | --- | | A. Maximum benefit, fully offline | llama.cpp | `KVB_ENGINE=llamacpp`, Connect, run _Document Q&A_ | First question prefills (seconds); later ones restore KV in tens of ms; _Engine restart_ shows an L2 restore | | B. Observe an existing Ollama setup | Ollama | `KVB_ENGINE=ollama`, Connect, run _Shared system prompt_ | Predicted reuse next to `prompt_eval_cached_count`; SHADOW label | | C. Measure a hosted provider's prefix cache | Cloud | Preset + model + key + token budget, Connect, run _Shared system prompt_ with a long input document | `cached_tokens` reported by the provider per request; `cold` adds a nonce and shows no reuse | | D. Try the cloud path for free | Cloud | Preset `ollama-local-openai`, model `llama3.2:latest` | Same GUI and metrics, no key, no cost | | E. Switch engines without editing files | any | Change the engine radio, edit fields, Connect, optionally _Save settings to .env_ | Status line states engine, location and FULL/SHADOW mode | - **Programmatic equivalent:** ```python from kvbridge.config import Settingsfrom kvbridge.backends import make_backendfrom kvbridge.engine import CachedEngine s = Settings.from_env() # honours KVB_ENGINE and .envassert not (problems := s.validate()), problemsengine = CachedEngine(s, make_backend(s.engine, s))engine.start() ``` Enter fullscreen mode Exit fullscreen mode * * * ## Final Disclaimer and findings after the tests 🚩 **As noted earlier, this application was designed to demystify LMCache functionality and practical usage. Operating as a local, GPU-free implementation, it offers a purely educational pathway to observe firsthand how tiered caching and prefix reuse enhance inference efficiency.** | Question | Answer | | --- | --- | | Does LMCache officially support Ollama / llama.cpp? | No Supported: vLLM, SGLang, TRT-LLM. Issue #923 (Ollama / LM Studio) was closed “not planned”. | | Can the PyPI wheel run on macOS? | No Wheels are manylinux-only and require CUDA (`CUDA_HOME not set`). | | Can LMCache's hasher be used standalone? | Partly With `NO_GPU_EXT=1` plus extra dependencies the native `TokenHasher` builds. | | Can llama.cpp export/import KV state? | Yes \`/slots/{id}?action=save | | Can a hosted (cloud) API export/import KV state? | No Never. Providers run their own prefix cache and report {% raw %}`cached_tokens`, enabling the same accounting-only _shadow mode_. | | Can Ollama export/import KV state? | No No public KV API. It does report `prompt_eval_cached_count`, enabling an accounting-only _shadow mode_. | * * * ## Conclusion Building a practical testbed around LMCache concepts provides valuable insights into modern prompt caching mechanics. Key takeaways from the project include: 1. **Measurable TTFT Reduction**: In benchmark scenarios (e.g., ~1.9k-token document Q&A), average TTFT dropped from 4,554 ms (cold) to 999 ms (lmcache)—achieving a 4.6x overall speedup across a 12-request workload. On warm hits, TTFT fell as low as 11 ms (L1 RAM) and 13 ms (L2 Disk). 2. **Persistence vs. Native In-Slot Caching**: While `native` engine caching can be faster on a single warm slot (e.g., 64 ms), the primary value of an LMCache-style architecture is capacity and persistence—snapshots survive process restarts, span arbitrary numbers of prefixes, and remain usable across multiple sessions. 3. **Observability & Control**: Through an intuitive Gradio GUI, automated test suites (over 180 unit/live tests), and robust SIGTERM lifecycle handling, developers can reliably test, visualize, and optimize prompt reuse strategies on any workstation. **Thanks for reading 🌒** ## Links - Code repository for this post: [https://github.com/aairom/KVBridge-lmcache-test]() - LMCache repository: [https://github.com/LMCache/LMCache]()