Skip to content
myny2005Public
forked from HMS-lab/memcache

About

Fork of https://gitcode.com/Ascend/memcache

Resources

Code of conduct

Contributing

Stars

0 stars

Watchers

0 watching

Forks

 
 

Repository files navigation


High-performance distributed key-value cache

Docs PyPI PyPI - Python Version PyPI - Downloads GitCode commit activity license


🔄 Latest News

  • [2026/06] MemCache enables PrefixCache acceleration for inference. See the wiki case library for the latest examples.

  • [2025/12] MemCache is available as a vllm-ascend backend to accelerate LLM inference. See the vllm-ascend open-source community and the usage example.

  • [2025/11] MemCache was open-sourced in November 2025. The community repository is https://gitcode.com/Ascend/memcache.

🔜 Roadmap and Release Strategy

See the MemCache roadmap: Roadmap. See the MemCache branch release strategy.

🎉 Overview

MemCache is a high-performance distributed KVCache storage engine designed for LLM and GR inference. Its main features include:

  • Object-based API: Supports batched and non-batched put/get/exist/remove operations, as well as read/write APIs for multi-layer KV blocks.
  • Tiered cache pool: Supports HBM, DDR, and SSD tiers, moving data between tiers through eviction and prefetch.
  • High bandwidth and low latency: Uses MemFabric as the foundation for tiered memory and heterogeneous network transfers. On Ascend hardware, paths including device_rdma (A2), device_sdma (A3), host_rdma (A2/A3), and device_urma (A5) provide OneCopy direct data access across machines and media for high-bandwidth, low-latency reads and writes. On Kunpeng hardware, host_urma (K5) is supported. host_shm enables shared-memory communication on the same node.
  • Scaling: LocalService instances can join and leave dynamically.
  • High availability: In Kubernetes clusters, MetaService supports active-active operation and metadata recovery, providing best-effort HA.
architecture

🧩 Core Components

MemCache consists of two core components: LocalService and MetaService.

  • MetaService:

    • Manages memory-pool allocation across the cluster and handles LocalService join and leave events.
    • Runs as a standalone process and can be started through the Python API or as a binary. See wheel installation and usage and run-package installation and usage.
    • Supports two deployment modes: 1. Standalone mode: A single MetaService process is easy to deploy, but creates a single point of failure. If the process crashes or becomes unreachable, the system cannot provide service until it recovers. 2. HA mode: Uses Kubernetes ClusterIP Services and Lease resources. This more complex deployment runs multiple MetaService instances for active-active high availability. See how to deploy a MemCache HA cluster.
  • LocalService: Provides the following functions:

    • Client: Exposes APIs to application processes through a shared library distributed as a wheel or .so file.
    • Memory provider: Provides a contiguous memory region for the memory pool. Other LocalService instances can access it directly by address.

📁 Project Structure

MemCache's core code follows a modular design, with the main implementation under src/memcache/csrc.

src/memcache/csrc/                        # C++ core implementation
  ├── meta_service/                       # MetaService: cluster metadata, allocation, and eviction
  │    ├── mmc_meta_service.cpp           # Main MetaService logic
  │    ├── mmc_meta_manager.cpp           # Metadata manager (global key-to-blob map)
  │    ├── mmc_blob_allocator.cpp         # Memory-pool allocator
  │    ├── mmc_meta_container_lru.cpp     # LRU metadata container (temperature tracking and eviction)
  │    └── mmc_meta_net_server.cpp        # RPC network server
  ├── client/                             # Local client loaded by inference processes as a wheel/shared library
  │    ├── mmc_client_default.cpp         # Core client implementation (batch put/get/copy)
  │    ├── mmc_meta_net_client.cpp        # RPC network client
  ├── local_service/                      # LocalService: memory provider
  │    ├── mmc_local_service_default.cpp  # Main LocalService implementation
  │    └── mmc_bm_proxy.cpp               # MemFabric BM semantic proxy
  └── under_api/                          # Foundation integration layer (MemFabric / UBSIO)
       ├── mf_smem/                       # MemFabric shared-library integration (HBM / DRAM tiers)
       │    └── smem_bm_api.cpp           # Batch memory-copy API
       └── ubs_io/                        # UBSIO shared-library integration (SSD tier)
            ├── mmc_ubs_io_proxy.cpp      # UBSIO proxy wrapper
            └── dl_ubsio_api.cpp          # Dynamic loading of UBSIO

🔥 Performance

MemCache provides large memory pools and high-performance H2D, D2H, D2RH, and RH2D data access. Because MemCache uses MemFabric as its pooling foundation, it supports OneCopy direct data access across machines and media, including RH2D and D2RH. The figure below compares RH2D with other transfer paths.

RH2D

Based on OneCopy direct access across machines and media, the following performance tests were run on A2/A3. The test simulates KV-sized blocks for the DeepSeek-R1 model. Each block is 61×128K + 61×16K = 8784KB ≈ 8.57MB across 122 discontiguous addresses.

  • Read/write performance using a two-node memory pool with two Ascend A2 nodes (8 cards per node):
A2 perf
  • Read/write performance using a two-node memory pool with two Ascend A3 nodes (8 cards and 16 dies per node):
A3 perf

🚀 Quick Start

See the following guides to get started:

📑 Tutorials

📦 Software and Hardware Compatibility

📌 FAQ

See the FAQ for common questions.

📝 Related Information

Note

This open-source project is not a Huawei product and provides limited support.

About

Fork of https://gitcode.com/Ascend/memcache

Resources

Code of conduct

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages