π Documentation |
π Quick Start |
π¦ Installation |
π¬ Slack
- [2026/07] π§ v0.25.1 under developmentβ Added Qwen3.5 / Qwen3.5-MoE, Gemma4 (text and multimodal), GLM MoE DSA, and DFlash speculative decoding
- [2026/02] β‘ Performance optimizationsβ Fused MoE with small batches, optimized attention metadata building, Multi-LoRA inference achieves 80%+ of non-LoRA performance
- [2026/02] π§ DeepSeek-V3.2 MTP supportβ Added MTP (Multi-Token Prediction) for DeepSeek-V3.2, with RoPE and decoding stage kernel optimizations
- [2026/01] π’ New quantization methodsβ Support for compressed-tensors W4A16, AWQ MoE W4A16, and DeepSeek-V3.2 W8A8 quantization
- [2026/01] π οΈ CI/CD overhaulβ Added E2E tests, unit test CI, ruff format checks, and modular CI workflow refactoring
- [2025/12] π v0.11.0 releasedβ Added Qwen3-Omni, Qwen3-Next, Seed-OSS support (Release Notes)
- [2025/12] π¦ v0.10.1.1 releasedβ 5+ multimodal models, AWQ/GPTQ quantization for dense models, Piecewise Kunlun Graph, vLLM V1 engine, Flash-Infer Top-K/Top-P sampling with 10-100Γ speedup (Release Notes)
- [2025/12] π Initial release of vLLM Kunlun β Open sourced on Dec 8, 2025
vLLM Kunlun (vllm-kunlun) is a community-maintained hardware plugin designed to seamlessly run vLLM on the Kunlun XPU. It is the recommended approach for integrating the Kunlun backend within the vLLM community, adhering to the principles outlined in the RFC Hardware Pluggable.
This plugin provides a hardware-pluggable interface that decouples the integration of the Kunlun XPU with vLLM. By utilizing vLLM Kunlun, popular open-source models β including Transformer-like, Mixture-of-Expert (MoE), Embedding, and Multi-modal LLMs β can run effortlessly on the Kunlun XPU.
- Seamless Plugin Integrationβ Works as a standard vLLM platform plugin via Python entry points, no need to modify vLLM source code
- Broad Model Supportβ Supports 20+ mainstream LLMs including Qwen, Llama, DeepSeek, GLM, Gemma4, Kimi-K2, and multimodal models
- Quantization Supportβ W8A8 (INT8), AWQ, GPTQ, and compressed-tensors W4A16 for MoE and dense models
- LoRA Fine-Tuningβ LoRA and Multi-LoRA adapter support for Qwen series models
- Piecewise Kunlun Graphβ Hardware-accelerated graph optimization for high-performance inference
- FlashMLA Attentionβ Optimized multi-head latent attention for DeepSeek MLA architectures
- Speculative Decodingβ MTP (Multi-Token Prediction) for DeepSeek-V3.2 and DFlash/EAGLE-style proposers
- Tensor Parallelismβ Multi-device parallel inference with distributed execution support
-
OpenAI-Compatible APIβ Serve models with the standard OpenAI API interface
-
Hardware: Kunlun3 P800
- OS: Ubuntu 20.04
- Software:- Python >= 3.10
- PyTorch >= 2.5.1 (KL3-customized
xpytorchbuild, see Installation) - vLLM (matching version, see requirements.txt)
- transformers == 5.2.0 (Qwen3.5 requires transformers 5.x)
| Model | Support | Quantization | LoRA | Kunlun Graph |
|---|---|---|---|---|
| Qwen2 | β
| β
| β
| β
|
| Qwen2.5 | β
| β
| β
| β
|
| Qwen3 | β
| β
| β
| β
|
| Qwen3-Moe | β
| β
| β
| β
|
| Qwen3-Next | β
| β
| β
| β
|
| Qwen3.5 | β
| β
| β
| |
| Qwen3.5-Moe | β
| β
| β
| |
| MiMo-V2-Flash | β
| β
| β
| |
| Llama2 | β
| β
| β
| β
|
| Llama3 | β
| β
| β
| β
|
| Llama3.1 | β
| β
| β
| |
| gpt-oss | β
| β
| ||
| Gemma4 | β
| β
| ||
| GLM4.5 | β
| β
| β
| |
| GLM4.5Air | β
| β
| β
| |
| GLM5 | β
| β
| β
| |
| InternLM2 | β
| β
| ||
| Seed-OSS | β
| β
| ||
| DeepSeek-R1 | β
| β
| β
| |
| DeepSeek-V3 | β
| β
| β
| |
| DeepSeek-V3.2 | β
| β
| β
| |
| Kimi-K2 | β
| β
| β
|
| Model | Support | Quantization | LoRA | Kunlun Graph |
|---|---|---|---|---|
| Qwen2-VL | β
| β
| β
| |
| Qwen2.5-VL | β
| β
| β
| |
| Qwen3-VL | β
| β
| β
| |
| Qwen3-VL-MoE | β
| β
| β
| |
| Gemma4 | β
| β
| ||
| InternVL-2.5 | β
| β
| ||
| InternVL-3.5 | β
| β
| ||
| InternS1 | β
| β
|
Current environment: 16-way concurrency, input/output size 2048.
python -m vllm.entrypoints.openai.api_server \
--host 0.0.0.0 \
--port 8356 \
--model <your-model-path> \
--gpu-memory-utilization 0.9 \
--trust-remote-code \
--max-model-len 32768 \
--tensor-parallel-size 1 \
--dtype float16 \
--max_num_seqs 128 \
--max_num_batched_tokens 32768 \
--block-size 128 \
--no-enable-prefix-caching \
--no-enable-chunked-prefill \
--distributed-executor-backend mp \
--served-model-name <your-model-name>
curl http://localhost:8356/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "<your-model-name>",
"messages": [{"role": "user", "content": "Hello!"}],
"max_tokens": 512
}'
| Version | Release Type | Documentation |
|---|---|---|
| v0.25.1 | Latest development version ( main) | Quick Start Β· Installation |
| v0.11.0 | Latest stable release | Quick Start Β· Installation |
vllm-kunlun/
βββ vllm_kunlun/ # Core plugin package
β βββ platforms/ # Kunlun XPU platform implementation
β βββ models/ # Model implementations (DeepSeek, Qwen, Gemma4, InternVL, etc.)
β βββ ops/ # Custom operators
β β βββ attention/ # FlashMLA, paged attention, merge attention states
β β βββ fla/ # Flash linear attention operations
β β βββ fused_moe/ # Fused MoE kernels
β β βββ mamba/ # Mamba / linear-attention state ops
β β βββ rotary_embedding/ # RoPE variants
β βββ quantization/ # AWQ, GPTQ, compressed-tensors, moe_wna16
β βββ lora/ # LoRA / Multi-LoRA support
β βββ v1/ # vLLM V1 engine adaptations (incl. spec decode: MTP, DFlash)
β βββ distributed/ # Communicators and distributed helpers
β βββ entrypoints/ # OpenAI-compatible serving overrides and tool parsers
β βββ reasoning/ # Reasoning parsers (Qwen3, Gemma4)
β βββ compilation/ # Torch compile wrapper for Kunlun Graph
β βββ transformers_utils/ # transformers config/tokenizer adaptations
β βββ csrc/ # C++ extensions (custom kernels)
β βββ config/ # Model configuration overrides
βββ tests/ # Test suite
βββ docs/ # Documentation (Sphinx-based, ReadTheDocs hosted)
βββ ci/ # CI pipeline configurations
βββ setup.py # Legacy build script (with C++ extensions)
βββ pyproject.toml # Modern Python build configuration (hatchling)
We welcome contributions from the community! Please read our Contributing Guide before submitting a PR.
Use the following prefixes for PR titles:
[Attention]β Attention mechanism features/optimizations[Core]β Core vllm-kunlun logic (platform, attention, communicators, model runner)[Kernel]β Compute kernels and ops[Bugfix]β Bug fixes[Doc]β Documentation improvements[Test]β Tests[CI]β CI/CD improvements[Misc]β Other changes
We opened the project at Dec 8, 2025. We love open source and collaboration β€οΈ