Qwen3.5 Deployment

Ollama bundles a complete Qwen3.5 model manager and inference engine into a single tool. After installation, one command pulls and runs any model size and automatically starts a local OpenAI-compatible API on port 11434. Zero configuration needed.

① Install Ollama

brew install ollama

curl -fsSL https://ollama.com/install.sh | sh

② Choose your Qwen3.5 version based on VRAM

# No discrete GPU / CPU-only — 0.8B (file size ~1.0 GB)
ollama run qwen3.5:0.8b

# 8 GB VRAM (RTX 3060, etc.) — 4B (file size ~3.4 GB)
ollama run qwen3.5:4b

# 16 GB VRAM (RTX 3090, etc.) — 9B (file size ~6.6 GB, recommended)
ollama run qwen3.5:9b

# 24 GB+ VRAM (RTX 4090, etc.) — 27B (file size ~17 GB)
ollama run qwen3.5:27b

③ Optional: Add Open-WebUI for a Chat Interface

# Requires Docker Desktop to be installed
docker run -d -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  --name open-webui ghcr.io/open-webui/open-webui:main
# Open http://localhost:3000 in your browser

④ OpenAI-Compatible API Call

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:9b",
    "messages": [{"role":"user","content":"Introduce yourself briefly."}],
    "temperature": 0.6,
    "top_k": 20,
    "top_p": 0.95,
    "max_tokens": 2048
  }'

llama.cpp is the most widely supported local Qwen3.5 inference engine, running on Windows, macOS, and Linux. It can run purely on CPU, and the -ngl flag offloads any number of layers to a GPU — enabling flexible CPU+GPU hybrid inference when your VRAM is tight.

① Install llama.cpp

# macOS — simplest method
brew install llama.cpp

# Linux / Windows — compile from source for best performance
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j 8
# Binaries will be in ./build/bin/ — add to PATH for convenience

② Download a Qwen3.5 GGUF model file

# From Hugging Face (international users)
pip install huggingface_hub
huggingface-cli download Qwen/Qwen3.5-4B-GGUF \
  qwen3.5-4b-q4_k_m.gguf --local-dir ./models

# From ModelScope (China mirror, faster)
pip install modelscope
modelscope download Qwen/Qwen3.5-4B-GGUF \
  qwen3.5-4b-q4_k_m.gguf --local_dir ./models

Quantization guide: Q4_K_M (balanced, recommended) · Q6_K (higher quality, +50% VRAM) · Q2_K (extreme compression, quality loss). The Q4_K_M 4B file is about 2.5 GB; 9B is about 5.5 GB.

③ Run the model

# Interactive chat — CPU only (-t = thread count, use physical cores)
llama-cli -m ./models/qwen3.5-4b-q4_k_m.gguf \
  --jinja --color -t 8 \
  --temp 0.6 --top-k 20 --top-p 0.95 \
  -c 40960 --no-context-shift

# Full GPU mode (-ngl 99 = all layers on GPU, -fa = FlashAttention)
llama-cli -m ./models/qwen3.5-4b-q4_k_m.gguf \
  --jinja --color -ngl 99 -fa \
  --temp 0.6 --top-k 20 --top-p 0.95 \
  -c 40960 --no-context-shift

# CPU+GPU hybrid (e.g., -ngl 20 = offload first 20 layers to GPU)
llama-cli -m ./models/qwen3.5-4b-q4_k_m.gguf \
  --jinja --color -ngl 20 \
  --temp 0.6 --top-k 20 --top-p 0.95 -c 40960

④ Start an HTTP server (OpenAI-compatible API)

llama-server -m ./models/qwen3.5-4b-q4_k_m.gguf \
  --jinja -ngl 99 -fa -c 40960 \
  --host 0.0.0.0 --port 8080
# Web UI:  http://localhost:8080
# API:     http://localhost:8080/v1/chat/completions

vLLM uses PagedAttention memory management and continuous batching to multiply Qwen3.5 inference throughput by several times over vanilla Transformers. It is the recommended choice for GPU server deployments that need to serve multiple concurrent users. Requires CUDA 11.8+.

① Install vLLM

pip install "vllm>=0.8.5"

② Start the Qwen3.5 inference server

# Single 16 GB GPU — deploy Qwen3.5-9B (32K context)
vllm serve Qwen/Qwen3.5-9B-Instruct \
  --port 8000 \
  --reasoning-parser qwen3 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9

# Tensor-parallel across 2 GPUs — deploy Qwen3.5-27B
vllm serve Qwen/Qwen3.5-27B-Instruct \
  --port 8000 \
  --tensor-parallel-size 2 \
  --reasoning-parser qwen3 \
  --max-model-len 32768

# 8× A100 cluster — flagship Qwen3.5-397B-A17B MoE
vllm serve Qwen/Qwen3.5-397B-A17B-Instruct \
  --port 8000 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --reasoning-parser qwen3

③ Test the API (OpenAI-compatible format)

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3.5-9B-Instruct",
    "messages": [{"role":"user","content":"Hello, introduce yourself."}],
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "max_tokens": 2048
  }'

For Apple M1 through M4 chip users, MLX is the definitive local deployment solution. The framework is fully optimized for Apple Silicon's unified memory architecture — the CPU and GPU share the same memory pool — so Qwen3.5 inference is both faster and more power-efficient than llama.cpp on any M-series Mac.

① Install mlx-lm

pip install mlx-lm

② Choose model based on unified memory size

# 8 GB unified RAM (M1 base) — 4B recommended
mlx_lm.chat --model mlx-community/Qwen3.5-4B-Instruct-4bit

# 16 GB unified RAM (M1 Pro / M2) — 9B recommended
mlx_lm.chat --model mlx-community/Qwen3.5-9B-Instruct-4bit

# 32 GB unified RAM (M1 Max / M3 Pro) — can run 27B
mlx_lm.chat --model mlx-community/Qwen3.5-27B-Instruct-4bit

# One-shot generation (no interactive chat)
mlx_lm.generate \
  --model mlx-community/Qwen3.5-4B-Instruct-4bit \
  --max-tokens 2048 \
  --prompt "Describe the architecture of Qwen3.5 in detail."

③ Start an OpenAI-compatible API server

mlx_lm.server \
  --model mlx-community/Qwen3.5-4B-Instruct-4bit \
  --port 8000

# Test the API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"mlx-community/Qwen3.5-4B-Instruct-4bit",
       "messages":[{"role":"user","content":"Hello"}],
       "temperature":0.6,"max_tokens":1024}'