Fastest GPU For Text Generation

Running an LLM locally: tokens per second on the models people actually run.

Qwen3-4B

Weights: Qwen/Qwen3-4B-GGUF

Measured in tok/s. Longer is faster.

Llama-3.1-8B

Weights: unsloth/Llama-3.1-8B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Qwen3 8B

Weights: Qwen/Qwen3-8B-GGUF

Measured in tok/s. Longer is faster.

Qwen2.5-Coder-14B

Weights: Qwen/Qwen2.5-Coder-14B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Gemma 4 12B

Weights: bartowski/google_gemma-4-12b-it-GGUF

Measured in tok/s. Longer is faster.

Qwen3 14B

Weights: Qwen/Qwen3-14B-GGUF

Measured in tok/s. Longer is faster.

Granite 4.1 3B

Weights: ibm-granite/granite-4.1-3b-GGUF

Measured in tok/s. Longer is faster.

LFM2.5 2.6B

Weights: LiquidAI/LFM2.5-2.6B-GGUF

Measured in tok/s. Longer is faster.

MiniCPM5 2B

Weights: openbmb/MiniCPM5-2B-GGUF

Measured in tok/s. Longer is faster.

Nemotron 3 Nano 4B

Weights: lmstudio-community/NVIDIA-Nemotron-3-Nano-4B-GGUF

Measured in tok/s. Longer is faster.

Ornith 1.5 9B

Weights: ornith-ai/Ornith-1.5-9B-GGUF

Measured in tok/s. Longer is faster.

DeepSeek Coder 7B Instruct v1.5

Weights: mradermacher/deepseek-coder-7b-instruct-v1.5-GGUF

Measured in tok/s. Longer is faster.

Llama 3 8B

Weights: bartowski/Meta-Llama-3-8B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Nemotron Nano 9B v2

Weights: bartowski/nvidia_NVIDIA-Nemotron-Nano-9B-v2-GGUF

Measured in tok/s. Longer is faster.

Qwen3 32B

Weights: unsloth/Qwen3-32B-GGUF

Measured in tok/s. Longer is faster.

Qwen3 30B A3B

Weights: bartowski/Qwen_Qwen3-30B-A3B-GGUF

Measured in tok/s. Longer is faster.

Llama 3.2 1B

Weights: bartowski/Llama-3.2-1B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Qwen2.5-7B

Weights: bartowski/Qwen2.5-7B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Qwen2.5-Coder 7B

Weights: bartowski/Qwen2.5-Coder-7B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Qwen3 0.6B

Weights: Qwen/Qwen3-0.6B-GGUF

Measured in tok/s. Longer is faster.

gpt-oss-20b

Weights: unsloth/gpt-oss-20b-GGUF

Measured in tok/s. Longer is faster.

Gemma 4 26B A4B

Weights: bartowski/google_gemma-4-26B-A4B-it-GGUF

Measured in tok/s. Longer is faster.

Llama 3.3 70B

Weights: bartowski/Llama-3.3-70B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Qwen3-Coder 30B A3B

Weights: bartowski/Qwen_Qwen3-Coder-30B-A3B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Gemma 4 31B

Weights: unsloth/gemma-4-31B-it-GGUF

Measured in tok/s. Longer is faster.

DeepSeek-R1 Distill Llama 8B

Weights: bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF

Measured in tok/s. Longer is faster.

DeepSeek-R1 Distill 1.5B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF

Measured in tok/s. Longer is faster.

DeepSeek-R1 Distill 14B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF

Measured in tok/s. Longer is faster.

DeepSeek-R1 Distill 7B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF

Measured in tok/s. Longer is faster.

Gemma 3 12B

Weights: bartowski/google_gemma-3-12b-it-GGUF

Measured in tok/s. Longer is faster.

Gemma 3 4B

Weights: bartowski/google_gemma-3-4b-it-GGUF

Measured in tok/s. Longer is faster.

Llama 3.2 3B

Weights: bartowski/Llama-3.2-3B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Mistral 7B v0.3

Weights: bartowski/Mistral-7B-Instruct-v0.3-GGUF

Measured in tok/s. Longer is faster.

Mistral Small 24B

Weights: bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF

Measured in tok/s. Longer is faster.

Phi-4 14B

Weights: bartowski/phi-4-GGUF

Measured in tok/s. Longer is faster.

Phi-4 Mini 3.8B

Weights: bartowski/microsoft_Phi-4-mini-instruct-GGUF

Measured in tok/s. Longer is faster.

Qwen3 1.7B

Weights: Qwen/Qwen3-1.7B-GGUF

Measured in tok/s. Longer is faster.

SmolLM3 3B

Weights: bartowski/HuggingFaceTB_SmolLM3-3B-GGUF

Measured in tok/s. Longer is faster.

Codestral 22B

Weights: bartowski/Codestral-22B-v0.1-GGUF

Measured in tok/s. Longer is faster.

Devstral Small 24B

Weights: bartowski/mistralai_Devstral-Small-2507-GGUF

Measured in tok/s. Longer is faster.

Dolphin Mistral 24B Venice

Weights: dphn/Dolphin-Mistral-24B-Venice-Edition-GGUF

Measured in tok/s. Longer is faster.

Dolphin X1 8B

Weights: dphn/Dolphin-X1-8B-GGUF

Measured in tok/s. Longer is faster.

Dolphin 3.0 Llama 3.1 8B

Weights: dphn/Dolphin3.0-Llama3.1-8B-GGUF

Measured in tok/s. Longer is faster.

Dolphin 3.0 R1 Mistral 24B

Weights: bartowski/cognitivecomputations_Dolphin3.0-R1-Mistral-24B-GGUF

Measured in tok/s. Longer is faster.

StarCoder2 15B

Weights: bartowski/starcoder2-15b-instruct-v0.1-GGUF

Measured in tok/s. Longer is faster.

Dolphin X1 Trinity Nano 6B

Weights: mradermacher/Dolphin-X1-Trinity-Nano-GGUF

Measured in tok/s. Longer is faster.

Dolphin-Mistral-24B-Venice-Edition

Weights: bartowski/cognitivecomputations_Dolphin-Mistral-24B-Venice-Edition-GGUF

Measured in tok/s. Longer is faster.

DeepSeek-Coder-V2-Lite

Weights: bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF

Measured in tok/s. Longer is faster.

DeepSeek-R1-0528-Qwen3-8B

Weights: lmstudio-community/DeepSeek-R1-0528-Qwen3-8B-GGUF

Measured in tok/s. Longer is faster.

EVA-Qwen2.5-14B-v0.2

Weights: bartowski/EVA-Qwen2.5-14B-v0.2-GGUF

Measured in tok/s. Longer is faster.

gemma-2-2b-it-abliterated

Weights: bartowski/gemma-2-2b-it-abliterated-GGUF

Measured in tok/s. Longer is faster.

gemma-2-2b

Weights: bartowski/gemma-2-2b-it-GGUF

Measured in tok/s. Longer is faster.

gemma-2-9b

Weights: bartowski/gemma-2-9b-it-GGUF

Measured in tok/s. Longer is faster.

gemma-3-1b

Weights: lmstudio-community/gemma-3-1b-it-GGUF

Measured in tok/s. Longer is faster.

gemma-3-270m

Weights: unsloth/gemma-3-270m-it-GGUF

Measured in tok/s. Longer is faster.

SmolLM3-3B

Weights: bartowski/HuggingFaceTB_SmolLM3-3B-GGUF

Measured in tok/s. Longer is faster.

LFM2.5-1.2B

Weights: unsloth/LFM2.5-1.2B-Instruct-GGUF

Measured in tok/s. Longer is faster.

Llama-2-7B

Weights: TheBloke/Llama-2-7B-Chat-GGUF

Measured in tok/s. Longer is faster.