Mistral Small 24B or Gemma 3 27B?

Our data and experience both say Mistral: 28% faster measured (119 vs 93 tok/s), ~3GB lighter floor, and task results that didn't favor the bigger model. Gemma's rebuttal is vision and multilingual breadth. Pick it only if you need those.

What GPU does it need?

~15GB measured peak at Q4_K_M: a 16GB card fits with little headroom; 24GB is comfortable. On rentals, the H200 measured 108 tok/s at 208W.

Where does it sit in the size hierarchy?

Just under our 27-32B sweet spot, and for many tasks, effectively in it. It's the strongest dense generalist below the 24GB hardware line, which is exactly what makes it interesting.

Is it good for coding?

Capable, but its Mistral siblings specialize better: Codestral 22B for completions, Devstral 24B for agent loops, both on the same base and hardware class. Keep this one as the generalist of the trio.

How does it fit a consensus stack?

As the fast dense anchor: direct answers at 119 tok/s peak while reasoning models deliberate. Add a DeepSeek distill for the thinking seat and an uncensored Dolphin as judge and the whole panel shares one 24GB card.

Mistral Small 24B · 10 GPUs measured first-party · llama.cpp Q4_K_M · Updated July 2026

What GPU Do You Need for Mistral Small 24B?

Mistral Small 24B is the quiet winner of the mid-size class, the model our testing keeps preferring over Google's bigger Gemma 3 27B. Measured on 10 GPUs (llama.cpp, Q4_K_M): 119 tok/s on the B300, ~15GB peak VRAM, 28% faster than the Gemma it competes with.

Benchmarked weights: bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF

119.3tok/s

Fastest: NVIDIA B300

measured, 3-run llama-bench

~15GB

VRAM needed (measured peak)

GPU-independent, applies to every card

GPUs measured

same pinned harness

0.52tok/W

Most efficient: NVIDIA H200

real power sampling, not TDP

What GPU Do You Need for Mistral Small 24B?, tok/s, fastest 10

NVIDIA B300

119.3 tok/s

NVIDIA B200

114.04 tok/s

NVIDIA H200

107.84 tok/s

NVIDIA H100 80GB HBM3

105.62 tok/s

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

93.55 tok/s

NVIDIA A100 40GB SXM4

62.92 tok/s

NVIDIA A100 80GB SXM4

61.38 tok/s

NVIDIA L40S

48.35 tok/s

NVIDIA A10G

33.04 tok/s

NVIDIA L4

17.26 tok/s

Measured on our own bench. A card absent from this chart has not been run on this model yet, or cannot fit it.

Mistral Small 24B. Measured generation speed by GPU

NVIDIA B300119.3

NVIDIA B200114.04

NVIDIA H200107.84

NVIDIA H100 80GB HBM3105.62

NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.55

NVIDIA A100 40GB SXM462.92

NVIDIA A100 80GB SXM461.38

NVIDIA L40S48.35

NVIDIA A10G33.04

NVIDIA L417.26

GPU	tok/s	Prompt t/s	tok/W	Avg power
NVIDIA B300	119.3	1993	0.35	344.7 W
NVIDIA B200	114.04	3789.1	0.34	331.0 W
NVIDIA H200	107.84	3408.5	0.52	207.6 W
NVIDIA H100 80GB HBM3	105.62	3562.5	0.44	240.6 W
NVIDIA RTX PRO 6000 Blackwell Workstation Edition	93.55	4891	0.41	230.3 W
NVIDIA A100 40GB SXM4	62.92	1632.2	0.41	152.0 W
NVIDIA A100 80GB SXM4	61.38	1644.6	0.4	154.2 W
NVIDIA L40S	48.35	3468	0.3	163.1 W
NVIDIA A10G	33.04	1494.5	0.27	122.2 W
NVIDIA L4	17.26	967	0.31	55.9 W

Why it beats a bigger rival. The head-to-head that defines this model: against Gemma 3 27B it generates 28% faster (119 vs 93 tok/s peak), needs three fewer gigabytes (~15 vs ~18GB), and in our day-to-day results its output quality holds its own or better. That's the whole case in one sentence, more speed, less VRAM, no quality concession we could detect. It sits just under our 27-32B sweet spot, close enough that for many workloads it is the sweet spot: the most model you can run fast without committing to 24GB-class hardware pressure.

Deployment picture. The ~15GB floor means 16GB cards technically fit it (snug, like its Devstral and Dolphin-Venice siblings on the same base), with 24GB the comfortable home. Speed holds up down the chart, 108 on the H200, 33 on an A10G, and it pays no reasoning-token tax, so measured tok/s is what you actually feel. Pair it with a Dolphin judge or a DeepSeek distill and you have a fast, unpretentious core for a consensus stack.

Our verdict

Mistral Small 24B: 119 tok/s peak, ~15GB floor, and a won head-to-head against Gemma 3 27B: faster, lighter, and in our use at least as good. If you're choosing one dense mid-size generalist and don't need vision, this is the pick our own testing keeps making.

FAQ

Mistral Small 24B or Gemma 3 27B?: Our data and experience both say Mistral: 28% faster measured (119 vs 93 tok/s), ~3GB lighter floor, and task results that didn't favor the bigger model. Gemma's rebuttal is vision and multilingual breadth. Pick it only if you need those.
What GPU does it need?: ~15GB measured peak at Q4_K_M: a 16GB card fits with little headroom; 24GB is comfortable. On rentals, the H200 measured 108 tok/s at 208W.
Where does it sit in the size hierarchy?: Just under our 27-32B sweet spot, and for many tasks, effectively in it. It's the strongest dense generalist below the 24GB hardware line, which is exactly what makes it interesting.
Is it good for coding?: Capable, but its Mistral siblings specialize better: Codestral 22B for completions, Devstral 24B for agent loops, both on the same base and hardware class. Keep this one as the generalist of the trio.
How does it fit a consensus stack?: As the fast dense anchor: direct answers at 119 tok/s peak while reasoning models deliberate. Add a DeepSeek distill for the thinking seat and an uncensored Dolphin as judge and the whole panel shares one 24GB card.