Mistral Small 24B · 11 GPUs measured first-party · llama.cpp Q4_K_M · Updated October 2026

What GPU Do You Need for Mistral Small 24B?

Mistral Small 24B is the quiet winner of the mid-size class, the model our testing keeps preferring over Google's bigger Gemma 3 27B. Measured on 11 GPUs (llama.cpp, Q4_K_M): 119 tok/s on the B300, ~15GB peak VRAM, 28% faster than the Gemma it competes with.

Benchmarked weights: bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF

Fastest we measured

NVIDIA B300

119.3 tok/s on Mistral Small 24B, the ceiling. Measured on our bench. 288GB of VRAM, $40,000 at launch.

Pros

• 119.3 tok/s on Mistral Small 24B
• 288GB, clears the Mistral Small 24B floor
• Rentable by the hour rather than bought

Cons

• 1400W board rating
• Datacenter or workstation hardware, not a retail purchase

Cheapest card that runs it

NVIDIA RTX PRO 6000 Blackwell Workstation Edition

93.55 tok/s on Mistral Small 24B, lowest launch price that still fits. Measured on our bench. 96GB of VRAM, $8,565 at launch.

Pros

• 93.55 tok/s on Mistral Small 24B
• 96GB, clears the Mistral Small 24B floor
• Rentable by the hour rather than bought

Cons

• 600W board rating
• Datacenter or workstation hardware, not a retail purchase

Why it beats a bigger rival. The head-to-head that defines this model: against Gemma 3 27B it generates 28% faster (119 vs 93 tok/s peak), needs three fewer gigabytes (~15 vs ~18GB), and in our day-to-day results its output quality holds its own or better. That's the whole case in one sentence, more speed, less VRAM, no quality concession we could detect. It sits just under our 27-32B sweet spot, close enough that for many workloads it is the sweet spot: the most model you can run fast without committing to 24GB-class hardware pressure.

Deployment picture. The ~15GB floor means 16GB cards technically fit it (snug, like its Devstral and Dolphin-Venice siblings on the same base), with 24GB the comfortable home. Speed holds up down the chart, 108 on the H200, 33 on an A10G, and it pays no reasoning-token tax, so measured tok/s is what you actually feel. Pair it with a Dolphin judge or a DeepSeek distill and you have a fast, unpretentious core for a consensus stack.

How it compares. H100 80GB HBM3: Mistral Small 24B 105.6 tok/s, Dolphin-Mistral-24B-Venice-Edition 105.6, Cydonia-24B-v4.3 105.5, Devstral Small 24B 107.8, Dolphin 3.0 R1 Mistral 24B 107.8. 3 of 4 beat Mistral Small 24B here.

Cost on a rented GPU. 1M generated tokens of Mistral Small 24B: $2.10 on a A100 40GB SXM4 ($0.47/hr, 4.5 hours), $16.16 on a B300 ($6.94/hr, 2.3 hours, 7.7x the cost).

Speed tiers for Mistral Small 24B. 30+ tok/s: 9 (B300, B200, H200); 10-30 tok/s: 2 (L4, T4). 30 tok/s is roughly where replies outpace reading.

Reading your prompt. Before Mistral Small 24B writes anything it reads the input: 4891.0 tok/s on the RTX PRO 6000 Blackwell Workstation Edition (0.8s for a 4,000-token prompt), 419.0 on the T4 (9.5s). Long documents and big code files feel this number more than the generation speed.

VRAM for Mistral Small 24B. Measured peak 13.9GB, so 16GB is the smallest common card size; smallest card it ran on: T4 (16GB). With long context: Q4_K_M 16GB (tested), Q2_K 10GB, Q3_K_M 13GB, Q5_K_M 19GB, Q6_K 22GB.

Power on Mistral Small 24B. Most efficient: H200, 208W, 0.53 kWh per 1M generated tokens. Hungriest: B300, 345W, 0.80 kWh. At $0.15/kWh: $0.080 per 1M generated tokens.

Our verdict

Mistral Small 24B: 119 tok/s peak, ~15GB floor, and a won head-to-head against Gemma 3 27B: faster, lighter, and in our use at least as good. If you're choosing one dense mid-size generalist and don't need vision, this is the pick our own testing keeps making.

FAQ

Mistral Small 24B or Gemma 3 27B?

Our data and experience both say Mistral: 28% faster measured (119 vs 93 tok/s), ~3GB lighter floor, and task results that didn't favor the bigger model. Gemma's rebuttal is vision and multilingual breadth. Pick it only if you need those.

What GPU does it need?

~15GB measured peak at Q4_K_M: a 16GB card fits with little headroom; 24GB is comfortable. On rentals, the H200 measured 108 tok/s at 208W.

Where does it sit in the size hierarchy?

Just under our 27-32B sweet spot, and for many tasks, effectively in it. It's the strongest dense generalist below the 24GB hardware line, which is exactly what makes it interesting.

Is it good for coding?

Capable, but its Mistral siblings specialize better: Codestral 22B for completions, Devstral 24B for agent loops, both on the same base and hardware class. Keep this one as the generalist of the trio.

How does it fit a consensus stack?

As the fast dense anchor: direct answers at 119 tok/s peak while reasoning models deliberate. Add a DeepSeek distill for the thinking seat and an uncensored Dolphin as judge and the whole panel shares one 24GB card.