Fastest GPU For LLM Serving

Throughput with 32 requests in flight at once, which is the question when a model sits behind an API rather than on your desk.

Qwen2.5 1.5B served

Weights: Qwen/Qwen2.5-1.5B

NVIDIA B2009186.4
NVIDIA H2007541.6
NVIDIA H100 80GB HBM36794.8
NVIDIA RTX PRO 6000 Blackwell Workstation Edition5786.7
NVIDIA L40S4686.8
NVIDIA A100 80GB SXM44564.7
NVIDIA A100 40GB SXM44111.1
NVIDIA A10G2702.5
NVIDIA L41827
NVIDIA T41337.1

Measured in serve tok/s. Longer is faster.

#GPUserve tok/s
1NVIDIA B2009186.4
2NVIDIA H2007541.6
3NVIDIA H100 80GB HBM36794.8
4NVIDIA RTX PRO 6000 Blackwell Workstation Edition5786.7
5NVIDIA L40S4686.8
6NVIDIA A100 80GB SXM44564.7
7NVIDIA A100 40GB SXM44111.1
8NVIDIA A10G2702.5
9NVIDIA L41827
10NVIDIA T41337.1

SmolLM2 1.7B served

Weights: HuggingFaceTB/SmolLM2-1.7B

NVIDIA B2009672.9
NVIDIA H2007107.8
NVIDIA H100 80GB HBM36509.9
NVIDIA RTX PRO 6000 Blackwell Workstation Edition5642
NVIDIA A100 80GB SXM44351.4
NVIDIA A100 40GB SXM43865.2
NVIDIA L40S3525.9
NVIDIA A10G2242.7
NVIDIA L41524.9
NVIDIA T41222.5

Measured in serve tok/s. Longer is faster.

#GPUserve tok/s
1NVIDIA B2009672.9
2NVIDIA H2007107.8
3NVIDIA H100 80GB HBM36509.9
4NVIDIA RTX PRO 6000 Blackwell Workstation Edition5642
5NVIDIA A100 80GB SXM44351.4
6NVIDIA A100 40GB SXM43865.2
7NVIDIA L40S3525.9
8NVIDIA A10G2242.7
9NVIDIA L41524.9
10NVIDIA T41222.5

TinyLlama 1.1B served

Weights: TinyLlama/TinyLlama-1.1B-Chat-v1.0

NVIDIA B20011562.2
NVIDIA H2009137.7
NVIDIA H100 80GB HBM38336.7
NVIDIA RTX PRO 6000 Blackwell Workstation Edition7678.7
NVIDIA L40S6278.4
NVIDIA A100 80GB SXM45701.5
NVIDIA A10G3558.2
NVIDIA A100 40GB SXM43221.3
NVIDIA L42584.3
NVIDIA T41897.8

Measured in serve tok/s. Longer is faster.

#GPUserve tok/s
1NVIDIA B20011562.2
2NVIDIA H2009137.7
3NVIDIA H100 80GB HBM38336.7
4NVIDIA RTX PRO 6000 Blackwell Workstation Edition7678.7
5NVIDIA L40S6278.4
6NVIDIA A100 80GB SXM45701.5
7NVIDIA A10G3558.2
8NVIDIA A100 40GB SXM43221.3
9NVIDIA L42584.3
10NVIDIA T41897.8

Qwen2.5 7B served

Weights: Qwen/Qwen2.5-7B

NVIDIA B2005799.1
NVIDIA H2004394.8
NVIDIA H100 80GB HBM33601.1
NVIDIA RTX PRO 6000 Blackwell Workstation Edition2282.8
NVIDIA A100 80GB SXM42187.2
NVIDIA A100 40GB SXM41769.6
NVIDIA L40S1406.8
NVIDIA A10G858.7
NVIDIA L4469.9

Measured in serve tok/s. Longer is faster.

#GPUserve tok/s
1NVIDIA B2005799.1
2NVIDIA H2004394.8
3NVIDIA H100 80GB HBM33601.1
4NVIDIA RTX PRO 6000 Blackwell Workstation Edition2282.8
5NVIDIA A100 80GB SXM42187.2
6NVIDIA A100 40GB SXM41769.6
7NVIDIA L40S1406.8
8NVIDIA A10G858.7
9NVIDIA L4469.9

Every number is a first-party run on our own bench. Cards missing from a board have not been measured on that model yet, or cannot fit it.