Fastest GPU For Text Generation

Running an LLM locally: tokens per second on the models people actually run.

Qwen3 4B

Weights: Qwen/Qwen3-4B-GGUF

NVIDIA GeForce RTX 5090375.6
NVIDIA RTX PRO 6000 Blackwell Workstation Edition362.81
NVIDIA B300333.34
NVIDIA H200318.84
NVIDIA B200317.93
NVIDIA RTX PRO 6000 Blackwell Server Edition315.72
NVIDIA H100 80GB HBM3310.26
NVIDIA RTX PRO 5000 Blackwell298.78
NVIDIA GeForce RTX 4090260.54
NVIDIA RTX 6000 Ada Generation241.33
GeForce RTX 5070 Ti231.69
NVIDIA GeForce RTX 3090 Ti227.54
NVIDIA RTX PRO 4500 Blackwell221.92
GeForce RTX 5080215.89
NVIDIA L40S212.05

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA GeForce RTX 5090375.6
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition362.81
3NVIDIA B300333.34
4NVIDIA H200318.84
5NVIDIA B200317.93
6NVIDIA RTX PRO 6000 Blackwell Server Edition315.72
7NVIDIA H100 80GB HBM3310.26
8NVIDIA RTX PRO 5000 Blackwell298.78
9NVIDIA GeForce RTX 4090260.54
10NVIDIA RTX 6000 Ada Generation241.33
11GeForce RTX 5070 Ti231.69
12NVIDIA GeForce RTX 3090 Ti227.54
13NVIDIA RTX PRO 4500 Blackwell221.92
14GeForce RTX 5080215.89
15NVIDIA L40S212.05

Llama 3.1 8B

Weights: unsloth/Llama-3.1-8B-Instruct-GGUF

NVIDIA B300287.23
NVIDIA B200274.41
NVIDIA H200268.31
NVIDIA GeForce RTX 5090268.14
NVIDIA H100 80GB HBM3261.83
NVIDIA RTX PRO 6000 Blackwell Workstation Edition258.37
NVIDIA RTX PRO 6000 Blackwell Server Edition233.8
NVIDIA RTX PRO 5000 Blackwell208.3
NVIDIA GeForce RTX 4090171.29
NVIDIA A100 80GB SXM4162.57
NVIDIA A100 80GB PCIe161.75
NVIDIA GeForce RTX 3090 Ti161.02
NVIDIA RTX 6000 Ada Generation157.89
NVIDIA A100 40GB SXM4157.26
GeForce RTX 5070 Ti153.23

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300287.23
2NVIDIA B200274.41
3NVIDIA H200268.31
4NVIDIA GeForce RTX 5090268.14
5NVIDIA H100 80GB HBM3261.83
6NVIDIA RTX PRO 6000 Blackwell Workstation Edition258.37
7NVIDIA RTX PRO 6000 Blackwell Server Edition233.8
8NVIDIA RTX PRO 5000 Blackwell208.3
9NVIDIA GeForce RTX 4090171.29
10NVIDIA A100 80GB SXM4162.57
11NVIDIA A100 80GB PCIe161.75
12NVIDIA GeForce RTX 3090 Ti161.02
13NVIDIA RTX 6000 Ada Generation157.89
14NVIDIA A100 40GB SXM4157.26
15GeForce RTX 5070 Ti153.23

Qwen2.5-Coder 14B

Weights: Qwen/Qwen2.5-Coder-14B-Instruct-GGUF

NVIDIA B300158.49
NVIDIA B200150.96
NVIDIA GeForce RTX 5090149.75
NVIDIA H200148.43
NVIDIA RTX PRO 6000 Blackwell Workstation Edition147.74
NVIDIA H100 80GB HBM3144.84
NVIDIA RTX PRO 6000 Blackwell Server Edition135.07
NVIDIA RTX PRO 5000 Blackwell114.63
NVIDIA GeForce RTX 409095.12
NVIDIA A100 80GB SXM489.45
NVIDIA A100 80GB PCIe88.36
NVIDIA GeForce RTX 3090 Ti88.25
NVIDIA RTX 6000 Ada Generation87.51
NVIDIA A100 40GB SXM486.2
GeForce RTX 5070 Ti83.44

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300158.49
2NVIDIA B200150.96
3NVIDIA GeForce RTX 5090149.75
4NVIDIA H200148.43
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition147.74
6NVIDIA H100 80GB HBM3144.84
7NVIDIA RTX PRO 6000 Blackwell Server Edition135.07
8NVIDIA RTX PRO 5000 Blackwell114.63
9NVIDIA GeForce RTX 409095.12
10NVIDIA A100 80GB SXM489.45
11NVIDIA A100 80GB PCIe88.36
12NVIDIA GeForce RTX 3090 Ti88.25
13NVIDIA RTX 6000 Ada Generation87.51
14NVIDIA A100 40GB SXM486.2
15GeForce RTX 5070 Ti83.44

Qwen3 32B

Weights: unsloth/Qwen3-32B-GGUF

NVIDIA B30083.68
NVIDIA B20078.56
NVIDIA H20076.58
NVIDIA H100 80GB HBM374.07
NVIDIA GeForce RTX 509071.15
NVIDIA RTX PRO 6000 Blackwell Workstation Edition70.13
NVIDIA RTX PRO 6000 Blackwell Server Edition63.9
NVIDIA RTX PRO 5000 Blackwell54.27
NVIDIA A100 80GB SXM445.53
NVIDIA GeForce RTX 409044.28
NVIDIA A100 40GB SXM443.8
NVIDIA A100 80GB PCIe43.77
NVIDIA GeForce RTX 3090 Ti42.18
NVIDIA RTX 6000 Ada Generation39.87
NVIDIA GeForce RTX 309037.95

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30083.68
2NVIDIA B20078.56
3NVIDIA H20076.58
4NVIDIA H100 80GB HBM374.07
5NVIDIA GeForce RTX 509071.15
6NVIDIA RTX PRO 6000 Blackwell Workstation Edition70.13
7NVIDIA RTX PRO 6000 Blackwell Server Edition63.9
8NVIDIA RTX PRO 5000 Blackwell54.27
9NVIDIA A100 80GB SXM445.53
10NVIDIA GeForce RTX 409044.28
11NVIDIA A100 40GB SXM443.8
12NVIDIA A100 80GB PCIe43.77
13NVIDIA GeForce RTX 3090 Ti42.18
14NVIDIA RTX 6000 Ada Generation39.87
15NVIDIA GeForce RTX 309037.95

Llama 3.3 70B

Weights: bartowski/Llama-3.3-70B-Instruct-GGUF

NVIDIA B30047.97
NVIDIA B20044.54
NVIDIA H20042.66
NVIDIA H100 80GB HBM341
NVIDIA RTX PRO 6000 Blackwell Workstation Edition34.87
NVIDIA RTX PRO 6000 Blackwell Server Edition32.08
NVIDIA RTX PRO 5000 Blackwell26.29
NVIDIA A100 80GB SXM424.4
NVIDIA A100 80GB PCIe22.89
NVIDIA RTX 6000 Ada Generation18.4
NVIDIA L40S16.49
NVIDIA RTX A600015.73

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30047.97
2NVIDIA B20044.54
3NVIDIA H20042.66
4NVIDIA H100 80GB HBM341
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition34.87
6NVIDIA RTX PRO 6000 Blackwell Server Edition32.08
7NVIDIA RTX PRO 5000 Blackwell26.29
8NVIDIA A100 80GB SXM424.4
9NVIDIA A100 80GB PCIe22.89
10NVIDIA RTX 6000 Ada Generation18.4
11NVIDIA L40S16.49
12NVIDIA RTX A600015.73

DeepSeek-R1 Distill Llama 8B

Weights: bartowski/DeepSeek-R1-Distill-Llama-8B-GGUF

NVIDIA B300283.77
NVIDIA B200273.9
NVIDIA H200265.25
NVIDIA H100 80GB HBM3261.15
NVIDIA RTX PRO 6000 Blackwell Workstation Edition235.07
NVIDIA A100 80GB SXM4159.37
NVIDIA A100 40GB SXM4157.16
NVIDIA L40S135.55
NVIDIA A10G86.83
NVIDIA L450.39
NVIDIA T436.36

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300283.77
2NVIDIA B200273.9
3NVIDIA H200265.25
4NVIDIA H100 80GB HBM3261.15
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition235.07
6NVIDIA A100 80GB SXM4159.37
7NVIDIA A100 40GB SXM4157.16
8NVIDIA L40S135.55
9NVIDIA A10G86.83
10NVIDIA L450.39
11NVIDIA T436.36

DeepSeek-R1 Distill 1.5B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-1.5B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition635.1
NVIDIA B300562.14
NVIDIA H200542.95
NVIDIA H100 80GB HBM3537.32
NVIDIA B200456.7
NVIDIA L40S426.38
NVIDIA A100 80GB SXM4327.66
NVIDIA A100 40GB SXM4320.78
NVIDIA A10G265.75
NVIDIA L4191.38
NVIDIA T4148.72

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition635.1
2NVIDIA B300562.14
3NVIDIA H200542.95
4NVIDIA H100 80GB HBM3537.32
5NVIDIA B200456.7
6NVIDIA L40S426.38
7NVIDIA A100 80GB SXM4327.66
8NVIDIA A100 40GB SXM4320.78
9NVIDIA A10G265.75
10NVIDIA L4191.38
11NVIDIA T4148.72

DeepSeek-R1 Distill 14B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF

NVIDIA B300156.33
NVIDIA B200150.69
NVIDIA H200146.82
NVIDIA H100 80GB HBM3144.82
NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.41
NVIDIA A100 80GB SXM487
NVIDIA A100 40GB SXM486.39
NVIDIA L40S74.6
NVIDIA A10G46.96
NVIDIA L427.39
NVIDIA T419.21

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300156.33
2NVIDIA B200150.69
3NVIDIA H200146.82
4NVIDIA H100 80GB HBM3144.82
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.41
6NVIDIA A100 80GB SXM487
7NVIDIA A100 40GB SXM486.39
8NVIDIA L40S74.6
9NVIDIA A10G46.96
10NVIDIA L427.39
11NVIDIA T419.21

DeepSeek-R1 Distill 7B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-7B-GGUF

NVIDIA B300284.77
NVIDIA B200277.02
NVIDIA H200267.1
NVIDIA H100 80GB HBM3264.42
NVIDIA RTX PRO 6000 Blackwell Workstation Edition256.04
NVIDIA A100 80GB SXM4162.96
NVIDIA A100 40GB SXM4159.09
NVIDIA L40S143.97
NVIDIA A10G89.79
NVIDIA L453.27
NVIDIA T437.58

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300284.77
2NVIDIA B200277.02
3NVIDIA H200267.1
4NVIDIA H100 80GB HBM3264.42
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition256.04
6NVIDIA A100 80GB SXM4162.96
7NVIDIA A100 40GB SXM4159.09
8NVIDIA L40S143.97
9NVIDIA A10G89.79
10NVIDIA L453.27
11NVIDIA T437.58

Gemma 3 12B

Weights: bartowski/google_gemma-3-12b-it-GGUF

NVIDIA B300161.27
NVIDIA H200153.26
NVIDIA B200151.41
NVIDIA H100 80GB HBM3150.58
NVIDIA RTX PRO 6000 Blackwell Workstation Edition138.09
NVIDIA A100 80GB SXM491.45
NVIDIA A100 40GB SXM488.97
NVIDIA L40S80.92
NVIDIA A10G52.07
NVIDIA L431.01
NVIDIA T423.12

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300161.27
2NVIDIA H200153.26
3NVIDIA B200151.41
4NVIDIA H100 80GB HBM3150.58
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition138.09
6NVIDIA A100 80GB SXM491.45
7NVIDIA A100 40GB SXM488.97
8NVIDIA L40S80.92
9NVIDIA A10G52.07
10NVIDIA L431.01
11NVIDIA T423.12

Gemma 3 4B

Weights: bartowski/google_gemma-3-4b-it-GGUF

NVIDIA B300301.07
NVIDIA RTX PRO 6000 Blackwell Workstation Edition297.73
NVIDIA B200291.82
NVIDIA H200291.76
NVIDIA H100 80GB HBM3289.61
NVIDIA L40S198.87
NVIDIA A100 80GB SXM4176.94
NVIDIA A100 40GB SXM4174.14
NVIDIA A10G124.85
NVIDIA L482.03
NVIDIA T465.12

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300301.07
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition297.73
3NVIDIA B200291.82
4NVIDIA H200291.76
5NVIDIA H100 80GB HBM3289.61
6NVIDIA L40S198.87
7NVIDIA A100 80GB SXM4176.94
8NVIDIA A100 40GB SXM4174.14
9NVIDIA A10G124.85
10NVIDIA L482.03
11NVIDIA T465.12

Gemma 4 12B

Weights: bartowski/google_gemma-4-12b-it-GGUF

NVIDIA B300155.96
NVIDIA H200151.05
NVIDIA H100 80GB HBM3148.17
NVIDIA B200146.36
NVIDIA RTX PRO 6000 Blackwell Workstation Edition137.73
NVIDIA A100 80GB SXM492.43
NVIDIA A100 40GB SXM487.14
NVIDIA L40S80.82
NVIDIA A10G52.67
NVIDIA L431.49
NVIDIA T424.03

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300155.96
2NVIDIA H200151.05
3NVIDIA H100 80GB HBM3148.17
4NVIDIA B200146.36
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition137.73
6NVIDIA A100 80GB SXM492.43
7NVIDIA A100 40GB SXM487.14
8NVIDIA L40S80.82
9NVIDIA A10G52.67
10NVIDIA L431.49
11NVIDIA T424.03

Llama 3.2 1B

Weights: bartowski/Llama-3.2-1B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition892.74
NVIDIA B300889.19
NVIDIA B200881.78
NVIDIA H100 80GB HBM3880.64
NVIDIA H200875.53
NVIDIA L40S618.87
NVIDIA A100 40GB SXM4532.08
NVIDIA A100 80GB SXM4525.32
NVIDIA A10G400.6
NVIDIA L4259.48
NVIDIA T4207.71

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition892.74
2NVIDIA B300889.19
3NVIDIA B200881.78
4NVIDIA H100 80GB HBM3880.64
5NVIDIA H200875.53
6NVIDIA L40S618.87
7NVIDIA A100 40GB SXM4532.08
8NVIDIA A100 80GB SXM4525.32
9NVIDIA A10G400.6
10NVIDIA L4259.48
11NVIDIA T4207.71

Llama 3.2 3B

Weights: bartowski/Llama-3.2-3B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition435.63
NVIDIA B300432.24
NVIDIA H200424.61
NVIDIA H100 80GB HBM3422.87
NVIDIA B200422.09
NVIDIA L40S270.85
NVIDIA A100 80GB SXM4264.25
NVIDIA A100 40GB SXM4261.24
NVIDIA A10G170.47
NVIDIA L4107.1
NVIDIA T485.94

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition435.63
2NVIDIA B300432.24
3NVIDIA H200424.61
4NVIDIA H100 80GB HBM3422.87
5NVIDIA B200422.09
6NVIDIA L40S270.85
7NVIDIA A100 80GB SXM4264.25
8NVIDIA A100 40GB SXM4261.24
9NVIDIA A10G170.47
10NVIDIA L4107.1
11NVIDIA T485.94

Mistral 7B v0.3

Weights: bartowski/Mistral-7B-Instruct-v0.3-GGUF

NVIDIA B300298.95
NVIDIA B200287.42
NVIDIA H200278.5
NVIDIA H100 80GB HBM3275.85
NVIDIA RTX PRO 6000 Blackwell Workstation Edition247.33
NVIDIA A100 80GB SXM4171.92
NVIDIA A100 40GB SXM4165.9
NVIDIA L40S144.74
NVIDIA A10G92.82
NVIDIA L454.02
NVIDIA T440.01

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300298.95
2NVIDIA B200287.42
3NVIDIA H200278.5
4NVIDIA H100 80GB HBM3275.85
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition247.33
6NVIDIA A100 80GB SXM4171.92
7NVIDIA A100 40GB SXM4165.9
8NVIDIA L40S144.74
9NVIDIA A10G92.82
10NVIDIA L454.02
11NVIDIA T440.01

Mistral Small 24B

Weights: bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF

NVIDIA B300119.3
NVIDIA B200114.04
NVIDIA H200107.84
NVIDIA H100 80GB HBM3105.62
NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.55
NVIDIA A100 40GB SXM462.42
NVIDIA A100 80GB SXM461.38
NVIDIA L40S48.43
NVIDIA A10G30.94
NVIDIA L417.33
NVIDIA T411.33

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300119.3
2NVIDIA B200114.04
3NVIDIA H200107.84
4NVIDIA H100 80GB HBM3105.62
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.55
6NVIDIA A100 40GB SXM462.42
7NVIDIA A100 80GB SXM461.38
8NVIDIA L40S48.43
9NVIDIA A10G30.94
10NVIDIA L417.33
11NVIDIA T411.33

Phi-4 14B

Weights: bartowski/phi-4-GGUF

NVIDIA B300177.41
NVIDIA H200168.2
NVIDIA H100 80GB HBM3165.17
NVIDIA B200163.85
NVIDIA RTX PRO 6000 Blackwell Workstation Edition142.68
NVIDIA A100 40GB SXM498.65
NVIDIA A100 80GB SXM495.22
NVIDIA L40S74.98
NVIDIA A10G48.18
NVIDIA L427.24
NVIDIA T417.46

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300177.41
2NVIDIA H200168.2
3NVIDIA H100 80GB HBM3165.17
4NVIDIA B200163.85
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition142.68
6NVIDIA A100 40GB SXM498.65
7NVIDIA A100 80GB SXM495.22
8NVIDIA L40S74.98
9NVIDIA A10G48.18
10NVIDIA L427.24
11NVIDIA T417.46

Phi-4 Mini 3.8B

Weights: bartowski/microsoft_Phi-4-mini-instruct-GGUF

NVIDIA B300398.16
NVIDIA H200392.38
NVIDIA RTX PRO 6000 Blackwell Workstation Edition391.04
NVIDIA H100 80GB HBM3388.86
NVIDIA B200375.74
NVIDIA A100 80GB SXM4239.17
NVIDIA A100 40GB SXM4236.71
NVIDIA L40S228.93
NVIDIA A10G144.42
NVIDIA L488.48
NVIDIA T464.15

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300398.16
2NVIDIA H200392.38
3NVIDIA RTX PRO 6000 Blackwell Workstation Edition391.04
4NVIDIA H100 80GB HBM3388.86
5NVIDIA B200375.74
6NVIDIA A100 80GB SXM4239.17
7NVIDIA A100 40GB SXM4236.71
8NVIDIA L40S228.93
9NVIDIA A10G144.42
10NVIDIA L488.48
11NVIDIA T464.15

Qwen2.5-Coder 7B

Weights: bartowski/Qwen2.5-Coder-7B-Instruct-GGUF

NVIDIA B300286.5
NVIDIA B200277.98
NVIDIA H200266.36
NVIDIA H100 80GB HBM3263.91
NVIDIA RTX PRO 6000 Blackwell Workstation Edition256.11
NVIDIA A100 80GB SXM4164.6
NVIDIA A100 40GB SXM4159.9
NVIDIA L40S143.93
NVIDIA A10G90.05
NVIDIA L453.26
NVIDIA T437.54

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300286.5
2NVIDIA B200277.98
3NVIDIA H200266.36
4NVIDIA H100 80GB HBM3263.91
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition256.11
6NVIDIA A100 80GB SXM4164.6
7NVIDIA A100 40GB SXM4159.9
8NVIDIA L40S143.93
9NVIDIA A10G90.05
10NVIDIA L453.26
11NVIDIA T437.54

Qwen3 0.6B

Weights: Qwen/Qwen3-0.6B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition788.97
NVIDIA H200718.67
NVIDIA B300718.58
NVIDIA H100 80GB HBM3713.53
NVIDIA L40S655.75
NVIDIA B200599.63
NVIDIA A10G451.38
NVIDIA A100 80GB SXM4428.39
NVIDIA A100 40GB SXM4417.88
NVIDIA L4357.79
NVIDIA T4263.47

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition788.97
2NVIDIA H200718.67
3NVIDIA B300718.58
4NVIDIA H100 80GB HBM3713.53
5NVIDIA L40S655.75
6NVIDIA B200599.63
7NVIDIA A10G451.38
8NVIDIA A100 80GB SXM4428.39
9NVIDIA A100 40GB SXM4417.88
10NVIDIA L4357.79
11NVIDIA T4263.47

Qwen3 1.7B

Weights: Qwen/Qwen3-1.7B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition575.77
NVIDIA H200563.52
NVIDIA H100 80GB HBM3556.48
NVIDIA B300555.1
NVIDIA B200551.76
NVIDIA L40S405.52
NVIDIA A100 80GB SXM4343.58
NVIDIA A100 40GB SXM4340.76
NVIDIA A10G267.08
NVIDIA L4177.5
NVIDIA T4139.92

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition575.77
2NVIDIA H200563.52
3NVIDIA H100 80GB HBM3556.48
4NVIDIA B300555.1
5NVIDIA B200551.76
6NVIDIA L40S405.52
7NVIDIA A100 80GB SXM4343.58
8NVIDIA A100 40GB SXM4340.76
9NVIDIA A10G267.08
10NVIDIA L4177.5
11NVIDIA T4139.92

Qwen3 14B

Weights: Qwen/Qwen3-14B-GGUF

NVIDIA B300165.44
NVIDIA B200158.34
NVIDIA H200154.51
NVIDIA H100 80GB HBM3151.16
NVIDIA RTX PRO 6000 Blackwell Workstation Edition139.01
NVIDIA A100 80GB SXM490.64
NVIDIA A100 40GB SXM490.01
NVIDIA L40S75.12
NVIDIA A10G48.12
NVIDIA L427.61
NVIDIA T419.31

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300165.44
2NVIDIA B200158.34
3NVIDIA H200154.51
4NVIDIA H100 80GB HBM3151.16
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition139.01
6NVIDIA A100 80GB SXM490.64
7NVIDIA A100 40GB SXM490.01
8NVIDIA L40S75.12
9NVIDIA A10G48.12
10NVIDIA L427.61
11NVIDIA T419.31

Qwen3 8B

Weights: Qwen/Qwen3-8B-GGUF

NVIDIA B300261.4
NVIDIA B200253.45
NVIDIA H200247.98
NVIDIA H100 80GB HBM3244.22
NVIDIA RTX PRO 6000 Blackwell Workstation Edition225.95
NVIDIA A100 80GB SXM4149.96
NVIDIA A100 40GB SXM4146.21
NVIDIA L40S130.27
NVIDIA A10G84.09
NVIDIA L448.95
NVIDIA T436.51

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300261.4
2NVIDIA B200253.45
3NVIDIA H200247.98
4NVIDIA H100 80GB HBM3244.22
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition225.95
6NVIDIA A100 80GB SXM4149.96
7NVIDIA A100 40GB SXM4146.21
8NVIDIA L40S130.27
9NVIDIA A10G84.09
10NVIDIA L448.95
11NVIDIA T436.51

SmolLM3 3B

Weights: bartowski/HuggingFaceTB_SmolLM3-3B-GGUF

NVIDIA B300411.25
NVIDIA RTX PRO 6000 Blackwell Workstation Edition406.45
NVIDIA H200402.24
NVIDIA B200400.73
NVIDIA H100 80GB HBM3397.99
NVIDIA L40S272.88
NVIDIA A100 80GB SXM4249.76
NVIDIA A100 40GB SXM4246.25
NVIDIA A10G172.15
NVIDIA L4112.07
NVIDIA T486.08

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300411.25
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition406.45
3NVIDIA H200402.24
4NVIDIA B200400.73
5NVIDIA H100 80GB HBM3397.99
6NVIDIA L40S272.88
7NVIDIA A100 80GB SXM4249.76
8NVIDIA A100 40GB SXM4246.25
9NVIDIA A10G172.15
10NVIDIA L4112.07
11NVIDIA T486.08

Codestral 22B

Weights: bartowski/Codestral-22B-v0.1-GGUF

NVIDIA B300119.81
NVIDIA B200112.16
NVIDIA H200111.34
NVIDIA H100 80GB HBM3110.05
NVIDIA RTX PRO 6000 Blackwell Workstation Edition92.7
NVIDIA A100 40GB SXM463.74
NVIDIA A100 80GB SXM463.26
NVIDIA L40S50.06
NVIDIA A10G32.09
NVIDIA L418.14
NVIDIA T412.59

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300119.81
2NVIDIA B200112.16
3NVIDIA H200111.34
4NVIDIA H100 80GB HBM3110.05
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition92.7
6NVIDIA A100 40GB SXM463.74
7NVIDIA A100 80GB SXM463.26
8NVIDIA L40S50.06
9NVIDIA A10G32.09
10NVIDIA L418.14
11NVIDIA T412.59

Devstral Small 24B

Weights: bartowski/mistralai_Devstral-Small-2507-GGUF

NVIDIA B300121.18
NVIDIA B200113.91
NVIDIA H200109.05
NVIDIA H100 80GB HBM3107.79
NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.52
NVIDIA A100 40GB SXM462.4
NVIDIA A100 80GB SXM461.63
NVIDIA L40S48.43
NVIDIA A10G30.98
NVIDIA L417.34
NVIDIA T411.4

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300121.18
2NVIDIA B200113.91
3NVIDIA H200109.05
4NVIDIA H100 80GB HBM3107.79
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.52
6NVIDIA A100 40GB SXM462.4
7NVIDIA A100 80GB SXM461.63
8NVIDIA L40S48.43
9NVIDIA A10G30.98
10NVIDIA L417.34
11NVIDIA T411.4

Dolphin Mistral 24B Venice

Weights: dphn/Dolphin-Mistral-24B-Venice-Edition-GGUF

NVIDIA B300120.93
NVIDIA B200113.93
NVIDIA H200109.14
NVIDIA H100 80GB HBM3109.13
NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.67
NVIDIA A100 80GB SXM462.72
NVIDIA A100 40GB SXM462.46
NVIDIA L40S48.43
NVIDIA A10G30.95
NVIDIA L417.34
NVIDIA T411.36

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300120.93
2NVIDIA B200113.93
3NVIDIA H200109.14
4NVIDIA H100 80GB HBM3109.13
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.67
6NVIDIA A100 80GB SXM462.72
7NVIDIA A100 40GB SXM462.46
8NVIDIA L40S48.43
9NVIDIA A10G30.95
10NVIDIA L417.34
11NVIDIA T411.36

Dolphin X1 8B

Weights: dphn/Dolphin-X1-8B-GGUF

NVIDIA B300287.02
NVIDIA B200273.85
NVIDIA H200268.36
NVIDIA H100 80GB HBM3266
NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.4
NVIDIA A100 80GB SXM4158.62
NVIDIA A100 40GB SXM4156.3
NVIDIA L40S135.48
NVIDIA A10G86.61
NVIDIA L450.12
NVIDIA T435.66

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300287.02
2NVIDIA B200273.85
3NVIDIA H200268.36
4NVIDIA H100 80GB HBM3266
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.4
6NVIDIA A100 80GB SXM4158.62
7NVIDIA A100 40GB SXM4156.3
8NVIDIA L40S135.48
9NVIDIA A10G86.61
10NVIDIA L450.12
11NVIDIA T435.66

Dolphin 3.0 Llama 3.1 8B

Weights: dphn/Dolphin3.0-Llama3.1-8B-GGUF

NVIDIA B300285.88
NVIDIA B200274.05
NVIDIA H200267.84
NVIDIA H100 80GB HBM3266.19
NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.28
NVIDIA A100 80GB SXM4158.63
NVIDIA A100 40GB SXM4156.55
NVIDIA L40S135.55
NVIDIA A10G86.54
NVIDIA L450.24
NVIDIA T435.68

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300285.88
2NVIDIA B200274.05
3NVIDIA H200267.84
4NVIDIA H100 80GB HBM3266.19
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.28
6NVIDIA A100 80GB SXM4158.63
7NVIDIA A100 40GB SXM4156.55
8NVIDIA L40S135.55
9NVIDIA A10G86.54
10NVIDIA L450.24
11NVIDIA T435.68

Dolphin 3.0 R1 Mistral 24B

Weights: bartowski/cognitivecomputations_Dolphin3.0-R1-Mistral-24B-GGUF

NVIDIA B300120.95
NVIDIA B200113.95
NVIDIA H200109.06
NVIDIA H100 80GB HBM3107.83
NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.62
NVIDIA A100 40GB SXM462.4
NVIDIA A100 80GB SXM461.17
NVIDIA L40S48.4
NVIDIA A10G30.9
NVIDIA L417.32
NVIDIA T411.32

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300120.95
2NVIDIA B200113.95
3NVIDIA H200109.06
4NVIDIA H100 80GB HBM3107.83
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.62
6NVIDIA A100 40GB SXM462.4
7NVIDIA A100 80GB SXM461.17
8NVIDIA L40S48.4
9NVIDIA A10G30.9
10NVIDIA L417.32
11NVIDIA T411.32

StarCoder2 15B

Weights: bartowski/starcoder2-15b-instruct-v0.1-GGUF

NVIDIA B300144.05
NVIDIA B200138.64
NVIDIA H200136.29
NVIDIA H100 80GB HBM3134.92
NVIDIA RTX PRO 6000 Blackwell Workstation Edition122.09
NVIDIA A100 40GB SXM480.25
NVIDIA A100 80GB SXM479.22
NVIDIA L40S66.88
NVIDIA A10G42.19
NVIDIA L424.4
NVIDIA T415.66

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300144.05
2NVIDIA B200138.64
3NVIDIA H200136.29
4NVIDIA H100 80GB HBM3134.92
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition122.09
6NVIDIA A100 40GB SXM480.25
7NVIDIA A100 80GB SXM479.22
8NVIDIA L40S66.88
9NVIDIA A10G42.19
10NVIDIA L424.4
11NVIDIA T415.66

Dolphin X1 Trinity Nano 6B

Weights: mradermacher/Dolphin-X1-Trinity-Nano-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition282.02
NVIDIA H200259.81
NVIDIA B300257.93
NVIDIA H100 80GB HBM3247.94
NVIDIA L40S246.45
NVIDIA B200209.31
NVIDIA A10G171.23
NVIDIA L4166.6
NVIDIA A100 80GB SXM4158.41
NVIDIA A100 40GB SXM4153.27
NVIDIA T4114.21

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition282.02
2NVIDIA H200259.81
3NVIDIA B300257.93
4NVIDIA H100 80GB HBM3247.94
5NVIDIA L40S246.45
6NVIDIA B200209.31
7NVIDIA A10G171.23
8NVIDIA L4166.6
9NVIDIA A100 80GB SXM4158.41
10NVIDIA A100 40GB SXM4153.27
11NVIDIA T4114.21

gpt-oss-20b

Weights: unsloth/gpt-oss-20b-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition375.09
NVIDIA B300355.56
NVIDIA H200354.23
NVIDIA B200351.42
NVIDIA H100 80GB HBM3346.8
NVIDIA L40S233.82
NVIDIA A100 80GB SXM4212.63
NVIDIA A100 40GB SXM4205.1
NVIDIA A10G144.61
NVIDIA L491.78
NVIDIA T463.63

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition375.09
2NVIDIA B300355.56
3NVIDIA H200354.23
4NVIDIA B200351.42
5NVIDIA H100 80GB HBM3346.8
6NVIDIA L40S233.82
7NVIDIA A100 80GB SXM4212.63
8NVIDIA A100 40GB SXM4205.1
9NVIDIA A10G144.61
10NVIDIA L491.78
11NVIDIA T463.63

Dolphin-Mistral-24B-Venice-Edition

Weights: bartowski/cognitivecomputations_Dolphin-Mistral-24B-Venice-Edition-GGUF

NVIDIA B300121.87
NVIDIA B200113.8
NVIDIA H200109.2
NVIDIA H100 80GB HBM3105.63
NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.71
NVIDIA A100 80GB SXM465.77
NVIDIA A100 40GB SXM462.56
NVIDIA L40S48.43
NVIDIA A10G31
NVIDIA L417.2
NVIDIA T411.33

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300121.87
2NVIDIA B200113.8
3NVIDIA H200109.2
4NVIDIA H100 80GB HBM3105.63
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.71
6NVIDIA A100 80GB SXM465.77
7NVIDIA A100 40GB SXM462.56
8NVIDIA L40S48.43
9NVIDIA A10G31
10NVIDIA L417.2
11NVIDIA T411.33

DeepSeek-Coder-V2-Lite

Weights: bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition352.52
NVIDIA B300340.53
NVIDIA B200325.85
NVIDIA H200312.1
NVIDIA H100 80GB HBM3309.14
NVIDIA L40S257.64
NVIDIA A100 80GB SXM4213.91
NVIDIA A100 40GB SXM4205.42
NVIDIA A10G160.13
NVIDIA L4110.11
NVIDIA T483.49

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition352.52
2NVIDIA B300340.53
3NVIDIA B200325.85
4NVIDIA H200312.1
5NVIDIA H100 80GB HBM3309.14
6NVIDIA L40S257.64
7NVIDIA A100 80GB SXM4213.91
8NVIDIA A100 40GB SXM4205.42
9NVIDIA A10G160.13
10NVIDIA L4110.11
11NVIDIA T483.49

DeepSeek-R1-0528-Qwen3-8B

Weights: lmstudio-community/DeepSeek-R1-0528-Qwen3-8B-GGUF

NVIDIA B300270.35
NVIDIA B200252.98
NVIDIA H200250.61
NVIDIA H100 80GB HBM3245.26
NVIDIA RTX PRO 6000 Blackwell Workstation Edition226.85
NVIDIA A100 80GB SXM4152.76
NVIDIA A100 40GB SXM4145.84
NVIDIA L40S130.23
NVIDIA A10G83.22
NVIDIA L449.07
NVIDIA T437.7

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300270.35
2NVIDIA B200252.98
3NVIDIA H200250.61
4NVIDIA H100 80GB HBM3245.26
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition226.85
6NVIDIA A100 80GB SXM4152.76
7NVIDIA A100 40GB SXM4145.84
8NVIDIA L40S130.23
9NVIDIA A10G83.22
10NVIDIA L449.07
11NVIDIA T437.7

EVA-Qwen2.5-14B-v0.2

Weights: bartowski/EVA-Qwen2.5-14B-v0.2-GGUF

NVIDIA B300159.46
NVIDIA B200150.67
NVIDIA H200148.74
NVIDIA H100 80GB HBM3145.15
NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.83
NVIDIA A100 80GB SXM490.35
NVIDIA A100 40GB SXM486.48
NVIDIA L40S74.63
NVIDIA A10G47
NVIDIA L427.3
NVIDIA T419.92

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300159.46
2NVIDIA B200150.67
3NVIDIA H200148.74
4NVIDIA H100 80GB HBM3145.15
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.83
6NVIDIA A100 80GB SXM490.35
7NVIDIA A100 40GB SXM486.48
8NVIDIA L40S74.63
9NVIDIA A10G47
10NVIDIA L427.3
11NVIDIA T419.92

gemma-2-2b-it-abliterated

Weights: bartowski/gemma-2-2b-it-abliterated-GGUF

NVIDIA B300418.18
NVIDIA RTX PRO 6000 Blackwell Workstation Edition416.68
NVIDIA H200397.43
NVIDIA H100 80GB HBM3392.65
NVIDIA B200392.32
NVIDIA L40S278.14
NVIDIA A100 80GB SXM4244.41
NVIDIA A100 40GB SXM4232.27
NVIDIA A10G174.56
NVIDIA L4113
NVIDIA T491.91

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300418.18
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition416.68
3NVIDIA H200397.43
4NVIDIA H100 80GB HBM3392.65
5NVIDIA B200392.32
6NVIDIA L40S278.14
7NVIDIA A100 80GB SXM4244.41
8NVIDIA A100 40GB SXM4232.27
9NVIDIA A10G174.56
10NVIDIA L4113
11NVIDIA T491.91

gemma-2-2b

Weights: bartowski/gemma-2-2b-it-GGUF

NVIDIA B300417.84
NVIDIA RTX PRO 6000 Blackwell Workstation Edition417.07
NVIDIA H200398.48
NVIDIA H100 80GB HBM3392.26
NVIDIA B200391.94
NVIDIA L40S278.1
NVIDIA A100 80GB SXM4244.52
NVIDIA A100 40GB SXM4234.68
NVIDIA A10G175.03
NVIDIA L4114.17
NVIDIA T492.13

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300417.84
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition417.07
3NVIDIA H200398.48
4NVIDIA H100 80GB HBM3392.26
5NVIDIA B200391.94
6NVIDIA L40S278.1
7NVIDIA A100 80GB SXM4244.52
8NVIDIA A100 40GB SXM4234.68
9NVIDIA A10G175.03
10NVIDIA L4114.17
11NVIDIA T492.13

gemma-2-9b

Weights: bartowski/gemma-2-9b-it-GGUF

NVIDIA B300202.22
NVIDIA B200187.45
NVIDIA H200180.47
NVIDIA H100 80GB HBM3174.7
NVIDIA RTX PRO 6000 Blackwell Workstation Edition156.29
NVIDIA A100 80GB SXM4108.68
NVIDIA A100 40GB SXM4101.97
NVIDIA L40S88.87
NVIDIA A10G57.37
NVIDIA L433.8
NVIDIA T428.62

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300202.22
2NVIDIA B200187.45
3NVIDIA H200180.47
4NVIDIA H100 80GB HBM3174.7
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition156.29
6NVIDIA A100 80GB SXM4108.68
7NVIDIA A100 40GB SXM4101.97
8NVIDIA L40S88.87
9NVIDIA A10G57.37
10NVIDIA L433.8
11NVIDIA T428.62

gemma-3-1b

Weights: lmstudio-community/gemma-3-1b-it-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition567.1
NVIDIA B300533.32
NVIDIA B200519.19
NVIDIA H200513.58
NVIDIA H100 80GB HBM3504.17
NVIDIA L40S426.49
NVIDIA A100 80GB SXM4324.08
NVIDIA A100 40GB SXM4310.69
NVIDIA A10G284.78
NVIDIA L4216.27
NVIDIA T4156.88

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition567.1
2NVIDIA B300533.32
3NVIDIA B200519.19
4NVIDIA H200513.58
5NVIDIA H100 80GB HBM3504.17
6NVIDIA L40S426.49
7NVIDIA A100 80GB SXM4324.08
8NVIDIA A100 40GB SXM4310.69
9NVIDIA A10G284.78
10NVIDIA L4216.27
11NVIDIA T4156.88

gemma-3-270m

Weights: unsloth/gemma-3-270m-it-GGUF

NVIDIA H200967.35
NVIDIA RTX PRO 6000 Blackwell Workstation Edition965
NVIDIA H100 80GB HBM3961.29
NVIDIA B300926.67
NVIDIA B200889.52
NVIDIA L40S822.52
NVIDIA A100 80GB SXM4617.58
NVIDIA A10G597.49
NVIDIA A100 40GB SXM4574.24
NVIDIA L4493.31
NVIDIA T4362.8

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA H200967.35
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition965
3NVIDIA H100 80GB HBM3961.29
4NVIDIA B300926.67
5NVIDIA B200889.52
6NVIDIA L40S822.52
7NVIDIA A100 80GB SXM4617.58
8NVIDIA A10G597.49
9NVIDIA A100 40GB SXM4574.24
10NVIDIA L4493.31
11NVIDIA T4362.8

SmolLM3-3B

Weights: bartowski/HuggingFaceTB_SmolLM3-3B-GGUF

NVIDIA B300420.3
NVIDIA RTX PRO 6000 Blackwell Workstation Edition409.73
NVIDIA H200404.27
NVIDIA H100 80GB HBM3399.14
NVIDIA B200399.05
NVIDIA L40S272.96
NVIDIA A100 80GB SXM4254.6
NVIDIA A100 40GB SXM4247.66
NVIDIA A10G171.66
NVIDIA L4110.77
NVIDIA T486.86

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300420.3
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition409.73
3NVIDIA H200404.27
4NVIDIA H100 80GB HBM3399.14
5NVIDIA B200399.05
6NVIDIA L40S272.96
7NVIDIA A100 80GB SXM4254.6
8NVIDIA A100 40GB SXM4247.66
9NVIDIA A10G171.66
10NVIDIA L4110.77
11NVIDIA T486.86

LFM2.5-1.2B

Weights: unsloth/LFM2.5-1.2B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition989.91
NVIDIA B300955.13
NVIDIA H200946.53
NVIDIA H100 80GB HBM3926.61
NVIDIA B200916.79
NVIDIA L40S661.54
NVIDIA A100 80GB SXM4576.95
NVIDIA A100 40GB SXM4572.69
NVIDIA A10G429.96
NVIDIA L4278.53
NVIDIA T4221.66

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition989.91
2NVIDIA B300955.13
3NVIDIA H200946.53
4NVIDIA H100 80GB HBM3926.61
5NVIDIA B200916.79
6NVIDIA L40S661.54
7NVIDIA A100 80GB SXM4576.95
8NVIDIA A100 40GB SXM4572.69
9NVIDIA A10G429.96
10NVIDIA L4278.53
11NVIDIA T4221.66

Llama-2-7B

Weights: TheBloke/Llama-2-7B-Chat-GGUF

NVIDIA B300315.9
NVIDIA B200292
NVIDIA H200290.81
NVIDIA H100 80GB HBM3284.62
NVIDIA RTX PRO 6000 Blackwell Workstation Edition263.84
NVIDIA A100 80GB SXM4180.09
NVIDIA A100 40GB SXM4172.53
NVIDIA L40S150.35
NVIDIA A10G96.69
NVIDIA L456.06
NVIDIA T442.98

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300315.9
2NVIDIA B200292
3NVIDIA H200290.81
4NVIDIA H100 80GB HBM3284.62
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition263.84
6NVIDIA A100 80GB SXM4180.09
7NVIDIA A100 40GB SXM4172.53
8NVIDIA L40S150.35
9NVIDIA A10G96.69
10NVIDIA L456.06
11NVIDIA T442.98

Llama-3.2-3B-Instruct-uncensored

Weights: bartowski/Llama-3.2-3B-Instruct-uncensored-GGUF

NVIDIA B300448.94
NVIDIA RTX PRO 6000 Blackwell Workstation Edition440.15
NVIDIA H200429.78
NVIDIA H100 80GB HBM3424.25
NVIDIA B200419.24
NVIDIA L40S271.01
NVIDIA A100 80GB SXM4269.13
NVIDIA A100 40GB SXM4260.83
NVIDIA A10G167.63
NVIDIA L4101.16
NVIDIA T486.29

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300448.94
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition440.15
3NVIDIA H200429.78
4NVIDIA H100 80GB HBM3424.25
5NVIDIA B200419.24
6NVIDIA L40S271.01
7NVIDIA A100 80GB SXM4269.13
8NVIDIA A100 40GB SXM4260.83
9NVIDIA A10G167.63
10NVIDIA L4101.16
11NVIDIA T486.29

Meta-Llama-3.1-8B

Weights: bartowski/Meta-Llama-3.1-8B-Instruct-GGUF

NVIDIA B300293
NVIDIA B200274.17
NVIDIA H200262.93
NVIDIA H100 80GB HBM3261.7
NVIDIA RTX PRO 6000 Blackwell Workstation Edition237.69
NVIDIA A100 80GB SXM4164.24
NVIDIA A100 40GB SXM4156.68
NVIDIA L40S135.46
NVIDIA A10G86.69
NVIDIA L450.36
NVIDIA T437.8

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300293
2NVIDIA B200274.17
3NVIDIA H200262.93
4NVIDIA H100 80GB HBM3261.7
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition237.69
6NVIDIA A100 80GB SXM4164.24
7NVIDIA A100 40GB SXM4156.68
8NVIDIA L40S135.46
9NVIDIA A10G86.69
10NVIDIA L450.36
11NVIDIA T437.8

Phi-4-mini

Weights: bartowski/microsoft_Phi-4-mini-instruct-GGUF

NVIDIA B300409.58
NVIDIA H200395.3
NVIDIA RTX PRO 6000 Blackwell Workstation Edition393.81
NVIDIA H100 80GB HBM3381.52
NVIDIA B200375.06
NVIDIA A100 80GB SXM4243.38
NVIDIA A100 40GB SXM4237.38
NVIDIA L40S229.28
NVIDIA A10G144.27
NVIDIA L487.21
NVIDIA T466

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300409.58
2NVIDIA H200395.3
3NVIDIA RTX PRO 6000 Blackwell Workstation Edition393.81
4NVIDIA H100 80GB HBM3381.52
5NVIDIA B200375.06
6NVIDIA A100 80GB SXM4243.38
7NVIDIA A100 40GB SXM4237.38
8NVIDIA L40S229.28
9NVIDIA A10G144.27
10NVIDIA L487.21
11NVIDIA T466

Mistral-7B-Instruct-v0.1

Weights: TheBloke/Mistral-7B-Instruct-v0.1-GGUF

NVIDIA B300306.44
NVIDIA B200286.89
NVIDIA H200282.64
NVIDIA H100 80GB HBM3276.29
NVIDIA RTX PRO 6000 Blackwell Workstation Edition252.55
NVIDIA A100 80GB SXM4173.87
NVIDIA A100 40GB SXM4166.39
NVIDIA L40S144.82
NVIDIA A10G92.45
NVIDIA L453.6
NVIDIA T440.63

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300306.44
2NVIDIA B200286.89
3NVIDIA H200282.64
4NVIDIA H100 80GB HBM3276.29
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition252.55
6NVIDIA A100 80GB SXM4173.87
7NVIDIA A100 40GB SXM4166.39
8NVIDIA L40S144.82
9NVIDIA A10G92.45
10NVIDIA L453.6
11NVIDIA T440.63

Mistral-7B-Instruct-v0.2

Weights: TheBloke/Mistral-7B-Instruct-v0.2-GGUF

NVIDIA B300306.79
NVIDIA B200286.37
NVIDIA H200282.32
NVIDIA H100 80GB HBM3276.73
NVIDIA RTX PRO 6000 Blackwell Workstation Edition253.34
NVIDIA A100 80GB SXM4173.8
NVIDIA A100 40GB SXM4166.8
NVIDIA L40S144.82
NVIDIA A10G92.74
NVIDIA L453.31
NVIDIA T441.31

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300306.79
2NVIDIA B200286.37
3NVIDIA H200282.32
4NVIDIA H100 80GB HBM3276.73
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition253.34
6NVIDIA A100 80GB SXM4173.8
7NVIDIA A100 40GB SXM4166.8
8NVIDIA L40S144.82
9NVIDIA A10G92.74
10NVIDIA L453.31
11NVIDIA T441.31

Mistral-7B-Instruct-v0.3

Weights: bartowski/Mistral-7B-Instruct-v0.3-GGUF

NVIDIA B300306.46
NVIDIA B200286.48
NVIDIA H200282.39
NVIDIA H100 80GB HBM3276.61
NVIDIA RTX PRO 6000 Blackwell Workstation Edition253.2
NVIDIA A100 80GB SXM4173.27
NVIDIA A100 40GB SXM4167.06
NVIDIA L40S144.68
NVIDIA A10G92.45
NVIDIA L453.66
NVIDIA T439.87

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300306.46
2NVIDIA B200286.48
3NVIDIA H200282.39
4NVIDIA H100 80GB HBM3276.61
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition253.2
6NVIDIA A100 80GB SXM4173.27
7NVIDIA A100 40GB SXM4167.06
8NVIDIA L40S144.68
9NVIDIA A10G92.45
10NVIDIA L453.66
11NVIDIA T439.87

Ornith-1.0-9B

Weights: unsloth/Ornith-1.0-9B-GGUF

NVIDIA B300240.67
NVIDIA B200226.2
NVIDIA H200222.94
NVIDIA H100 80GB HBM3216.89
NVIDIA RTX PRO 6000 Blackwell Workstation Edition201.69
NVIDIA A100 80GB SXM4134.67
NVIDIA A100 40GB SXM4126.79
NVIDIA L40S114.57
NVIDIA A10G73.78
NVIDIA L443.35
NVIDIA T432.93

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300240.67
2NVIDIA B200226.2
3NVIDIA H200222.94
4NVIDIA H100 80GB HBM3216.89
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition201.69
6NVIDIA A100 80GB SXM4134.67
7NVIDIA A100 40GB SXM4126.79
8NVIDIA L40S114.57
9NVIDIA A10G73.78
10NVIDIA L443.35
11NVIDIA T432.93

Phi-3.5-mini

Weights: bartowski/Phi-3.5-mini-instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition362.79
NVIDIA B300352.25
NVIDIA H200348.14
NVIDIA H100 80GB HBM3345.07
NVIDIA B200325
NVIDIA A100 80GB SXM4229.46
NVIDIA L40S229.06
NVIDIA A100 40GB SXM4222.53
NVIDIA A10G145.97
NVIDIA L490.56
NVIDIA T472.07

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition362.79
2NVIDIA B300352.25
3NVIDIA H200348.14
4NVIDIA H100 80GB HBM3345.07
5NVIDIA B200325
6NVIDIA A100 80GB SXM4229.46
7NVIDIA L40S229.06
8NVIDIA A100 40GB SXM4222.53
9NVIDIA A10G145.97
10NVIDIA L490.56
11NVIDIA T472.07

Qwen3-0.6B

Weights: bartowski/Qwen_Qwen3-0.6B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition806.77
NVIDIA B300742.64
NVIDIA H200724.67
NVIDIA H100 80GB HBM3710.97
NVIDIA L40S656.26
NVIDIA B200596.73
NVIDIA A10G446.74
NVIDIA A100 80GB SXM4432.66
NVIDIA A100 40GB SXM4408.39
NVIDIA L4356.93
NVIDIA T4259.91

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition806.77
2NVIDIA B300742.64
3NVIDIA H200724.67
4NVIDIA H100 80GB HBM3710.97
5NVIDIA L40S656.26
6NVIDIA B200596.73
7NVIDIA A10G446.74
8NVIDIA A100 80GB SXM4432.66
9NVIDIA A100 40GB SXM4408.39
10NVIDIA L4356.93
11NVIDIA T4259.91

Qwen3-1.7B

Weights: bartowski/Qwen_Qwen3-1.7B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition593.5
NVIDIA B300575.97
NVIDIA H200568.85
NVIDIA H100 80GB HBM3557.09
NVIDIA B200551.93
NVIDIA L40S405.3
NVIDIA A100 80GB SXM4350.4
NVIDIA A100 40GB SXM4341.85
NVIDIA A10G262.49
NVIDIA L4176.08
NVIDIA T4135.21

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition593.5
2NVIDIA B300575.97
3NVIDIA H200568.85
4NVIDIA H100 80GB HBM3557.09
5NVIDIA B200551.93
6NVIDIA L40S405.3
7NVIDIA A100 80GB SXM4350.4
8NVIDIA A100 40GB SXM4341.85
9NVIDIA A10G262.49
10NVIDIA L4176.08
11NVIDIA T4135.21

Qwen3-14B

Weights: bartowski/Qwen_Qwen3-14B-GGUF

NVIDIA B300169.52
NVIDIA B200158.19
NVIDIA H200156.3
NVIDIA H100 80GB HBM3152.2
NVIDIA RTX PRO 6000 Blackwell Workstation Edition139.45
NVIDIA A100 80GB SXM494.35
NVIDIA A100 40GB SXM490.07
NVIDIA L40S75.01
NVIDIA A10G47.8
NVIDIA L427.48
NVIDIA T419.44

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300169.52
2NVIDIA B200158.19
3NVIDIA H200156.3
4NVIDIA H100 80GB HBM3152.2
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition139.45
6NVIDIA A100 80GB SXM494.35
7NVIDIA A100 40GB SXM490.07
8NVIDIA L40S75.01
9NVIDIA A10G47.8
10NVIDIA L427.48
11NVIDIA T419.44

Qwen3-8B

Weights: bartowski/Qwen_Qwen3-8B-GGUF

NVIDIA B300270.35
NVIDIA B200253.14
NVIDIA H200251.04
NVIDIA H100 80GB HBM3244.2
NVIDIA RTX PRO 6000 Blackwell Workstation Edition226.88
NVIDIA A100 80GB SXM4152.8
NVIDIA A100 40GB SXM4145.96
NVIDIA L40S130.28
NVIDIA A10G83.04
NVIDIA L448.34
NVIDIA T437.37

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300270.35
2NVIDIA B200253.14
3NVIDIA H200251.04
4NVIDIA H100 80GB HBM3244.2
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition226.88
6NVIDIA A100 80GB SXM4152.8
7NVIDIA A100 40GB SXM4145.96
8NVIDIA L40S130.28
9NVIDIA A10G83.04
10NVIDIA L448.34
11NVIDIA T437.37

Qwen2.5-0.5B

Weights: Qwen/Qwen2.5-0.5B-Instruct-GGUF

NVIDIA H200914.58
NVIDIA RTX PRO 6000 Blackwell Workstation Edition905.27
NVIDIA H100 80GB HBM3890
NVIDIA B300843.4
NVIDIA B200818.27
NVIDIA L40S738.06
NVIDIA A100 80GB SXM4571.49
NVIDIA A100 40GB SXM4517.09
NVIDIA A10G508.78
NVIDIA L4399.4
NVIDIA T4295.47

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA H200914.58
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition905.27
3NVIDIA H100 80GB HBM3890
4NVIDIA B300843.4
5NVIDIA B200818.27
6NVIDIA L40S738.06
7NVIDIA A100 80GB SXM4571.49
8NVIDIA A100 40GB SXM4517.09
9NVIDIA A10G508.78
10NVIDIA L4399.4
11NVIDIA T4295.47

Qwen2.5-1.5B

Weights: Qwen/Qwen2.5-1.5B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition636.39
NVIDIA B300584.96
NVIDIA H200549.36
NVIDIA H100 80GB HBM3537.24
NVIDIA B200454.69
NVIDIA L40S427.23
NVIDIA A100 80GB SXM4335.29
NVIDIA A100 40GB SXM4315.59
NVIDIA A10G265.7
NVIDIA L4191.39
NVIDIA T4148.14

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition636.39
2NVIDIA B300584.96
3NVIDIA H200549.36
4NVIDIA H100 80GB HBM3537.24
5NVIDIA B200454.69
6NVIDIA L40S427.23
7NVIDIA A100 80GB SXM4335.29
8NVIDIA A100 40GB SXM4315.59
9NVIDIA A10G265.7
10NVIDIA L4191.39
11NVIDIA T4148.14

Qwen2.5-14B

Weights: bartowski/Qwen2.5-14B-Instruct-GGUF

NVIDIA B300159.33
NVIDIA B200150.57
NVIDIA H200148.76
NVIDIA H100 80GB HBM3144.7
NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.66
NVIDIA A100 80GB SXM490.46
NVIDIA A100 40GB SXM486.07
NVIDIA L40S74.62
NVIDIA A10G47.02
NVIDIA L427.37
NVIDIA T419.84

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300159.33
2NVIDIA B200150.57
3NVIDIA H200148.76
4NVIDIA H100 80GB HBM3144.7
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.66
6NVIDIA A100 80GB SXM490.46
7NVIDIA A100 40GB SXM486.07
8NVIDIA L40S74.62
9NVIDIA A10G47.02
10NVIDIA L427.37
11NVIDIA T419.84

Qwen2.5-3B

Weights: Qwen/Qwen2.5-3B-Instruct-GGUF

NVIDIA B300409.56
NVIDIA H200400.48
NVIDIA RTX PRO 6000 Blackwell Workstation Edition398.72
NVIDIA H100 80GB HBM3395.52
NVIDIA B200388.14
NVIDIA L40S270.04
NVIDIA A100 80GB SXM4248.47
NVIDIA A100 40GB SXM4240.48
NVIDIA A10G168.23
NVIDIA L4110.84
NVIDIA T486.04

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300409.56
2NVIDIA H200400.48
3NVIDIA RTX PRO 6000 Blackwell Workstation Edition398.72
4NVIDIA H100 80GB HBM3395.52
5NVIDIA B200388.14
6NVIDIA L40S270.04
7NVIDIA A100 80GB SXM4248.47
8NVIDIA A100 40GB SXM4240.48
9NVIDIA A10G168.23
10NVIDIA L4110.84
11NVIDIA T486.04

Qwen2.5-7B

Weights: bartowski/Qwen2.5-7B-Instruct-GGUF

NVIDIA B300293.63
NVIDIA B200277.17
NVIDIA H200265.15
NVIDIA H100 80GB HBM3264.17
NVIDIA RTX PRO 6000 Blackwell Workstation Edition256.63
NVIDIA A100 80GB SXM4166.92
NVIDIA A100 40GB SXM4159.81
NVIDIA L40S143.94
NVIDIA A10G89.68
NVIDIA L453.32
NVIDIA T439.01

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300293.63
2NVIDIA B200277.17
3NVIDIA H200265.15
4NVIDIA H100 80GB HBM3264.17
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition256.63
6NVIDIA A100 80GB SXM4166.92
7NVIDIA A100 40GB SXM4159.81
8NVIDIA L40S143.94
9NVIDIA A10G89.68
10NVIDIA L453.32
11NVIDIA T439.01

Qwen2.5-Coder-1.5B

Weights: Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition650.4
NVIDIA B300584.5
NVIDIA H200545.54
NVIDIA H100 80GB HBM3538.44
NVIDIA B200455.35
NVIDIA L40S427.12
NVIDIA A100 80GB SXM4333.87
NVIDIA A100 40GB SXM4322.96
NVIDIA A10G264.79
NVIDIA L4184.6
NVIDIA T4148.51

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition650.4
2NVIDIA B300584.5
3NVIDIA H200545.54
4NVIDIA H100 80GB HBM3538.44
5NVIDIA B200455.35
6NVIDIA L40S427.12
7NVIDIA A100 80GB SXM4333.87
8NVIDIA A100 40GB SXM4322.96
9NVIDIA A10G264.79
10NVIDIA L4184.6
11NVIDIA T4148.51

Qwen2.5-Coder-3B

Weights: Qwen/Qwen2.5-Coder-3B-Instruct-GGUF

NVIDIA B300410.06
NVIDIA RTX PRO 6000 Blackwell Workstation Edition401.24
NVIDIA H200401.11
NVIDIA H100 80GB HBM3394
NVIDIA B200388.17
NVIDIA L40S269.9
NVIDIA A100 80GB SXM4248.7
NVIDIA A100 40GB SXM4241.51
NVIDIA A10G168
NVIDIA L4109.42
NVIDIA T485.74

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300410.06
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition401.24
3NVIDIA H200401.11
4NVIDIA H100 80GB HBM3394
5NVIDIA B200388.17
6NVIDIA L40S269.9
7NVIDIA A100 80GB SXM4248.7
8NVIDIA A100 40GB SXM4241.51
9NVIDIA A10G168
10NVIDIA L4109.42
11NVIDIA T485.74

AI21-Jamba-Reasoning-3B

Weights: bartowski/ai21labs_AI21-Jamba-Reasoning-3B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition400.02
NVIDIA B300380.88
NVIDIA H200370.66
NVIDIA H100 80GB HBM3363.09
NVIDIA B200360.54
NVIDIA L40S258.41
NVIDIA A100 80GB SXM4239.23
NVIDIA A100 40GB SXM4231.15
NVIDIA A10G166.29
NVIDIA L4107.59
NVIDIA T486.91

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition400.02
2NVIDIA B300380.88
3NVIDIA H200370.66
4NVIDIA H100 80GB HBM3363.09
5NVIDIA B200360.54
6NVIDIA L40S258.41
7NVIDIA A100 80GB SXM4239.23
8NVIDIA A100 40GB SXM4231.15
9NVIDIA A10G166.29
10NVIDIA L4107.59
11NVIDIA T486.91

Codestral 22B (Q3_K_M)

Weights: bartowski/Codestral-22B-v0.1-GGUF

NVIDIA B300107.98
NVIDIA RTX PRO 6000 Blackwell Workstation Edition107.67
NVIDIA B20089.9
NVIDIA H20088.03
NVIDIA H100 80GB HBM387.21
NVIDIA L40S59.53
NVIDIA A100 80GB SXM449.52
NVIDIA A100 40GB SXM448.37
NVIDIA A10G28.01
NVIDIA L417.96
NVIDIA T411.3

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA B300107.98
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition107.67
3NVIDIA B20089.9
4NVIDIA H20088.03
5NVIDIA H100 80GB HBM387.21
6NVIDIA L40S59.53
7NVIDIA A100 80GB SXM449.52
8NVIDIA A100 40GB SXM448.37
9NVIDIA A10G28.01
10NVIDIA L417.96
11NVIDIA T411.3

DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored

Weights: bartowski/DarkIdol-Llama-3.1-8B-Instruct-1.2-Uncensored-GGUF

NVIDIA B300292.55
NVIDIA B200274.3
NVIDIA H200267.82
NVIDIA H100 80GB HBM3262.76
NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.45
NVIDIA A100 80GB SXM4164.46
NVIDIA A100 40GB SXM4156.62
NVIDIA L40S134.78
NVIDIA A10G87.84
NVIDIA L448.82
NVIDIA T435.73

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300292.55
2NVIDIA B200274.3
3NVIDIA H200267.82
4NVIDIA H100 80GB HBM3262.76
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.45
6NVIDIA A100 80GB SXM4164.46
7NVIDIA A100 40GB SXM4156.62
8NVIDIA L40S134.78
9NVIDIA A10G87.84
10NVIDIA L448.82
11NVIDIA T435.73

DeepSeek-R1 Distill 14B (Q3_K_M)

Weights: bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition149.81
NVIDIA B300144.74
NVIDIA B200120.47
NVIDIA H200119.66
NVIDIA H100 80GB HBM3118.43
NVIDIA L40S85.85
NVIDIA A100 80GB SXM467.13
NVIDIA A100 40GB SXM465.33
NVIDIA A10G41.72
NVIDIA L427.87
NVIDIA T416.86

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition149.81
2NVIDIA B300144.74
3NVIDIA B200120.47
4NVIDIA H200119.66
5NVIDIA H100 80GB HBM3118.43
6NVIDIA L40S85.85
7NVIDIA A100 80GB SXM467.13
8NVIDIA A100 40GB SXM465.33
9NVIDIA A10G41.72
10NVIDIA L427.87
11NVIDIA T416.86

dolphin-2.9-llama3-8b

Weights: bartowski/dolphin-2.9-llama3-8b-GGUF

NVIDIA B300290.47
NVIDIA B200274.48
NVIDIA H200267.96
NVIDIA H100 80GB HBM3262.47
NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.29
NVIDIA A100 80GB SXM4164.09
NVIDIA A100 40GB SXM4153.34
NVIDIA L40S134.94
NVIDIA A10G87.98
NVIDIA L450.04
NVIDIA T435.1

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300290.47
2NVIDIA B200274.48
3NVIDIA H200267.96
4NVIDIA H100 80GB HBM3262.47
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.29
6NVIDIA A100 80GB SXM4164.09
7NVIDIA A100 40GB SXM4153.34
8NVIDIA L40S134.94
9NVIDIA A10G87.98
10NVIDIA L450.04
11NVIDIA T435.1

Gemma 3 12B (Q3_K_M)

Weights: bartowski/google_gemma-3-12b-it-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition153.93
NVIDIA B300149.61
NVIDIA H200127.04
NVIDIA H100 80GB HBM3125.92
NVIDIA B200125.31
NVIDIA L40S93.05
NVIDIA A100 80GB SXM471.88
NVIDIA A100 40GB SXM469.7
NVIDIA A10G46.29
NVIDIA L432.29
NVIDIA T419.79

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition153.93
2NVIDIA B300149.61
3NVIDIA H200127.04
4NVIDIA H100 80GB HBM3125.92
5NVIDIA B200125.31
6NVIDIA L40S93.05
7NVIDIA A100 80GB SXM471.88
8NVIDIA A100 40GB SXM469.7
9NVIDIA A10G46.29
10NVIDIA L432.29
11NVIDIA T419.79

GLM-4.7-Flash-REAP-23B-A3B

Weights: unsloth/GLM-4.7-Flash-REAP-23B-A3B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition199.14
NVIDIA B300178.49
NVIDIA H200171.45
NVIDIA H100 80GB HBM3168.41
NVIDIA B200167.76
NVIDIA L40S146.51
NVIDIA A100 80GB SXM4112.75
NVIDIA A100 40GB SXM4109.1
NVIDIA A10G97.48
NVIDIA L470.87
NVIDIA T453.83

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition199.14
2NVIDIA B300178.49
3NVIDIA H200171.45
4NVIDIA H100 80GB HBM3168.41
5NVIDIA B200167.76
6NVIDIA L40S146.51
7NVIDIA A100 80GB SXM4112.75
8NVIDIA A100 40GB SXM4109.1
9NVIDIA A10G97.48
10NVIDIA L470.87
11NVIDIA T453.83

Josiefied-Qwen3-8B-abliterated-v1

Weights: bartowski/Goekdeniz-Guelmez_Josiefied-Qwen3-8B-abliterated-v1-GGUF

NVIDIA B300270.09
NVIDIA B200253.28
NVIDIA H200250.88
NVIDIA H100 80GB HBM3245
NVIDIA RTX PRO 6000 Blackwell Workstation Edition227.15
NVIDIA A100 80GB SXM4152.29
NVIDIA A100 40GB SXM4146.8
NVIDIA L40S130.25
NVIDIA A10G83.26
NVIDIA L448.72
NVIDIA T435.62

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300270.09
2NVIDIA B200253.28
3NVIDIA H200250.88
4NVIDIA H100 80GB HBM3245
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition227.15
6NVIDIA A100 80GB SXM4152.29
7NVIDIA A100 40GB SXM4146.8
8NVIDIA L40S130.25
9NVIDIA A10G83.26
10NVIDIA L448.72
11NVIDIA T435.62

Hermes-3-Llama-3.2-3B

Weights: bartowski/Hermes-3-Llama-3.2-3B-GGUF

NVIDIA B300446.78
NVIDIA RTX PRO 6000 Blackwell Workstation Edition439.65
NVIDIA H200430.79
NVIDIA H100 80GB HBM3423.72
NVIDIA B200420.6
NVIDIA A100 80GB SXM4269.13
NVIDIA L40S265.87
NVIDIA A100 40GB SXM4261
NVIDIA A10G169.23
NVIDIA L4106.14
NVIDIA T484.99

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300446.78
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition439.65
3NVIDIA H200430.79
4NVIDIA H100 80GB HBM3423.72
5NVIDIA B200420.6
6NVIDIA A100 80GB SXM4269.13
7NVIDIA L40S265.87
8NVIDIA A100 40GB SXM4261
9NVIDIA A10G169.23
10NVIDIA L4106.14
11NVIDIA T484.99

L3-8B-Stheno-v3.2

Weights: bartowski/L3-8B-Stheno-v3.2-GGUF

NVIDIA B300293
NVIDIA B200274.48
NVIDIA H200267.92
NVIDIA H100 80GB HBM3261.83
NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.84
NVIDIA A100 80GB SXM4164.51
NVIDIA A100 40GB SXM4156.55
NVIDIA L40S135.53
NVIDIA A10G86.8
NVIDIA L450.07
NVIDIA T436.28

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300293
2NVIDIA B200274.48
3NVIDIA H200267.92
4NVIDIA H100 80GB HBM3261.83
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition239.84
6NVIDIA A100 80GB SXM4164.51
7NVIDIA A100 40GB SXM4156.55
8NVIDIA L40S135.53
9NVIDIA A10G86.8
10NVIDIA L450.07
11NVIDIA T436.28

LFM2.5-8B-A1B

Weights: unsloth/LFM2.5-8B-A1B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition617.09
NVIDIA B300596.16
NVIDIA H200583.96
NVIDIA H100 80GB HBM3581.98
NVIDIA B200575.15
NVIDIA L40S394.48
NVIDIA A100 80GB SXM4369.01
NVIDIA A100 40GB SXM4351.47
NVIDIA A10G268.04
NVIDIA L4170.2
NVIDIA T4139.03

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition617.09
2NVIDIA B300596.16
3NVIDIA H200583.96
4NVIDIA H100 80GB HBM3581.98
5NVIDIA B200575.15
6NVIDIA L40S394.48
7NVIDIA A100 80GB SXM4369.01
8NVIDIA A100 40GB SXM4351.47
9NVIDIA A10G268.04
10NVIDIA L4170.2
11NVIDIA T4139.03

Mistral-Nemo-Instruct-2407

Weights: bartowski/Mistral-Nemo-Instruct-2407-GGUF

NVIDIA B300196.92
NVIDIA B200183.92
NVIDIA H200181.3
NVIDIA H100 80GB HBM3177.38
NVIDIA RTX PRO 6000 Blackwell Workstation Edition164.28
NVIDIA A100 80GB SXM4111.07
NVIDIA A100 40GB SXM4106.33
NVIDIA L40S89.72
NVIDIA A10G57
NVIDIA L433.01
NVIDIA T422.98

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300196.92
2NVIDIA B200183.92
3NVIDIA H200181.3
4NVIDIA H100 80GB HBM3177.38
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition164.28
6NVIDIA A100 80GB SXM4111.07
7NVIDIA A100 40GB SXM4106.33
8NVIDIA L40S89.72
9NVIDIA A10G57
10NVIDIA L433.01
11NVIDIA T422.98

Mistral Small 24B (Q3_K_M)

Weights: bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition108.88
NVIDIA B300108.37
NVIDIA B20087.72
NVIDIA H20085.1
NVIDIA H100 80GB HBM384
NVIDIA L40S58.21
NVIDIA A100 80GB SXM446.63
NVIDIA A100 40GB SXM445.34
NVIDIA A10G26.8
NVIDIA L416.62
NVIDIA T410.13

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition108.88
2NVIDIA B300108.37
3NVIDIA B20087.72
4NVIDIA H20085.1
5NVIDIA H100 80GB HBM384
6NVIDIA L40S58.21
7NVIDIA A100 80GB SXM446.63
8NVIDIA A100 40GB SXM445.34
9NVIDIA A10G26.8
10NVIDIA L416.62
11NVIDIA T410.13

NemoMix-Unleashed-12B

Weights: bartowski/NemoMix-Unleashed-12B-GGUF

NVIDIA B300197.06
NVIDIA B200183.93
NVIDIA H200181.53
NVIDIA H100 80GB HBM3177.28
NVIDIA RTX PRO 6000 Blackwell Workstation Edition164.21
NVIDIA A100 80GB SXM4110.99
NVIDIA A100 40GB SXM4105.05
NVIDIA L40S88.85
NVIDIA A10G57.52
NVIDIA L433.02
NVIDIA T423.19

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300197.06
2NVIDIA B200183.93
3NVIDIA H200181.53
4NVIDIA H100 80GB HBM3177.28
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition164.21
6NVIDIA A100 80GB SXM4110.99
7NVIDIA A100 40GB SXM4105.05
8NVIDIA L40S88.85
9NVIDIA A10G57.52
10NVIDIA L433.02
11NVIDIA T423.19

Hermes-4-14B

Weights: bartowski/NousResearch_Hermes-4-14B-GGUF

NVIDIA B300169.51
NVIDIA B200158.22
NVIDIA H200156.26
NVIDIA H100 80GB HBM3152
NVIDIA RTX PRO 6000 Blackwell Workstation Edition139.56
NVIDIA A100 80GB SXM494.48
NVIDIA A100 40GB SXM489.97
NVIDIA L40S75.1
NVIDIA A10G47.8
NVIDIA L427.48
NVIDIA T419.02

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300169.51
2NVIDIA B200158.22
3NVIDIA H200156.26
4NVIDIA H100 80GB HBM3152
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition139.56
6NVIDIA A100 80GB SXM494.48
7NVIDIA A100 40GB SXM489.97
8NVIDIA L40S75.1
9NVIDIA A10G47.8
10NVIDIA L427.48
11NVIDIA T419.02

phi-2

Weights: TheBloke/phi-2-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition410.12
NVIDIA B300356.65
NVIDIA B200348.72
NVIDIA H200347.08
NVIDIA H100 80GB HBM3340.09
NVIDIA L40S283.26
NVIDIA A100 80GB SXM4235.55
NVIDIA A100 40GB SXM4228.1
NVIDIA A10G172.45
NVIDIA L4114.99
NVIDIA T489.6

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition410.12
2NVIDIA B300356.65
3NVIDIA B200348.72
4NVIDIA H200347.08
5NVIDIA H100 80GB HBM3340.09
6NVIDIA L40S283.26
7NVIDIA A100 80GB SXM4235.55
8NVIDIA A100 40GB SXM4228.1
9NVIDIA A10G172.45
10NVIDIA L4114.99
11NVIDIA T489.6

Phi-4 14B (Q3_K_M)

Weights: bartowski/phi-4-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition165.43
NVIDIA B300165.06
NVIDIA H200139.97
NVIDIA H100 80GB HBM3138.87
NVIDIA B200135.21
NVIDIA L40S89.18
NVIDIA A100 80GB SXM482.68
NVIDIA A100 40GB SXM480.71
NVIDIA A10G43.56
NVIDIA L427.86
NVIDIA T416.04

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition165.43
2NVIDIA B300165.06
3NVIDIA H200139.97
4NVIDIA H100 80GB HBM3138.87
5NVIDIA B200135.21
6NVIDIA L40S89.18
7NVIDIA A100 80GB SXM482.68
8NVIDIA A100 40GB SXM480.71
9NVIDIA A10G43.56
10NVIDIA L427.86
11NVIDIA T416.04

Qwen3-4B-Instruct-2507

Weights: bartowski/Qwen_Qwen3-4B-Instruct-2507-GGUF

NVIDIA B300339.78
NVIDIA RTX PRO 6000 Blackwell Workstation Edition333.54
NVIDIA H200318.67
NVIDIA B200316.66
NVIDIA H100 80GB HBM3310.58
NVIDIA L40S210.24
NVIDIA A100 80GB SXM4199.39
NVIDIA A100 40GB SXM4192.89
NVIDIA A10G130.83
NVIDIA L483.03
NVIDIA T466.89

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300339.78
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition333.54
3NVIDIA H200318.67
4NVIDIA B200316.66
5NVIDIA H100 80GB HBM3310.58
6NVIDIA L40S210.24
7NVIDIA A100 80GB SXM4199.39
8NVIDIA A100 40GB SXM4192.89
9NVIDIA A10G130.83
10NVIDIA L483.03
11NVIDIA T466.89

Qwen1.5-0.5B

Weights: Qwen/Qwen1.5-0.5B-Chat-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition1025.58
NVIDIA B300894.84
NVIDIA H200855.22
NVIDIA H100 80GB HBM3847.02
NVIDIA L40S774.78
NVIDIA B200699.73
NVIDIA A10G542.19
NVIDIA A100 80GB SXM4540.32
NVIDIA A100 40GB SXM4497.73
NVIDIA L4431.41
NVIDIA T4328.29

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition1025.58
2NVIDIA B300894.84
3NVIDIA H200855.22
4NVIDIA H100 80GB HBM3847.02
5NVIDIA L40S774.78
6NVIDIA B200699.73
7NVIDIA A10G542.19
8NVIDIA A100 80GB SXM4540.32
9NVIDIA A100 40GB SXM4497.73
10NVIDIA L4431.41
11NVIDIA T4328.29

Qwen2-1.5B

Weights: Qwen/Qwen2-1.5B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition648.49
NVIDIA B300583.79
NVIDIA H200546.47
NVIDIA H100 80GB HBM3536.52
NVIDIA B200454.55
NVIDIA L40S416.57
NVIDIA A100 80GB SXM4333.79
NVIDIA A100 40GB SXM4322.09
NVIDIA A10G265.45
NVIDIA L4189.18
NVIDIA T4147.68

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition648.49
2NVIDIA B300583.79
3NVIDIA H200546.47
4NVIDIA H100 80GB HBM3536.52
5NVIDIA B200454.55
6NVIDIA L40S416.57
7NVIDIA A100 80GB SXM4333.79
8NVIDIA A100 40GB SXM4322.09
9NVIDIA A10G265.45
10NVIDIA L4189.18
11NVIDIA T4147.68

Uncensored

Weights: bartowski/Qwen2.5-14B_Uncensored_Instruct-GGUF

NVIDIA B300159.48
NVIDIA B200150.51
NVIDIA H200148.7
NVIDIA H100 80GB HBM3144.92
NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.68
NVIDIA A100 80GB SXM489.43
NVIDIA A100 40GB SXM485.82
NVIDIA L40S74.31
NVIDIA A10G47.45
NVIDIA L427.29
NVIDIA T418.94

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300159.48
2NVIDIA B200150.51
3NVIDIA H200148.7
4NVIDIA H100 80GB HBM3144.92
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.68
6NVIDIA A100 80GB SXM489.43
7NVIDIA A100 40GB SXM485.82
8NVIDIA L40S74.31
9NVIDIA A10G47.45
10NVIDIA L427.29
11NVIDIA T418.94

Qwen2.5-Coder-0.5B

Weights: Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF

NVIDIA H200920.36
NVIDIA RTX PRO 6000 Blackwell Workstation Edition910.13
NVIDIA H100 80GB HBM3892.81
NVIDIA B300843.44
NVIDIA B200828.48
NVIDIA L40S715.81
NVIDIA A100 80GB SXM4562.59
NVIDIA A100 40GB SXM4542.36
NVIDIA A10G513.36
NVIDIA L4397.1
NVIDIA T4294.96

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA H200920.36
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition910.13
3NVIDIA H100 80GB HBM3892.81
4NVIDIA B300843.44
5NVIDIA B200828.48
6NVIDIA L40S715.81
7NVIDIA A100 80GB SXM4562.59
8NVIDIA A100 40GB SXM4542.36
9NVIDIA A10G513.36
10NVIDIA L4397.1
11NVIDIA T4294.96

Qwen2.5-Coder-14B-Instruct-abliterated

Weights: bartowski/Qwen2.5-Coder-14B-Instruct-abliterated-GGUF

NVIDIA B300159.5
NVIDIA B200150.67
NVIDIA H200148.48
NVIDIA H100 80GB HBM3145.01
NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.88
NVIDIA A100 80GB SXM490.28
NVIDIA A100 40GB SXM486.34
NVIDIA L40S74.63
NVIDIA A10G46.91
NVIDIA L427.29
NVIDIA T418.86

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300159.5
2NVIDIA B200150.67
3NVIDIA H200148.48
4NVIDIA H100 80GB HBM3145.01
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition135.88
6NVIDIA A100 80GB SXM490.28
7NVIDIA A100 40GB SXM486.34
8NVIDIA L40S74.63
9NVIDIA A10G46.91
10NVIDIA L427.29
11NVIDIA T418.86

Qwen2.5-Coder-7B-Instruct-abliterated

Weights: bartowski/Qwen2.5-Coder-7B-Instruct-abliterated-GGUF

NVIDIA B300292.21
NVIDIA B200276.57
NVIDIA H200270.92
NVIDIA H100 80GB HBM3264.24
NVIDIA RTX PRO 6000 Blackwell Workstation Edition257.07
NVIDIA A100 80GB SXM4166.84
NVIDIA A100 40GB SXM4159.76
NVIDIA L40S142.97
NVIDIA A10G91.41
NVIDIA L453.02
NVIDIA T437.31

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300292.21
2NVIDIA B200276.57
3NVIDIA H200270.92
4NVIDIA H100 80GB HBM3264.24
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition257.07
6NVIDIA A100 80GB SXM4166.84
7NVIDIA A100 40GB SXM4159.76
8NVIDIA L40S142.97
9NVIDIA A10G91.41
10NVIDIA L453.02
11NVIDIA T437.31

Qwen3 30B A3B (Q3_K_M)

Weights: bartowski/Qwen_Qwen3-30B-A3B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition309.61
NVIDIA B300268.71
NVIDIA H200246.35
NVIDIA H100 80GB HBM3242.6
NVIDIA B200229.58
NVIDIA L40S224.95
NVIDIA A100 80GB SXM4146.72
NVIDIA A100 40GB SXM4142.78
NVIDIA A10G120.38
NVIDIA L494.03
NVIDIA T461.82

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition309.61
2NVIDIA B300268.71
3NVIDIA H200246.35
4NVIDIA H100 80GB HBM3242.6
5NVIDIA B200229.58
6NVIDIA L40S224.95
7NVIDIA A100 80GB SXM4146.72
8NVIDIA A100 40GB SXM4142.78
9NVIDIA A10G120.38
10NVIDIA L494.03
11NVIDIA T461.82

Qwen3-4B-Instruct-2507

Weights: lmstudio-community/Qwen3-4B-Instruct-2507-GGUF

NVIDIA B300339.72
NVIDIA RTX PRO 6000 Blackwell Workstation Edition334.71
NVIDIA H200318.27
NVIDIA B200316.7
NVIDIA H100 80GB HBM3311.47
NVIDIA L40S212.05
NVIDIA A100 80GB SXM4199.65
NVIDIA A100 40GB SXM4193.39
NVIDIA A10G129.9
NVIDIA L483.13
NVIDIA T467.45

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300339.72
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition334.71
3NVIDIA H200318.27
4NVIDIA B200316.7
5NVIDIA H100 80GB HBM3311.47
6NVIDIA L40S212.05
7NVIDIA A100 80GB SXM4199.65
8NVIDIA A100 40GB SXM4193.39
9NVIDIA A10G129.9
10NVIDIA L483.13
11NVIDIA T467.45

Qwen3-4B-Thinking-2507

Weights: lmstudio-community/Qwen3-4B-Thinking-2507-GGUF

NVIDIA B300339.74
NVIDIA RTX PRO 6000 Blackwell Workstation Edition334.65
NVIDIA H200318.94
NVIDIA B200317.03
NVIDIA H100 80GB HBM3310.84
NVIDIA L40S211.88
NVIDIA A100 80GB SXM4199.71
NVIDIA A100 40GB SXM4189.89
NVIDIA A10G129.99
NVIDIA L483.12
NVIDIA T467

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300339.74
2NVIDIA RTX PRO 6000 Blackwell Workstation Edition334.65
3NVIDIA H200318.94
4NVIDIA B200317.03
5NVIDIA H100 80GB HBM3310.84
6NVIDIA L40S211.88
7NVIDIA A100 80GB SXM4199.71
8NVIDIA A100 40GB SXM4189.89
9NVIDIA A10G129.99
10NVIDIA L483.12
11NVIDIA T467

SmolLM2-135M

Weights: bartowski/SmolLM2-135M-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition1252.77
NVIDIA L40S916.56
NVIDIA H200905.38
NVIDIA H100 80GB HBM3898.53
NVIDIA B300855.87
NVIDIA B200815.3
NVIDIA A10G680.37
NVIDIA L4652.15
NVIDIA A100 80GB SXM4554.44
NVIDIA A100 40GB SXM4544.06
NVIDIA T4416.24

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition1252.77
2NVIDIA L40S916.56
3NVIDIA H200905.38
4NVIDIA H100 80GB HBM3898.53
5NVIDIA B300855.87
6NVIDIA B200815.3
7NVIDIA A10G680.37
8NVIDIA L4652.15
9NVIDIA A100 80GB SXM4554.44
10NVIDIA A100 40GB SXM4544.06
11NVIDIA T4416.24

Cydonia-24B-v4.3

Weights: bartowski/TheDrummer_Cydonia-24B-v4.3-GGUF

NVIDIA B300121.9
NVIDIA B200113.81
NVIDIA H200109.2
NVIDIA H100 80GB HBM3105.49
NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.76
NVIDIA A100 80GB SXM465.82
NVIDIA A100 40GB SXM462.41
NVIDIA L40S48.43
NVIDIA A10G31.04
NVIDIA L417.3
NVIDIA T411.21

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B300121.9
2NVIDIA B200113.81
3NVIDIA H200109.2
4NVIDIA H100 80GB HBM3105.49
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition93.76
6NVIDIA A100 80GB SXM465.82
7NVIDIA A100 40GB SXM462.41
8NVIDIA L40S48.43
9NVIDIA A10G31.04
10NVIDIA L417.3
11NVIDIA T411.21

Qwen3 30B A3B

Weights: bartowski/Qwen_Qwen3-30B-A3B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition308.74
NVIDIA H200292.11
NVIDIA B300284.66
NVIDIA H100 80GB HBM3283.78
NVIDIA B200270.84
NVIDIA L40S213.66
NVIDIA A100 80GB SXM4177.32
NVIDIA A100 40GB SXM4169.1
NVIDIA A10G140.12
NVIDIA L496.15

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition308.74
2NVIDIA H200292.11
3NVIDIA B300284.66
4NVIDIA H100 80GB HBM3283.78
5NVIDIA B200270.84
6NVIDIA L40S213.66
7NVIDIA A100 80GB SXM4177.32
8NVIDIA A100 40GB SXM4169.1
9NVIDIA A10G140.12
10NVIDIA L496.15

DeepSeek-R1 Distill 32B

Weights: bartowski/DeepSeek-R1-Distill-Qwen-32B-GGUF

NVIDIA B30082.9
NVIDIA B20078.55
NVIDIA H100 80GB HBM375.8
NVIDIA H20075.8
NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.06
NVIDIA A100 40GB SXM443.13
NVIDIA A100 80GB SXM443.11
NVIDIA L40S34.6
NVIDIA A10G21.85
NVIDIA L412.28

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30082.9
2NVIDIA B20078.55
3NVIDIA H100 80GB HBM375.8
4NVIDIA H20075.8
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.06
6NVIDIA A100 40GB SXM443.13
7NVIDIA A100 80GB SXM443.11
8NVIDIA L40S34.6
9NVIDIA A10G21.85
10NVIDIA L412.28

Dolphin 2.9.1 Yi 1.5 34B

Weights: bartowski/dolphin-2.9.1-yi-1.5-34b-GGUF

NVIDIA B30085.15
NVIDIA B20079.35
NVIDIA H100 80GB HBM376.76
NVIDIA H20076.7
NVIDIA RTX PRO 6000 Blackwell Workstation Edition63.74
NVIDIA A100 80GB SXM443.47
NVIDIA A100 40GB SXM442.97
NVIDIA L40S33.08
NVIDIA A10G21.12
NVIDIA L411.92

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30085.15
2NVIDIA B20079.35
3NVIDIA H100 80GB HBM376.76
4NVIDIA H20076.7
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition63.74
6NVIDIA A100 80GB SXM443.47
7NVIDIA A100 40GB SXM442.97
8NVIDIA L40S33.08
9NVIDIA A10G21.12
10NVIDIA L411.92

Gemma 3 27B

Weights: bartowski/google_gemma-3-27b-it-GGUF

NVIDIA B30093.44
NVIDIA B20086.08
NVIDIA H100 80GB HBM384.77
NVIDIA H20084.75
NVIDIA RTX PRO 6000 Blackwell Workstation Edition72.1
NVIDIA A100 80GB SXM448.43
NVIDIA A100 40GB SXM447.47
NVIDIA L40S38.48
NVIDIA A10G24.8
NVIDIA L414.18

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30093.44
2NVIDIA B20086.08
3NVIDIA H100 80GB HBM384.77
4NVIDIA H20084.75
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition72.1
6NVIDIA A100 80GB SXM448.43
7NVIDIA A100 40GB SXM447.47
8NVIDIA L40S38.48
9NVIDIA A10G24.8
10NVIDIA L414.18

Qwen2.5-Coder 32B

Weights: bartowski/Qwen2.5-Coder-32B-Instruct-GGUF

NVIDIA B30082.9
NVIDIA B20078.53
NVIDIA H100 80GB HBM375.81
NVIDIA H20075.77
NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.07
NVIDIA A100 80GB SXM443.36
NVIDIA A100 40GB SXM443.14
NVIDIA L40S34.59
NVIDIA A10G21.92
NVIDIA L412.29

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30082.9
2NVIDIA B20078.53
3NVIDIA H100 80GB HBM375.81
4NVIDIA H20075.77
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.07
6NVIDIA A100 80GB SXM443.36
7NVIDIA A100 40GB SXM443.14
8NVIDIA L40S34.59
9NVIDIA A10G21.92
10NVIDIA L412.29

Qwen3-Coder 30B A3B

Weights: bartowski/Qwen_Qwen3-Coder-30B-A3B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition317.68
NVIDIA H200297.9
NVIDIA H100 80GB HBM3296.22
NVIDIA B300284.88
NVIDIA B200277.68
NVIDIA L40S219.04
NVIDIA A100 80GB SXM4182.26
NVIDIA A100 40GB SXM4173.47
NVIDIA A10G142.77
NVIDIA L499.14

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition317.68
2NVIDIA H200297.9
3NVIDIA H100 80GB HBM3296.22
4NVIDIA B300284.88
5NVIDIA B200277.68
6NVIDIA L40S219.04
7NVIDIA A100 80GB SXM4182.26
8NVIDIA A100 40GB SXM4173.47
9NVIDIA A10G142.77
10NVIDIA L499.14

QwQ 32B

Weights: Qwen/QwQ-32B-GGUF

NVIDIA B30082.91
NVIDIA B20078.47
NVIDIA H100 80GB HBM375.82
NVIDIA H20075.77
NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.07
NVIDIA A100 80GB SXM443.27
NVIDIA A100 40GB SXM443.02
NVIDIA L40S34.59
NVIDIA A10G21.87
NVIDIA L412.28

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30082.91
2NVIDIA B20078.47
3NVIDIA H100 80GB HBM375.82
4NVIDIA H20075.77
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.07
6NVIDIA A100 80GB SXM443.27
7NVIDIA A100 40GB SXM443.02
8NVIDIA L40S34.59
9NVIDIA A10G21.87
10NVIDIA L412.28

Olmo-3.1-32B-Think

Weights: bartowski/allenai_Olmo-3.1-32B-Think-GGUF

NVIDIA B30087.85
NVIDIA B20079.75
NVIDIA H20078.46
NVIDIA H100 80GB HBM375.53
NVIDIA RTX PRO 6000 Blackwell Workstation Edition65.71
NVIDIA A100 80GB SXM446.49
NVIDIA A100 40GB SXM443.67
NVIDIA L40S34.11
NVIDIA A10G22.13
NVIDIA L412.27

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30087.85
2NVIDIA B20079.75
3NVIDIA H20078.46
4NVIDIA H100 80GB HBM375.53
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition65.71
6NVIDIA A100 80GB SXM446.49
7NVIDIA A100 40GB SXM443.67
8NVIDIA L40S34.11
9NVIDIA A10G22.13
10NVIDIA L412.27

GLM-4.7-Flash

Weights: unsloth/GLM-4.7-Flash-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition215.61
NVIDIA B300196.52
NVIDIA H200188.57
NVIDIA H100 80GB HBM3186.05
NVIDIA B200183.82
NVIDIA L40S157.89
NVIDIA A100 80GB SXM4124.08
NVIDIA A100 40GB SXM4119.1
NVIDIA A10G103.24
NVIDIA L475.03

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition215.61
2NVIDIA B300196.52
3NVIDIA H200188.57
4NVIDIA H100 80GB HBM3186.05
5NVIDIA B200183.82
6NVIDIA L40S157.89
7NVIDIA A100 80GB SXM4124.08
8NVIDIA A100 40GB SXM4119.1
9NVIDIA A10G103.24
10NVIDIA L475.03

KAT-Coder-V2.5-Dev

Weights: bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition253.35
NVIDIA B300239.66
NVIDIA H200236.31
NVIDIA H100 80GB HBM3231.83
NVIDIA B200230.68
NVIDIA L40S173.06
NVIDIA A100 80GB SXM4146.38
NVIDIA A100 40GB SXM4137.31
NVIDIA A10G113.6
NVIDIA L479.01

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition253.35
2NVIDIA B300239.66
3NVIDIA H200236.31
4NVIDIA H100 80GB HBM3231.83
5NVIDIA B200230.68
6NVIDIA L40S173.06
7NVIDIA A100 80GB SXM4146.38
8NVIDIA A100 40GB SXM4137.31
9NVIDIA A10G113.6
10NVIDIA L479.01

Ornith-1.0-35B

Weights: unsloth/Ornith-1.0-35B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition237.7
NVIDIA B300232.36
NVIDIA H100 80GB HBM3221.65
NVIDIA B200221.13
NVIDIA H200208.51
NVIDIA L40S157.39
NVIDIA A100 80GB SXM4138.8
NVIDIA A100 40GB SXM4131.77
NVIDIA A10G104.9
NVIDIA L470.31

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition237.7
2NVIDIA B300232.36
3NVIDIA H100 80GB HBM3221.65
4NVIDIA B200221.13
5NVIDIA H200208.51
6NVIDIA L40S157.39
7NVIDIA A100 80GB SXM4138.8
8NVIDIA A100 40GB SXM4131.77
9NVIDIA A10G104.9
10NVIDIA L470.31

Qwen-AgentWorld-35B-A3B

Weights: unsloth/Qwen-AgentWorld-35B-A3B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition234.82
NVIDIA B300233.04
NVIDIA H200225.31
NVIDIA B200221.78
NVIDIA H100 80GB HBM3219.1
NVIDIA L40S156.98
NVIDIA A100 80GB SXM4139.69
NVIDIA A100 40GB SXM4131.16
NVIDIA A10G104.36
NVIDIA L470.23

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition234.82
2NVIDIA B300233.04
3NVIDIA H200225.31
4NVIDIA B200221.78
5NVIDIA H100 80GB HBM3219.1
6NVIDIA L40S156.98
7NVIDIA A100 80GB SXM4139.69
8NVIDIA A100 40GB SXM4131.16
9NVIDIA A10G104.36
10NVIDIA L470.23

Qwen2.5-32B

Weights: bartowski/Qwen2.5-32B-Instruct-GGUF

NVIDIA B30083.41
NVIDIA B20078.55
NVIDIA H20075.76
NVIDIA H100 80GB HBM373.5
NVIDIA RTX PRO 6000 Blackwell Workstation Edition66
NVIDIA A100 80GB SXM445.6
NVIDIA A100 40GB SXM443.23
NVIDIA L40S34.59
NVIDIA A10G21.98
NVIDIA L412.24

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30083.41
2NVIDIA B20078.55
3NVIDIA H20075.76
4NVIDIA H100 80GB HBM373.5
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition66
6NVIDIA A100 80GB SXM445.6
7NVIDIA A100 40GB SXM443.23
8NVIDIA L40S34.59
9NVIDIA A10G21.98
10NVIDIA L412.24

DeepSeek-R1-Distill-Qwen-32B-abliterated

Weights: bartowski/DeepSeek-R1-Distill-Qwen-32B-abliterated-GGUF

NVIDIA B30083.19
NVIDIA B20078.48
NVIDIA H20075.71
NVIDIA H100 80GB HBM373.48
NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.03
NVIDIA A100 80GB SXM443.48
NVIDIA A100 40GB SXM443.07
NVIDIA L40S34.43
NVIDIA A10G22.14
NVIDIA L412.26

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30083.19
2NVIDIA B20078.48
3NVIDIA H20075.71
4NVIDIA H100 80GB HBM373.48
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition66.03
6NVIDIA A100 80GB SXM443.48
7NVIDIA A100 40GB SXM443.07
8NVIDIA L40S34.43
9NVIDIA A10G22.14
10NVIDIA L412.26

Qwen3-30B-A3B

Weights: bartowski/Qwen_Qwen3-30B-A3B-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition311.76
NVIDIA H200293.47
NVIDIA B300291.82
NVIDIA H100 80GB HBM3285.44
NVIDIA B200271.69
NVIDIA L40S214.05
NVIDIA A100 80GB SXM4178.14
NVIDIA A100 40GB SXM4167.1
NVIDIA A10G140.14
NVIDIA L495.02

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition311.76
2NVIDIA H200293.47
3NVIDIA B300291.82
4NVIDIA H100 80GB HBM3285.44
5NVIDIA B200271.69
6NVIDIA L40S214.05
7NVIDIA A100 80GB SXM4178.14
8NVIDIA A100 40GB SXM4167.1
9NVIDIA A10G140.14
10NVIDIA L495.02

Qwen2.5-Coder 32B (Q3_K_M)

Weights: bartowski/Qwen2.5-Coder-32B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition75.78
NVIDIA B30074.28
NVIDIA B20060.24
NVIDIA H20058.87
NVIDIA H100 80GB HBM358.05
NVIDIA L40S41.06
NVIDIA A100 80GB SXM431.99
NVIDIA A100 40GB SXM431.04
NVIDIA A10G18.78
NVIDIA L412.05

Measured in Q3_K_M. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition75.78
2NVIDIA B30074.28
3NVIDIA B20060.24
4NVIDIA H20058.87
5NVIDIA H100 80GB HBM358.05
6NVIDIA L40S41.06
7NVIDIA A100 80GB SXM431.99
8NVIDIA A100 40GB SXM431.04
9NVIDIA A10G18.78
10NVIDIA L412.05

Nemotron-3-Nano-30B-A3B

Weights: unsloth/Nemotron-3-Nano-30B-A3B-GGUF

NVIDIA B200345.93
NVIDIA B300345.31
NVIDIA H200327.48
NVIDIA RTX PRO 6000 Blackwell Workstation Edition325.43
NVIDIA H100 80GB HBM3316.51
NVIDIA A100 80GB SXM4204.06
NVIDIA A100 40GB SXM4192.44
NVIDIA L40S190.96

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B200345.93
2NVIDIA B300345.31
3NVIDIA H200327.48
4NVIDIA RTX PRO 6000 Blackwell Workstation Edition325.43
5NVIDIA H100 80GB HBM3316.51
6NVIDIA A100 80GB SXM4204.06
7NVIDIA A100 40GB SXM4192.44
8NVIDIA L40S190.96

DeepSeek-R1-Distill-Llama-70B

Weights: unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF

NVIDIA B30048.11
NVIDIA B20044.49
NVIDIA H20042.76
NVIDIA H100 80GB HBM341.17
NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.52
NVIDIA A100 80GB SXM422.99
NVIDIA L40S16.49

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30048.11
2NVIDIA B20044.49
3NVIDIA H20042.76
4NVIDIA H100 80GB HBM341.17
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.52
6NVIDIA A100 80GB SXM422.99
7NVIDIA L40S16.49

Llama-3.3-70B-Instruct-abliterated

Weights: bartowski/Llama-3.3-70B-Instruct-abliterated-GGUF

NVIDIA B30048.1
NVIDIA B20044.48
NVIDIA H20042.75
NVIDIA H100 80GB HBM341.15
NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.53
NVIDIA A100 80GB SXM422.92
NVIDIA L40S16.49

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30048.1
2NVIDIA B20044.48
3NVIDIA H20042.75
4NVIDIA H100 80GB HBM341.15
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.53
6NVIDIA A100 80GB SXM422.92
7NVIDIA L40S16.49

Meta-Llama-3.1-70B

Weights: bartowski/Meta-Llama-3.1-70B-Instruct-GGUF

NVIDIA B30048.12
NVIDIA B20044.48
NVIDIA H20042.72
NVIDIA H100 80GB HBM341.19
NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.52
NVIDIA A100 80GB SXM422.92
NVIDIA L40S16.49

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30048.12
2NVIDIA B20044.48
3NVIDIA H20042.72
4NVIDIA H100 80GB HBM341.19
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.52
6NVIDIA A100 80GB SXM422.92
7NVIDIA L40S16.49

Hermes-4-70B

Weights: lmstudio-community/Hermes-4-70B-GGUF

NVIDIA B30048.1
NVIDIA B20044.49
NVIDIA H20042.77
NVIDIA H100 80GB HBM341.2
NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.51
NVIDIA A100 80GB SXM424.47
NVIDIA L40S16.45

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30048.1
2NVIDIA B20044.49
3NVIDIA H20042.77
4NVIDIA H100 80GB HBM341.2
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition32.51
6NVIDIA A100 80GB SXM424.47
7NVIDIA L40S16.45

Qwen3-Coder-Next-abliterated

Weights: bartowski/huihui-ai_Qwen3-Coder-Next-abliterated-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition210.47
NVIDIA B300199.12
NVIDIA H200197.14
NVIDIA H100 80GB HBM3190.32
NVIDIA B200186.57
NVIDIA A100 80GB SXM4119.56

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition210.47
2NVIDIA B300199.12
3NVIDIA H200197.14
4NVIDIA H100 80GB HBM3190.32
5NVIDIA B200186.57
6NVIDIA A100 80GB SXM4119.56

Qwen3-Coder-Next

Weights: bartowski/Qwen_Qwen3-Coder-Next-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition205.69
NVIDIA B300196.34
NVIDIA H200195.21
NVIDIA H100 80GB HBM3188.21
NVIDIA B200186.39
NVIDIA A100 80GB SXM4118.62

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition205.69
2NVIDIA B300196.34
3NVIDIA H200195.21
4NVIDIA H100 80GB HBM3188.21
5NVIDIA B200186.39
6NVIDIA A100 80GB SXM4118.62

Qwen3-Next-80B-A3B-Thinking

Weights: bartowski/Qwen_Qwen3-Next-80B-A3B-Thinking-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition203.29
NVIDIA B300194.98
NVIDIA H200194.82
NVIDIA H100 80GB HBM3188.14
NVIDIA B200186.03
NVIDIA A100 80GB SXM4118.93

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition203.29
2NVIDIA B300194.98
3NVIDIA H200194.82
4NVIDIA H100 80GB HBM3188.14
5NVIDIA B200186.03
6NVIDIA A100 80GB SXM4118.93

Qwen2.5-72B

Weights: bartowski/Qwen2.5-72B-Instruct-GGUF

NVIDIA B30048.28
NVIDIA B20045.34
NVIDIA H20043.4
NVIDIA H100 80GB HBM340.78
NVIDIA RTX PRO 6000 Blackwell Workstation Edition29.65
NVIDIA A100 80GB SXM423.95

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA B30048.28
2NVIDIA B20045.34
3NVIDIA H20043.4
4NVIDIA H100 80GB HBM340.78
5NVIDIA RTX PRO 6000 Blackwell Workstation Edition29.65
6NVIDIA A100 80GB SXM423.95

Qwen3-Coder-Next

Weights: unsloth/Qwen3-Coder-Next-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition211.5
NVIDIA B300199.39
NVIDIA H200196.65
NVIDIA H100 80GB HBM3193.79
NVIDIA B200189.76
NVIDIA A100 80GB SXM4120.71

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition211.5
2NVIDIA B300199.39
3NVIDIA H200196.65
4NVIDIA H100 80GB HBM3193.79
5NVIDIA B200189.76
6NVIDIA A100 80GB SXM4120.71

Qwen3-Next-80B-A3B-Thinking

Weights: unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition214.04
NVIDIA H200201.12
NVIDIA B300200.55
NVIDIA H100 80GB HBM3191.08
NVIDIA B200188.78
NVIDIA A100 80GB SXM4120.98

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition214.04
2NVIDIA H200201.12
3NVIDIA B300200.55
4NVIDIA H100 80GB HBM3191.08
5NVIDIA B200188.78
6NVIDIA A100 80GB SXM4120.98

Qwen3-Next-80B-A3B

Weights: Qwen/Qwen3-Next-80B-A3B-Instruct-GGUF

NVIDIA RTX PRO 6000 Blackwell Workstation Edition210.17
NVIDIA H200192.52
NVIDIA B300191.32
NVIDIA H100 80GB HBM3186.22
NVIDIA B200179.71
NVIDIA A100 80GB SXM4117.44

Measured in tok/s. Longer is faster.

#GPUtok/s
1NVIDIA RTX PRO 6000 Blackwell Workstation Edition210.17
2NVIDIA H200192.52
3NVIDIA B300191.32
4NVIDIA H100 80GB HBM3186.22
5NVIDIA B200179.71
6NVIDIA A100 80GB SXM4117.44

Every number is a first-party run on our own bench. Cards missing from a board have not been measured on that model yet, or cannot fit it.