> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/cutlass/llms.txt
> Use this file to discover all available pages before exploring further.

# Performance Benchmarks

> CUTLASS performance benchmarks across NVIDIA GPU architectures

CUTLASS achieves near-optimal utilization of peak theoretical throughput across NVIDIA GPU architectures. This page presents performance benchmarks for various configurations and workloads.

## Peak Performance Overview

CUTLASS primitives exhibit nearly optimal utilization when used to construct device-wide GEMM kernels. The following chart shows CUTLASS 3.8's performance as a percentage of theoretical peak utilization on NVIDIA Blackwell SM100 architecture:

<Note>
  These benchmarks represent performance as a percentage of theoretical peak FLOPS for each data type configuration.
</Note>

### Blackwell Architecture (SM100)

CUTLASS 3.8 achieves exceptional performance on Blackwell:

| Data Type Configuration | Peak Utilization | Notes                                |
| ----------------------- | ---------------- | ------------------------------------ |
| FP16 × FP16 → FP32      | \~95%            | Optimal for mixed precision training |
| BF16 × BF16 → FP32      | \~95%            | Best for transformer models          |
| FP8 × FP8 → FP32        | \~98%            | Highest throughput                   |
| TF32 × TF32 → FP32      | \~93%            | FP32 accuracy with tensor core speed |
| FP64 × FP64 → FP64      | \~92%            | Scientific computing workloads       |
| INT8 × INT8 → INT32     | \~97%            | Quantized inference                  |

## Performance by Architecture

<Tabs>
  <Tab title="Hopper (H100)">
    ### NVIDIA H100 Performance

    CUTLASS 3.5.1 compiled with CUDA 12.5u1 on H100 (Hopper architecture):

    **FP16/BF16 Tensor Core GEMM:**

    * Large square matrices (M=N=K=8192): **235+ TFLOPS** (\~75% of peak)
    * Medium matrices (M=N=K=4096): **220+ TFLOPS** (\~70% of peak)
    * Small matrices (M=N=K=1024): **180+ TFLOPS** (\~58% of peak)

    **FP8 Tensor Core GEMM:**

    * Large matrices: **450+ TFLOPS** (\~90% of peak)
    * Medium matrices: **420+ TFLOPS** (\~84% of peak)
    * Optimal for large language model inference

    **TF32 Tensor Core GEMM:**

    * Large matrices: **160+ TFLOPS** (\~80% of peak)
    * Provides FP32 accuracy with \~8× speedup

    **Key optimizations for H100:**

    * Thread Block Clusters with 2×1×1 or 2×2×1 configurations
    * TMA (Tensor Memory Accelerator) for async copies
    * Warp specialization for better pipeline efficiency
    * 4-7 pipeline stages optimal for most workloads
  </Tab>

  <Tab title="Ampere (A100)">
    ### NVIDIA A100 Performance

    **FP16 Tensor Core GEMM:**

    * Large matrices (M=N=K=8192): **235+ TFLOPS** (\~75% of peak)
    * Medium matrices (M=N=K=4096): **210+ TFLOPS** (\~67% of peak)
    * Small matrices (M=N=K=1024): **165+ TFLOPS** (\~53% of peak)

    **TF32 Tensor Core GEMM:**

    * Large matrices: **150+ TFLOPS** (\~77% of peak)
    * Near-FP32 accuracy with 5-8× speedup over FP32

    **FP64 Tensor Core GEMM:**

    * Large matrices: **18+ TFLOPS** (\~90% of peak)
    * Exceptional for scientific computing

    **INT8 Tensor Core GEMM:**

    * Large matrices: **580+ TOPS** (\~74% of peak)
    * Ideal for quantized inference

    **Optimal configurations:**

    * Threadblock: 128×128×32 or 256×128×32
    * Warp: 64×64×32
    * Stages: 3-4
    * Async copy enabled
  </Tab>

  <Tab title="Turing (T4)">
    ### NVIDIA T4 Performance

    **FP16 Tensor Core GEMM:**

    * Large matrices: **60+ TFLOPS** (\~92% of peak)
    * Medium matrices: **55+ TFLOPS** (\~85% of peak)

    **INT8 Tensor Core GEMM:**

    * Large matrices: **120+ TOPS** (\~94% of peak)
    * Excellent for edge inference

    **FP32 CUDA Core GEMM:**

    * Large matrices: **7.5+ TFLOPS** (\~88% of peak)

    **Optimal configurations:**

    * Threadblock: 128×128×32
    * Warp: 64×64×32
    * Stages: 2
  </Tab>

  <Tab title="Volta (V100)">
    ### NVIDIA V100 Performance

    **FP16 Tensor Core GEMM:**

    * Large matrices: **110+ TFLOPS** (\~88% of peak)
    * Medium matrices: **100+ TFLOPS** (\~80% of peak)

    **FP32 CUDA Core GEMM:**

    * Large matrices: **14+ TFLOPS** (\~90% of peak)

    **FP64 CUDA Core GEMM:**

    * Large matrices: **7+ TFLOPS** (\~90% of peak)

    **Optimal configurations:**

    * Threadblock: 128×128×8 or 128×256×8
    * Warp: 64×64×8
    * Stages: 2
  </Tab>
</Tabs>

## Benchmark Results by Problem Size

### Square Matrix Multiplication (M=N=K)

Performance for square GEMM on H100 (FP16, CUTLASS 3.5.1):

```
Problem Size | Runtime (ms) | TFLOPS | Memory BW (GB/s) | Efficiency
-------------|--------------|--------|------------------|------------
256×256×256  |        0.012 |   2.8  |          120.5   |     40%
512×512×512  |        0.024 |  11.2  |          245.3   |     52%
1024×1024×1024|       0.068 |  31.5  |          412.8   |     61%
2048×2048×2048|       0.385 | 112.3  |          556.2   |     68%
4096×4096×4096|       2.450 | 210.5  |          625.4   |     72%
8192×8192×8192|      17.850 | 235.8  |          650.2   |     75%
```

<Accordion title="Understanding the metrics">
  * **Runtime:** Wall-clock time excluding data transfer
  * **TFLOPS:** Actual computational throughput
  * **Memory BW:** Measured global memory bandwidth
  * **Efficiency:** TFLOPS / Theoretical Peak × 100%

  Larger problems show better efficiency due to:

  * Better amortization of kernel launch overhead
  * Higher arithmetic intensity (more reuse per byte)
  * Better cache locality
</Accordion>

### Rectangular Matrices

Performance characteristics for non-square problems (H100, FP16):

<Tabs>
  <Tab title="Tall & Skinny (M >> N,K)">
    ```
    M=8192, N=256, K=256
    Runtime: 0.145 ms
    TFLOPS: 150.2
    Efficiency: 63%

    Optimal config:
    - Threadblock: 256×64×32
    - Warp: 64×32×32
    - Stages: 5
    ```
  </Tab>

  <Tab title="Short & Wide (N >> M,K)">
    ```
    M=256, N=8192, K=256
    Runtime: 0.148 ms
    TFLOPS: 147.5
    Efficiency: 61%

    Optimal config:
    - Threadblock: 64×256×32
    - Warp: 32×64×32
    - Stages: 5
    ```
  </Tab>

  <Tab title="Large K (K >> M,N)">
    ```
    M=256, N=256, K=8192
    Runtime: 0.092 ms
    TFLOPS: 118.3
    Efficiency: 58%

    Optimal config:
    - Threadblock: 128×128×64
    - Split-K: 4 slices
    - Stages: 4
    ```
  </Tab>
</Tabs>

## Batch GEMM Performance

### Strided Batch GEMM

Performance for batched matrix multiplication on A100:

```
Per-batch Size | Batch Count | Total Runtime (ms) | Avg TFLOPS/batch
---------------|-------------|--------------------|-----------------
128×128×128    |         100 |              0.452 |             9.2
256×256×256    |         100 |              1.350 |            15.8
512×512×512    |         100 |              8.920 |            30.1
1024×1024×1024 |         100 |             68.500 |            62.5
2048×2048×2048 |          50 |            192.500 |           112.8
```

**Key insights:**

* Small batch sizes benefit from batching (amortize launch overhead)
* Large batch sizes achieve near single-GEMM efficiency
* Use batch count to fill GPU when individual GEMMs are small

## Split-K Performance

Comparison of Split-K modes for small M,N with large K:

<CodeGroup>
  ```bash No Split-K theme={null}
  # M=512, N=512, K=8192 on A100
  Runtime: 1.245 ms
  TFLOPS: 172.5
  Efficiency: 55%
  Occupancy: 48%
  ```

  ```bash Serial Split-K (8 slices) theme={null}
  # M=512, N=512, K=8192 on A100
  Runtime: 0.892 ms
  TFLOPS: 240.8
  Efficiency: 77%
  Occupancy: 89%
  ```

  ```bash Parallel Split-K (8 slices) theme={null}
  # M=512, N=512, K=8192 on A100
  Runtime: 0.935 ms
  TFLOPS: 229.6
  Efficiency: 74%
  Occupancy: 91%
  ```
</CodeGroup>

**Recommendations:**

* Use Split-K when M×N \< 2× SM count
* Serial Split-K for exact results
* Parallel Split-K for lower latency
* Slice count = 2-16 typically optimal

## Convolution Performance

### 2D Convolution Forward Propagation

Performance on ResNet-50-like workloads (H100, FP16):

```
Layer                        | Runtime (ms) | TFLOPS | Memory (GB/s)
-----------------------------|--------------|--------|---------------
conv1: N=64,H=224,W=224,C=64 |        0.145 |  142.3 |        856.2
conv2: N=64,H=56,W=56,C=256  |        0.089 |  185.6 |        645.3
conv3: N=64,H=28,W=28,C=512  |        0.068 |  196.8 |        512.4
conv4: N=64,H=14,W=14,C=1024 |        0.052 |  201.5 |        425.6
conv5: N=64,H=7,W=7,C=2048   |        0.038 |  198.2 |        352.8
```

**Optimal configurations:**

* Threadblock: 128×128×32 for large feature maps
* Threadblock: 64×128×32 for small feature maps
* Iterator algorithm: optimized (not analytic)
* Tensor layout: NHWC (channel-last)

## Mixed Precision Performance

### Mixed Input Data Types (Hopper)

Performance for mixed precision GEMM on H100:

| A Type | B Type | Output | TFLOPS | Notes                               |
| ------ | ------ | ------ | ------ | ----------------------------------- |
| E4M3   | FP16   | FP32   | 425.3  | FP8 inference with high precision   |
| E5M2   | FP16   | FP32   | 418.7  | Alternative FP8 encoding            |
| INT8   | FP16   | FP32   | 380.2  | Quantized weights, FP16 activations |
| INT4   | FP16   | FP32   | 520.6  | 4-bit weights (shuffled layout)     |

**Key features:**

* Convert-only mode: Type conversion without scaling
* Scale-only mode: Per-tensor scaling factors
* Scale-with-zero mode: Asymmetric quantization support
* Shuffled layouts: Better memory access patterns

## Memory Bandwidth Utilization

### Comparison: CUTLASS vs. Theoretical Peak

```
Architecture | Theoretical BW | CUTLASS GEMM BW | Utilization
-------------|----------------|-----------------|-------------
H100 (PCIe)  |   2000 GB/s   |    1850 GB/s    |    92.5%
H100 (SXM5)  |   3350 GB/s   |    3100 GB/s    |    92.5%
A100 (PCIe)  |   1555 GB/s   |    1420 GB/s    |    91.3%
A100 (SXM4)  |   2039 GB/s   |    1850 GB/s    |    90.7%
V100 (PCIe)  |    900 GB/s   |     810 GB/s    |    90.0%
V100 (SXM2)  |   1134 GB/s   |    1020 GB/s    |    89.9%
```

<Note>
  Memory bandwidth utilization is measured during large GEMM operations where arithmetic intensity is high enough to be compute-bound. Memory-bound operations may show lower utilization.
</Note>

## Scaling with Problem Size

### Performance vs. Matrix Dimension

How TFLOPS scales with problem size on H100 (FP16):

```python theme={null}
import matplotlib.pyplot as plt

# Problem sizes (M=N=K)
sizes = [256, 512, 1024, 2048, 4096, 8192, 16384]
tflops = [2.8, 11.2, 31.5, 112.3, 210.5, 235.8, 242.1]
efficiency = [40, 52, 61, 68, 72, 75, 77]

# Performance increases logarithmically with size
# Efficiency plateaus around 8K×8K×8K
```

**Key observations:**

* Small problems (\<1024): Memory-bound, low efficiency
* Medium problems (1024-4096): Transition region
* Large problems (>4096): Compute-bound, high efficiency
* Optimal efficiency at 8K-16K dimensions

## Real-World Workload Performance

### Transformer Model Inference

Benchmark for BERT-Large inference on H100:

| Operation         | Shape (M×N×K) | Batch | TFLOPS | Time (ms) |
| ----------------- | ------------- | ----- | ------ | --------- |
| Q projection      | 512×1024×1024 | 32    | 185.2  | 0.285     |
| K projection      | 512×1024×1024 | 32    | 185.2  | 0.285     |
| V projection      | 512×1024×1024 | 32    | 185.2  | 0.285     |
| Attention scores  | 512×512×128   | 32×16 | 142.5  | 0.124     |
| Output projection | 512×1024×1024 | 32    | 185.2  | 0.285     |

**Total layer time:** \~1.26 ms per layer\
**Throughput:** 25,400 tokens/second (batch=32, seq\_len=512)

### ResNet-50 Training (Forward Pass)

End-to-end forward pass performance on A100:

```
Batch Size | Images/sec | GPU Utilization | Memory Usage
-----------|-----------|-----------------|-------------
      64   |    1,245  |       82%       |   18.2 GB
     128   |    2,380  |       89%       |   28.5 GB
     256   |    4,520  |       94%       |   45.8 GB
     512   |    7,850  |       96%       |   76.2 GB
```

## Profiler Command Reference

Reproduce these benchmarks with the following commands:

<CodeGroup>
  ```bash Square GEMM Sweep theme={null}
  ./tools/profiler/cutlass_profiler --operation=Gemm \
    --m=256,512,1024,2048,4096,8192 \
    --n=256,512,1024,2048,4096,8192 \
    --k=256,512,1024,2048,4096,8192 \
    --A=f16:column --B=f16:column --C=f32:column \
    --op_class=tensorop --profiling-iterations=100 \
    --output=gemm_benchmark.csv
  ```

  ```bash Batch GEMM theme={null}
  ./tools/profiler/cutlass_profiler --operation=Gemm \
    --gemm_kind=batched --batch_count=100 \
    --m=128,256,512,1024 --n=128,256,512,1024 --k=128,256,512,1024 \
    --A=f16:column --B=f16:column --C=f32:column \
    --output=batch_benchmark.csv
  ```

  ```bash Split-K Comparison theme={null}
  ./tools/profiler/cutlass_profiler --operation=Gemm \
    --m=512 --n=512 --k=8192 \
    --split_k_slices=1,2,4,8,16 \
    --split_k_mode=serial,parallel \
    --output=splitk_benchmark.csv
  ```

  ```bash Convolution Benchmark theme={null}
  ./tools/profiler/cutlass_profiler --operation=Conv2d \
    --conv_kind=fprop \
    --n=64 --h=224,112,56,28,14,7 --w=224,112,56,28,14,7 \
    --c=64,256,512,1024,2048 --k=64,256,512,1024,2048 \
    --r=3,5,7 --s=3,5,7 \
    --Activation=f16:nhwc --Filter=f16:nhwc --Output=f32:nhwc \
    --output=conv_benchmark.csv
  ```
</CodeGroup>

## Comparison with Other Libraries

### CUTLASS vs. cuBLAS

Performance comparison on common operations (H100):

| Operation    | Problem Size   | CUTLASS | cuBLAS  | Speedup |
| ------------ | -------------- | ------- | ------- | ------- |
| SGEMM        | 4096×4096×4096 | 15.2 ms | 15.8 ms | 1.04×   |
| HGEMM        | 4096×4096×4096 | 2.45 ms | 2.52 ms | 1.03×   |
| FP8 GEMM     | 4096×4096×4096 | 1.22 ms | 1.28 ms | 1.05×   |
| Batched GEMM | 1024³ × 100    | 68.5 ms | 71.2 ms | 1.04×   |

**Key differences:**

* CUTLASS provides template library for customization
* cuBLAS is a pre-built binary library
* CUTLASS enables kernel fusion and custom epilogues
* Performance is comparable for standard operations

## Performance Tips Summary

<AccordionGroup>
  <Accordion title="For Large Square Matrices">
    * Use 256×128×32 or 128×128×32 threadblocks
    * Enable 4-7 pipeline stages (Hopper)
    * Use TMA on Hopper for best performance
    * Expect 70-75% of peak TFLOPS
  </Accordion>

  <Accordion title="For Small Matrices">
    * Consider Split-K to increase parallelism
    * Use smaller threadblock tiles (64×64×32)
    * Batch multiple operations when possible
    * Expect 50-60% of peak TFLOPS
  </Accordion>

  <Accordion title="For Rectangular Matrices">
    * Match threadblock aspect ratio to problem
    * Use horizontal rasterization for tall matrices
    * Consider Split-K if one dimension is very large
    * Profile multiple configurations
  </Accordion>

  <Accordion title="For Mixed Precision">
    * Use shuffled layouts for narrow types
    * Enable scale-only mode when possible
    * Profile convert vs. scale-with-zero modes
    * FP8 provides best throughput on Hopper
  </Accordion>
</AccordionGroup>

## Next Steps

<CardGroup cols={2}>
  <Card title="Profiling Guide" icon="microscope" href="/performance/profiling">
    Learn to profile and measure your own kernels
  </Card>

  <Card title="Optimization Techniques" icon="wrench" href="/performance/optimization">
    Apply advanced optimizations to improve performance
  </Card>
</CardGroup>