Convolution

Implement high-performance 2D convolution using CUTLASS Tensor Cores

Convolution Example

This example demonstrates how to run 2D convolution kernels using CUTLASS with Tensor Core acceleration on NVIDIA Turing GPUs.

Overview

CUTLASS implements convolution as an implicit GEMM operation, transforming the convolution into a matrix multiplication that can leverage highly optimized GEMM kernels and Tensor Cores.

The convolution operation computes:

Output[n,p,q,k] = alpha * Conv2D(Input[n,h,w,c], Filter[k,r,s,c]) + beta * Output[n,p,q,k]

Where:

• Input: NHWC layout (batch, height, width, channels)
• Filter: KRSC layout (output channels, filter height, filter width, input channels)
• Output: NPQK layout (batch, output height, output width, output channels)

Key Concepts

• Implicit GEMM: Convolution implemented as matrix multiplication
• Tensor Cores: Using specialized hardware for int4 operations
• Tile sizes: Hierarchical tiling at threadblock, warp, and instruction levels
• Pipeline stages: Overlapping data movement with computation
• NHWC layout: Optimized tensor layout for modern GPUs

Implementation

Building and Running

Build the example

cd /path/to/cutlass
mkdir build && cd build
cmake .. -DCUTLASS_NVCC_ARCHS='75;80;86'
make 09_turing_tensorop_conv2dfprop

Run the example

# Run with default parameters
./examples/09_turing_tensorop_conv2dfprop/09_turing_tensorop_conv2dfprop

# Run with custom dimensions
./examples/09_turing_tensorop_conv2dfprop/09_turing_tensorop_conv2dfprop \
  --n=1 --h=224 --w=224 --c=32 --k=32 --r=3 --s=3

# Run with reference check
./examples/09_turing_tensorop_conv2dfprop/09_turing_tensorop_conv2dfprop \
  --n=32 --h=224 --w=224 --c=128 --k=256 --r=1 --s=1 --ref-check

# Benchmark mode
./examples/09_turing_tensorop_conv2dfprop/09_turing_tensorop_conv2dfprop --benchmark

Command-line options

• --n=<int>: Batch size (N)
• --h=<int>: Input height (H)
• --w=<int>: Input width (W)
• --c=<int>: Input channels (C)
• --k=<int>: Output channels (K)
• --r=<int>: Filter height (R)
• --s=<int>: Filter width (S)
• --alpha=<float>: Scaling factor alpha
• --beta=<float>: Scaling factor beta
• --ref-check: Enable reference verification
• --perf-check: Enable performance measurement
• --benchmark: Run performance benchmarks on multiple layer configurations
• --iterations=<int>: Number of profiling iterations

Source Code Location

The complete source code for this example is available at:

• examples/09_turing_tensorop_conv2dfprop/turing_tensorop_conv2dfprop.cu

What This Example Demonstrates

1. Implicit GEMM convolution: How CUTLASS transforms convolution into matrix multiplication
2. Tensor Core usage: Leveraging int4 Tensor Core operations for high performance
3. Hierarchical tiling: Composing threadblock, warp, and instruction-level tiles
4. Pipeline optimization: Using multiple stages to hide memory latency
5. NHWC layout: Working with channel-last tensor format
6. Alignment requirements: Handling int4 data type alignment constraints

Performance Optimization

Key factors for optimal performance:

Tile Sizes

• Threadblock: 128×128×128 balances occupancy and reuse
• Warp: 64×64×128 fits well within register file limits
• Instruction: 8×8×32 matches Tensor Core operation size

Pipeline Stages

• NumStages=2 overlaps global memory loads with computation
• Higher values may improve performance but increase shared memory usage

Alignment

• int4 operations require 32-element (128-bit) alignment
• Input channels (C) and output channels (K) must be divisible by 32

Benchmark Results

The --benchmark mode tests various convolution layer configurations typical in ResNet:

./examples/09_turing_tensorop_conv2dfprop/09_turing_tensorop_conv2dfprop --benchmark

Example layers tested:

• 56×56×64 → 56×56×256 (1×1 convolution)
• 56×56×64 → 56×56×64 (3×3 convolution)
• 28×28×128 → 28×28×512 (1×1 convolution)
• And many more…

Key Takeaways

• CUTLASS implements convolution as implicit GEMM for maximum performance
• Tensor Cores provide massive acceleration for low-precision convolution
• Hierarchical tiling (threadblock → warp → instruction) enables efficient data reuse
• Pipeline stages overlap data movement with computation to hide latency
• Proper alignment is critical for int4 Tensor Core operations
• NHWC layout is optimal for modern GPU architectures

Next Steps

• Learn about Basic GEMM to understand the underlying GEMM operation
• Explore Fused Operations to combine convolution with activations
• Check out examples/16_ampere_tensorop_conv2dfprop/ for Ampere-specific optimizations
• See examples/42_ampere_tensorop_group_conv/ for grouped convolutions