Fused Operations

Combine GEMM with element-wise operations like bias addition and ReLU activation

Fused Operations Example

This example demonstrates how to fuse element-wise operations (bias and ReLU) with GEMM computation in a single kernel, improving performance by eliminating intermediate memory operations.

Overview

Instead of computing GEMM and then applying bias and ReLU in separate kernel launches:

// Inefficient: Multiple kernel launches
D = alpha * (A x B)           // GEMM kernel
D = D + bias                  // Bias kernel
D = max(0, D)                 // ReLU kernel

CUTLASS allows you to fuse these operations into a single kernel:

// Efficient: Single fused kernel
D = max(0, alpha * (A x B) + bias)

Key Concepts

• Epilogue: The final stage of a GEMM kernel that processes the accumulator
• Fused operations: Combining multiple operations to reduce memory bandwidth
• Custom epilogue: Using CUTLASS epilogue templates for different output transformations
• Bias broadcasting: Applying per-row or per-column bias efficiently

Implementation

Building and Running

Build the example

cd /path/to/cutlass
mkdir build && cd build
cmake .. -DCUTLASS_NVCC_ARCHS='75;80;86'
make 12_gemm_bias_relu

Run the example

./examples/12_gemm_bias_relu/12_gemm_bias_relu

Expected output:

Passed

Source Code Location

The complete source code for this example is available at:

• examples/12_gemm_bias_relu/gemm_bias_relu.cu

What This Example Demonstrates

1. Epilogue customization: Using custom epilogue operations for fused kernels
2. Bias broadcasting: Efficiently applying per-row bias using stride-0 tensors
3. Activation functions: Fusing ReLU activation with GEMM
4. Performance optimization: Eliminating intermediate memory operations
5. Mixed precision: Using FP16 inputs with FP32 accumulation and output

Performance Benefits

Fusing operations provides significant benefits:

• Reduced memory bandwidth: No need to write/read intermediate results
• Lower latency: Single kernel launch instead of multiple
• Better cache utilization: Data stays in cache between operations
• Higher throughput: Overlapped computation and memory operations

For typical workloads, fused GEMM+Bias+ReLU can be 2-3x faster than separate kernels.

Bias Layout Requirements

Important notes about bias layout:

• For column-major output: bias must be M×1 (per-row bias)
• For row-major output: bias would be 1×N (per-column bias)
• Use stride 0 in the non-broadcast dimension to efficiently broadcast the bias vector

Key Takeaways

• CUTLASS epilogues enable efficient fusion of element-wise operations
• LinearCombinationRelu combines scaling, bias addition, and ReLU activation
• Bias broadcasting is achieved using stride-0 tensor references
• Fused operations significantly reduce memory traffic and improve performance
• The same pattern works for other epilogue operations (GELU, sigmoid, etc.)

Next Steps

• Explore Basic GEMM for simple matrix multiplication
• Learn about Batched GEMM for multiple independent operations
• Check out other epilogue operations in cutlass/epilogue/thread/