The Skill Explained
What Are CUDA and GPU Programming?
CUDA is NVIDIA's platform for programming graphics processors (GPUs) to do general calculations. GPU programming means writing code, called kernels, that runs on thousands of GPU cores at once. Triton is a language that makes writing such kernels easier from Python.
Why AI Employers Ask for Them
Training and running AI models is mostly huge amounts of matrix arithmetic, and GPUs do it fastest. Engineers who can write or tune GPU code make models train faster and serve cheaper, which saves AI companies large sums. It is a specialist skill asked for in infrastructure, performance and research engineering roles.
How It Works
A GPU has many simple cores grouped into blocks. A CUDA kernel is a function that thousands of threads run in parallel, each on its own piece of the data. The programmer decides how work is split into threads and blocks and how data moves between the GPU's memories.
Speed depends mostly on memory, not arithmetic: reading data from the GPU's main memory is slow compared to computing on it, so good kernels reuse data in fast on-chip memory and combine several operations into one (kernel fusion). Libraries such as cuBLAS and cuDNN provide tuned kernels for common operations.
Most AI engineers use GPUs through PyTorch without writing kernels. Specialists profile models, find the slow operations and write custom kernels, increasingly in Triton, for attention, quantized matrix multiplication or other hot spots.
What You Do With It at Work
- Profile a model to find where GPU time is spent.
- Write or tune a custom kernel for a slow operation.
- Fuse several operations to reduce memory traffic.
- Make quantized or sparse operations run fast.
- Benchmark kernels on different GPU types.
Tools and Terms You Will Meet
- Kernel
- A function that runs in parallel on many GPU threads.
- Thread block
- A group of threads that share fast memory and cooperate.
- Memory bandwidth
- How fast data can be moved; often the real limit on speed.
- Kernel fusion
- Combining operations into one kernel to avoid extra memory trips.
- Triton
- A Python-like language for writing GPU kernels with less effort.
Industry Example: Pharma
A computational chemistry team speeds up molecular simulation code by moving its heaviest loops to custom GPU kernels, turning an overnight run into one that finishes in under an hour.
A Common Mix-Up
Using a GPU from PyTorch is not CUDA programming. Ads that ask for CUDA usually mean writing or optimising the low-level GPU code itself.
How Deep Do You Need to Go?
- Know itExplain why GPUs speed up AI and what a kernel is.
- Use itProfile models, write simple kernels and use Triton for custom operations.
- Master itWrite highly optimised kernels and tune them for specific GPU architectures.