The vocabulary of the Compute layer.

The abstract description of a GPU that programmers write against: a hierarchy of threads grouped into blocks and grids, a hierarchy of memory with different speeds and scopes, and kernels that run across that hierarchy. The model hides the physical layout of SMs and cores so that the same code runs on different GPUs. It is the layer that gives NVIDIA its software moat: two decades of libraries and skills are written against it.

See also Thread hierarchy, Memory hierarchy, Kernel

A version number, such as 9.0 for Hopper or 10.0 for Blackwell, that identifies which features of the programming model a GPU supports. Compiled code targets a compute capability, and code compiled for a newer one will not run on older hardware. In practice it is the field to check when asking whether a given software stack can run on a given fleet.

See also Streaming Multiprocessor architecture, nvcc

The way the programming model groups work: individual threads are grouped into thread blocks, blocks are grouped into a grid, and on recent architectures blocks can also be grouped into clusters. Each level maps to a level of the hardware, with threads running on cores, blocks on SMs and the grid across the whole GPU. Getting this mapping right is most of what GPU programming consists of.

See also Thread, Thread Block, Thread Block Grid

The smallest unit of work in the programming model: one sequence of instructions operating on one piece of data. A GPU kernel is written as if for a single thread and then launched across thousands or millions of them. Threads are cheap to create and are not scheduled individually; the hardware handles them in warps.

See also Warp, Kernel

A group of 32 threads that the hardware executes together, issuing the same instruction to all of them at once. The warp is the real unit of scheduling on an NVIDIA GPU, even though programmers write in terms of threads. When threads within a warp need to take different paths through the code, the warp must run each path in turn, which is the source of warp divergence.

See also Warp Scheduler, Warp divergence, Warpgroup

A set of four consecutive warps, 128 threads, introduced with the Hopper architecture as the unit that issues instructions to the newer Tensor Cores. Operations at the warpgroup level allow much larger matrix tiles per instruction than warp-level ones. It is an example of the programming model growing a new level to match the hardware.

See also Warp, Tensor Core

A group of threads, up to 1,024, that run on the same SM and can cooperate through shared memory and synchronisation barriers. The thread block is the unit in which work is handed to an SM, and a block never moves once placed. Choosing block size well decides how many blocks fit on an SM at once and therefore how well latency is hidden.

See also Cooperative Thread Array, Shared Memory, Occupancy

The hardware-level name for a thread block, used in the lower-level PTX and SASS documentation. The two terms describe the same thing from different sides: programmers say thread block, the compiler and hardware say CTA. On recent architectures, CTAs can be grouped into clusters that span several SMs.

See also Thread Block, Parallel Thread Execution

The full set of thread blocks launched for one kernel, arranged in one, two or three dimensions to match the shape of the data. The grid can be far larger than the GPU can run at once; the hardware works through it block by block as SMs become free. Grid size is set per launch, which is how one kernel scales across GPUs of different sizes.

See also Thread Block, Kernel

A function written to run on the GPU, launched from the host across a grid of threads. Everything a GPU computes, from a matrix multiply to an attention layer, is one or more kernels. Kernel quality determines how much of a GPU's theoretical throughput is actually delivered, which is why a facility's realised output depends on software as well as hardware.

See also Thread Block Grid, CUDA C++, Peak rate

The ordered set of memory types a thread can use, from fastest and smallest to slowest and largest: registers, shared memory and L1 cache, L2 cache, and global memory in GPU RAM. Each step down is roughly an order of magnitude slower. Most performance work on a GPU is about keeping data as high in this hierarchy as possible for as long as possible.

See also Registers, Shared Memory, Global Memory

Per-thread storage, the fastest memory on the device, allocated from the SM's register file. Each thread gets its own registers and they are private to it. The compiler decides how many a kernel needs; when that number is high, fewer threads fit on the SM, and when it exceeds the limit, values spill to much slower memory.

See also Register File, Register pressure

A block of fast on-chip memory, inside the SM, that all threads in a thread block can read and write. It is explicitly managed by the programmer and is the main tool for reusing data across threads without returning to global memory. Well-written matrix kernels stage tiles of their inputs in shared memory, and this staging is where much of their speed comes from.

See also L1 Data Cache, Bank conflict, Tensor Memory Accelerator

The memory space backed by GPU RAM, visible to every thread on the device and to the host. It is the largest memory on the GPU and by far the slowest to access from a thread, with latency measured in hundreds of cycles. Model weights, activations and datasets live here, so global memory bandwidth is the ceiling for any memory-bound workload.

See also GPU RAM, Memory bandwidth, Memory coalescing

NVIDIA's intermediate instruction set: a stable, virtual assembly language that compilers target and that the driver translates into the actual machine code for a specific GPU. PTX lets code compiled years ago run on hardware released later. It is also where much of NVIDIA's compatibility guarantee lives, and the layer that competing chip makers find hardest to replicate.

See also Streaming Assembler, nvcc, NVRTC

The real machine code that an NVIDIA GPU executes, specific to each architecture. PTX is translated into SASS either at compile time or by the driver at load time. SASS is not publicly documented in full, which keeps the lowest layer of the stack proprietary; performance engineers read it through NVIDIA's disassembly tools when PTX does not explain what the hardware is doing.

See also Parallel Thread Execution, CUDA Binary Utilities

A newer programming model, introduced alongside Blackwell, in which the programmer describes operations on tiles of data rather than on individual threads, and the compiler maps those tiles onto warps, Tensor Cores and memory. It raises the level of abstraction in the same way the thread model once did for graphics hardware. Its purpose is to make high Tensor Core utilisation reachable without hand-tuned kernels.

See also cuTile, Tensor Core, CUDA programming model

The resource that limits how fast a workload can run: arithmetic throughput, memory bandwidth, latency, host overhead or something else. Every workload has one, and speeding up anything other than the bottleneck changes nothing. Finding it is most of performance engineering, and it is the first question to ask when a facility's measured output falls short of its specification.

See also Roofline model, Compute-bound, Memory-bound

The theoretical maximum throughput of a GPU for a given operation and precision, quoted on specification sheets in floating-point operations per second or bytes per second. No real workload reaches it; well-tuned matrix multiplications reach a large fraction and most other work reaches far less. Revenue models built on peak rates rather than achieved rates are a recurring source of disappointment.

See also Arithmetic bandwidth, Memory bandwidth, Pipe utilisation

A simple chart that shows the best performance a workload can achieve on a given GPU as a function of its arithmetic intensity. Below a certain intensity the ceiling is set by memory bandwidth and rises with intensity; above it the ceiling is flat at the arithmetic peak. Placing a workload on the roofline shows immediately whether it is memory-bound or compute-bound and how far it is from the limit.

See also Arithmetic intensity, Arithmetic bandwidth, Memory bandwidth

The number of arithmetic operations a workload performs per byte of data it moves from memory. High intensity means the GPU can do a lot of work on each value it fetches and is likely to be compute-bound; low intensity means it is fetching more than it computes and is likely to be memory-bound. Large-batch training has high intensity; token-by-token inference has low intensity.

See also Roofline model, Compute-bound, Memory-bound

The rate at which a GPU can perform arithmetic, usually expressed in floating-point operations per second for a given precision. It is the flat part of the roofline. For AI workloads the relevant figure is the Tensor Core rate at the precision actually used.

See also Peak rate, Tensor Core, Compute-bound

The rate at which data can be moved between GPU RAM and the processor, measured in bytes per second. It is the sloped part of the roofline and the binding constraint for memory-bound work such as inference decoding, where each generated token requires reading the full set of model weights. Memory bandwidth, more than arithmetic peak, often determines how many tokens per second a GPU can serve.

See also GPU RAM, Memory-bound, Little's law

A workload whose speed is limited by how fast the GPU can perform arithmetic rather than by how fast it can fetch data. Compute-bound work sits under the flat part of the roofline and benefits from faster or more numerous Tensor Cores and from lower-precision formats. Training large models is the main compute-bound workload in AI.

See also Memory-bound, Arithmetic intensity

A workload whose speed is limited by memory bandwidth, because it performs too few operations per byte fetched to keep the arithmetic units busy. Most inference serving at small batch sizes is memory-bound. For such work, a GPU with more memory bandwidth outperforms one with a higher arithmetic peak, which matters when choosing hardware for an inference-heavy facility.

See also Compute-bound, Memory bandwidth

Time spent on work that is neither computation nor useful data movement: launching kernels, synchronising, waiting on the host, or moving data across the slower link between host and device. Overhead is why a workload made of many small kernels can leave the GPU mostly idle even when each kernel is efficient. CUDA Graphs and larger batch sizes are the usual remedies.

See also CUDA Graph, Nsight Systems

The technique of keeping a GPU busy by switching between many resident warps so that while one waits for memory another computes. GPUs are designed around this rather than around the large caches and prediction logic that CPUs use. It depends on having enough independent work resident at once, which is what occupancy measures.

See also Warp Scheduler, Occupancy, Little's law

A result from queueing theory: the average number of items in a system equals their arrival rate multiplied by the time each spends there. Applied to a GPU, it says how many memory requests must be in flight to sustain a given bandwidth at a given latency. It explains why high memory bandwidth requires high occupancy, and why a GPU with few resident warps cannot use the bandwidth it paid for.

See also Latency hiding, Memory bandwidth, Occupancy

The ratio of warps actually resident on an SM to the maximum the SM could hold. It is limited by how many registers and how much shared memory each thread block uses. Higher occupancy gives the warp schedulers more to choose from and so hides latency better, though beyond a point more occupancy does not help and the fastest kernels often run below the maximum.

See also Register pressure, Latency hiding, Thread Block

The demand a kernel places on the register file. When each thread needs many registers, fewer threads fit on the SM, which lowers occupancy; when the demand exceeds what is available, the compiler spills values to local memory, which is far slower. Managing register pressure is a central trade-off in kernel tuning.

See also Register File, Registers, Occupancy

The hardware's ability to combine the memory requests of the 32 threads in a warp into as few transactions as possible when those threads access adjacent addresses. Coalesced access uses the full width of the memory bus; scattered access wastes most of it. Data layout decisions that preserve coalescing can change effective bandwidth by an order of magnitude.

See also Global Memory, Warp, Memory bandwidth

A slowdown that occurs when several threads in a warp try to access shared memory addresses that map to the same memory bank in the same cycle, forcing the accesses to be serialised. Shared memory is divided into 32 banks precisely to allow 32 simultaneous accesses, so conflicts undo its main advantage. Tile layouts are often padded specifically to avoid them.

See also Shared Memory, Warp

What happens when threads within one warp take different branches of a conditional. Because a warp issues one instruction at a time, the hardware runs each branch in turn with the non-participating threads idle, so a fully divergent warp can run at a fraction of its potential. Code is structured so that threads in the same warp follow the same path wherever possible.

See also Warp, Branch efficiency

The proportion of branch instructions in which all threads of a warp took the same path. It is the profiler's measure of warp divergence: an efficiency of 100 percent means no divergence, and anything much below it points to control flow that should be restructured.

See also Warp divergence

The condition of a resident warp at a given cycle: selected and issuing an instruction, eligible but not selected, or stalled waiting on something. Profilers break stalls down by cause, such as waiting on memory, on a barrier or on a dependent instruction. The distribution of these states across a kernel shows directly why the SM is not issuing every cycle.

See also Scoreboard stall, Issue efficiency, Warp Scheduler

A stall in which a warp cannot issue its next instruction because that instruction depends on a result, usually a memory load, that has not yet arrived. The scoreboard is the hardware structure that tracks these dependencies. Long scoreboard stalls are the signature of a memory-bound kernel with too little latency hiding.

See also Warp execution state, Memory-bound, Latency hiding

A clock cycle in which an SM has at least one warp resident and so is doing or waiting on work, as opposed to a cycle in which it is empty. Counting active cycles against elapsed cycles separates time when the SM had nothing to do from time when it had work but was stalled. The two problems have different fixes.

See also Issue efficiency, Streaming Multiprocessor utilisation

The fraction of active cycles in which a warp scheduler actually issued an instruction. It measures how well latency is being hidden: low issue efficiency means the scheduler frequently found every resident warp stalled. Along with pipe utilisation it is the most direct hardware measure of whether a GPU's time is being used.

See also Active cycle, Pipe utilisation, Warp execution state

How busy each class of execution unit within the SM is: the Tensor Cores, the CUDA Cores, the load/store units and the special function units, each reported separately. For AI workloads the figure that matters is Tensor Core pipe utilisation, which is routinely far lower than the coarse GPU utilisation number that monitoring tools display. This is the metric closest to what a customer is actually paying for.

See also Tensor Core, Streaming Multiprocessor utilisation, nvidia-smi

The share of time, across the GPU, that SMs are active. It is a more meaningful figure than the single utilisation percentage from nvidia-smi, which only reports whether any kernel was running on the device at all, but it still does not say how hard active SMs were working. A GPU can show high SM utilisation while its Tensor Cores sit mostly idle. Pipe utilisation, one level down, shows whether the work is productive.

See also Pipe utilisation, nvidia-smi, Active cycle