Overview with affiliate links. We receive a commission if you click through via us. Read how
Compare

GPU Specs Explained: What Do All Those Numbers Mean?

An explanation of the key GPU specifications and how to compare them safely.

Blog6 min read

Short answer

The key specifications of a GPU are its memory (VRAM), memory bandwidth, number of compute units, theoretical compute power (TFLOPS or TOPS) and thermal design power (TDP). No single number tells the whole story: what matters depends on your application. So compare the same properties, and pay attention to the precision at which a figure is quoted.

The key specifications

The key GPU specifications at a glance
SpecificationWhat it tells youKeep in mind
Memory (VRAM)How much data the GPU can hold at onceMainly determines the maximum model or scene size
Memory bandwidthHow quickly the GPU can reach its dataMatters a lot for AI and for very large datasets
Compute unitsNumber of cores or similar unitsNames and layouts differ per manufacturer; do not compare them one to one
TFLOPS or TOPSTheoretical peak compute at a given precisionWithout the precision (FP32, FP16, FP8) the number cannot be interpreted
TDPDesign power for power supply and coolingNot measured consumption
InterfaceConnection to the rest of the system, such as PCIeA faster interface mainly helps with heavy data exchange

Memory and bandwidth

Memory determines what fits. If a model does not fit in memory, a faster card will not help much. Bandwidth determines how quickly data moves back and forth. A lot of AI work is not waiting for compute power but for data from memory; in that case bandwidth is the limit. See also what VRAM is.

Compute power: TFLOPS and TOPS

TFLOPS stands for trillions of floating-point operations per second, TOPS for trillions of operations per second, often with integers. Manufacturers quote peaks at a particular precision. A figure at a lower precision looks higher, but it is only relevant if your software uses that precision. See our article on compute precision.

How do you compare safely?

A practical example

Say you want to use a language model that needs a lot of memory. First you rule out all cards whose memory is too small. Then you look at bandwidth, because that determines how quickly the model responds. Only after that do you look at compute power and power consumption. That way you avoid picking a card on the strength of one impressive number.

Do the same with the interface and cooling: a card that does not fit in your server or does not get enough airflow will perform worse than the specifications suggest. So also check the dimensions, power supply and supported connections.

What does ‘with sparsity’ mean in a GPU’s TFLOPS figures?

Product pages for data centre GPUs often list several compute figures one below the other, per precision. Some lines carry an asterisk with the addition ‘with sparsity’. For example, on the H100 page NVIDIA marks the Tensor Core lines for TF32, BFLOAT16, FP16, FP8 and INT8, while the FP64 and FP32 lines have no asterisk.

Sparsity means that some of the values in a compute matrix are zero, in a fixed pattern. For the A100, NVIDIA explains that with a 2:4 pattern (two non-zero values in every four) the Tensor Cores skip the calculations on zero values, which doubles throughput. The matching figures in the A100 article are therefore twice as high as the figures without sparsity.

What that means for you:

How do you check for yourself whether a GPU specification fits your situation?

A manufacturer’s datasheet is the starting point, but you do need to be able to read it. Work through these steps:

  1. Find the official product page or datasheet, and not just an overview from a third party.
  2. Read the footnotes below the table. They contain conditions such as ‘with sparsity’ or the mode in which a figure was measured.
  3. Note the precision for each compute figure: FP64, FP32, TF32, FP16, BF16, FP8 or INT8.
  4. Check which precision your software uses. Much AI software lets you configure this; look it up in your program’s documentation.
  5. Check whether variants of the same card exist, for example with a different form factor or cooling. The figures can differ per variant, as the H100 SXM and the H100 NVL show.
  6. Compare the result with the requirements of your application, such as the memory your model needs.

Which GPU specifications do people often forget to check?

Compute power and memory get the most attention, but a card may still not fit your system for other reasons. This table helps you get started.

SpecificationWhy it mattersWhere to find it
Form factor and dimensionsDoes the card physically fit in the case or server bay?Product page or manual
Power connectorYour power supply needs the right cables and sufficient capacitySpecifications of the card and power supply
Cooling designSome cards expect airflow from the case itselfManual of the card or server
PCIe version and number of lanesThe motherboard must offer the connection usedSpecifications of the card and motherboard
Supported software or driversNot every program uses every card equally wellDocumentation of your software

For a first check you can use the check whether a graphics card fits in your PC. In addition, PCIe versions explained covers what the interface means, and what TDP means has more on power.

Frequently asked questions

Is more TFLOPS better by definition?

Not automatically. It is a theoretical peak. In practice, memory, bandwidth and software determine how much of it you notice.

Why are there no prices?

We do not show prices. Use the button on a product to see the price at our partner.

What is the difference between TFLOPS and TOPS for a GPU?

TFLOPS counts trillions of floating-point operations per second, while TOPS counts trillions of operations per second, often with integers such as INT8. So check which precision the figure refers to.

Further reading

Last updated on

Ask us
Compare Get in touch

Are you sure?