GPU Specs Explained: What Do All Those Numbers Mean?
An explanation of the key GPU specifications and how to compare them safely.
The key specifications of a GPU are its memory (VRAM), memory bandwidth, number of compute units, theoretical compute power (TFLOPS or TOPS) and thermal design power (TDP). No single number tells the whole story: what matters depends on your application. So compare the same properties, and pay attention to the precision at which a figure is quoted.
The key specifications
| Specification | What it tells you | Keep in mind |
|---|---|---|
| Memory (VRAM) | How much data the GPU can hold at once | Mainly determines the maximum model or scene size |
| Memory bandwidth | How quickly the GPU can reach its data | Matters a lot for AI and for very large datasets |
| Compute units | Number of cores or similar units | Names and layouts differ per manufacturer; do not compare them one to one |
| TFLOPS or TOPS | Theoretical peak compute at a given precision | Without the precision (FP32, FP16, FP8) the number cannot be interpreted |
| TDP | Design power for power supply and cooling | Not measured consumption |
| Interface | Connection to the rest of the system, such as PCIe | A faster interface mainly helps with heavy data exchange |
Memory and bandwidth
Memory determines what fits. If a model does not fit in memory, a faster card will not help much. Bandwidth determines how quickly data moves back and forth. A lot of AI work is not waiting for compute power but for data from memory; in that case bandwidth is the limit. See also what VRAM is.
Compute power: TFLOPS and TOPS
TFLOPS stands for trillions of floating-point operations per second, TOPS for trillions of operations per second, often with integers. Manufacturers quote peaks at a particular precision. A figure at a lower precision looks higher, but it is only relevant if your software uses that precision. See our article on compute precision.
How do you compare safely?
- Only compare products from the same segment: comparing a graphics card with a data centre GPU rarely gives a fair picture.
- Look at the properties that determine your work, not at a single number.
- Check whether figures from different manufacturers are quoted in the same way.
- Test with your own software before you make a choice.
A practical example
Say you want to use a language model that needs a lot of memory. First you rule out all cards whose memory is too small. Then you look at bandwidth, because that determines how quickly the model responds. Only after that do you look at compute power and power consumption. That way you avoid picking a card on the strength of one impressive number.
Do the same with the interface and cooling: a card that does not fit in your server or does not get enough airflow will perform worse than the specifications suggest. So also check the dimensions, power supply and supported connections.
What does ‘with sparsity’ mean in a GPU’s TFLOPS figures?
Product pages for data centre GPUs often list several compute figures one below the other, per precision. Some lines carry an asterisk with the addition ‘with sparsity’. For example, on the H100 page NVIDIA marks the Tensor Core lines for TF32, BFLOAT16, FP16, FP8 and INT8, while the FP64 and FP32 lines have no asterisk.
Sparsity means that some of the values in a compute matrix are zero, in a fixed pattern. For the A100, NVIDIA explains that with a 2:4 pattern (two non-zero values in every four) the Tensor Cores skip the calculations on zero values, which doubles throughput. The matching figures in the A100 article are therefore twice as high as the figures without sparsity.
What that means for you:
- The figure with sparsity only applies if your model and software use that pattern.
- Without sparsity, the peak is lower. Look in the documentation or the datasheet for the figure without an asterisk.
- If you compare two cards, compare both with sparsity or both without.
How do you check for yourself whether a GPU specification fits your situation?
A manufacturer’s datasheet is the starting point, but you do need to be able to read it. Work through these steps:
- Find the official product page or datasheet, and not just an overview from a third party.
- Read the footnotes below the table. They contain conditions such as ‘with sparsity’ or the mode in which a figure was measured.
- Note the precision for each compute figure: FP64, FP32, TF32, FP16, BF16, FP8 or INT8.
- Check which precision your software uses. Much AI software lets you configure this; look it up in your program’s documentation.
- Check whether variants of the same card exist, for example with a different form factor or cooling. The figures can differ per variant, as the H100 SXM and the H100 NVL show.
- Compare the result with the requirements of your application, such as the memory your model needs.
Which GPU specifications do people often forget to check?
Compute power and memory get the most attention, but a card may still not fit your system for other reasons. This table helps you get started.
| Specification | Why it matters | Where to find it |
|---|---|---|
| Form factor and dimensions | Does the card physically fit in the case or server bay? | Product page or manual |
| Power connector | Your power supply needs the right cables and sufficient capacity | Specifications of the card and power supply |
| Cooling design | Some cards expect airflow from the case itself | Manual of the card or server |
| PCIe version and number of lanes | The motherboard must offer the connection used | Specifications of the card and motherboard |
| Supported software or drivers | Not every program uses every card equally well | Documentation of your software |
For a first check you can use the check whether a graphics card fits in your PC. In addition, PCIe versions explained covers what the interface means, and what TDP means has more on power.
Frequently asked questions
Is more TFLOPS better by definition?
Not automatically. It is a theoretical peak. In practice, memory, bandwidth and software determine how much of it you notice.
Why are there no prices?
We do not show prices. Use the button on a product to see the price at our partner.
What is the difference between TFLOPS and TOPS for a GPU?
TFLOPS counts trillions of floating-point operations per second, while TOPS counts trillions of operations per second, often with integers such as INT8. So check which precision the figure refers to.
Further reading
Last updated on
