What Is an NPU and What Does TOPS Mean?
An explanation of the NPU, the TOPS unit and its role alongside the CPU and GPU.
An NPU (neural processing unit) is a compute unit made for AI calculations, mainly for running models efficiently on laptops, phones and small devices. TOPS is the unit in which manufacturers quote peak performance: trillions of operations per second. The number says little without the precision and software support.
CPU, GPU and NPU side by side
| Chip | Strong at | Typical use |
|---|---|---|
| CPU | Varied tasks and control | Operating system, programs, servers |
| GPU | Many similar calculations at once | Graphics, AI training, heavy inference |
| NPU | Running AI models efficiently | AI features on laptops and phones |
What does TOPS mean?
TOPS stands for tera operations per second. Manufacturers quote it for their NPU or GPU, often with integers and low precision. A higher number means a higher theoretical peak, but not automatically a better experience: software also has to be able to use the NPU. See also our article on compute precision.
Do you need an NPU for cloud work?
For AI in the cloud you usually look at GPU instances; some providers have their own AI chips, but these are not usually called NPUs. NPUs are more relevant for devices on which you run AI locally. Read more in private or public AI.
Where do you come across an NPU?
NPUs are found in many new laptops and phones for features such as speech recognition, photo editing and summarising text on the device itself. The advantage is that it uses less power and that data can stay on your device. The drawback is that the model has to be small enough for the device.
For heavier tasks you switch to a GPU, locally or in the cloud. That way the CPU, GPU and NPU complement each other.
How do you read a TOPS figure on a spec sheet?
TOPS stands for trillions of operations per second. In its documentation for Windows developers, Microsoft describes an NPU of more than 40 TOPS as a chip that can perform more than 40 trillion operations per second. Such a figure is a theoretical maximum. So ask yourself a few questions when looking at a spec sheet:
- Which precision does the figure apply to? It is often about calculating with integers at low precision. If your model runs at a different precision, practice may turn out differently.
- Does the figure belong to the NPU alone, or to several components together? Manufacturers may present this differently. Read the footnotes to the figure.
- Is it a peak value? How long a chip sustains that peak depends on things like cooling and power settings.
- Does your software actually use the NPU? A program has to support the NPU explicitly, otherwise it goes unused.
Microsoft also writes that many new Windows AI features require an NPU of 40+ TOPS. Note the word ‘many’: not every feature and not every program sets the same requirement. So look at the system requirements of the software you want to use.
A thought experiment with your own figures: a manufacturer states X TOPS for a chip and a program asks for Y TOPS. Comparing only makes sense if both figures measure the same thing, so with the same precision and the same component as the starting point. If that is not stated anywhere, ask the manufacturer or the maker of the software.
How to check for yourself whether your PC has and uses an NPU
- Look up the exact type of your laptop or processor and open the manufacturer’s specification page. Search for NPU or for the mention of TOPS.
- Open Task Manager in Windows 11 and go to the Performance tab. According to Microsoft, Task Manager can show NPU usage. If your device has an NPU, it appears there among components such as CPU, GPU and memory. If it is not there, your device does not have one, or your Windows version is too old.
- Start the feature or app you want to use and see whether the NPU graph starts moving. If it stays flat, the work may be running on the CPU or GPU.
- Read the app’s documentation to see which hardware is needed for local AI features.
On phones and tablets you usually will not find this kind of overview in the same form. There, the manufacturer’s specifications offer the most to go on.
Which misconceptions do you often see about NPUs and TOPS?
- ‘More TOPS means a faster chatbot.’ Not necessarily. The software, the memory and the size of the model also determine what you notice. You can read more about running models locally in Running AI at home or in the cloud.
- ‘Without an NPU I cannot use AI.’ Many AI services run on the provider’s servers. Your own device then does little processing.
- ‘TOPS is the same as FLOPS.’ The units differ: FLOPS counts floating-point operations, TOPS usually counts integer operations. Do not compare them directly. For GPUs, read more in GPU specifications explained.
- ‘An NPU makes everything faster.’ Ordinary programs use the CPU and sometimes the GPU. The NPU only helps with work that has been written for it.
Why might an NPU be interesting when choosing a device?
Microsoft says an NPU can perform AI tasks using less energy than a CPU or GPU, which can benefit battery life. For people who often use local AI features, such as speech recognition or photo editing, that can be a reason to look at the NPU. If you mainly use web services or ordinary programs, the NPU plays a smaller role.
So first ask yourself which features you really want to use and check whether that software supports the NPU. Only then does it make sense to compare devices on the figure.
Frequently asked questions
Does an NPU replace a GPU?
No. An NPU is more efficient for certain AI tasks but less flexible than a GPU.
Can I train models on an NPU?
An NPU is usually made for running models, not for training large models.
How many TOPS do I need?
That depends on the application. Microsoft says many new Windows AI features require an NPU of 40+ TOPS, but other programs set other requirements. Check the system requirements of the software you want to use.
Further reading
Last updated on
