ERR_NVGPUCTRPERM: causes and fix

The profiler is fine and your program is fine; the NVIDIA driver is refusing to let a non-admin user read the GPU's hardware performance counters.

This page is not one of the 28 in the original inventory. It earned its slot the way the inventory's own refresh rule says a new page must: a lesson generated it, on real hardware, and then every profiling session after that had to work around it.

Enum, code	none. This is a tools and driver error, not a cudaError_t
Raised by	Nsight Compute, Nsight Graphics, nvprof, Visual Profiler, CUPTI
Source	https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters (checked 2026-09-01)

Strings a reader might paste

The exact failure, captured 2026-08-29 on a Tesla T4, driver 595.84, ncu 2024.3.2, in code/day11-coalescing/evidence/run-2026-08-29.txt:

==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access
NVIDIA GPU Performance Counters on the target device 0.

NVIDIA's solutions page phrases the generic form as "The user running <tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters or the Hardware Event System on the target device" (source above, checked 2026-09-01).

Cause 1: the driver's default restricts counters to admin users

This is the cause almost every time, because it is the shipped default. NVIDIA restricted counter access to admin users after the side-channel research of 2019, from driver 418.43 on Linux and 419.17 on Windows (source above, checked 2026-09-01). The same evidence file pins the root cause on our node to the driver parameter itself:

  ncu as a normal user FAILS:
    ==ERROR== ERR_NVGPUCTRPERM - The user does not have permission to access
    NVIDIA GPU Performance Counters on the target device 0.
  ncu under sudo WORKS.
  Root cause is the driver parameter, /proc/driver/nvidia/params:
    RmProfilingAdminOnly: 1

Two fixes, in order of how long they last:

1. For one session: sudo ncu .... This is what every Module 5 run on our T4 uses; the day 42 evidence header records "sudo ncu; plain ncu fails with ERR_NVGPUCTRPERM" (code/day42-ncu/evidence/run-2026-09-01.txt).
2. For good: set the module option NVreg_RestrictProfilingToAdminUsers=0. Put options nvidia NVreg_RestrictProfilingToAdminUsers=0 in a file under /etc/modprobe.d/, rebuild the initramfs if your distribution needs it, and reboot. Both the option name and the admin-only meaning of 1 are on NVIDIA's solutions page (checked 2026-09-01). Confirm it took by reading RmProfilingAdminOnly back out of /proc/driver/nvidia/params.

Cause 2: a container without the profiling capability

The host allows profiling, but the container cannot reach the counters. Fix: start it with --cap-add=SYS_ADMIN, or fix cause 1 on the host (same source, checked 2026-09-01).

Cause 3: a hosted notebook where you will never have root

On Colab and Kaggle you cannot use sudo on the driver and cannot reload the module, so if the platform's image restricts counters there is no fix from inside the notebook. The day 11 evidence file draws the practical conclusion: on a box where you have root this is a non-issue, and on a hosted tier where you do not it is fatal, which is why every profiling lesson in this course ships its own captured report instead of assuming you can reproduce it. Note the blast radius is counters, not tracing: on the same node, nsys profile -t cuda,nvtx ran as a normal user in the day 41 session while plain ncu failed, so a timeline is still available to you even when a per-kernel counter report is not.

Confirm which one you have

cat /proc/driver/nvidia/params | grep RmProfilingAdminOnly

RmProfilingAdminOnly: 1 plus a failing plain ncu plus a working sudo ncu is cause 1, exactly the three-line proof in the transcript above. If the value is 0 and the error persists inside Docker, it is cause 2.

Related errors

No sibling cudaError_t codes; your program is not failing, your profiler is. The neighbours are practical ones: a kernel that dies under the profiler is usually an illegal memory access was encountered, and a profile that shows a huge first launch is not an error at all, just context creation, which day 9 measures at 255.966 ms on this same T4. A refused launch during a profiling session is invalid configuration argument.

The lesson

Day 42 reads a full Nsight Compute report and is where you will first want counters; day 11 is where this course first hit the error. GPU-less and hosted setups are covered at /setup/learn-cuda-without-a-gpu. The full runtime error table is on the error hub.

---

Written 2026-09-01. Transcripts quoted from code/day11-coalescing/evidence/run-2026-08-29.txt (captured 2026-08-29) and code/day42-ncu/evidence/run-2026-09-01.txt (captured 2026-09-01), both on a Tesla T4 (sm_75), driver 595.84, CUDA 12.6 (V12.6.85), ncu 2024.3.2. Author and reviewer: not yet assigned; this page does not publish until both are named.