What is CUPTI?

The profiling interface underneath Nsight and the PyTorch profiler, which is where the counter permission requirement comes from.

CUPTI, the CUDA Profiling Tools Interface, is the library profilers are built on rather than a tool you run. It exposes two different kinds of information, and the split explains most profiling surprises. Activity tracing (which kernels ran, when, on which stream) comes from the driver's own records and is cheap. Hardware performance counters (sectors, stalls, throughput percentages) come from the GPU's profiling hardware, and access to those counters is restricted to admin users on stock drivers, a restriction NVIDIA documents together with the ERR_NVGPUCTRPERM error it produces (https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters , checked 2026-08-29).

That is why every tool in the stack hits the same wall at the same place. Nsight Systems mostly traces, so it runs unprivileged. Nsight Compute is counters end to end, so it needs the permission. And a framework profiler like PyTorch's sits on the same interface, so what it can report on a locked-down machine is bounded by the same rule, not by anything PyTorch decided. When a hosted notebook shows you kernel timings but no memory chart, this layer is the reason.

The fix is one of three, in order of invasiveness: run the counter tool as root, set the driver option NVreg_RestrictProfilingToAdminUsers=0 and reload, or do what this course does for readers without either power, ship the .ncu-rep files so the reports can be read anywhere.

Measured

Tesla T4, driver 595.84, CUDA 12.6 (V12.6.85), Nsight Systems and Nsight Compute 2024.3.2, captured 2026-09-01. On the project's verification node, the split was observed directly across day 41 and day 42:

nsys profile -t cuda,nvtx -o day41-nvtx ./pagerank_nvtx    # worked as a normal user
ncu ./matmul_profile                                       # ERR_NVGPUCTRPERM
sudo ncu --set full ...                                    # worked

The node's driver carries the stock default RmProfilingAdminOnly: 1, so the tracing path produced day 41's full NVTX timeline unprivileged while every counter read in day 42 needed root. Same machine, same driver, same underlying interface; the difference is which half of CUPTI each tool asks for.

Related terms

• Nsight Compute
• Nsight Systems
• NVTX
• PyTorch profiler

Where you meet this

• Day 41, Nsight Systems, the tracing half, run without root.
• Day 42, Nsight Compute, the counter half, and the lesson that documents the permission error in full.
• Colab setup, where the restriction is the reason the course ships its reports.

Sources

• NVIDIA, "ERR_NVGPUCTRPERM: Permission issue with Performance Counters", for the restriction, the error and the driver-option fix: https://developer.nvidia.com/nvidia-development-tools-solutions-err_nvgpuctrperm-permission-issue-performance-counters (checked 2026-08-29)
• Nsight Compute Profiling Guide, for what the counter-based sections report: https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html (checked 2026-08-29)

Byline

Written by: pending. Reviewed by: pending. Written on: pending. Last checked on: pending. The observations came off the verification node on 2026-09-01, and this entry stays a draft until a named author and a different named reviewer sign it.