Day 94Module 10

in-technical-review

Sharing a GPU: green contexts, MPS and MIG

Two claims follow this topic around. The first is that green contexts are a Hopper feature. The second is that MPS requires Volta. The first is wrong: green contexts run wherever the driver has the API, and what moves with compute capability is only how coarse a partition has to be. The second is a sentence nobody can find in NVIDIA's own Multi-Process Service document, and this page went and read it rather than repeating it.

Underneath both is a real question. Put a long batch kernel and a stream of short ones on one card and the short ones get slower. By the end of this page you will have asked a T4's driver how it is willing to divide its own 40 SMs, run those two workloads three ways, and be able to say which way a tenant should want. The answer is not the fast one.

What a green context owns

A normal CUDA context sees the whole device. Two processes, or two streams in one process, submit into the same pool of SMs and the hardware decides who gets a slot as blocks retire. That is fine when both jobs are the same shape and miserable when one of them is a 16,384 block batch kernel and the other is a 256 block kernel somebody is waiting on.

A green context owns a named subset of the SMs and nothing else. Work on its stream cannot spill into a neighbour's SMs, and a neighbour's work cannot take its. Four driver API calls build one, and each stage is a different kind of object: cuDeviceGetDevResource asks the device for its SM resource, cuDevSmResourceSplitByCount cuts that into groups, cuDevResourceGenerateDesc freezes a group into a descriptor, and cuGreenCtxCreate provisions it. Only the last touches hardware.

The split is the interesting call, because you do not get to choose the number of SMs. You ask for a minimum per group and the driver tells you what it will do:

On Compute Architecture 6.X: The minimum count is 1 SM. On Compute Architecture 7.X: The minimum count is 2 SMs and must be a multiple of 2. On Compute Architecture 8.X: The minimum count is 4 SMs and must be a multiple of 2. On Compute Architecture 9.0+: The minimum count is 8 SMs and must be a multiple of 8.

That is the CUDA 12.6 Green Contexts reference verbatim, and the sentence directly above it calls the list "a guideline for each architecture and may be subject to change" (https://docs.nvidia.com/cuda/archive/12.6.0/cuda-driver-api/group__CUDA__GREEN__CONTEXTS.html , checked 2026-09-01). Hopper is not the entry ticket. Hopper is the row where the grain gets coarser, because a partition there has to be able to hold a thread block cluster, which day 77 built.

Diagram: one card, three ways to divide it. Three horizontal bands, each the same strip of 40 SM cells with a time axis underneath, and two shaded jobs on top: a wide batch job and a narrow short one. Band 1, taking turns: the batch job fills all 40 cells for the whole strip, then the short job fills all 40 for a sliver at the end. Caption "One stream: 40 SMs each, one at a time, and the short job waits for all of it." Band 2, sharing: both jobs drawn across all 40 cells at once, the short job's slices wedged into gaps as batch blocks retire. Caption "Two streams: 40 SMs each, overlapping, neither one promised anything." Band 3, partitioned: a line at cell 20 splits the strip, the batch job fills cells 0 to 19 for longer than in band 2, the short job fills 20 to 39 and finishes early. Caption "Two green contexts: 20 SMs each, and the line is the promise." Alt text: "One forty-SM card divided three ways. Taking turns gives each job all forty and makes the short job wait. Sharing overlaps them with no guarantee. Two green contexts give twenty each, and the short job finishes first."

The three mechanisms, and which one you can reach

Green contexts are one of three answers to "two workloads, one GPU", and they are the only one an ordinary user can drive from inside a process. The other two, MPS and MIG, are worth knowing exactly, because both are widely misquoted.

MIG has a hard floor and it is the one people state correctly. It "allows GPUs (starting with NVIDIA Ampere architecture) to be securely partitioned" (https://docs.nvidia.com/datacenter/tesla/mig-user-guide/introduction.html , checked 2026-08-29), and only on the datacenter parts. It is also the strongest isolation of the three: separate memory, separate cache slices, a device that appears in nvidia-smi on its own. Getting one needs root and a device reconfiguration.

MPS is the one with the folklore. Its document's only stated platform limit is not a GPU at all: "MPS is only supported on the Linux and QNX operating systems." No minimum compute capability appears anywhere. What the document does instead is split its own feature set in two and name the halves. The daemon commands that set an active thread percentage sit under the heading "Commands available to Volta MPS control", and CUDA_MPS_ACTIVE_THREAD_PERCENTAGE is introduced with "On Volta GPUs, this environment variable sets the portion of the available threads that can be used by the client contexts". The line that settles the question is about mixed machines: "The MPS control daemon will further filter-out any pre-Volta devices, if any visible device is Volta+." A daemon that filters pre-Volta devices out is a daemon that expects them. MPS runs below Volta; the SM-limiting knob this page cares about is documented for Volta and newer. Those are two claims and only one of them is true (https://docs.nvidia.com/deploy/mps/index.html and its Architecture and Appendix pages, checked 2026-09-01).

MPS has since grown a spatial partitioner, and that one does carry a floor: "Static partitioning mode is only supported on NVIDIA Ampere architecture and newer GPUs", in chunks of 4 SMs before Hopper and 8 after, handed to clients through CUDA_MPS_SM_PARTITION (same document, checked 2026-09-01). It needs a daemon, so it is a deployment decision rather than a line of code. Green contexts are the same idea inside one process, on older cards, with no daemon.

One version note, because it decides which spelling you write. Green contexts arrived as a driver API in CUDA 12.4, cuGreenCtxStreamCreate in 12.5, and a runtime-API spelling (cudaGreenCtxCreate, cudaExecutionCtxStreamCreate) in CUDA 13.1, which changes nothing else: "No need to modify any code in this function or in your kernel(s)" (https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/green-contexts.html , checked 2026-09-01). This lesson keeps the driver-API version and links -lcuda, so the same source covers both recorded CUDA 12.6 and 13.0 runs; the runtime-API spelling starts later, at CUDA 13.1.

Holding the work still and moving the SMs

Full program in code/day94-gpu-sharing/gpu_sharing.cu.

One kernel, two workloads. Both jobs are the same dependent-FMA chain, differing only in element count and chain length: 4 Mi elements at 8192 steps for the batch job, 64 Ki elements at 1024 steps launched 64 times for the short one. Nothing in the timing can be arithmetic, because there is only one piece of arithmetic.

The driver is asked, never assumed. The program simulates a split before performing it, and reads back what each green context actually got rather than trusting what it requested.

The answers are checked after every schedule. Two tenants sharing a card must not change each other's results, so all three schedules are compared against a closed-form reference and then against each other, bit for bit.

The split call is two calls, and the first one performs nothing. Passing a null result array asks the driver how many groups it would make, which is how a program learns this card's grain without carrying a table keyed on compute capability:

    unsigned int groups = 0;
    if (!driverOk(cuDevSmResourceSplitByCount(nullptr, &groups, input, nullptr,
                                              useFlags, minCount),
                  "cuDevSmResourceSplitByCount (simulate)")) {
        return false;
    }
    row->groups = groups;
    if (groups == 0) {
        return true;
    }
    std::vector<CUdevResource> parts(groups);
    CUdevResource rest{};
    unsigned int made = groups;
    if (!driverOk(cuDevSmResourceSplitByCount(parts.data(), &made, input, &rest,
                                              useFlags, minCount),
                  "cuDevSmResourceSplitByCount")) {
        return false;
    }

Turning one of those groups into something you can launch on is four calls and no cleverness:

static bool makeClient(CUdevice dev, CUdevResource* part, Client* c) {
    CUdevResourceDesc desc = nullptr;
    if (!driverOk(cuDevResourceGenerateDesc(&desc, part, 1),
                  "cuDevResourceGenerateDesc")) {
        return false;
    }
    if (!driverOk(
            cuGreenCtxCreate(&c->green, desc, dev, CU_GREEN_CTX_DEFAULT_STREAM),
            "cuGreenCtxCreate")) {
        return false;
    }
    if (!driverOk(cuCtxFromGreenCtx(&c->ctx, c->green), "cuCtxFromGreenCtx")) {
        return false;
    }
    return driverOk(
        cuGreenCtxStreamCreate(&c->stream, c->green, CU_STREAM_NON_BLOCKING, 0),
        "cuGreenCtxStreamCreate");
}

cudaStream_t and CUstream are the same typedef, so the stream the driver API hands back goes straight into a <<< >>> launch with no cast and no wrapper. That is the whole ergonomic story: the partition is set up once and the kernels never learn about it.

Timing needs one trick. Both working streams are non-blocking, so the NULL stream never waits for them, and an event recorded there completes the moment the GPU reaches it. Three such events give the schedule's start, the moment the short job finished, and the moment everything finished, all on the GPU clock. Day 59 used the same idea to time a leg that ran entirely on the CPU.

    CUDA_CHECK(cudaEventRecord(evStart, 0));
    bool ok = submit(bulkOn, b, true) && submit(shortOn, b, false);
    ok = ok && waitOn(shortOn);
    CUDA_CHECK(cudaEventRecord(evShort, 0));
    ok = ok && waitOn(bulkOn);
    CUDA_CHECK(cudaEventRecord(evAll, 0));
    CUDA_CHECK(cudaEventSynchronize(evAll));
    CUDA_CHECK(cudaGetLastError());

What this program does not do is MPS or MIG. One needs a daemon and the other needs root and a device reconfiguration, so neither fits inside an unprivileged process. The page covers them from the documentation and measures only the mechanism it can actually drive.

Results

Re-verified on hardware. The CUDA program ran on the same Tesla T4 with driver 580.173.02 and CUDA 13.0 (V13.0.88), passed every gate and exited 0. The split table reproduced exactly, but schedule timing moved materially: partitioning still protected the short job from sharing, yet no longer beat its alone latency and had the largest total. Both run transcripts are listed in front matter. The accompanying nvidia-smi -q -d compute capture records Compute Mode as Default. On the same node, nvidia-cuda-mps-control -d and the subsequent quit command both exited 0. The daemon warned that it could not write logs under /var/log/nvidia-mps; that affected log capture, not daemon startup.

Part 1, the split the driver is willing to make:

minCount	groups	SMs per group
1	20	2
2	20	2
3	10	4
4	10	4
8	5	8
20	2	20

Part 2, two workloads and five schedules:

schedule	batch SMs	short SMs	short done	everything done
alone-batch	40	-	0.024 ms	21.922 ms
alone-short	-	40	3.332 ms	3.335 ms
serial	40	40	25.222 ms	25.228 ms
shared	40	40	25.192 ms	25.196 ms
partitioned	20	20	5.930 ms	39.528 ms

Five predictions, each of which the run can kill.

One held. Asking for 1 SM produced 20 groups of 2; asking for 3 produced 10 groups of 4.

Two was refuted. CU_DEV_SM_RESOURCE_SPLIT_IGNORE_SM_COSCHEDULING changed none of the printed rows on this T4; the top row stayed 20 groups of 2 rather than 40 groups of 1.

Three held. Serial's columns were 25.222 and 25.228 ms, and its total remained the two alone workloads laid end to end.

Four held in direction, with a different magnitude. Shared edged serial by only 0.032 ms, effectively a tie at this noise level, while the short job took 25.192 ms: 7.56x its alone time and 4.25x the partitioned result.

Five held under CUDA 13.0. Partitioning bought the predicted short-job protection at 5.930 ms against shared's 25.192 ms, while its 39.528 ms total was the largest of the three. It did not reproduce CUDA 12.6's unusually better-than-alone short latency.

The partitioned columns were not equal, so the two green-context streams did overlap in this run. The program used context pushes and proves that conservative path works; it cannot prove the pushes are unnecessary without an A/B run. A partition is a promise about the worst case, not the average. Read the short-job and total columns as two different customers.

Run it yourself

A free Colab T4 runs the whole thing, and so does any card you have, because min_cc here is 7.5 and the only unusual build flag is -lcuda. The driver library is the thing to watch: a conda or micromamba toolkit ships only a stub libcuda.so, which links fine and cannot run, so the real driver library has to be first on LD_LIBRARY_PATH. From the README:

nvcc -std=c++17 -O3 -arch=sm_75 -o gpu_sharing gpu_sharing.cu -lcuda

There is no profiler here and no root. Everything on the page comes out of printf, which is deliberate: a page about who gets which SMs should not require the permission that day 42 had to fight for.

Exercise

Split the card unevenly. Give the short job the smallest partition the driver will make and the batch job everything else, then say which of the two columns in part 2 moved and by how much.

Time: 30 to 40 minutes. Submit: the partitioned row you measured with its two SM counts, beside the even-split row, and one sentence saying which customer you just made worse.

Check: the program already gates every schedule against the closed-form reference and against the alone runs bit for bit, so an uneven split that changes one output value is caught with the first bad index and both values printed. Two mistakes account for most failures. Building a descriptor from groups that came from different calls to cuDevSmResourceSplitByCount returns CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION, because the resources in one descriptor must be outputs of the same split. And handing the leftover set to a green context works but is not the exercise: the reference says the remainder "does not have the same functional or performance guarantees as the groups", so its row is not comparable to the one beside it.

Hint 1

You do not need two calls. One split at the smallest legal minimum gives you every group at once, and a descriptor may be built from more than one of them.

Hint 2

Which column should be able to move at all? Neither job's work changed. Only the number of SMs each is allowed to use changed, and only one of those changed a lot.

Solution

Split at the granularity floor, then pass one group to the short job's descriptor and the rest of the array to the batch job's. On a 7.5 card that is groups of 2, so the short job gets 2 SMs and the batch job gets 19 groups in one cuDevResourceGenerateDesc call.

Expect the batch job's total to come most of the way back toward its alone time, because it got nearly the whole card, while the short job's finish gets much worse, roughly in proportion to the SMs taken off it. The ratio between the two columns is the dial, and the even split was one setting of it. An SM partition is not a performance feature. It is a way to choose which of two customers absorbs the contention.

Pitfalls

You created two green contexts and the two kernels still take turns. A disjoint SM partition is not a concurrency guarantee. The reference names the reason: other resources, hardware work queues among them, can serialise the two, and it points at CUDA_DEVICE_MAX_CONNECTIONS. Raise that environment variable before you conclude the partition did nothing.

cuGreenCtxStreamCreate fails and you passed 0 for the flags. There is one legal value and it is not the default: the reference says of CU_STREAM_NON_BLOCKING that "This must be specified". The same call also ignores whatever context is current, which is the point of it, so a failure here is about the flag and not about your context.

You asked for 1 SM per group and got 2. That is the 7.x row of the granularity guideline doing its job. Simulate the split with a null result array first and read the group count back, rather than computing SM counts on the host and expecting the driver to agree. On a 9.0 card the same code gets groups of 8.

The program builds and then will not start. -lcuda links against the driver library, and a toolkit installed without a driver ships only a stub version of it. The stub satisfies the linker and resolves nothing at run time, so put the real libcuda.so.1 ahead of the stub directory on LD_LIBRARY_PATH. This is a packaging problem, not a CUDA one, and it bites hardest in conda environments.

You reached for MPS on a Turing card to get SM partitions. MPS will run there, but the partitioning mode will not: static SM partitioning is Ampere and newer. Green contexts are what a 7.5 card has, and unlike MPS they need no daemon and no second process. Day 51 covers the case where you did not need either, which is most cases.

Your partitioned numbers move every run. Both jobs are compute bound, so occupancy and clocks matter, and a half-empty card boosts differently from a full one. Report the shape and the ratio between columns, not the milliseconds, which is why part 2 prints five schedules rather than one.

Go deeper

CUDA Programming Guide 4.6, "Green Contexts", for the runtime-API spelling that arrived in CUDA 13.1 and the worked before-and-after example: https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/green-contexts.html (checked 2026-09-01)
CUDA Driver API 6.35, "Green Contexts", for every call this program makes, the granularity guideline and the no-concurrency-guarantee paragraph: https://docs.nvidia.com/cuda/archive/12.6.0/cuda-driver-api/group__CUDA__GREEN__CONTEXTS.html (checked 2026-09-01)
Multi-Process Service, "When to Use MPS" and "Architecture", for what MPS actually promises, the Volta split, and static SM partitioning: https://docs.nvidia.com/deploy/mps/index.html (checked 2026-09-01)
MIG User Guide, "Introduction", for the Ampere floor and what hardware isolation buys that an SM partition does not: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/introduction.html (checked 2026-08-29)

Day 95 starts the last stretch, where the kernels stop being demonstrations and start being the ones a language model actually runs: softmax, layer norm and a fused RMS norm measured against PyTorch. The SM budget you just learned to hand out is the same budget those kernels compete for when an inference server puts two requests on one card.