#884

RTX 4080 / NVIDIA 621.01: the corpus's two largest compute pipelines fail to compile (NVVM), a few take 8-16 GB and up to 26 min, and --jobs 14 crashed the driver

OpenProudnoob4opened 9h ago

The full shader corpus from my own 1.0.2.1 copy, translated at 91b36cf with --wave64-emulate (26503/26503 pass spirv-val, see #539), run through vk_pipeline_check on an RTX 4080 (NVIDIA 621.01, Windows 10 22H2, 32 GB RAM):

1. Two compute pipelines fail on NVIDIA: NVVM compilation failed: 3. They are #539's two shaders, the two largest modules in the corpus:

sorted-index positionwordsblocksOpSwitchlargest switchresult alone (--jobs 1)
2094615,086,95944,048416,000 casesNVVM compilation failed: 3 after 411 s, peak 2.8 GB
2242815,054,92343,829416,000 casesNVVM compilation failed: 3 after 401 s, peak 11.2 GB

Corpus for scale: median 43,243 words, p99 1,241,960. These are the modules where d7cbc3e now splits the dispatch OpSwitch into chunks of 16,000; they pass spirv-val but NVIDIA's compiler gives up on them. (On main 4a36bb1 they no longer reach the driver at all: since #755 they exceed the SPIR-V id bound, see #881. So these driver numbers are from 91b36cf.)

2. A few compute pipelines need minutes and gigabytes each:

positionwordsblocksOpSwitchtime alonepeak RAM
207677,980,74746,11201,585 s15.9 GB
224962,010,0924,2511 (3,279 cases)772 s9.0 GB
210971,765,8034,1091 (3,137 cases)622 s8.0 GB

In the full run, 78 pipelines took 60 s or more (45 compute, 33 vertex), 29 took 120 s or more, 8 took 300 s, 6 took 600 s, and 2 took 900 s.

3. With --jobs 14 the driver crashed. After 23,250 of 26,503 pipelines (106 min), vk_pipeline_check.exe exited with 0xC0000005. The Windows event log puts the fault in nvgpucomp64.dll 32.0.16.2101. At that moment the four heavy compute pipelines above were compiling side by side; together they need about 44 GB alone, on a 32 GB machine. So running out of memory inside the driver is the likely cause. That's an inference: memory wasn't recorded at the crash, and none of them crashed when compiled one at a time. A checkpoint cache (3.0 GB) had been saved 4 minutes earlier.

Possible directions, for whoever owns them: a different lowering for these dispatch-loop giants; or, in vk_pipeline_check and the pipelines stage, fewer concurrent compiles for very large modules (#641 made --jobs reach the checker). Positions are in vk_pipeline_check's sorted index order; names and shader data stay on my machine. Happy to rerun anything on this GPU.

AI-assisted (Claude Code in OpenCode): it ran these on this machine.

Activity

8
ItsNotRexed7h ago

This is very likely the two indirect_diffuse_lighting ScreenProbes_ActivateProbes compute shaders. Each decoded every typed-buffer format inline at 3,281 loads, which made 50–60 MB modules (past SPIR-V's id bound under --wave64-emulate). dcd8ecf emits that decode once per module: 6.16 M to 2.20 M instructions, about 41.6 MB, and they validate again. Could you retry on the RTX 4080 / 621.01 with a fresh translation? vk_pipeline_check now also compiles modules over 4 MiB alone (#903).

poke900310-lgtm4h ago

AMD numbers for items 2 and 3, including a retry with dcd8ecf. RX 9070 XT (RDNA4), AMD 26.11.1 (LLPC, 32.0.32015.16001), Windows 11, 64 GB RAM.

  • Modules come from my own 1.0.2.1 copy. vk_pipeline_check is at 4a36bb1, run with --jobs 1, one module at a time, and no pipeline cache file.
  • Each run used a copy with a different SPIR-V generator word, so the driver's disk cache couldn't serve an earlier compile.
  • The GPU was otherwise idle. Repeat runs of the same module varied by up to about 10%.
positiontranslationwordsAMD compileAMD peak RAMyour RTX 4080 (91b36cf, emulated)
207674a36bb1, native16,773,035797 s9.1 GB7,980,747 words, 1,585 s, 15.9 GB
20767+ dcd8ecf, native5,312,541769 s9.0 GB
224964a36bb1, --wave64-emulate2,044,480908 s5.3 GB2,010,092 words, 772 s, 9.0 GB
210974a36bb1, --wave64-emulate1,797,423702 s4.8 GB1,765,803 words, 622 s, 8.0 GB
21097+ dcd8ecf, --wave64-emulate1,773,224663 s4.8 GB
224964a36bb1, native1,856,42316 s0.65 GB
210974a36bb1, native1,619,88313 s0.55 GB

What this shows:

  • dcd8ecf makes 20767 3.2x smaller, but on AMD it compiles in about the same time and RAM. So for this one the cost isn't the duplicated decode.
  • The two switch modules are just as slow on AMD as on your 4080 when emulated, at roughly half the RAM. Without emulation, which is what AMD picks by default, they have no OpSwitch and take 13-16 s. On both vendors the cost looks like the emulated path's ~3k-case dispatch switch, not the shaders themselves.
  • --jobs 4 at 4a36bb1, with 20767 and 22560 native plus emulated 22496 and 21097 side by side, like your crash case: 1,110 s, no crash, peak 16.6 GB working set (19.0 GB private). On a 32 GB machine that's already tight, so a lower concurrency limit for very large or switch-heavy modules looks worthwhile on AMD too.

AI-assisted run (Claude Code); counts only, no game data.

dcastell13h ago

NVIDIA numbers for item 1 on a different box, with the current module size: both modules now die with LLVM ERROR: out of memory, not NVVM compilation failed: 3.

RTX 5090 (32 GB), NVIDIA 621.24, Windows 11, 64 GB RAM, vk_pipeline_check at aae4d99, Vulkan SDK 1.4.363.1 with the implicit layers disabled. Modules translated from 426023a with --wave64-emulate: cs_screenprobes_activate 42,815,556 B and cs_screenprobes_target 42,663,088 B, 2/2 pass spirv-val.

modulerunresulttimelast working set
cs_screenprobes_targetalone (--jobs 1)LLVM ERROR: out of memory / Allocation failed1,740 s3.0 GB
both, one process eachconcurrentLLVM ERROR: out of memory5,820 s10.0 GB

(activate alone has not finished a run yet; the numbers above come from the concurrent pass, so the 5,820 s is the wall clock of that process, not a clean measurement.)

What this says so far:

  • Neither module is close to the box's RAM (64 GB) or VRAM, so this is not a capacity OOM in the usual sense. It looks like an allocation-size or address-space limit inside the driver's LLVM, which would also explain why the 4080/621.01 box stops earlier with NVVM compilation failed: 3 instead of reaching the allocator.
  • A retry of target alone is running to rule out that the concurrent pass caused it. I will post the result.
  • Both modules are re-translated into smaller shapes for the same test: without --wave64-emulate 34 MB / 33 MB, and --dispatch-loop 33 MB / 31 MB, all pass spirv-val. If one of those compiles, the 2.2 M-instruction size after dcd8ecf is not the whole story and the remaining blow-up is worth finding before the next corpus pass.
Proudnoob42h ago

@ItsNotRexed on it: a fresh run on the RTX 4080 / 621.01 (32 GB) at main bb2e1842. That's the shaders stage from scratch on my copy, then the two ScreenProbes modules alone (time, peak RAM, result), then the full pipelines stage with #903. I'll post the numbers here. AI-assisted.

dcastell12h ago

The retry with cs_screenprobes_target alone (--jobs 1, nothing else compiling) also ends in LLVM ERROR: out of memory / Allocation failed, so the concurrent pass was not the cause:

runresulttimelast working set seen
target aloneLLVM ERROR: out of memory4,445 s9.6 GB
target + activate, one process eachLLVM ERROR: out of memory1,740 s (target), 5,820 s (activate)3.0 / 10.0 GB

Both on the RTX 5090, 621.24, 64 GB RAM, policy 0, implicit layers disabled. Alone it ran 2.5x longer and got to 9.6 GB before the allocator gave up.

So on this driver the failure is a memory ceiling inside the compiler, not total-RAM pressure: 9.6 GB used out of 64 GB available, and the process had been growing steadily (2.8 GB at 20 min, 6.3 GB at 52 min, 9.6 GB at 74 min). The 4080/621.01 box stopped earlier, at 11.2 GB, with NVVM compilation failed: 3 instead of reaching the allocator, which is consistent with the same limit hit at a different point in the pipeline.

Next: the same module retranslated without --wave64-emulate (34 MB instead of 42.7 MB) is compiling now. It is not a correct translation for a 32-wide device — this is a compile-only probe to see whether the emulation's extra instructions are what pushes the compiler over, or whether the 44k block bodies are enough on their own. I will post that result too.

dcastell12h ago

The wave64 emulation is what makes NVVM give up. The same cs_screenprobes_target, retranslated without --wave64-emulate, compiles:

moduletranslationsizeresulttime
cs_screenprobes_target--wave64-emulate42,663,088 BLLVM ERROR: out of memory, 9.6 GB4,445 s (alone)
cs_screenprobes_targetplain35,229,388 Bok450 s

RTX 5090, 621.24, 64 GB, --jobs 1 --slow 1, implicit layers off. 450 s is also what the RTX 4080 / 621.01 box saw for the same shape (401 s, 11.2 GB), so the plain module is consistently compilable; only the emulated one dies, and it dies in the allocator rather than at pipeline creation.

This is a compile-only probe: without --wave64-emulate the module assumes a native 64-wide wave, so on this 32-wide GPU its cross-lane results would be wrong. The point is to identify the trigger, and it is the emulation's structure, not the 44k block bodies: it wraps the whole shader in a two-round loop with OpControlBarrier between rounds (Gen.x_round / x_split, workgroup slots for the exchanged values), and that loop around a 16,000-case dispatch switch is what the driver's structurizer cannot size.

Practical consequence: on NVIDIA, these two ScreenProbes pipelines are unreachable as long as they need the emulation, even though the plain module compiles in 450 s on the same driver. Making them compile would mean making the emulation cheaper — for example hoisting the exchange out of the dispatch loop, or emitting the two rounds as straight-line code instead of a loop — not shrinking the shader.

cs_screenprobes_activate (35,372,420 B plain) is compiling now for the second data point.

dcastell12h ago

Second data point for the plain shapes, and a correction to what I said above about the plain module being a possible way out.

moduleplain (no emulation)emulated
cs_screenprobes_target35,229,388 B, ok in 450 s42,663,088 B, OOM at 4,445 s / 9.6 GB
cs_screenprobes_activate35,372,420 B, ok in 466 sOOM at 5,820 s (concurrent pass)

So on this driver the plain shapes are reliably compilable (450/466 s, matching the 4080's ~400 s) and only the emulated ones die. But the plain shape is not a usable translation for these two shaders, so this is a size probe only: both are full of wave-wide EXEC tests.

Opcode counts in cs_screenprobes_target (85,141 instructions): 9,896 s_cbranch_execz, 3,292 s_and_saveexec_b64, 3,295 s_andn2_b64, 4,113 s_or_b64, 3 s_cbranch_vccnz, and no s_barrier, no ds_bpermute/v_readlane. cs_screenprobes_activate is the same shape. Each s_cbranch_execz is a vote on the whole 64-bit EXEC, so on a 32-wide subgroup the two halves would branch independently and the per-lane model would stop matching hardware; the emulation is required here, not optional.

That also says what the compiler is chewing on: ~9,900 wave-wide votes per module means ~9,900 segments in the round scheduler, so it is a state machine with ~10k states wrapping the dispatch switch, not just a 21%-larger straight-line body. Shrinking the module will not fix it; the scheduler would have to stop being one switch over ~10k segment starts (for instance, keeping the original CFG inside a round and only cutting at the votes that actually read another half's mask, which for execz after s_and_saveexec_b64 VCC is every lane's own comparison ORed across the wave).

I am not proposing a patch for that blind; happy to prototype if a maintainer wants the numbers from a variant.

Thatters1h ago

@dcastell1 this is the larger module (41.7 MB) alone, which your table doesn't have yet. On this box it was a real capacity OOM: the compiler ran the machine out of commit, not into a small cap.

GPU:       NVIDIA GeForce RTX 5070 Ti (subgroup 32..32)
Driver:    621.24 (32.0.16.2124)
OS:        Windows 11 Pro 25H2, 64 GB RAM + 30 GB page file
Translate: tools/rdna2spirv.py --wave64-emulate at main 48ad23f (includes dcd8ecf); 2/2 pass spirv-val,
           41,709,696 B and 41,557,100 B (10.43 M / 10.39 M words), 44 s for both
Compile:   vk_pipeline_check --jobs 1 on just these two (binary from 43d0015; with --jobs 1, #903 changes nothing)
  • The larger module, alone: it compiled for 4,860 s (81 min) on one thread, and the working set kept growing: 2.8 GB at 30 min, 8.6 GB at 61 min, peak 49 GB. Then LLVM ERROR: out of memory / Allocation failed.
  • At that moment Windows logged a low-virtual-memory event (Resource-Exhaustion-Detector 2004) with vk_pipeline_check.exe at 80.3 GB committed, against 94 GB of RAM plus page file. The process faulted in nvgpucomp64.dll 32.0.16.2124 with exception 0x80000003. The pipeline cache stayed empty.
  • The smaller module: never reached, because the process died first.

On "a memory ceiling, not total-RAM pressure": my sampled working set followed yours almost exactly (8.2 GB at 61 min, vs your 6.3 GB at 52 min and 9.6 GB at 74 min). But the process's peak working set counter read 49 GB when it died, and commit reached 80 GB. So the growth wasn't steady to the end: in the last ~20 minutes it jumped from about 9 GB to everything the machine had, and the allocator failed then. A "last seen" sample taken before that jump would show ~10 GB. If you can, check PeakWorkingSet64, Task Manager's commit size, or a System-log event 2004 around the failure time. If they show the same jump, then on 32 GB (the 4080) the same jump would hit the limit sooner, which might explain its earlier stop.

That doesn't change your conclusion. The plain module compiling in 450 s points at the emulation's round loop as the trigger, and it fits @poke900310-lgtm's AMD numbers (native dcd8ecf module 5.3 M words, 769 s / 9 GB). What it adds is that the failure mode looks like a runaway allocation at the end, not a fixed cap.

For the rest of the corpus (same PC, main 43d0015, where these two still failed spirv-val and were left out): vk_pipeline_check --jobs 12 compiled 26,501 of 26,501 with 0 failures and no driver crash in 95.4 min. The slowest pipeline (compute) took 19 min.

AI-assisted.