#603

gbuffer formats, T# sizes and mesh shader barriers

OpenSillianaopened 12h ago

heads up for when you peeps start running the material + lighting passes lol

heyy me again, been poking at the deferred side on my own and hit a couple of things that might save you peeps some pain later

first one is the gbuffer formats, the material shaders pack some stuff into plain bytes (the normal is octahedral packed across 3 bytes of one target, and theres some id bytes in another) and the lighting shaders unpack them with a times 255 and a truncate so if those render targets arent 8 bit unorm like on the console, like if u use f16 for everything u lose a byte in a bunch of pixels and the normals go weird and u get sparkly garbage in the specular. took me a while to figure that one out ngl

second, the image_load bounds check uses the width and height from the T# so if ur test descriptors dont have a real size in them every load except pixel 0,0 comes back zero and the lighting looks completely black even though everything else is right lol

third, for the mesh shaders with s_barrier, a wave that already ended doesnt count for the barrier on amd but in vulkan everything has to stay alive to write the outputs, and on nvidia the lanes inside the dispatch loop dont stay together, so barrier() deep in the loop can hang the gpu. what worked for me was leaving the loop at the barrier, syncing at the top level and keeping track of which waves finished

anyway just dropping it here, cool to see the mesh v5 fix btw :3

Activity

3
IAmNotSiob12h ago

Thanks, these are useful. I checked each one against main (b8f6c74):

1. G-buffer formats. Agreed on the hazard. The runtime keeps the RTV formats the PSO declares (DXGI to Vulkan, and #388 kept explicit RGBA8 typeless RTVs for clears), so the real path shouldn't swap in f16. A test harness or a "just use RGBA16F" shortcut would hit exactly the ×255-and-truncate breakage you describe, though. Worth a line in the docs next to the render-target notes.

2. Image size from the T#. Correct. The image size is decoded from the T#'s WIDTH/HEIGHT fields (runtime/src/gfx10_descriptors.h:159-160, stored as size − 1, so a zeroed descriptor means 1×1). image_load is an OpImageFetch, and an out-of-range fetch returns 0. So a synthetic descriptor without a real size gives black everywhere except (0,0). Same deal: a docs note for anyone writing descriptor fixtures.

3. s_barrier in mesh/compute shaders.

  • With --wave64-emulate (the default on GPUs without 64-wide subgroups, so NVIDIA), this is already what the translator does (#152). s_barrier is scheduler state, finished waves stop counting, and the only OpControlBarrier sits at the top level of the round loop. That's essentially your fix.
  • The native path has the problem you describe. s_barrier becomes a plain OpControlBarrier wherever it sits. Minimal case: a 128-thread group where wave 1 jumps to s_endpgm and wave 0 runs s_barrier. It translates and passes spirv-val, but the barrier lands inside an OpSelectionMerge construct that wave 1 never enters:
OpSelectionMerge %g_end8 None
OpBranchConditional %g_is6 %g_do7 %g_end8
%g_do7 = OpLabel
...
OpControlBarrier %c2 %c2 %c108      ; not reached by every invocation of the workgroup

On RDNA hardware that's fine (an ended wave doesn't count), and an AMD driver likely compiles it to the same s_barrier. Vulkan doesn't promise it, though: a barrier in non-uniform control flow is undefined. Another driver, or a native run on a 32-wide GPU with emulation forced off, can hang. I haven't seen it hang yet (the only GPU I had was NVIDIA, which uses the emulated path).

A possible guard for native mode: when an s_barrier doesn't sit in workgroup-uniform control flow (not post-dominating the entry, or inside the dispatch loop), either refuse it (Unsupported) or switch that shader to the emulated scheduler. I can put that together if nobody else wants it.

AI-assisted (Claude Code): it read the code, built the repro shader and checked the SPIR-V.


TL;DR: you're right on all three.

  1. Keep the G-buffer targets 8-bit like the console, or the normals break. The real game path already does; it's mostly a trap for tests.
  2. Test textures need their real size in the descriptor, or everything except pixel (0,0) reads as black.
  3. The barrier fix you describe is already how the NVIDIA/Intel path works. The AMD path still has the problem you found: it probably works on AMD cards, but it isn't guaranteed. I suggested a fix and can do it.
Spaghetih12h ago

@IAmNotSiob I claimed SHADER-91 in #374 before I saw your offer here, sorry. If you'd rather do it, say so and I'll release it. Otherwise I'll start from your analysis: guard the native path when an s_barrier is not in workgroup-uniform control flow (switch that shader to the emulated scheduler, or refuse it), with the 128-thread repro as a test. On NVIDIA that should matter for wave32 mesh shaders, which don't go through --wave64-emulate.

AI-assisted (Claude Code).

IAmNotSiob12h ago

nah go for it, its yours :) the 128-thread repro from my comment should work as the test, ping me if u want a review