#358

Translator: v_cvt_pk_u8_f32 rounds to nearest even; KytyPS5 truncates. Which matches RDNA2 hardware?

OpenDiazE117opened 15h ago

tools/rdna2spirv.py translates v_cvt_pk_u8_f32 as clamp to [0, 255], round to nearest even, convert, then insert the byte (FClamp → RoundEven → OpConvertFToU). That rule has been there since the initial commit. It has no GPU test, no Mesa/LLVM reference and no row in the docs/design.md contract table. KytyPS5's RDNA2 recompiler does the opposite: it truncates toward zero, clamps, and sends NaN to 0. One of them disagrees with the hardware by 1 for every fraction ≥ 0.5. The instruction is the usual way to pack 8-bit colour channels, so the error would show up as off-by-one colour.

Measured (Leonida's SPIR-V, NVIDIA)

The probe below converts each value into byte 0 of 0x11223344. Translated on main 8bd0087 without emulation, spirv-val clean, vk_compute_test <spv> 1 dump, three identical runs. NVIDIA GeForce RTX 4070 SUPER, driver 610.88, Vulkan SDK 1.4.363.0:

inputLeonida (measured)KytyPS5 (its own test, CvtPkU8F32PacksSelectedByte)
12.750x1122330d (13)12 ("truncates")
13.50x1122330e (14)13 (truncation)
12.50x1122330c (12)12
254.50x112233fe (254)254
0.50x11223300 (0)0
300.00x112233ff (255)255
-1.00x11223300 (0)0
NaN (0x7fc00000)0x11223300 (0)0

The NaN result is 0 here only by chance. FClamp with a NaN operand is undefined in GLSL.std.450, and so is OpConvertFToU of NaN. Another driver may return something else.

Kyty's test only checks Kyty's own translator, so it is not hardware evidence either. I could not settle the question from public sources. Mesa's GitLab refuses automated fetches. LLVM has no constant-folding rule for llvm.amdgcn.cvt.pk.u8.f32, and I could not fetch the RDNA3 ISA PDF to read its pseudocode.

What would settle it

The instruction needs to run natively on an AMD GPU. Running Leonida's SPIR-V there would only measure the translation again. Two ways:

  • Read the RDNA2/RDNA3 ISA pseudocode for V_CVT_PK_U8_F32 and the helper it uses (f32_to_u8 or similar), or grep a local Mesa tree for how ACO uses v_cvt_pk_u8_f32, e.g. whether it rounds the value first.
  • Run it with HIP inline asm on any RDNA GPU. This is an untested sketch:
__global__ void k(const float* in, unsigned* out) {
    unsigned r, base = 0x11223344u;
    asm volatile("v_cvt_pk_u8_f32 %0, %1, 0, %2" : "=v"(r) : "v"(in[threadIdx.x]), "v"(base));
    out[threadIdx.x] = r;
}
// in = {12.75f, 300.0f, -1.0f, NaN, 12.5f, 13.5f, 0.5f, 254.5f}

If the hardware truncates, the fix is to replace RoundEven with a truncating conversion and to select 0 explicitly for NaN. Either way the probe can become runtime/test/shaders/cs_cvt_pk_u8.asm with the hardware's words, plus a contract-table row.

Probe (hand-written, no game data)
; v_cvt_pk_u8_f32 on 12.75, 300.0, -1.0, NaN, 12.5, 13.5, 0.5, 254.5 into byte 0 of 0x11223344
v_mbcnt_lo_u32_b32 v1, -1, 0
v_mov_b32_e32 v4, 0x11223344
v_mov_b32_e32 v0, 0x414c0000
v_cvt_pk_u8_f32 v10, v0, 0, v4
v_mov_b32_e32 v0, 0x43960000
v_cvt_pk_u8_f32 v11, v0, 0, v4
v_mov_b32_e32 v0, 0xbf800000
v_cvt_pk_u8_f32 v12, v0, 0, v4
v_mov_b32_e32 v0, 0x7fc00000
v_cvt_pk_u8_f32 v13, v0, 0, v4
v_mov_b32_e32 v0, 0x41480000
v_cvt_pk_u8_f32 v14, v0, 0, v4
v_mov_b32_e32 v0, 0x41580000
v_cvt_pk_u8_f32 v15, v0, 0, v4
v_mov_b32_e32 v0, 0x3f000000
v_cvt_pk_u8_f32 v16, v0, 0, v4
v_mov_b32_e32 v0, 0x437e8000
v_cvt_pk_u8_f32 v17, v0, 0, v4
v_cmpx_eq_u32_e32 0, v1
v_mov_b32_e32 v21, 0
buffer_store_dword v10, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 4
buffer_store_dword v11, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 8
buffer_store_dword v12, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 12
buffer_store_dword v13, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 16
buffer_store_dword v14, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 20
buffer_store_dword v15, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 24
buffer_store_dword v16, v21, s[0:3], 0 offen
v_mov_b32_e32 v21, 28
buffer_store_dword v17, v21, s[0:3], 0 offen
s_endpgm

It is straight-line code, so it translates without llvm-mc encodings (as in tests/semantic/regen_fixtures.py).

AI-assisted (Claude Code): it found the difference while comparing the two translators, ran the probe and wrote this issue.

Activity

6
YaBulochka14h ago

Native hardware measurement, not SPIR-V: the instruction executed as raw RDNA on an AMD GPU gives round-to-nearest-even, then clamp. So the current rdna2spirv.py rule is right and KytyPS5's truncation is wrong.

Setup: AMD Radeon RX 7600 (Navi 33, gfx1102, RDNA3), the hardware oracle from americas-recomp (tools/hw-oracle, HSA on /dev/kfd), 32 lanes, all float modes 0, body:

v_cvt_pk_u8_f32 v10, v4, v5, v6

with v5 = 0x11223344, v6 = 0 (byte 0), v4 = the input. Output is v10.

inputhardwarenotes
12.7513nearest, not truncation
13.514
12.512tie to even
254.5254tie to even
0.50tie to even
1.5 / 2.5 / 3.52 / 2 / 4tie to even each time
127.5 / 128.5128 / 128tie to even
200.5 / 201.5 / 202.5 / 203.5200 / 202 / 202 / 204tie to even
254.9255nearest
0.9991
300 / 65535 / 1e9255clamp
-1 / -1e90clamp
NaN0
-0.00

So the hardware result is exactly clamp(roundToNearestEven(x), 0, 255), with NaN giving 0. Leonida's FClamp -> RoundEven -> OpConvertFToU matches on every row; Kyty's truncate-and-clamp disagrees by 1 for every fraction >= 0.5 that is not a tie, and truncate is what its own test asserts.

Caveat: this is RDNA3 (gfx1102), not RDNA2 (gfx10.3). I do not have RDNA2 hardware. The tie cases are the discriminating ones here and they are RNE, which is also what the RDNA ISA's v_cvt_pk_u8_f32 is documented to do on the generation I can run; if someone has a gfx10.3 card, the same probe under tools/hw-oracle settles it in a minute.

One separate point from the same run: NaN gave 0, but the translation relies on undefined behaviour for it (FClamp and OpConvertFToU of NaN). If exactness for NaN matters, the translator could select 0 explicitly instead of inheriting the driver's choice.

BetterJdue14h ago

Measured natively on RDNA2 (SHADER-48 in #374). Short answer: round to nearest even is right for Leonida's contract; truncation is wrong. The one real gap is NaN.

Setup: AMD Radeon RX 6700 (gfx1031, RDNA2), AMD Windows driver 32.0.21045.29003, Windows 11, run through the HIP runtime of AnyPS5's hardware oracle (assembles and dispatches the raw instruction; no translator involved). Body: v_cvt_pk_u8_f32 v10, v4, v5, v6. 2,890 rows (k/8 for k = -16..2099, every n+0.5 tie for n = 0..256 and its two f32 neighbours, ±0, denormals, ±inf, quiet/signaling/negative NaNs, ±1e10, ±FLT_MAX, plus byte-select rows) × IEEE 0/1 × f32 denorm modes 0–3 × f32 round modes 0–3 = 92,480 results. For reference, gfx1031 and gfx1013 assemble this instruction to the same encoding (D55E000A 041A0B04); the PS5 itself is not measured.

Result: every one of the 92,480 results matches this rule:

  • src0 is rounded in the MODE register's f32 round mode, then clamped to 0..255. With the default mode (round to nearest even) that is exactly FClamp + RoundEven.
  • Every NaN (quiet, signaling, negative) gives 0. ±inf gives 255 / 0.
  • A denormal src0 is flushed when the f32 denormal field's input bit is clear (modes 0 and 2). It only shows in round toward +inf: the smallest positive denormal gives 1 in modes 1 and 3 and 0 in modes 0 and 2.
  • IEEE mode changes nothing.
  • The byte index is src1[1:0]; higher bits are ignored (src1 = 4, 0x100, 0xffffffff behave as 0, 0, 3). The other bytes of src2 are kept.
src0nearest eventoward +inftoward -inftoward zero
0.50100
1.52211
2.52322
12.7513131212
13.514141313
254.5254255254254
255.5255255255255
-0.750000
300.0255255255255
NaN0000

Against the three models over all 92,480 results:

ModelMismatches, default round modeMismatches, directed round modes
Leonida today (FClamp, RoundEven)0 (NaN rows excluded, see below)1,152–1,203 per mode
KytyPS5 (truncate, NaN 0)1,203 per run0 toward -inf and toward zero, 2,355–2,359 toward +inf
Round mode from MODE, flush per denorm mode, NaN 000

The directed-mode mismatches don't matter for Leonida today: docs/design.md already lists the MODE register as ignored. So RoundEven is the right choice. Switching to truncation would make the default mode wrong on 1,203 of 2,890 rows per run, including 12.75 → 12 and 13.5 → 13.

The gap is NaN. Hardware gives 0. Leonida gets that only by luck, because FClamp leaves a NaN x undefined. NClamp is defined through NMin/NMax, so NClamp(NaN, 0, 255) is 0, which is exactly the hardware result. That is a one-word change in tools/rdna2spirv.py (FClamp → NClamp in the v_cvt_pk_u8_f32 branch). A GPU fixture can pin the rows above, as you suggested with runtime/test/shaders/cs_cvt_pk_u8.asm.

Independent corroboration: AnyPS5 measured the same rule on an RX 6800 (gfx1030, RDNA2) over 2,133 sources (docs/dev/debt/shader.md, entry v_cvt_pk_u8_f32).

I can open the NClamp change with the fixture if that's wanted (I'll claim it in #374 first). The full measurement JSON (inputs, every output word, all 32 runs) is available on request.

AI-assisted (Claude); the measurement ran on my GPU.

aslan214214h ago

Measured natively on RDNA2: v_cvt_pk_u8_f32 rounds by the shader's rounding mode. In the default mode that is round to nearest even, as the translator does.

Hardware: the integrated GPU of a Ryzen 9 7950X3D (Raphael, RDNA2, 1002:164e), Linux, amdgpu.

Method: the probe from this issue, assembled with llvm-mc and run as a raw compute kernel through libdrm_amdgpu. The command sequence is libdrm's own dispatch test for this GPU generation, and its memset shader ran first as a check.

Default rounding mode

inputresultbyte
12.750x1122330d13
13.50x1122330e14
12.50x1122330c12
254.50x112233fe254
0.50x112233000
300.00x112233ff255
-1.00x112233000
NaN0x112233000
  • 28 more inputs agree: ties go to even (1.5 → 2, 2.5 → 2, 3.5 → 4), values just off a tie round the right way, and ±infinity clamp.
  • Quiet, negative and signalling NaNs all give 0.
  • The byte select works for all four positions and keeps the other bytes.

Other rounding modes (FLOAT_MODE in COMPUTE_PGM_RSRC1; a v_add_f32 control confirms the mode took effect)

mode12.7513.512.250.5254.52.50.75
nearest even131412025421
towards +infinity131413125531
towards -infinity121312025420
towards zero121312025420

So each translation is right for one mode: nearest even in the default mode, truncation under round-towards-zero. What stays open is which mode the game's shaders run in.

For the translator

  • The rounding rule is confirmed for the default mode.
  • NaN gives 0 on hardware. FClamp leaves that undefined; NClamp would make it 0.

I can send a PR with a contract-table row, a GPU test with these words, the NClamp change and the native runner (Linux, libdrm), if that is wanted.

AI-assisted (Claude Code, Claude Fable 5.2).

ItsNotRexed14h ago

Good question. The translator follows the RDNA2 ISA reference, whose pseudocode converts with round-to-nearest-even before clamping to 0..255. Another emulator truncating doesn't prove the hardware does. This needs a measurement on real RDNA2 (or RDNA1: the conversion is the same) with values exactly halfway between integers, e.g. 0.5, 1.5, 2.5, 254.5, plus -0.5 and 255.5 for the clamp. A small GPU test in runtime/test/shaders that runs on an AMD card would settle it; please claim it in #374. If the hardware truncates, the fix is one line in the translator.

DiazE11710h ago

Thanks @YaBulochka, @BetterJdue and @aslan2142: three native measurements (RDNA2 and RDNA3) agree, so the question is answered. The hardware rounds to nearest even and then clamps, as the translator does, and KytyPS5's truncation is wrong. The remaining gap is NaN → 0. @BetterJdue, please go ahead with the NClamp change and its GPU fixture (claim it in #634). If your PR says Fixes #358, this issue closes when it merges.

AI-assisted (Claude Code).

DiazE1179h ago

Correction to my last comment: the NClamp fix already has a PR, #409 by @JustDownTV (SHADER-48, Fixes #358), opened before my comment. @BetterJdue, no need to open a second one. Your RX 6700 rows would make a good check of #409 once it's rebased (it conflicts with main right now).

AI-assisted (Claude Code).