#294

@vgpu/wgsl: full Unicode (XID) identifier support in scanner/printer

Openmatiasngfopened 3mo ago

Context

WGSL identifiers may use Unicode (XID_Start/XID_Continue), but @vgpu/wgsl's scanner and token printer are ASCII-only. A fuzzing campaign (two independent lanes, ~1,500 cases) found that identifiers whose first character is non-ASCII corrupt minified output silently (let Ω = 1.0; → letΩ — keyword fused with the identifier; fails at CreateShaderModule with no resolve-time diagnostic), and non-ASCII identifiers in declaration positions produce misleading diagnostics (Expected identifier, bogus unknown-type errors). ASCII-first identifiers with non-ASCII continuation (café) happen to survive by token re-joining.

Current state

Rather than shipping half-broken Unicode support, @vgpu/wgsl now rejects identifiers that start with a non-ASCII character loudly at scan time (VGPU-WGSL-IDENT-NONASCII), pointing at this issue. All processing paths are protected, including bundler loaders (which run with validate: false and are not covered by device validation).

This issue

Implement full WGSL-conformant Unicode identifier support:

  • Scanner: identifier start/continue per XID_Start/XID_Continue + _ (JS \p{ID_Start}/\p{ID_Continue} property escapes are close; check WGSL spec deltas).
  • Token printer: isIdentStart/isIdentContinue separator predicates must agree with the scanner exactly.
  • Mangler/DCE/scope-walker: verify identifier-class assumptions downstream (the scope graph consumes scanner tokens).
  • Perf: keep an ASCII fast path — the scanner is per-character and perf-sensitive.
  • Tests: the fuzzing harnesses' unicode batteries (currently red) should flip green; add printer round-trip and device-differential cases for Greek/Cyrillic/CJK/combining-mark identifiers.

Prioritize on demand — no known user has requested non-ASCII shader identifiers; the loud rejection makes the limitation honest in the meantime.

Activity

0

No activity yet.