Test262 survey (static-adapted-v5): per-chapter results on main vs integration branch

Summary

This is a full run of the pinned Test262 snapshot through the scriptc Test262 runner (tests/test262, profile static-adapted-v5). It covers main and an integration branch that combines main with the currently open PRs. Every admitted variant was compiled and executed on both. These are adapted-profile results, not whole-suite ECMAScript conformance. Pass counts describe the selected adapted profile only. Exclusions are runner capabilities, not scriptc implementation classifications.

Headline numbers for main:

• 37.5% of admitted variants pass (28,066 of 74,793), which is 27.3% of all 102,956 variants in the snapshot.
• 5,131 wrong results (6.9% of admitted): assertion failures and unexpected exceptions that are not refusals, plus 6 crashes. This is the real spec-divergence figure.
• 41,367 refusals (55.3% of admitted): explicit compile, runtime and harness refusals, plus 5,168 runner fail results that are refusals underneath (see "Runner issues" below).
• 229 other outcomes: 220 internal compiler errors, 7 build errors, 2 timeouts.

The runner reports 10,293 fail + 6 crash. About half of those fail results are refusals rather than wrong results; the classification is explained under Methodology.

On the integration branch, passes rise by 189 to 37.8%, but wrong results rise by 729 to 5,860. The extra wrong results come mostly from built-in members that main refuses with SC2020 and that now compile but read as undefined (details below).

Methodology

• Test262: tc39/test262 at 7ab7fafa0003f73fc85c1b95d88094d33f7eb8bd, fetched with pnpm test:test262:fetch (snapshot digest verified). 53,597 test files, which expand to 102,956 variants: sloppy and strict for scripts without a flag, plus module and raw variants.
• scriptc:
  • main at ab70d3d36 (2026-10-19).
  • The integration branch is main at bf2afee9f plus the open PRs. It lacks only main's most recent commit (#1432).
  • Both trees were built fresh, including the LLVM helper and the runtime pack.
• Runner: pnpm test:test262 --root <snapshot>, profile static-adapted-v5, default backend policy. The actual backend was LLVM for every variant that produced a binary (48,689 on main, 55,618 on the integration branch). No sanitizers.
• Timeouts: runner defaults, 120 s per compile and 10 s per program.
• Host: Linux x86_64, 8 vCPU / 16 GB, Node 24.15, LLVM 22.1.8.
• Execution:
  • Variants were split 20 ways with SCRIPTC_TEST_SHARD=i/20, using --workers 8 and --journal.
  • Some shards were interrupted by dropped remote sessions. Their remaining variants were rerun individually with the unmodified runSource/matchesParseNegative logic, so the union covers every admitted variant exactly once on both trees.
  • Nothing was sampled.
• Exclusions (28,163 variants, 27.4%) come from the runner's capability checks. They are the same on both trees:
  • harness-includes (15,338): the test includes a harness file other than compareArray.js/propertyHelper.js. The largest are temporalHelpers.js 5,618, testTypedArray.js 4,128, isConstructor.js 1,288, regExpUtils.js 1,172 and asyncHelpers.js 756.
  • harness:assert-surface 3,780.
  • host:eval 2,183, host:script-environment 1,548, host:Function 1,372 and host:arguments 1,268.
  • harness:propertyHelper-surface 643, execution:module 596, host:$262 540 and harness:Test262Error-surface 387.
  • Negative parse tests in module or raw variants 227; runtime negatives 66 and resolution negatives 34.
  • host:globalThis 144, host:agents 18, and smaller reasons.
• Outcome classification:
  • The runner's own categories (pass, compile refusal, runtime refusal, harness refusal, fail, build error, internal compiler error, crash, timeout) are reported unchanged.
  • The runner's fail category mixes real divergences with refusals. To separate them, every non-refusal-shaped fail was rerun with a scratch copy of the host harness that prints the error caught by assert.throws, the reason passed to $DONE, and the values compared by assert.sameValue/notSameValue. Runner semantics were otherwise unchanged; the rerun outcomes matched the original runs.
  • A fail counts as a refusal in three cases:
    1. Its stderr is an unlocated scriptc refusal (… is not supported yet, [SCnnnn] without a source location).
    2. The error caught by assert.throws or passed to $DONE is a scriptc refusal.
    3. It is ReferenceError: Temporal is not defined. On the integration branch, this replaces main's compile refusal "Cannot find name 'Temporal'".
  • Everything else in fail, plus crashes, is a wrong result.

Totals

Per-chapter results: requested chapters

Columns are variant counts. "Refused" covers all refusal kinds above; "wrong" is wrong results.

Per-chapter results: all top-level chapters with at least 100 admitted variants

The full per-chapter and second-level tables, with every outcome column, were generated with this survey (tables.md per branch, available on request). Second-level chapters with the most wrong results on main:

• language/statements/class 502 and language/expressions/class 456
• language/statements/for-await-of 202
• language/expressions/object 194
• built-ins/Object/getOwnPropertyDescriptor 172
• staging/sm 141
• language/statements/for-of 134
• language/expressions/async-generator 132
• language/expressions/compound-assignment 116
• built-ins/RegExp (top level) 106
• built-ins/Number/prototype/toString 84
• built-ins/Array/prototype/slice 80

Top wrong-result clusters (main), by root cause

These were grouped heuristically from test paths and diagnostic stderr. Counts are variants (sloppy and strict counted separately).

Unclassified long tail: 802 of 5131 wrong results.

The integration branch has the same clusters with similar counts. Two clusters grow: "missing member reads as undefined" goes from 205 to 596 and "built-in property descriptors read as undefined" goes from 380 to 582. Both trace to the change described in the next section.

Integration branch vs main

Per-variant transitions (coarse categories):

• Refusal → wrong (769 variants).
  • On main, these variants hit the SC2020 fence "'X' is part of the standard library types but has no scriptc lowering yet". The most common subjects are Object.getOwnPropertyDescriptor (174), Date.prototype (134), RegExp.prototype (54), Map.prototype (30), Object.defineProperty (26), Set.prototype, WeakMap.prototype, WeakSet.prototype, Object.isExtensible/isSealed/isFrozen and Promise.prototype.
  • On the integration branch these compile, but the value reads as undefined. Tests then fail with "Cannot convert undefined or null to object" or "Cannot read properties of undefined".
  • Examples: annexB/built-ins/Date/prototype/getYear/B.2.4.js, built-ins/Map/prototype/size/length.js, built-ins/Map/properties-of-the-map-prototype-object.js.
  • In these cases an explicit refusal has become a silent divergence.
• Refusal → pass (169) and wrong → pass (34). These are concentrated in built-ins/Date (+70 passes), built-ins/WeakMap (+24), language/arguments-object (+18), staging (+21) and built-ins/RegExp (+12). Examples: built-ins/Date/name.js, built-ins/Date/parse/without-utc-offset.js, built-ins/Array/prototype/exotic-array.js.
• Pass → wrong (10). Examples: built-ins/Promise/resolve/S25.4.4.5_A1.1_T1.js and built-ins/RegExp/escape/is-function.js.
• Pass → refusal (4). built-ins/Array/from/source-object-iterator-2.js and built-ins/Array/from/source-object-missing.js now hit an unlocated refusal.
• Reclassified refusals.
  • 5,736 Temporal variants move from compile refusal ("Cannot find name 'Temporal'") to a runtime ReferenceError.
  • 1,166 variants move from compile refusal to a located runtime fence.

Top refusal messages (main)

More detail on the largest refusals:

• "Cannot find name" (11,411). Mostly Temporal (7,166). It also covers 1,199 negative parse tests where $DONOTEVALUATE is reported because the expected early error was not raised, and tests that deliberately reference undeclared identifiers (x, w, unresolvableReference, …).
• SC2020 (7,985). Led by Object.defineProperty (627), new String (564), Object.getOwnPropertyDescriptor (324), new Number (316), new Intl.NumberFormat (286) and new Boolean (261).

Runner issues found

1. fail results whose stderr is a refusal (1,918 on main, 2,112 on the integration branch).
   • The runner recognizes a runtime refusal only when stderr is exactly Uncaught Error: … [SCnnnn at <file>:<line>].
   • Unlocated fences are classified as fail. Examples: Object.defineProperty on an array index or length is not supported yet, Object.getPrototypeOf on this value is not supported yet, … [SC2020] without a location, and TypeError: scriptc: Object.prototype reflection and mutation are not supported yet.
   • These should be classified as runtime refusals.
2. Refusals caught by assert.throws become assertion failures (2,057 variants on main).
   • When the callback hits a scriptc refusal, the host assert.throws sees an Error whose constructor differs from the expected one. It then reports "Expected exception".
   • The host harness should detect refusal errors and exit with a refusal status, as it already does for reference assertions.
3. The async-completion cluster is mostly refusals, not a harness artifact.
   • "SCRIPTC_TEST262_HARNESS: async completion failed" accounts for 1,750 variants on main.
   • 1,193 (68%) are refusals passed to $DONE as the rejection reason, 1,187 of them located SC fences. Examples: yield* in async generators (SC1071), for await over unsupported sources (SC1070), and Promise.all over unsupported argument shapes (SC2020).
   • The remaining 557 are real assertion failures inside async tests, largely the same function-name and destructuring root causes listed above.
   • No double $DONE calls were observed.
   • $DONE should print the reason and classify refusal-shaped reasons as runtime refusals.
4. Missed early errors are reported as compile refusals.
   • About 1,199 admitted negative parse tests fail with "Cannot find name '$DONOTEVALUATE'". scriptc did not report the expected early error, and the type checker then flagged the harness sentinel.
   • These are accepted invalid programs, closer to wrong results than refusals. They deserve their own outcome.
5. The caught error is not reported.
   • assert.throws failures print only "Expected exception", and $DONE failures print only "async completion failed". The actual error or rejection reason is dropped.
   • Printing it would make fail triage possible without a diagnostic rerun.

🤖 Generated with Claude Code