#3115

Skip finish passes over functions that cannot hold their nodes

Opencramforcewants to merge intofrom

What this does

finishModule runs about fifteen whole-program IR walks back to back, and most of them look for a rare node kind or library entry point that a given function does not hold at all. On pi's build that is 42K functions walked fifteen times to find nothing, and every walk that rewrites copied each node it passed through. This is WP2-7 of the pi build-speed plan.

Census. A FunctionCensus records, per function and in one scan, the expression kinds, statement kinds, libCall entry points and intrinsics of the body. Each finisher asks the census whether a function can hold what it looks for and skips the rest. The filters list every kind the finisher reacts to, including constructor values that arrive through dynFrom, and routeWrapperConsumers now tests for the two entry points and two intrinsics it routes instead of for any library call at all (which almost every function holds). The census is validated the way DispatchSites validates its summaries: same statement-list array, same length, same first and last statement. A pass that changes a node inside an unchanged list calls forget, and a pass that rewrites nodes in place clears the census. SCRIPTC_CHECK_FUNCTION_CENSUS=1 rescans and compares on every reuse.

Sharing. Inside the module finish (withStructureSharing), a rewrite that leaves a node's children unchanged returns the node itself instead of a copy, and mapOf, mapValues, mapKeyValues and mapCases return their input when no element changed. Outside that window every case still rebuilds its node, because lowering needs that: settleSharedHelperLocations retags shared helper nodes' locations in place. Visitation order is unchanged, so local ids, IR and LLVM are unchanged.

Stack frame. mapExprChildren and mapStmtChildren are on the native stack once per nesting level of every recursive rewrite, and an unoptimized build gives every local, comparison and conditional result its own stack slot. Each case therefore computes its children into a few function-level slots and asks a leaf predicate (kept, kept2 to kept4) whether they came back unchanged. Measured in the sanitized dev build of the self-hosting emitter:

mapExprChildrenstack slotsframe
main265about 34 KB
a name and a comparison per case (first version of this change)68087 KB (measured)
this PR277about 36 KB

The first version failed the sanitized emitter bootstrap's fixed point with RangeError: Maximum call stack size exceeded after 70 nested frames; on main that run already uses between 3.8 and 4.6 MB of its 6 MB of usable stack.

Proof

Everything below ran on 0e9c39e73, this commit on 714482d49. The branch head is the same commit rebased onto 7954cf5d4 without conflicts; the four PRs in between touch the type mapper, lint rules, generic class fields and CI.

Zero-cost gate

bin/validate --skip=lanes,bench,t262,san-bootstrap --gate-corpus=all against 714482d49:

validate fix-pi-speed-wp2-5-7 @ 0e9c39e73 vs 714482d49: PASS in 23.4m
jobs: gate-1..6, gate-fixture, gate-self-1..4 (11 jobs, 32 vCPU each)

PASS  zero-cost gate
      time       bench geomean +0.63% [95% CI -0.43%, +1.51%]: within tolerance
      time       self -1.60% [95% CI -2.63%, -0.26%]: within tolerance
      time       fixtures geomean -0.47% [95% CI -0.81%, -0.12%]: within tolerance
      self       candidate compiler source builds with the candidate (671.14s)
      zero-cost gate: PASS - 5176 identical, 96 both-fail over 8 shards

Every program, JavaScript corpus and npm suite included, is byte-identical in IR, LLVM and release executable. There are no differing rows and no runtime-only rows, and no --gate-allow pattern was used. Compiling the compiler with itself got 1.6% faster.

Bootstraps

All four run on this commit's tree in 32 vCPU / 64 GB sandboxes:

bootstrapresult
tests/harness/self-hosting-llvm-emitter.test.ts2 passed, fixed point verified
tests/harness/self-hosting-native-driver.test.ts1 passed (3297 s)
SCRIPTC_SAN=1 self-hosting-llvm-emitter.test.ts2 passed, fixed point verified (1992 s)
SCRIPTC_SAN=1 self-hosting-native-driver.test.ts1 passed (4360 s)

The sanitized emitter fixed point is the one the first version failed. The emitter built the way that test builds it (dev, ASan) also emits its own IR with a 6144 kB stack, the smallest of the limits tried at which main's does (both fail at 5120 kB).

Tests

  • pnpm vitest run packages/compiler: 544 files passed, 2 skipped; 5087 tests passed, 29 skipped, none failed. That includes the new function-census.test.ts.
  • pnpm lint: 0 warnings, 0 errors. pnpm format:check: clean. pnpm test:ts7 --baselines-only: every fixture has a baseline.

pi build

pi e770f5efe, entry packages/coding-agent/src/cli.ts, --npm-static (15 packages), --node-islands=auto, compiler run under Node from packages/cli/dist. Two sandboxes of 32 vCPU / 64 GB, both Intel(R) Xeon(R) Processor @ 2.90GHz; trees alternate on each sandbox.

sandboxruntreewall speak RSS GiBCPU s (user+sys)load sisland reload slower s
w59m12main 714482d49868.888.17963.8120.2110.3636.2
w59c12this PR899.015.031022.3124.2111.7660.7
w59c13this PR898.095.451012.9119.9109.5666.6
w59m13main926.477.131087.9125.4114.1685.2
w59c14this PR852.695.60951.9121.3106.4623.2
w59m14main878.026.97986.5121.6111.3643.5
wp27-pipm1main910.037.141030.7128.1114.0665.2
wp27-pipc1this PR886.615.43991.0123.7108.8651.5
wp27-pipm2main968.417.631095.6134.4125.0707.1
wp27-pipc2this PR987.205.431103.4144.6127.3713.5

(A tree copy and a 20 s TypeScript build ran beside c14, three minutes into it. The second sandbox slowed down over its session, load phase included; its runs of a third tree, not part of this PR, sat between the ones shown.)

meanwall speak RSS GiBCPU slower s
w59, main891.1 (869 - 926)7.43 (6.97 - 8.17)1012.7655.0
w59, this PR883.3 (853 - 899)5.36 (5.03 - 5.60)995.7650.2
wp27-pi, main939.2 (910 - 968)7.39 (7.14 - 7.63)1063.2686.2
wp27-pi, this PR936.9 (887 - 987)5.43 (5.43 - 5.43)1047.2682.5
  • Peak memory is the effect: -28% on w59 (7.43 to 5.36 GiB) and -27% on the second sandbox (7.39 to 5.43 GiB). Every run of this PR (5.03 - 5.60 GiB) is below every run of main (6.97 - 8.17 GiB): the finish passes no longer copy every node they pass through.
  • Build time on pi is unchanged within the noise. The means differ by -0.9% wall on w59 and -0.2% on the second sandbox, and runs of the same tree differ by 6% to 11%. The plan estimated 20 to 30 s for this package; pi's wall clock does not show it.
  • Diagnostics are identical. Every run reports 66 errors, and the diagnostics text of every run of this PR is byte-identical to main's (everything the build prints except its timing lines).

The compiler compiling itself, which the gate times in four base/candidate pairs, is 1.6% faster (95% CI -2.63% to -0.26%).

SCRIPTC_CHECK_FUNCTION_CENSUS=1 ran one full pi build on the previous revision of this branch (same census code): every reuse of a census rescanned the body, none had gone stale, and the diagnostics were unchanged.

🤖 Generated with Claude Code

Activity

11
vercel[bot]6d ago

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated
scriptc ReadyPreview, v0Nov 7, 2026 7:15am UTC
vercel[bot]reviewed6d ago
vercel[bot]reviewed6d ago
cramforcereviewed6d ago
cramforce6d ago

Brought up to date with main (head b082601c8), in three commits on top of 0bb484605, nothing rebased:

  • 4d68e50af merges main at f2b336d18. #3158 added the constCell expression; this branch had restructured everyExprChild and mapExprChildren, so the merged traverse.ts adds the case to each (the index is its one child; mapExprChildren returns the node itself when the index is kept).
  • 27b57f97c fixes the review thread on receiver-returns.ts: the pass forgets a dispatch helper's census when it rewrites a branch in place. A fluent JavaScript class called through an untyped value stopped the build under SCRIPTC_CHECK_FUNCTION_CENSUS=1 before; test in receiver-returns.test.ts.
  • b082601c8 merges main at 078af302a (#3154 touched the import line next to this branch's in lowerer.ts; both imports stay).

Proof on b082601c8 against main 078af302a, bin/validate --skip=lanes,bench,t262,san-bootstrap --gate-corpus=all: PASS, 5,232 programs identical (JavaScript corpus and npm suite included), 102 both-fail (they need a libnode pack), no differing and no runtime-only rows, no --gate-allow. Timing within tolerance: bench geomean -0.52% [-1.98%, +0.64%], self +0.93% [-0.92%, +1.64%], fixtures -2.47% [-5.95%, +0.97%]. The same gate on 27b57f97c against f2b336d18 also passed (5,215 identical). Focused: traversal, function-reference, census, receiver-return, constant-table and unit-journal tests (214 pass), pnpm lint, pnpm format:check, the literal-table and fluent corpus programs under the census check.

Not rerun for the merges: the four bootstraps and the pi pair in the description (they ran on 0e9c39e73). The merges add one traversal case and one import line; the gate's self row (the candidate's compiler source builds with the candidate) passed on both heads.

cramforce6d ago

One more check, prompted by the review thread: every corpus entry (5,256 programs, TypeScript and JavaScript) was lowered with SCRIPTC_CHECK_FUNCTION_CENSUS=1 on 27b57f97c (scriptc build <program> --emit=ir, frontend only). All 5,256 lowered; none reported reused census … disagrees with a fresh scan and none ended in an internal compiler error. So the dispatch-helper rewrite was the only pass in the corpus's reach that changed a body inside an unchanged statement list without forgetting it.

cramforce5d ago

Hold: a cost finding, not a correctness one. Found while validating this PR together with three others (2026-11-07). Its own claims held on the merged tree (pi's peak memory from 8.0–10.8 GB down to 6.8 GB; zero-cost gate, both lanes, both native bootstraps, census check clean over 5,490 programs).

What it costs: the compiler's own seed build. Four builds on one machine, each alone, in the order main / with this PR / with this PR / main: 1,037.8 s, 1,264.3 s, 1,157.7 s, 1,046.7 s. Both builds with the PR are above both builds of main: +16% at the mean, +10.6% between the closest two. Lowering time is equal; the native-objects stage goes from 146 s to 201 s with the same cores busy.

Why the gate did not show it: the gate's self timing compiles main's source with both compilers; the seed build compiles each tree's own source. This PR's source by itself adds 2,089 functions and 3.0% to the compiler's own LLVM text: about 576 lifted lambdas, 86 array loops, 53 union retags, 67 record width conversions and 29 record types. It is not the dynamic tier and not a dynamic view of the lowerer.

What it needs before it lands: the same saving with code that does not multiply the compiler's own output (the lifted lambdas are the largest part), with a seed-build pair in its proof.

🤖 Generated with Claude Code