#625

rg and grep run RE2 on every line for alternations and -i, so a 20 MB search takes 4–9 s

Openasafyishopened 14d ago

just-bash version

3.6.0 (same on main at 9ab0e0d)

Reproduction

import { Bash } from "just-bash";

// About 20 MB of code-like text in 1,000 files.
const files = {};
for (let i = 0; i < 1000; i++) {
  const lines = [];
  for (let j = 0; j < 200; j++) {
    lines.push(`    if (record${j}.Status__c == 'Open' && record${j}.Amount__c > ${i * j}) { update record${j}; } // line ${j}`);
  }
  files[`/w/classes/File${i}.cls`] = `${lines.join("\n")}\n`;
}
const bash = new Bash({ files, cwd: "/w" });

for (const command of [
  'rg -c "Target_Date__c|VP_MC__c" classes',
  'rg -c "Target_Date__c" classes',
  'rg -c -i "target_date__c" classes',
  'rg -c "inactive|product.*update" classes',
]) {
  const start = performance.now();
  const result = await bash.exec(command);
  console.log(`${Math.round(performance.now() - start)} ms  exit ${result.exitCode}  ${command}`);
}

How you run it

node repro.mjs

Expected behavior

None of the files contain these words, so every search comes back empty. ripgrep 15.2.0 runs each of them over the same tree in about 30 ms, and just-bash already handles the plain literal in about 300 ms. I'd expect the other three to take about as long.

Actual behavior

4180 ms  exit 1  rg -c "Target_Date__c|VP_MC__c" classes
308 ms  exit 1  rg -c "Target_Date__c" classes
7130 ms  exit 1  rg -c -i "target_date__c" classes
9374 ms  exit 1  rg -c "inactive|product.*update" classes

Every line of every file goes through re2js (searchContent runs regex.exec per line), which gets through this kind of text at a few MB/s. A single case-sensitive literal is fast because re2js finds it by its prefix. An alternation or -i (see #435) loses that.

The search engine already has a substring preFilter that would skip almost all of these lines, but:

  • buildRegex only builds one when the whole pattern is plain literals joined by |, so product.*update gets none.
  • rg never passes it to searchContent, so even Target_Date__c|VP_MC__c doesn't use it. grep does.

Every match of product.*update contains both product and update, so a file or line without them can be skipped before RE2 sees it. I have a patch that works out those literals for any pattern and uses them in rg as well. I'll open a PR.

For context: we use just-bash as the shell of an LLM agent over a 120 MB Salesforce metadata checkout. Its recursive rg searches took one to ten minutes of CPU each, and a few at once starved the host.

Environment

  • Runtime (Node, Bun, browser, CLI): Node.js
  • Runtime version: 26.8.2
  • Operating system: macOS 15.7 (Apple Silicon); same behavior on Linux in production
  • Bundler or framework, when relevant: none

Activity

1
asafyish13d ago

The exact-literal-alternative follow-up is now draft PR #629. It avoids RE2 on a line where a whole literal alternative already proves the match, while retaining RE2 for offsets and replacements. This is useful for long/minified lines that survive #626's required-literal prefilter.

It depends on #626; the PR links the isolated follow-up diff and will stay draft until that prerequisite lands. Validation: 89 focused search tests, TypeScript, and banned-pattern checks pass.