#654

perf(rg): batch file-size admission after merging discovery roots

Draftasafyishwants to merge intofrom

Batch file-size admission after merging rg discovery roots, using the existing bounded batches. Retain early output, diagnostics, ordering and finite size limits. Explicit files already admitted during operand processing are not checked again by the discovery filter; read-time resource checks remain.

Draft while #631 remains unmerged. This branch is rebased onto current main with the shared prerequisite fixes. Focused follow-up diff. The full GitHub diff includes the prerequisite; it should be rebased again once that prerequisite lands.

Validation: focused behavior tests, TypeScript, scoped Biome, knip and banned-pattern checks pass. No manual workflow rerun or review-bot request was made.

Activity

12
vercel[bot]10d ago

@asafyish is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

auto-maintain[bot]9d ago

🤖 auto-maintain review

Automated, advisory triage for @asafyish's PR. Facts below are read from the GitHub API.

CheckResult
Author's merged PRs (this repo)1
Account established✅ (age 5392d · 13 followers · 7 public repos)
Commits signed/verified⚠️ 0/16
Changeset included✅ (.changeset/rg-streaming.md, .changeset/streaming-pipelines.md, .changeset/tiny-files-search-together.md)

Review panel: 🟡 medium highest severity

just-bash maintainer code review: 🟡 medium

The streaming path introduces an unbounded accumulation case for quiet files-without-match searches.

  • packages/just-bash/src/commands/rg/rg-search.ts:1468 — Quiet `--files-without-match` pipelines append every nonmatching filename to `stdout` but never flush or bound the string. Since streaming discovery bypasses the prior file-array cap, `rg -Lq ... | ...` can exceed `maxStringLength`/`maxLiveBytes` without accounting. Track only the success state or stop after the first miss when quiet.

General code review: 🟡 medium

The discovery refactor introduces several observable rg compatibility regressions and unnecessarily rechecks explicit operands.

  • packages/just-bash/src/commands/rg/rg-search.ts:529 — Explicit file operands now pass through recursive glob/type/ignore filtering, so commands such as `rg -g '*.js' pattern /src/a.txt` incorrectly skip the explicitly requested file.
  • packages/just-bash/src/commands/rg/rg-search.ts:555 — All ordinary operand stat failures are silently discarded. Missing or inaccessible paths now produce no diagnostic and exit as a normal no-match instead of reporting the path error.
  • packages/just-bash/src/commands/rg/rg-search.ts:338 — Explicit `.` is no longer normalized to `./`, making `rg pattern .` emit paths like `file` instead of `./file`; the same regression affects `rg --files .`.
  • packages/just-bash/src/commands/rg/rg-search.ts:560 — With a finite size limit and any directory operand, the merged size filter stats explicit files a second time even though they were already admitted above, adding an unnecessary metadata round trip and allowing a transient second failure to drop them.

Adversarial security: 🟢 low

No actionable adversarial security issues found in the complete diff.

Adversarial security (second opinion): 🟡 medium

No backdoors, exfiltration, or sandbox-escape vectors found; the streaming-pipeline gate is conservative (static words only, no builtins/functions/redirections/fds/groupStdin), capability revocation and the defense-in-depth trust scope are preserved for the new `ctx.stdio` surface, pipe buffers and final output stay under the shared execution budget, and the rg size-admission move keeps explicit-operand filtering intact — one resource-limit divergence in streaming `seq` should be fixed or consciously accepted before merge.

  • packages/just-bash/src/commands/seq/seq.ts:198 — The new streaming `seq` branch caps iterations with `ctx.limits.maxLoopIterations` instead of the `maxIterations = min(maxLoopIterations, maxArrayElements)` value computed 45 lines above and still used by the buffered branch (line 223). A host that lowers `maxArrayElements` to bound `seq` silently loses that bound as soon as `seq` is in an eligible streaming pipeline: `seq 1 100000000 | cat` is throttled only by `maxLoopIterations` and downstream output accounting, while the identical buffered invocation aborts with `seq: iteration limit exceeded`. Use `maxIterations` in both paths so the limit is enforced consistently.

Standard Bash and host portability: 🟢 low

No actionable Bash or host-portability issues found.

Posted by auto-maintain. This automated code review is advisory; a human maintainer makes the call.

asafyish9d ago

Verified the real findings and fixed them. The #631 prerequisite now preserves current main’s explicit-file filtering, diagnostics and ./ prefix, and quiet --files-without-match stops on its first miss without accumulating filenames. This PR’s size filter now skips explicit files already admitted during operand processing. One regression makes a redundant metadata read fail and verifies rg --files --max-filesize 4 /src /explicit still lists both operands. Read-time resource checks remain.

The seq array-limit claim does not apply to the streaming representation: it does not construct the buffered array. maxLoopIterations still limits generation, and pipeline/output budgets still apply; imposing maxArrayElements on a representation with no array would undo the intended streaming behavior. No seq change is included.

Rebased onto current main and #631; the focused size tests, rg/pipeline suite, typecheck, formatting, knip and banned-pattern checks pass.

vercel-security-reviewer[bot]9d ago

This pull request is a draft

Merging is unavailable until it is marked ready for review.