Addresses #415. Depends on the streaming infrastructure in #453.
Draft until #453 lands. GitHub currently shows the dependency commits too; the grep-only comparison isolates the 6-file follow-up. Once #453 lands, rebase the follow-up commits onto main; no separate pipeline implementation is proposed here.
grep -rn MATCH /src | head -40 currently reads all 2,000 files in the regression fixture. Stream per-file output through the existing ctx.stdio API and discover recursive files lazily, so downstream closure stops both file reads and traversal. egrep and fgrep use the same implementation.
Keep operand/traversal order, duplicate operands, filename prefixes, context/count/list output, and diagnostics already encountered before a broken pipe. Both buffered and streaming execution retain bounded 50-file parallel reads. Results are emitted before discovering the next batch, preserving full-search throughput while stopping unneeded later batches. Stdin and each individual file still use the existing whole-input matcher; no filesystem API, regex semantics, or pipeline infrastructure changes.
Validation: 503 grep/pipe and comparison tests pass (8 existing skips). The early-stop regression fails on the base branch. A parallel-read regression also protects full-consuming pipelines from serialized remote file reads. Build, typecheck, formatting, banned-pattern checks and knip pass. Broader suite: 15,702 pass; four unrelated xz/tar tests also fail on the unchanged base in this local environment (98 existing skips).
Activity
15🤖 auto-maintain review
Automated, advisory triage for @asafyish's PR. Facts below are read from the GitHub API.
| Check | Result |
|---|---|
| Author's merged PRs (this repo) | 1 |
| Account established | ✅ (age 5391d · 13 followers · 7 public repos) |
| Commits signed/verified | ⚠️ 0/10 |
| Changeset included | ✅ (.changeset/grep-file-streaming.md, .changeset/streaming-pipelines.md) |
Review panel: 🟡 medium highest severity
just-bash maintainer code review: 🟡 medium
The streaming cat path regresses mixed file/stdin pipelines by buffering stdin before earlier operands.
- packages/just-bash/src/commands/cat/cat.ts:124 — Mixed file/stdin operands eagerly drain all stdin before processing operands. Pipelines such as `seq 1000000000 | cat /file - | head -1` hit the input limit instead of emitting `/file` and stopping upstream; `cat - /file | head` likewise cannot stream. Read/write each operand in order.
General code review: 🟢 low
No actionable correctness or maintainability issues found in the complete diff.
Adversarial security: 🟢 low
No actionable adversarial security issues found in the complete diff.
Adversarial security (second opinion): 🟢 low
Streaming pipeline and grep discovery changes are tightly gated, resource-bounded, and state-isolated; the bundled defense-in-depth change is a necessary hardening for the new concurrency, and no backdoor, exfiltration, or sandbox-escape vector was found.
Standard Bash and host portability: 🟢 low
No actionable Bash or host-portability issues found.
Posted by auto-maintain. This automated code review is advisory; a human maintainer makes the call.
Verified and fixed the two ordering issues in 741413b6:
- With
/hitcontaining matches,seq 1000000000 | grep MATCH /hit - | head -n 1now emits/hit:MATCHbefore attempting to drain stdin. The regression previously hit the input limit. - A first matching recursive file followed by empty directories now reaches
headbefore searching for another file. The regression previously hit the traversal limit without emitting the first file.
The patch lazily reads stdin, skips unnecessary filename lookahead for recursive/no-filename output, and emits the first streaming file before filling subsequent batches. Ordinary full searches retain parallel batches of up to 50 files; mixed stdin operands are consumed in order. Two regressions added; the existing parallel-read regression still verifies concurrent reads after the initial output.
The comment saying the current code selects batch size 1 is stale: the reviewed head uses fileBatches(..., 50, ...). That read-ahead is intentional, not a reason to serialize all file searches.
Also carried the shared trust-scope fix from #453 in 8fa70e7e. 392 focused grep/pipeline tests pass (8 existing skips), along with type checking, formatting and banned-pattern checks. Private Slack-only review details remain inaccessible.
Carried the shared result-getter protection verified on #642 in b1b385c0. This extends the existing trust-scope fix: command-result accounting and copying remain inside the same restricted callback, and the existing overlapping-stage regression now covers returned getters. Grep behavior is unchanged. All 38 focused pipeline tests and typecheck pass.
Confirmed the mixed-operand cat finding. The shared foundation now processes operands in order: seq 1000000000 | cat /prefix - | head -n 1 emits prefix\n and stops without draining the producer. One regression covers this. Rebased onto current main and carried the shared result-accounting, UTF-8 failure-output and bounded pipe-read fixes. The grep follow-up remains isolated in the focused comparison in the PR description.
Fixed the new comparison CI failure in 45432475. The test itself passed; CI failed because record mode changed the macOS grep diagnostic for repeated - operands to Linux/GNU grep’s empty stderr. Recorded the Linux result and locked that platform-specific fixture using the existing fixture mechanism. The existing comparison now explicitly checks stderr too. All five cases pass in playback and ordinary record mode, which leaves the corrected fixture unchanged. No production code change or workflow rerun request.
This pull request is a draft
Merging is unavailable until it is marked ready for review.
@asafyish is attempting to deploy a commit to the Vercel Labs Team on Vercel.
A member of the Team first needs to authorize it.