#630

feat(grep): stream recursive discovery and per-file output

Draftasafyishwants to merge intofrom

Addresses #415. Depends on the streaming infrastructure in #453.

Draft until #453 lands. GitHub currently shows the dependency commits too; the grep-only comparison isolates the 6-file follow-up. Once #453 lands, rebase the follow-up commits onto main; no separate pipeline implementation is proposed here.

grep -rn MATCH /src | head -40 currently reads all 2,000 files in the regression fixture. Stream per-file output through the existing ctx.stdio API and discover recursive files lazily, so downstream closure stops both file reads and traversal. egrep and fgrep use the same implementation.

Keep operand/traversal order, duplicate operands, filename prefixes, context/count/list output, and diagnostics already encountered before a broken pipe. Both buffered and streaming execution retain bounded 50-file parallel reads. Results are emitted before discovering the next batch, preserving full-search throughput while stopping unneeded later batches. Stdin and each individual file still use the existing whole-input matcher; no filesystem API, regex semantics, or pipeline infrastructure changes.

Validation: 503 grep/pipe and comparison tests pass (8 existing skips). The early-stop regression fails on the base branch. A parallel-read regression also protects full-consuming pipelines from serialized remote file reads. Build, typecheck, formatting, banned-pattern checks and knip pass. Broader suite: 15,702 pass; four unrelated xz/tar tests also fail on the unchanged base in this local environment (98 existing skips).

Activity

15
vercel[bot]12d ago

@asafyish is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

vercel-security-reviewer[bot]12d ago
auto-maintain[bot]12d ago

🤖 auto-maintain review

Automated, advisory triage for @asafyish's PR. Facts below are read from the GitHub API.

CheckResult
Author's merged PRs (this repo)1
Account established✅ (age 5391d · 13 followers · 7 public repos)
Commits signed/verified⚠️ 0/10
Changeset included✅ (.changeset/grep-file-streaming.md, .changeset/streaming-pipelines.md)

Review panel: 🟡 medium highest severity

just-bash maintainer code review: 🟡 medium

The streaming cat path regresses mixed file/stdin pipelines by buffering stdin before earlier operands.

  • packages/just-bash/src/commands/cat/cat.ts:124 — Mixed file/stdin operands eagerly drain all stdin before processing operands. Pipelines such as `seq 1000000000 | cat /file - | head -1` hit the input limit instead of emitting `/file` and stopping upstream; `cat - /file | head` likewise cannot stream. Read/write each operand in order.

General code review: 🟢 low

No actionable correctness or maintainability issues found in the complete diff.

Adversarial security: 🟢 low

No actionable adversarial security issues found in the complete diff.

Adversarial security (second opinion): 🟢 low

Streaming pipeline and grep discovery changes are tightly gated, resource-bounded, and state-isolated; the bundled defense-in-depth change is a necessary hardening for the new concurrency, and no backdoor, exfiltration, or sandbox-escape vector was found.

Standard Bash and host portability: 🟢 low

No actionable Bash or host-portability issues found.

Posted by auto-maintain. This automated code review is advisory; a human maintainer makes the call.

asafyish11d ago

Verified and fixed the two ordering issues in 741413b6:

  • With /hit containing matches, seq 1000000000 | grep MATCH /hit - | head -n 1 now emits /hit:MATCH before attempting to drain stdin. The regression previously hit the input limit.
  • A first matching recursive file followed by empty directories now reaches head before searching for another file. The regression previously hit the traversal limit without emitting the first file.

The patch lazily reads stdin, skips unnecessary filename lookahead for recursive/no-filename output, and emits the first streaming file before filling subsequent batches. Ordinary full searches retain parallel batches of up to 50 files; mixed stdin operands are consumed in order. Two regressions added; the existing parallel-read regression still verifies concurrent reads after the initial output.

The comment saying the current code selects batch size 1 is stale: the reviewed head uses fileBatches(..., 50, ...). That read-ahead is intentional, not a reason to serialize all file searches.

Also carried the shared trust-scope fix from #453 in 8fa70e7e. 392 focused grep/pipeline tests pass (8 existing skips), along with type checking, formatting and banned-pattern checks. Private Slack-only review details remain inaccessible.

asafyish10d ago

Carried the shared result-getter protection verified on #642 in b1b385c0. This extends the existing trust-scope fix: command-result accounting and copying remain inside the same restricted callback, and the existing overlapping-stage regression now covers returned getters. Grep behavior is unchanged. All 38 focused pipeline tests and typecheck pass.

asafyish9d ago

Confirmed the mixed-operand cat finding. The shared foundation now processes operands in order: seq 1000000000 | cat /prefix - | head -n 1 emits prefix\n and stops without draining the producer. One regression covers this. Rebased onto current main and carried the shared result-accounting, UTF-8 failure-output and bounded pipe-read fixes. The grep follow-up remains isolated in the focused comparison in the PR description.

asafyish9d ago

Fixed the new comparison CI failure in 45432475. The test itself passed; CI failed because record mode changed the macOS grep diagnostic for repeated - operands to Linux/GNU grep’s empty stderr. Recorded the Linux result and locked that platform-specific fixture using the existing fixture mechanism. The existing comparison now explicitly checks stderr too. All five cases pass in playback and ordinary record mode, which leaves the corrected fixture unchanged. No production code change or workflow rerun request.

This pull request is a draft

Merging is unavailable until it is marked ready for review.