#483
Pull Requests/vxcontrol/pentagi/#483

feat(tools): add Outcrawl web search provider

Openheydryftwants to merge intofrom
1 commit24 files changed+715-17

Description of the Change

Problem

The unified web_search tool can fall back across Google, DuckDuckGo, Tavily, Firecrawl, Traversaal, Perplexity, Searxng and Sploitus. There is no option for Outcrawl, a web search and scrape API for AI agents that returns the markdown of each result page in the same search call.

Solution

Adds Outcrawl as one more search engine, following the Firecrawl engine (#373) file for file:

  • backend/pkg/tools/searchers/outcrawl.go: POST https://api.outcrawl.ai/v1/search with {"query", "limit", "scrape": true, "formats": ["markdown"]} and a Bearer key. Hits are rendered in the same # Links / ### Short content / ### Raw content layout as Firecrawl, and page markdown goes through the shared summarizer when one is set (one-based source ids aligned with the links, markdown capped at maxRawContentLength per page). HTTP statuses map to Fatal / Retryable like the other engines, and the message field of Outcrawl's {code, message} error body is kept in the error text.
  • backend/pkg/tools/searchers/outcrawl_test.go: request shape through the existing MITM test proxy, status classification, summarizer and raw-content paths, the per-page cap, IsAvailable. errors_test.go gets the outcrawl refuses row.
  • backend/migrations/sql/20261101_120000_add_outcrawl_search_type.sql: rebuilds SEARCHENGINE_TYPE with outcrawl after firecrawl; the down migration remaps logged outcrawl rows to firecrawl before narrowing.
  • backend/pkg/database/models.go: regenerated with sqlc generate (sqlc 1.27.0, against a PostgreSQL with all migrations applied); the only change is SearchengineTypeOutcrawl.
  • Wiring: config.go (OUTCRAWL_API_KEY, plus the secret-masking list), web_search.go (engine constant, registered in buildSearchEngines and in the link-engine set, placed right after Firecrawl in every mode's order), registry.go, server/models/searchlogs.go, ftester (args, executor, mock), installer wizard (key field, status line, env var plumbing, locale strings), docker-compose.yml, .env.example, README, backend/docs/config.md, searchers/README.md, searcher.tmpl, CLAUDE.md and the installation guide.

Only an API key is configurable; Outcrawl is a hosted API, so there is no URL override like FIRECRAWL_API_URL.

Type of Change

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • 📚 Documentation update
  • 🔧 Configuration change
  • 🧪 Test update
  • 🛡️ Security update

Areas Affected

  • Core Services (Frontend UI/Backend API)
  • AI Agents (Researcher/Developer/Executor)
  • Security Tools Integration
  • Memory System (Vector Store/Knowledge Base)
  • Monitoring Stack (Grafana/OpenTelemetry)
  • Analytics Platform (Langfuse)
  • External Integrations (LLM/Search APIs)
  • Documentation
  • Infrastructure/DevOps

Testing and Verification

Test Configuration

PentAGI Version: main @ cffed94 + this branch
Docker Version: 29.5.2 (PostgreSQL pgvector/pgvector:pg16 for the migration and sqlc)
Host OS: macOS (arm64), Go 1.27.3
LLM Provider: n/a (searcher tested directly, summarizer not set)
Enabled Features: none

Test Steps

  1. cd backend && go build ./...: OK.
  2. CGO_ENABLED=0 go test ./pkg/tools/searchers ./pkg/config ./pkg/server/models ./cmd/ftester/worker: all ok, including the new TestOutcrawl_* tests. (CGO_ENABLED=0 only because github.com/shoenig/go-m1cpu segfaults in its cgo init on this macOS build; that happens on main too.)
  3. CGO_ENABLED=0 go test ./pkg/tools ./cmd/installer/...: a few tests fail on this machine (file-system and compose-image checks in installer, two pkg/tools args tests). I ran the same packages on main with the change stashed and the list of failing tests is identical, so they are unrelated to this change.
  4. gofmt -l on every touched Go file: clean. go vet ./pkg/tools/... ./pkg/config/... ./pkg/server/models/... ./cmd/ftester/... ./cmd/installer/wizard/...: clean.
  5. Migration: goose up through 20261101_120000, then goose down and goose up again on PostgreSQL 16: OK.
  6. Live call: a throwaway test (not committed) ran NewOutcrawl(cfg, nil).Handle(ctx, Request{Query: "OWASP Top 10 2021 A03 injection", MaxResults: 2}) with a real key. It returned 11,619 characters: the # Links section with top10.owasp.org/2021/A03_2021-Injection and top10.owasp.org/2021, their snippets, and the raw markdown of both pages.

Test Results

All of the above passed as described.

Security Considerations

The API key is read from OUTCRAWL_API_KEY, added to the config secret-masking list and to the installer's sensitive-variable maps, same as the Firecrawl key. Requests go through system.GetHTTPClient, so the proxy and custom CA settings apply.

Performance Impact

None unless OUTCRAWL_API_KEY is set; an unconfigured engine returns ErrNotConfigured and is skipped.

Documentation Updates

  • README.md updates
  • API documentation updates
  • Configuration documentation updates
  • GraphQL schema updates
  • Other: installation guide, searchers README, CLAUDE.md

Deployment Notes

New optional env var OUTCRAWL_API_KEY (passed through docker-compose.yml). One new migration extends the SEARCHENGINE_TYPE enum.

Checklist

Code Quality

  • My code follows the project's coding standards
  • I have added/updated necessary documentation
  • I have added tests to cover my changes
  • All new and existing tests pass (new and touched-package tests pass; the pre-existing local failures above also fail on main)
  • I have run go fmt and go vet (for Go code)
  • I have run pnpm run lint (for TypeScript/JavaScript code) (no frontend changes)

Security

  • I have considered security implications
  • Changes maintain or improve the security model
  • Sensitive information has been properly handled

Compatibility

  • Changes are backward compatible
  • Breaking changes are clearly marked and documented (none)
  • Dependencies are properly updated (no new dependencies)

Documentation

  • Documentation is clear and complete
  • Comments are added for non-obvious code
  • API changes are documented (no API changes)

Additional Notes

This PR was prepared with AI assistance (omp with Claude): the code, tests, migration and this description were written with it, following the existing Firecrawl engine. The build, unit tests, migration up/down, sqlc regeneration and the live search call above were run as described. I work on Outcrawl.

Activity

1
N
feat(tools): add Outcrawl web search provider
b9fa024