#434

[Enhancement]: Add Webz.io News Search as a search engine

OpenShakedDeganiopened 1mo ago

Target Component

External Integrations (LLM/Search APIs)

Enhancement Description

Problem statement:

  • Current limitation: PentAGI can search the web through Tavily, Perplexity, Firecrawl, and SearXNG, but it has no global news index with publication dates, language, country, and source filters.
  • Use case: research and monitoring flows that need recent news articles, not only general web pages.

Proposed solution:

  • Add a Webz.io News Search searcher beside the existing engines.
  • Webz.io News Search is a global news index queried in natural language, with filters for date, language, country, domain, sentiment, and category. It is also available as a hosted MCP server at https://news-search-mcp.webz.io/mcp. The tool name is news_search_by_webz. The user supplies WEBZ_API_TOKEN.
  • Docs: https://docs.webz.io/docs/webz/news-search-api-mcp

I work on Webz.io News Search.

Technical Details

Follow backend/pkg/tools/searchers/tavily.go. Implement Searcher from searchers.go: Engine(), IsAvailable(), and Handle(). Read WEBZ_API_TOKEN from config, call the hosted MCP tool news_search_by_webz, and return the same result string shape the orchestrator already consumes. Add a searcher test next to tavily_test.go.

Alternative Solutions

Users can already attach the hosted MCP server themselves. A native searcher would put Webz in the same engine list and fallback order as Tavily, without a separate MCP setup.

Verification

  • I have checked that this enhancement hasn't been already proposed
  • This enhancement aligns with PentAGI's goal of autonomous penetration testing
  • I have considered the security implications of this enhancement
  • I have provided clear use cases and benefits

Activity

1
XinwangTao24d ago

Nice addition — a news-search engine fills a real gap for OSINT workflows, and MCP-based discovery keeps it decoupled from the core searcher list. Two practical questions: (1) is the plan to register it as a normal searcher provider (so it participates in existing per-agent searcher assignment), or as a dedicated MCP tool with its own agent exposure? (2) Webz.io requires an API token (WEBZ_API_TOKEN) — will it reuse the existing per-provider credential storage pattern from the other search providers, or does it expect an env var like the current LLM/search providers? If the latter, a Settings → Providers entry (similar to what #398 proposes for LLM providers) would make it usable for self-hosted users without touching .env.