#457
Issues/vxcontrol/pentagi/#457

[Bug]: primary agent can report a fabricated subtask result without delegating or running anything

Openlalith0407opened 22d ago

Affected Component

AI Agents (Researcher/Developer/...)

Describe the bug

What happens: For a simple flow (e.g. prompt "print the hostname of this machine"), the primary agent sometimes calls done with success: true and a result that contains command output it never actually produced — no specialist was delegated and no terminal command ran. The fabricated output is then stored as the subtask result and surfaced to the user by the refiner/reporter as if it were real.

What should happen: The primary agent should only report results that a specialist/tool actually produced in that subtask. A success that reports command output with no delegation and no tool execution should not be accepted at face value.

Additional context:

  • Observed firsthand on a "print the hostname" flow: the report claimed the hostname was debian, but the flow's termlogs table had no rows — no terminal command was ever run. Other flows that really executed hostname did have termlogs and reported the real sandbox container id.
  • Reproduced on a clean stack (OpenAI). Measured phantom rate ~2/5 on the simple hostname prompt.

Root cause: The primary executor's done barrier handler (backend/pkg/providers/provider.go, the FinalyToolName branch) only calls Done.Validate() (backend/pkg/tools/args.go), which merely checks that success is set. It never checks whether any delegation/tool call happened in the subtask, so a success that reports unverified output is accepted and stored.

Steps to Reproduce

  1. Start a flow with a simple execution prompt such as "print the hostname of this machine" (OpenAI provider).
  2. Repeat a handful of times.
  3. Observe occasional flows that finish "successfully" reporting a hostname while the flow's termlogs contain no entry — i.e. nothing was executed and the value is fabricated.

System Configuration

  • PentAGI Version: main (h843dfeb)
  • Deployment: Docker Compose (default local stack)
  • LLM Provider: OpenAI (default models)

Impact

Silent fabrication of results undermines trust in every flow report: an operator cannot tell a real finding from a hallucinated one.

Activity

1
XinwangTao22d ago

This is a serious trust issue — if a "successful" result can be stored without any delegation or tool execution, every downstream report/refiner step becomes unreliable. The root cause analysis makes sense: Done.Validate() only checks that success is set, so unverified output is accepted at face value.

A couple of thoughts on the fix direction:

  1. Requiring real delegation is good, but consider also requiring at least one recorded tool/delegation event in the same subtask scope (not just a flag on the final call), otherwise a caller could still pass validation with a stale or unrelated delegation.
  2. It may be worth marking such results explicitly in the flow telemetry (e.g. a "verified: false" field) so operators can audit which subtask results were actually produced by tools, rather than only rejecting them at the done barrier.

Thanks for including the phantom rate measurement (~2/5) — that makes the impact concrete.