#403

[Bug]: Resources page freezes with large resource libraries

Opentavgaropened 2mo ago

Affected Component

Core Services (Frontend UI/Backend API)

Describe the bug

With ~100,000 resource entries, the Resources page fetches the entire library recursively (~29 MB) and becomes unresponsive, appearing empty. The API returns successfully and the stored files remain intact. Loading a smaller subset works.

Steps to Reproduce

n/a

System Configuration

n/a

Logs and Artifacts

No response

Screenshots or Recordings

No response

Verification

  • I have checked that this issue hasn't been already reported
  • I have provided all relevant configuration files (with sensitive data removed)
  • I have included relevant logs and error messages
  • I am running the latest version of PentAGI

Activity

4
traderasim329-a11y1mo ago

What mean

XinwangTao24d ago

Classic large-list rendering issue, glad to see #410 tackling it. One suggestion beyond the freeze fix: consider server-side pagination/search on the Resources API rather than only client-side — shipping the full payload of thousands of resources to the browser each visit is what makes it lag in the first place. Batch actions (delete / regenerate / reseed) would also benefit from operating on server-side filtered sets instead of selecting everything at once.

AlaeddineMessadi15d ago

This is how I'd do the lazy loading and virtualization asked for when #410 was closed, in three PRs.

Right now ResourcesProvider fetches the whole library with recursive: true (resources-provider.tsx#L73-L75) and is mounted around every route (app.tsx#L117-L126). FileManager then mounts every visible row (file-manager.tsx#L433-L449). Most of the pieces exist. Without recursive, GET /api/v1/resources/ already returns one level for path/paths[] (resources.go#L2135-L2149), and useElementVirtualList is already used with a threshold in detail-navigation-sheet.tsx#L77-L92.

The catch is attaching a folder to a flow:

With lazy loading, ticking a collapsed folder would attach an empty folder.

  1. Virtualize FileManager rows above a threshold with useElementVirtualList. focusRow finds rows in the DOM (file-manager.tsx#L285-L295), so keyboard navigation has to scroll a row into view before focusing it. No API change.
  2. Expand directory ids to their descendants on the server, in both callers of CopyResourcesToFlow. One is the REST attach above; the other is the resourceIds path through flow.go#L761-L772.
  3. Load the root non-recursively and fetch children on expand via paths[]. Back the search box with a capped server-side search on GET /api/v1/resources/, so a short query can't pull the whole library back.

Does this direction work for you? If so, I'll start with 1.

XinwangTao12d ago

This direction works for me — starting with 1 (virtualize FileManager rows above a threshold) gives the biggest user-visible win with zero API risk, and 3 (expand directory ids to descendants on the server) is the piece that actually keeps attach semantics correct once lazy loading hides the descendants from the client; without it a collapsed folder would silently attach an empty folder, which is worse than the current freeze. For the capped server-side search, one detail worth guarding: make sure a short query on a huge library still returns quickly and the cap is explicit in the UI (e.g. "showing first N results, refine to narrow") so users don't mistake a capped result set for the full library.