Affected Component
Core Services (Frontend UI/Backend API)
Describe the bug
With ~100,000 resource entries, the Resources page fetches the entire library recursively (~29 MB) and becomes unresponsive, appearing empty. The API returns successfully and the stored files remain intact. Loading a smaller subset works.
Steps to Reproduce
n/a
System Configuration
n/a
Logs and Artifacts
No response
Screenshots or Recordings
No response
Verification
- I have checked that this issue hasn't been already reported
- I have provided all relevant configuration files (with sensitive data removed)
- I have included relevant logs and error messages
- I am running the latest version of PentAGI
Activity
4Classic large-list rendering issue, glad to see #410 tackling it. One suggestion beyond the freeze fix: consider server-side pagination/search on the Resources API rather than only client-side — shipping the full payload of thousands of resources to the browser each visit is what makes it lag in the first place. Batch actions (delete / regenerate / reseed) would also benefit from operating on server-side filtered sets instead of selecting everything at once.
This is how I'd do the lazy loading and virtualization asked for when #410 was closed, in three PRs.
Right now ResourcesProvider fetches the whole library with recursive: true (resources-provider.tsx#L73-L75) and is mounted around every route (app.tsx#L117-L126). FileManager then mounts every visible row (file-manager.tsx#L433-L449). Most of the pieces exist. Without recursive, GET /api/v1/resources/ already returns one level for path/paths[] (resources.go#L2135-L2149), and useElementVirtualList is already used with a threshold in detail-navigation-sheet.tsx#L77-L92.
The catch is attaching a folder to a flow:
- Ticking a folder selects only the descendants loaded on the client (use-file-manager-data.ts#L123-L141).
- The server copies exactly the ids it gets (flow_files.go#L1627-L1660).
- A directory id becomes only a
MkdirAll(files.go#L580-L589).
With lazy loading, ticking a collapsed folder would attach an empty folder.
- Virtualize
FileManagerrows above a threshold withuseElementVirtualList.focusRowfinds rows in the DOM (file-manager.tsx#L285-L295), so keyboard navigation has to scroll a row into view before focusing it. No API change. - Expand directory ids to their descendants on the server, in both callers of
CopyResourcesToFlow. One is the REST attach above; the other is theresourceIdspath through flow.go#L761-L772. - Load the root non-recursively and fetch children on expand via
paths[]. Back the search box with a capped server-sidesearchonGET /api/v1/resources/, so a short query can't pull the whole library back.
Does this direction work for you? If so, I'll start with 1.
This direction works for me — starting with 1 (virtualize FileManager rows above a threshold) gives the biggest user-visible win with zero API risk, and 3 (expand directory ids to descendants on the server) is the piece that actually keeps attach semantics correct once lazy loading hides the descendants from the client; without it a collapsed folder would silently attach an empty folder, which is worse than the current freeze. For the capped server-side search, one detail worth guarding: make sure a short query on a huge library still returns quickly and the cap is explicit in the UI (e.g. "showing first N results, refine to narrow") so users don't mistake a capped result set for the full library.
What mean