#3342
Large libraries: fixes from the 50k stress test
This issue tracks the problems that a stress test found on large libraries. The test ran qui develop (0e9989cc, go-qbittorrent 546d7b1c) against qBittorrent 5.2.4 with 50,000 synthetic torrents. Each sub-issue has the measurements.
Sub-issues
- #3001: the tracker-health fetch every 60 s freezes the live updates for about 10 s each minute, and the SSE loop rebuilds every view on each tick. This is the largest cost on large libraries. It needs a design decision first.
- #3365: with #3349, the torrent list polls every 3 s when no sync changes anything for 2 minutes. It needs a design decision first.
- autobrr/go-qbittorrent#145: a read of the torrent list waits for a running sync. After a bulk action on 30,000+ torrents, the list freezes for 10 to 24 s. Fixed by autobrr/go-qbittorrent#146.
- autobrr/go-qbittorrent#167: a filter by hash visits every torrent. It slows bulk actions, notifications after an add, and the cross-seed check in the Delete dialog.
- #3340: a mass import of 10,000 torrents drops notifications and makes SQLite reads fail.
- #3143: "Also delete these cross-seeded torrents" is ignored in select-all mode. Open PR #3170 fixes it.
- #3341: the Delete dialog checks cross-seeds only for the loaded rows. Blocked by #3143.
Related work, not sub-issues
- #2507 (open PR): pages after page 0 read from the cache while the stream runs. Together with autobrr/go-qbittorrent#146, it removes the list freeze on scrolled pages.
- #2823 and #2824 (merged): both pin go-qbittorrent commit
92372302, the squash merge of autobrr/go-qbittorrent#146. #2823 also changes the notification lookup that #3340 changes. - #2820 (merged): made the tracker-health fetch run every 60 s.
- #3328 (merged): a failed sync now asks for a full update. When the tracker-health fetch pushes a sync past its budget, the next sync downloads the full library (92 MB at 50,000 torrents).
- #2821 (closed by #2824): a slow sync left cross-seeds stopped after an add. The tracker-health stall is a likely trigger on large libraries.
Order
- Done. autobrr/go-qbittorrent#146 is merged, and #2823 and #2824 pin its commit
92372302. No release tag contains it yet. - Grill #3001, then build it. It is the largest cost on large libraries, so it moved up from the last step.
- Fix autobrr/go-qbittorrent#167 (in progress), then tag a release and move qui from the commit pin to that tag. It runs in parallel with step 2.
- Merge #3170, then fix #3341.
- Fix #3340. It can run in parallel with steps 2 to 4.
Not tracked here
The test also found smaller problems: slow export and backups, a backup that cannot be cancelled, each instance save reconnects, the client proxy shares the qui session, and a missing-files automation that reads all file lists. They do not have issues yet.
Activity
0No activity yet.