#3342

Large libraries: fixes from the 50k stress test

Opens0up4200opened 10d ago

This issue tracks the problems that a stress test found on large libraries. The test ran qui develop (0e9989cc, go-qbittorrent 546d7b1c) against qBittorrent 5.2.4 with 50,000 synthetic torrents. Each sub-issue has the measurements.

Sub-issues

  • #3001: the tracker-health fetch every 60 s freezes the live updates for about 10 s each minute, and the SSE loop rebuilds every view on each tick. This is the largest cost on large libraries. It needs a design decision first.
  • #3365: with #3349, the torrent list polls every 3 s when no sync changes anything for 2 minutes. It needs a design decision first.
  • autobrr/go-qbittorrent#145: a read of the torrent list waits for a running sync. After a bulk action on 30,000+ torrents, the list freezes for 10 to 24 s. Fixed by autobrr/go-qbittorrent#146.
  • autobrr/go-qbittorrent#167: a filter by hash visits every torrent. It slows bulk actions, notifications after an add, and the cross-seed check in the Delete dialog.
  • #3340: a mass import of 10,000 torrents drops notifications and makes SQLite reads fail.
  • #3143: "Also delete these cross-seeded torrents" is ignored in select-all mode. Open PR #3170 fixes it.
  • #3341: the Delete dialog checks cross-seeds only for the loaded rows. Blocked by #3143.

Related work, not sub-issues

  • #2507 (open PR): pages after page 0 read from the cache while the stream runs. Together with autobrr/go-qbittorrent#146, it removes the list freeze on scrolled pages.
  • #2823 and #2824 (merged): both pin go-qbittorrent commit 92372302, the squash merge of autobrr/go-qbittorrent#146. #2823 also changes the notification lookup that #3340 changes.
  • #2820 (merged): made the tracker-health fetch run every 60 s.
  • #3328 (merged): a failed sync now asks for a full update. When the tracker-health fetch pushes a sync past its budget, the next sync downloads the full library (92 MB at 50,000 torrents).
  • #2821 (closed by #2824): a slow sync left cross-seeds stopped after an add. The tracker-health stall is a likely trigger on large libraries.

Order

  1. Done. autobrr/go-qbittorrent#146 is merged, and #2823 and #2824 pin its commit 92372302. No release tag contains it yet.
  2. Grill #3001, then build it. It is the largest cost on large libraries, so it moved up from the last step.
  3. Fix autobrr/go-qbittorrent#167 (in progress), then tag a release and move qui from the commit pin to that tag. It runs in parallel with step 2.
  4. Merge #3170, then fix #3341.
  5. Fix #3340. It can run in parallel with steps 2 to 4.

Not tracked here

The test also found smaller problems: slow export and backups, a backup that cannot be cancelled, each instance save reconnects, the client proxy shares the qui session, and a missing-files automation that reads all file lists. They do not have issues yet.

Activity

0

No activity yet.