#3340

Mass import of 10k torrents drops notifications and makes SQLite reads fail

Opens0up4200opened 10d ago

This was generated by AI during a stress test.

Agent Brief

Category: bug Summary: When about 10,000 torrents complete or get added in one burst, qui drops every notification, skips the cross-seed completion check for thousands of torrents, and the notification settings page returns HTTP 500.

Current behavior: Test setup: qui develop 0e9989cc, qBittorrent 5.2.4, 50,000 torrents. 10,000 torrents were deleted in qBittorrent and added again while one UI tab was open.

  • qui logged 12k to 14k warnings that the notification queue is full and dropped each event. No notification target was configured.
  • qui logged 2.4k to 5.4k warnings that the cross-seed completion settings could not be loaded. The cause is a context deadline on the PRAGMA cache_size connection hook.
  • GET /api/notifications/targets returned HTTP 500 after 7 to 11 s.
  • qui CPU went above 800 %. The torrent list stalled for 7 to 23 s. RSS peaked at 3.65 GB in one sample.
  • An import of 5,000 torrents gave no warnings.

A CPU profile of 40 s showed two causes:

  1. The completion handler starts one goroutine per completed torrent, with no limit. Each goroutine calls InstanceCrossSeedCompletionStore.Get. The SQLite reader pool has no connection limit, so each call opens a new connection and runs the connection pragmas. This was 62 s of CPU.
  2. Each added torrent starts one goroutine that calls refreshTorrentForNotification. That function calls SyncManager.GetTorrents with a filter of one hash, which scans all 50,000 torrents. This was 58 s of CPU.

Desired behavior: A burst of 10,000 completed or added torrents does not drop notifications that a target listens for, does not skip cross-seed completion checks, and does not make other SQLite reads fail. The UI stays usable during the burst.

Key interfaces:

  • HandleTorrentCompletion in the cross-seed service: reads InstanceCrossSeedCompletionStore.Get once per torrent. The settings are per instance, so one read per burst is enough.
  • The completion and torrent-added dispatch in qbittorrent.Client: starts one goroutine per torrent with no limit.
  • refreshTorrentForNotification: must find one torrent without a scan of the full library. autobrr/go-qbittorrent#167 fixes the scan for every caller of a filter by hash. Open PR #2823 changes this function to sync twice before each read. The fix here and #2823 must agree.
  • notifications.Service.Notify: queues events also when no target listens for the event type.

Acceptance criteria:

  • A test with 10,000 completion events reads the completion settings at most once per instance per burst.
  • The number of goroutines for completion and added events has a fixed upper limit.
  • A torrent-added notification for one hash does not scan all torrents.
  • With no target for an event type, Notify does not queue that event.
  • On the 50k rig, a 10k import gives zero "queue full" and zero "completion settings" warnings, and GET /api/notifications/targets returns 200.

Out of scope:

  • The tracker-health fetch (#3001).
  • The scan of a filter by hash in go-qbittorrent (autobrr/go-qbittorrent#167).
  • The size of the SQLite reader pool for other callers.

Domain:

  • Glossary terms: none used.
  • ADRs: None.
  • Owner: the qbittorrent client event dispatch, the cross-seed service, and the notifications service.

Logs:

  • None

Activity

0

No activity yet.