Summary

This adds a warning in the documentation about the default (random) test order of execution possibly causing issues for MPI programs using Catch2 after version 3.9.1.

We recently update Catch2 from version 3.1.0 to the 3.15.1, and we were surprised that our tests started encountering MPI deadlock due the default test execution being changed to random. The fix is simple (use --lex or another non-default ordering option), but a google search was unhelpful. I hope that including this warning in the documentation helps improve the visibility of the fix.

Activity

add warning about default test order causing problems with MPI

d531948Aug 14

codecov[bot]2mo ago

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 91.16%. Comparing base (0aeb818) to head (0a4a310). :warning: Report is 11 commits behind head on devel.

Additional details and impacted files

@@            Coverage Diff             @@
##            devel    #3192      +/-   ##
==========================================
- Coverage   91.27%   91.16%   -0.11%     
==========================================
  Files         204      206       +2     
  Lines        8973     9016      +43     
==========================================
+ Hits         8190     8219      +29     
- Misses        783      797      +14

:rocket: New features to boost your workflow:

:snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

horenmarreviewed2mo ago

fix typo "indentical" -> "identical"

0a4a310Aug 24

kbthomp1reviewed2mo ago

horenmarrequested changes2mo ago

horenmar2mo ago

Warning is fine, but the overall text is wrong; you do want the option with MPI and friends, but using a fixed seed... or use the --order option.

Also I suggest implementing a stronger warning than just in docs, similar to here: https://github.com/catchorg/Catch2/commit/0aeb818520028ca05cc2539ed242402ef7a7ae41

horenmar2mo ago

Thinking about it a bit more, I mentioned the concept of non-deterministic test order in last release notes, which is also the underlying issue for MPI. The docs should probably explain that (maybe in ordering section? and probably also add a note about it in FAQ) and then mention that this is problem for MPI and similar.