Skip to content

TST: Skip tests that use a large amount of memory by default #31007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
QuLogic wants to merge 3 commits into
base: main
Choose a base branch
from
Open

TST: Skip tests that use a large amount of memory by default #31007

QuLogic wants to merge 3 commits into from

Conversation

@QuLogic
Copy link
Member

@QuLogic QuLogic commented Jan 21, 2026 •
edited
Loading

PR summary

According to memray, these tests use fairly large amounts of memory:

  • lib/matplotlib/tests/test_simplification.py::test_throw_rendering_complexity_exceeded: 2.2GiB
  • lib/matplotlib/tests/test_agg.py::test_chunksize_fails: 2.0GiB
  • lib/matplotlib/tests/test_image.py::test_large_image[png-col-16777216-2\*\*24 rows-upper]: 1.0GiB
  • lib/matplotlib/tests/test_image.py::test_large_image[png-col-16777216-2\*\*24 rows-lower]: 1.0GiB
  • lib/matplotlib/tests/test_image.py::test_downsample_interpolation_stage[png]: 1.0GiB
  • lib/matplotlib/tests/test_lines.py::test_invisible_Line_rendering: 822.0MiB
  • lib/matplotlib/tests/test_image.py::test_large_image[png-row-8388608-2\*\*23 columns-lower]: 537.8MiB
  • lib/matplotlib/tests/test_image.py::test_large_image[png-row-8388608-2\*\*23 columns-upper]: 537.8MiB

The next closest tests use just 216MiB, then 145.5MiB, etc. and then the majority under 100MiB.

If you're unlucky, with 4-way xdist you might use upwards of 6GiB RAM if you get all these on separate processes.

The above tests are all fairly low-level checks for Agg renderer limits that are not likely to change often, so skip them by default unless on CI.

PR checklist

@QuLogic QuLogic force-pushed the skip-big-tests branch 2 times, most recently from 8b5ae9d to a3edc5c Compare January 21, 2026 10:50
@QuLogic
Copy link
Member Author

QuLogic commented Jan 21, 2026

Hmm, looking at test_downsample_interpolation_stage, it passes with a 40x40 image instead of 4000x4000; does it even need to be that big?

@timhoffm
Copy link
Member

timhoffm commented Jan 21, 2026

I propose to delete lib/matplotlib/tests/test_lines.py::test_invisible_Line_rendering.

The test tries to confirm that visible == False returns early. It does so by creating large data, that should produce a noticable delay when being processed.
If one wanted to test this, a better strategy would be to assert that x, y are not touched during draw using mocking. But overall, what are we trying to prevent here? The test would only fail if the if not self.is_visible(): return code would be moved further down again. That is highly unlikely. If we would write draw() from scratch with this early return, we would not add a test for the early return. So let's KISS and just delete it.

@QuLogic
Copy link
Member Author

QuLogic commented Jan 22, 2026

Okay, I removed that test and reduced the memory usage for test_downsample_interpolation_stage.

@QuLogic QuLogic force-pushed the skip-big-tests branch 2 times, most recently from 60e2f0c to 7c98478 Compare January 22, 2026 05:12
@QuLogic
Copy link
Member Author

QuLogic commented Jan 22, 2026

So it turns out that pytest_addoption doesn't work if conftest.py is installed, so I've switched to checking an environment variable (but it should also continue to enable these tests on CI as well.)

timhoffm
timhoffm reviewed Jan 22, 2026
View reviewed changes
doc/devel/testing.rst Outdated
Comment on lines 55 to 56
Some tests may use a large amount of memory (>0.5GiB); to enable those tests, set the
environment variable ``MPL_TEST_HIGH_MEMORY``.
Copy link
Member

@timhoffm timhoffm Jan 22, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to generalize to "expensive tests"? This could include runtime as well as memory.
The 14 slowest tests together take 120s.

Copy link
Member Author

@QuLogic QuLogic Jan 22, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we have too many that take long; I guess just the sphinx docs one, maybe?

Copy link
Member

@timhoffm timhoffm Jan 22, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

============================= slowest 25 durations =============================
10.91s call     lib/matplotlib/tests/test_sphinxext.py::test_tinypages
10.24s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[mhip-ps-True]
9.48s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[mhip-pdf-False]
9.25s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[mhip-ps-False]
9.02s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[mhip-svg-False]
9.01s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[mhip-svg-True]
8.51s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[p-svg-False]
8.50s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[i-pdf-False]
8.50s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[h-pdf-False]
8.49s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[m-pdf-False]
8.34s call     lib/matplotlib/tests/test_determinism.py::test_determinism_check[-pdf-False]
7.84s call     lib/matplotlib/tests/test_sphinxext.py::test_srcset_version
5.32s call     lib/matplotlib/tests/test_matplotlib.py::test_importable_with__OO
4.74s call     lib/matplotlib/tests/test_backend_pgf.py::test_minus_signs_with_tex[lualatex-pdf]
4.58s call     lib/matplotlib/tests/test_subplots.py::test_shared
4.36s call     lib/matplotlib/tests/test_streamplot.py::test_integration_options[png]
4.20s call     lib/matplotlib/tests/test_sphinxext.py::test_plot_html_show_source_link
4.09s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_backend[toolbar2-MPLBACKEND=gtk4agg-BACKEND_DEPS=cairo,gi]
3.77s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_timers[MPLBACKEND=gtk4agg-BACKEND_DEPS=cairo,gi]
3.74s call     lib/matplotlib/tests/test_axes.py::test_axes_clear_reference_cycle
3.74s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_timers[MPLBACKEND=gtk4cairo-BACKEND_DEPS=cairo,gi]
3.59s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_timers[MPLBACKEND=qtagg-QT_API=PySide6-BACKEND_DEPS=PySide6]
3.59s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_timers[MPLBACKEND=gtk3agg-BACKEND_DEPS=cairo,gi]
3.56s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_timers[MPLBACKEND=qtcairo-QT_API=PySide6-BACKEND_DEPS=PySide6,cairocffi]
3.54s call     lib/matplotlib/tests/test_backends_interactive.py::test_interactive_timers[MPLBACKEND=wxagg-BACKEND_DEPS=wx]

The determinism ones are also quite significant

@QuLogic
TST: Reduce memory usage of test_downsample_interpolation_stage
f63f54a 
By cutting the grid by one-tenth in each dimension, this cuts memory
usage from 1.0GiB to 65.2MiB. I temporarily added an assert in
`_ImageBase._make_image` to confirm that the interpolation stage did
correctly switch to 'rgba' as well.
@QuLogic
TST: Delete test_invisible_Line_rendering
5f1498b 
This test tries to check for slowdowns, which is inherently flaky (as
noted by the comment), and we don't check `set_visible` for other
artists.
@QuLogic
TST: Skip tests that use a large amount of memory by default
a499ce9 
According to memray, these tests use fairly large amounts of memory:
- lib/matplotlib/tests/test_simplification.py::test_throw_rendering_complexity_exceeded: 2.2GiB
- lib/matplotlib/tests/test_agg.py::test_chunksize_fails: 2.0GiB
- lib/matplotlib/tests/test_image.py::test_large_image[png-col-16777216-2\\*\\*24 rows-upper]: 1.0GiB
- lib/matplotlib/tests/test_image.py::test_large_image[png-col-16777216-2\\*\\*24 rows-lower]: 1.0GiB
- lib/matplotlib/tests/test_image.py::test_large_image[png-row-8388608-2\\*\\*23 columns-lower]: 537.8MiB
- lib/matplotlib/tests/test_image.py::test_large_image[png-row-8388608-2\\*\\*23 columns-upper]: 537.8MiB

The next closest tests use just 216MiB, then 145.5MiB, etc. and then the
majority under 100MiB.

If you're unlucky, with 4-way xdist you might use upwards of 6GiB RAM if
you get all these on separate processes.

The above tests are all fairly low-level checks for Agg renderer limits
that are not likely to change often, so skip them by default unless on
CI.
timhoffm
timhoffm reviewed Jan 24, 2026
View reviewed changes
Copy link
Member

@timhoffm timhoffm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technical correct. Is there a specific reason to implement this via a fixture? I would have intuitively expected a mark or skipif decorator. It's a bit unconventional to use a fixture for test selection, they are typically rather for context.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timhoffm timhoffm timhoffm approved these changes

Assignees

No one assigned

Labels

Documentation: devdocs files in doc/devel Performance topic: mplot3d topic: testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@QuLogic @timhoffm