Page MenuHomePhabricator

Download combined phpunit.results.cache timing data and use it to create balanced split_groups
Closed, ResolvedPublic

Description

Since T378478, tests are allocated to split_groups in alphabetical order by filename rather than round-robin. This creates less time-balanced groups - the tests for some extensions take longer than for others, and the slower extensions then slow down the split_group that they are included in.

T384925 creates a service that makes combined timing information available for download. The service provides data for both unit (databaseless) and integration (database) tests, since the same PHPUnit test classes will have different test run durations depending on which PHPUnit test group is executed.

1113147 demonstrates a possible approach here, and shows how the duration information can be applied to the split group generation. In local testing for T378797, the current split group generation implementation (TestSuiteBuilder::buildSuites) was not able to generate balanced groups. The existing algorithm assumes that timing information will not always be available, and in the absence of timing data splits the groups so that they have the same number of test classes. Currently the fallback class counting behaviour dominates the splitting process so that the duration information never becomes relevant.

Investigate (possibly based on the existing patches) how to generate duration-balanced split groups using the available timing information, and create an implementation for mediawiki/core that implements balanced generation.

Acceptance Criteria

  • Make changes to mediawiki/core to download the combined timing data and apply it to the split_group generation
  • Ensure that the split_groups are more balanced (i.e. the difference between the maximum and minimum runtime of the split groups is lower) in the presence of timing data than in its absence.
  • Ensure that there is no regression in group duration balancing after whatever changes are made, even if timing data is no longer available.

Event Timeline

ArthurTaylor renamed this task from Download combined phpunit.results.cache timing data and use it to create balanced split_groups to [SW] Download combined phpunit.results.cache timing data and use it to create balanced split_groups.Jan 28 2025, 1:01 PM
ArthurTaylor renamed this task from [SW] Download combined phpunit.results.cache timing data and use it to create balanced split_groups to Download combined phpunit.results.cache timing data and use it to create balanced split_groups.Feb 5 2025, 9:55 AM
ArthurTaylor updated the task description. (Show Details)
hoo removed hoo as the assignee of this task.Feb 26 2025, 3:46 PM

I still have trouble getting this to run locally, thus I don't have much to show off right now.

One thing we still need to come up with: Where and how to wire up the configuration for the result fetching. Within the job, AFAICT the JOB_NAME environment variable should be set. Using that and a configurable base URL (where?), we can easily construct the URL to fetch the data from (like https://phpunit-results-cache.toolforge.org/results/JOB_NAME). But I'm not sure where this logic would fit, as this seems to be very specific to our infrastructure. Maybe pass the full URL in as an additional (optional) environment variable (which we could set from integration config)?

Change #1113147 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/core@master] Download latest phpunit results cache before parallel tests

https://gerrit.wikimedia.org/r/1113147

Change #1113983 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[integration/quibble@master] Update list of phpunit config files to copy to log directory

https://gerrit.wikimedia.org/r/1113983

Change #1128831 had a related patch set uploaded (by Arthur taylor; author: Arthur taylor):

[mediawiki/core@master] Improve PHPUnit parallel split_group generation algorithm

https://gerrit.wikimedia.org/r/1128831

Created T389189 as a follow-up - it's too complicated to address all three acceptance criteria in a single patch. The attached patch addresses only the first A/C.

Oh wait, if T389189 is a separate task now then I think this one can go into tech verification.

Change #1113147 merged by jenkins-bot:

[mediawiki/core@master] Download latest phpunit results cache before parallel tests

https://gerrit.wikimedia.org/r/1113147

Change #1113983 merged by jenkins-bot:

[integration/quibble@master] Update list of phpunit config files to copy to log directory

https://gerrit.wikimedia.org/r/1113983

Change #1140182 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] release: Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140182

Change #1140182 merged by jenkins-bot:

[integration/quibble@master] release: Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140182

Change #1140215 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch jobs to Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140215

Change #1140215 merged by jenkins-bot:

[integration/config@master] jjb: switch jobs to Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140215