Page MenuHomePhabricator

How to e2e/integration test simple Item search
Open, Needs TriagePublic13 Estimated Story Points

Description

Our implementation for simple Item search (T383132) currently relies on WikibaseCirrusSearch which uses ElasticSearch. This makes e2e/integration testing difficult because ElasticSearch is unavailable in our regular CI system at the moment.

Options:

  1. Cirrus Search extensions runs its integration tests, which might be close to, or not, to our end to end tests using a dedicated test runner, that is then automated and integrated with the Jenkins CI through a browser test bot. This is likely a "historical" approach, which might not be preferred, and it is unclear to me how much of Cirrus Search runner and bot could be reused, and feasible using this approach would be for Wikibase (Search API) CI
  2. Enable Quibble to run Elastic Search. This would like follow a way how it was made possible for Quibble to run Apache server using supervisord. Depending how much resources (memory) would it consume and how fast starting would take, using Elastic through Quibble on Jenkins CI might be acceptable. I figure that might be the most "right" way of addressing the need, and should start with "prototyping" adding Elastic to Quibble containers, and observing how it behaves.

Let's start with trying out approach 2, and if that turns out not to be feasible, we'll go with the workaround in option 1.

Event Timeline

Jakob_WMDE renamed this task from E2e testing simple Item search with WikibaseCirrusSearch to How to e2e/integration test simple Item search.Feb 19 2025, 11:47 AM
Jakob_WMDE updated the task description. (Show Details)

Change #1120963 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] Search: Add SqlTermStoreSearchEngine

https://gerrit.wikimedia.org/r/1120963

Change #1120963 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Search: Add SqlTermStoreSearchEngine

https://gerrit.wikimedia.org/r/1120963

Two potentially feasible possibilities that could be considered that I got through chatting with @hashar lately. Either requires further exploration to understand would it work for Wikibase Restful Search needs, and be feasible to integrate with its CI

  1. Cirrus Search extensions runs its integration tests, which might be close to, or not, to our end to end tests using a dedicated test runner, that is then automated and integrated with the Jenkins CI through a browser test bot. This is likely a "historical" approach, which might not be preferred, and it is unclear to me how much of Cirrus Search runner and bot could be reused, and feasible using this approach would be for Wikibase (Search API) CI
  2. Enable Quibble to run Elastic Search. This would like follow a way how it was made possible for Quibble to run Apache server using supervisord. Depending how much resources (memory) would it consume and how fast starting would take, using Elastic through Quibble on Jenkins CI might be acceptable. I figure that might be the most "right" way of addressing the need, and should start with "prototyping" adding Elastic to Quibble containers, and observing how it behaves.

Change #1137857 had a related patch set uploaded (by Jakob; author: Jakob):

[integration/quibble@master] [DNM] Add ElasticSearch

https://gerrit.wikimedia.org/r/1137857

Change #1137108 had a related patch set uploaded (by Jakob; author: Jakob):

[integration/config@master] [DNM] Elasticsearch in quibble

https://gerrit.wikimedia.org/r/1137108

Change #1137108 merged by jenkins-bot:

[integration/config@master] Include OpenSearch in quibble

https://gerrit.wikimedia.org/r/1137108

Change #1139866 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Add job to test Quibble with OpenSearch

https://gerrit.wikimedia.org/r/1139866

Change #1139866 merged by jenkins-bot:

[integration/config@master] Add job to test Quibble with OpenSearch

https://gerrit.wikimedia.org/r/1139866

Change #1139870 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] zuul: set QUIBBLE_OPENSEARCH for Quibble opensearch job

https://gerrit.wikimedia.org/r/1139870

Change #1139870 abandoned by Hashar:

[integration/config@master] zuul: set QUIBBLE_OPENSEARCH for Quibble opensearch job

Reason:

It is actually starting

https://gerrit.wikimedia.org/r/1139870

Change #1139870 restored by Hashar:

[integration/config@master] zuul: set QUIBBLE_OPENSEARCH for Quibble opensearch job

https://gerrit.wikimedia.org/r/1139870

Change #1139870 merged by jenkins-bot:

[integration/config@master] zuul: set QUIBBLE_OPENSEARCH for Quibble opensearch job

https://gerrit.wikimedia.org/r/1139870

Change #1137857 merged by jenkins-bot:

[integration/quibble@master] Add support for OpenSearch

https://gerrit.wikimedia.org/r/1137857

Change #1140156 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/Wikibase@master] Enable WikibaseCirrusSearch if CI has OpenSearch

https://gerrit.wikimedia.org/r/1140156

Change #1140182 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/quibble@master] release: Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140182

Change #1140182 merged by jenkins-bot:

[integration/quibble@master] release: Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140182

Change #1140215 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch jobs to Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140215

Change #1140215 merged by jenkins-bot:

[integration/config@master] jjb: switch jobs to Quibble 1.14.0

https://gerrit.wikimedia.org/r/1140215

Change #1142077 had a related patch set uploaded (by Jakob; author: Jakob):

[mediawiki/extensions/CirrusSearch@master] Allow writes in CI if OpenSearch is enabled

https://gerrit.wikimedia.org/r/1142077

Change #1142413 had a related patch set uploaded (by Jakob; author: Jakob):

[integration/config@master] Add missing OpenSearch plugins

https://gerrit.wikimedia.org/r/1142413

Change #1142413 merged by jenkins-bot:

[integration/config@master] Add missing OpenSearch plugins

https://gerrit.wikimedia.org/r/1142413

Change #1142544 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: update Quibble opensearch job for plugins

https://gerrit.wikimedia.org/r/1142544

Change #1142544 merged by jenkins-bot:

[integration/config@master] jjb: update Quibble opensearch job for plugins

https://gerrit.wikimedia.org/r/1142544

Last week @Jakob_WMDE asked about how to roll this. One issue is OpenSearch is we still have CI jobs for php 7.4 that comes with Debian Buster (T335766), and those images do not have OpenSearch.

Then the new feature is behind a feature flag (QUIBBLE_OPENSEARCH) introduced by https://gerrit.wikimedia.org/r/c/integration/config/+/1139870/2/zuul/parameter_functions.py :

zuul/parameter_functions.py
if job.name.startswith('integration-quibble-fullrun-opensearch'):
    params['QUIBBLE_OPENSEARCH'] = 'true'

We can add another guard in the condition to not add that parameter when the job name does not contain php74. The OpenSearch tests would still run with php 8.1 flavor of job though.

Change #1142077 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Allow writes in CI if OpenSearch is enabled

https://gerrit.wikimedia.org/r/1142077

Change #1145213 had a related patch set uploaded (by WMDE-leszek; author: WMDE-leszek):

[integration/config@master] zuul: Only set QUIBBLE_OPENSEARCH for PHP 8 jobs

https://gerrit.wikimedia.org/r/1145213

Pretty much, that is the idea.

The current condition match solely integration-quibble-fullrun-opensearch which was used to tests the patches send to integration/quibble. We would need another condition that match the repository, job and branch we want to enable the feature on. My understanding is that it is for Wikibase and the apitesting test suite, is that correct?

If so we could potentially add:

if (
    params['ZUUL_PROJECT'] == 'mediawiki/extensions/Wikibase'
    and params['ZUUL_BRANCH'] == 'master'
    and job.name.startswith('mediawiki-quibble-apitests')
    and 'php74' not in job.name
)

But the apitests job uses the php74 version: mediawiki-quibble-apitests-vendor-php74. We would need to add the php81 and for what we are kind of waiting for the phase out of php74 to have completed. Then the last thing remaining are moving maintenance tests, so we can switch that job for sure.

Change #1145257 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] zuul: upgrade apitests jobs from 7.4 to 8.1

https://gerrit.wikimedia.org/r/1145257

Change #1145257 merged by jenkins-bot:

[integration/config@master] Zuul: Upgrade all Quibble 'apitests' jobs from 7.4 to 8.1

https://gerrit.wikimedia.org/r/1145257

Mentioned in SAL (#wikimedia-releng) [2025-05-13T19:27:02Z] <James_F> Zuul: Upgrade all Quibble 'apitests' jobs from 7.4 to 8.1, for T386691, T328921, T328922

Change #1145213 abandoned by WMDE-leszek:

[integration/config@master] zuul: Only set QUIBBLE_OPENSEARCH for PHP 8 jobs

Reason:

was clearly a mistake/misunderstanding, a Wikibase specific condition is needed, and will be provided shortly

https://gerrit.wikimedia.org/r/1145213

Change #1145331 had a related patch set uploaded (by WMDE-leszek; author: WMDE-leszek):

[integration/config@master] Enable Open Search for Wikibase api tests jobs

https://gerrit.wikimedia.org/r/1145331

The current condition match solely integration-quibble-fullrun-opensearch which was used to tests the patches send to integration/quibble.

of course, silly me.
You got what we want right, so I've added the specific condition, this time right hopefully.
I'm not sure if with the above changes you were kind to make it would be possible to have apitests job that would run php74 but included this condition just in case. Would be easy to drop.

of course, silly me.

There are so many layers of configuration that it is easy to get lost (Jenkins job templates, Jenkins job, Zuul injecting parameters, the Zuul layout/workflow, Docker image, parameters passed by Zuul and the jobs to the container etc). It is easy to get lost :]

Change #1145331 merged by jenkins-bot:

[integration/config@master] Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs

https://gerrit.wikimedia.org/r/1145331

Mentioned in SAL (#wikimedia-releng) [2025-05-14T13:43:25Z] <hashar> Reloded Zuul for Zuul: [mediawiki/extensions/Wikibase] Enable Open Search for apitests jobs | https://gerrit.wikimedia.org/r/1145331 | T386691

I have deployed @WMDE-leszek patch https://gerrit.wikimedia.org/r/c/integration/config/+/1145331 which causes CI to inject QUIBBLE_OPENSEARCH=true for Wikibase / master / apitests job.

I then triggered a recheck on Gerrit 1140156 - Use WikibaseCirrusSearch in CI if it has OpenSearch which enables CirrusSearch whenever the env variable is set to true.

We will see how the mediawiki-quibble-apitests-vendor-php81 job behaves :]

We will see how the mediawiki-quibble-apitests-vendor-php81 job behaves :]

@hashar I guess T394376 is related?

@hashar I guess T394376 is related?

Apparently yes, enabling the OpenSearch integration tests caused one test to fail. That got quickly solved by reverting the feature flag \o/

Why? I do not know :-] Maybe I can add a job that has opensearch enabled and make it non voting (ie if it fails, that would not vote Verified-1/block).

Why? I do not know :-] Maybe I can add a job that has opensearch enabled and make it non voting (ie if it fails, that would not vote Verified-1/block).

That sounds like a very good idea, @hashar. A non-voting job will help us figuring out what's going wrong, without the need to enable every time we think there might be a fix and then disabling it again on the next unexpected error.