Strange statistics for language variants for languages other than zh and sr
Open, Needs TriagePublicBUG REPORT
Actions

Assigned To

None

Authored By

	Amire80
	Thu, Apr 24, 6:41 PM

Description

Data Platform Engineering Bug Report or Data Problem Form.

Please fill out the following

Please ensure you set priority

What kind of problem are you reporting?

Access related problem
Service related problem
Data related problem

As a follow-up to the Central Asian WikiCon 2025, which took place in Uzbekistan, and also as part of T364147, I am trying to understand the usage of the language variants for the Uzbek (uz) and Tajik (tg) languages.

I found that Turnilo includes language variants information, which could be useful, but several things about the results look very unusual.

First, if I filter out the "default" variant, which, expectedly, has the largest number of views by far, the most prominent variants are all from Chinese (zh) and Serbian (sr). Also, if I filter only by including zh-* variants and split by project, then zh.wikipedia and other zh.* projects are the most popular ones, and same is true for sr-* variants.

All of the above is sensible, because Chinese and Serbian are two prominent languages with very active editing communities and very high demand for all their variants (Traditional/Simplified, Cyrillic/Latin).

So far, so good. But if I try to look at variants for other languages, everything becomes weird.

If I go back to the first search, where I filter out "default" and split by project, then pretty much all, or at least most of the "variants" that come after zh and sr, are not actually variants, but mostly regular language codes, with "en" being the most prominent. Some of them are quite obscure, such as "gaz" or "pro", which I've never seen those anywhere in the Wikimedia world before.

What's even more interesting is that if I filter by excluding "default" and all the "zh-*" and "sr-*" variants, and then split by project, then the most popular project by far is Wikifunctions. And if I filter by including only tg, tg-cyrl, and tg-latn, then Wikifunctions is the only project that has any hits. The same happens if I include only uz, uz-cyrl, and uz-latn. Perhaps this happens because of Wikifunctions' URL structure, which is different from most other wikis, but I might be wrong.

Something about this seems quite wrong. For example, I'd expect at least some usage for variants of uz-* in uz.wikipedia.org, given that both Cyrillic and Latin alphabets are frequently used in Uzbekistan (I've just visited the country a few days ago and saw it a lot). I wouldn't be surprised if variants of Uzbek had lower usage than variants of Serbian, but currently, there's nothing at all; according to Turnilo, uz-* variants are used only in Wikifunctions.

Related Objects

Mentioned In: T364147: Remove tg-latn from Mediawiki
Mentioned Here: T364147: Remove tg-latn from Mediawiki

Event Timeline

Amire80 created this task.Thu, Apr 24, 6:41 PM

Restricted Application added subscribers: Stang, Aklapper. · View Herald TranscriptThu, Apr 24, 6:41 PM

Amire80 mentioned this in T364147: Remove tg-latn from Mediawiki.Thu, Apr 24, 8:07 PM

Adding some context from an internal discussion to hopefully help whoever picks this up:

The issue is that the initial approach to language variants was to incorporate them into the URL path -- e.g., https://sr.wikipedia.org/wiki/<page-title> for default and https://sr.wikipedia.org/<lang-variant>/<page-title> for the variants. The code for measuring language variants then looks for that /<lang-variant>/ pattern in the URL path and extracts it. The other language editions seem to use a different approach, which is the same thing for the default variant but then https://tg.wikipedia.org/w/index.php?title=<page-title>&variant=<lang-variant> for the variants. Capturing those would require not just looking at the URL path but also URL params.

In T392624#10766525, @Isaac wrote:

Adding some context from an internal discussion to hopefully help whoever picks this up:

The issue is that the initial approach to language variants was to incorporate them into the URL path -- e.g., https://sr.wikipedia.org/wiki/<page-title> for default and https://sr.wikipedia.org/<lang-variant>/<page-title> for the variants. The code for measuring language variants then looks for that /<lang-variant>/ pattern in the URL path and extracts it. The other language editions seem to use a different approach, which is the same thing for the default variant but then https://tg.wikipedia.org/w/index.php?title=<page-title>&variant=<lang-variant> for the variants. Capturing those would require not just looking at the URL path but also URL params.

Thanks for more context.

I'm also wondering why is there a difference in the first place. It's a similar functionality, so I'd expect it to work similarly across all languages. Perhaps @cscott knows?

nshahquinn-wmf added a project: Analytics-Data-Problem.Mon, Apr 28, 11:34 PM

nshahquinn-wmf subscribed.

@OSefu-WMF can you help assess the effort, impact and prioritize?

Ahoelzl moved this task from Incoming (new tickets) to Tag with Radar on the Data-Engineering board.Tue, Apr 29, 5:47 PM

Ahoelzl added a project: Data-Engineering-Radar.