Fallback Languages

[downstairsdev]

We should do a better job of loading translation files in the user's language if they are available.

For instance, if a Spanish speaker has their locale set as es_MX (Spanish Mexico) it would be preferable to load any available Spanish translations files (es_ES, es_CO, etc) before returning the default (generally English).

I wrote up a quick patch, tester plugin, and plugin that demonstrate this idea. If a $mofile is not available in the user's current locale, it will search for and return the first available translation that is also in the same language.

A better option might be to create a filterable stack rank of locales for WordPress to search for within the language before returning the default. Other suggestions are also welcome.

This idea was also discussed in an "ideas" thread:
​http://wordpress.org/ideas/topic/fallback-to-generic-language-file-when-specific-locale-file-absent

[downstairsdev]

WordPress Patch

[downstairsdev]

Tester plugin.

[downstairsdev]

The tester plugin displays two strings in the dashboard next to the "screen options" tab. The first one is the locale for the site, e.g. "en_MX". The second one display the translation for the string "default". So, if no translation file is loaded, "default" is displayed.

There are two bundled translation files, es_ES and es_MX. If those files are loaded, the translation of "default" is displayed- which I've just set as the locale for each file ('es_ES' and 'es_MX' respectively).

To test, set a Spanish locale other than one of the two translations (e.g. define('WPLANG', 'es_CO'); ). If everything is working correctly, the first string will return the locale ( 'es_CO' ), and the second string will display the translation for the actual translation file that got loaded. Since 'es_CO' is not available, it will be one of the other Spanish translations ( 'es_ES' ).

Hopefully this makes sense!

--

The filter also allows us to do this through a plugin. Suggestions and pull requests welcome there as well:
​https://github.com/devinsays/fallback-languages

[nacin]

This is a cool idea, though it's going to be really slow as written. We have some plans to do this on the side of language packs (and WordPress.org), versus at the system level.

This also doesn't necessarily work linguistically for every language out there. A proper mapping of language codes that can be fallen back to could help a bit with both this and performance.

[toscho]

I like the idea. I do the same in Multilingual Press for language negotiation.

substr( get_locale(), 0, 2) will fail. For many languages the code has three letters, because it is taken from ISO 639-2. Just use strtok( get_locale(), '_' ).

[cvedovini]

Hi, nice work.

However my initial proposition was to fallback to a generic language file, like if fr_FR is not present then fallback to fr, not fr_CA or any other random variation that happen to be there.

This is much simpler to code but this would also require some work at the language packs level in order to define a "default" language (tho I can see where that could become a political issue)

[downstairsdev]

Fallback languages (generic) patch.

[downstairsdev]

Plugin to test generic fallback languages.

[downstairsdev]

I attached an updated patch that implements @cvedovini idea. I was trying to work within the existing framework for translations- but a generic fallback could work better.

1) Translation teams from different locales could focus on the generic fallback version first- and then quickly copy it to make country specific localizations. Hopefully it would not get political as any translator who disagreed with a translation in the default could just change it for their specific locale.

2) This avoids having to build and maintain an ordered list of fallbacks for each language.

Instead of defaulting to "generic" we could default to the locale with simply the largest population. E.g. English > United States, Spanish > Mexico, Portuguese > Brazil. But then you might get might get a Spanish (es_ES) translation for an es_MX localizations simply because it would be used more. So for this reason, a generic fallback would probably be better.

Thanks for the coding tip @toscho. I am using that instead for this updated patch.

Nacin, could you outline where the performance issues are? (I don't have much experience in that area). Would it be helpful to cache translation file paths in a transient?

[grapplerulrich]

I am not sure a generic language fallback is best. I think you are best asking on the ​Polyglots make blog.

Who defines what is the generic form? Take the spelling of colour/color in English. How can you convince both parties that which is the most generic form? If we say for the sake of argument that the en_US form is more generic then we could just have en_UK fallback to en_US.
(I know there is no en_US translation file. This is just an example that most should be able to understand.)

I was told by @jenialaszlo at WordCamp Switzerland that Ukrainian could use a fallback to Russian. You would need to take that into account also.

By introducing a generic file, it creates extra work for core translators and also theme/plugin translators/developers who would need to create a generic file so to make use of this new feature.

[cvedovini]

@grapplerulrich There's no additional work required from the translators, on the contrary. First, that's a totally optional solution, the "en_US" (or "en_UK") files don't have to be renamed to "en" (and even then, that's not much of a work), and second, it simplifies work for plugins and themes developers who wants to provide localised versions.

I can see some theoretical or political reasons why locales should be kept separated but from a purely practical standpoint, me having to provide 4+ version of spanish files where one would be enough to make most people happy is just ludicrous and actually prevent them from using my plugins in their language.

Contrary to what I said earlier there's also no actual requirement for WP to decide on a default locale per language, it would just be better. And it can also be done on a per language basis. I can imagine people agreeing on en_UK being the default English and if Ukrainian is an issue then let things separated (Tho I should note that the idea is to fallback on a default locale, not another language, like in your Ukrainian/Russian exemple).

[vanillalounge]

Answered on the ​Polyglots' P2. tl;dr: either no fallback at all, or else let the user choose from *all* available languages.

[vanillalounge]

Replying to nacin:

A proper mapping of language codes that can be fallen back to could help a bit with both this and performance.

This might prove a task of epic proportions. ​Ethnologue has been trying, for years, and it's still not completely accurate, despite the thousands of entries. I understand the engineering drive to normalize this stuff, but suspect that there are simply too many shades and nuances (not to mention the fact that it's constantly changing) for a usable, discrete classification. I'm not even sure it's desirable.

[vanillalounge]

Replying to downstairsdev:

1) Translation teams from different locales could focus on the generic fallback version first- and then quickly copy it to make country specific localizations. Hopefully it would not get political as any translator who disagreed with a translation in the default could just change it for their specific locale.

The problem with this is that there is no such thing as a "generic" version everyone would agree on, for any language, even if vaguely, and you'd end up with a total of 0 volunteers to translate that version. And of course, it would immediately get political, before anything else.

[grapplerulrich]

I can see some theoretical or political reasons why locales should be kept separated but from a purely practical standpoint, me having to provide 4+ version of spanish files where one would be enough to make most people happy is just ludicrous and actually prevent them from using my plugins in their language.

You could just as easily rename the spanish files instead of having the rest of the users have to create a "es" file.

@vanillalounge mentioned that this should be optional. We could always allow users to define the fallback language when defining the language that they need. define ( 'WPLANG', 'es_ES', 'es_MX' ) Most people need to make the change in wp-config.php so they may just as well define the fallback languages. We can always provide a few examples in the codex that people can copy.

[cvedovini]

There's a misunderstanding I think. First my initial proposition is not to consider a "generic" language file but eventually to consider one of the locale to be the default for one given language, like es_ES would be the default if a es_MX file is not present. You do that by providing a es file (renaming the es_ES.mo file to es.mo)

Second there's no asking other people to change what they do (including WP translators) if they don't want. But including that fallback mechanism would allow me (and others) to provide a unique language file for all locales of that language without the user doing anything (they can still define es_MX and get MX locale when available), me having to copy the same file over with a different name (like es_ES, es_MX, es_CL, es_PE, es_VE, etc.) or adding the code I described here: ​http://vedovini.net/2013/12/smart-fallback-mechanism-for-loading-text-domains-in-wordpress/

[vanillalounge]

Replying to cvedovini:

...but eventually to consider one of the locale to be the default for one given language, like es_ES would be the default if a es_MX file is not present.

I'm sorry, but this is a recipe for disaster. Who gets to decide which one is the default? The es_ES translators? The es_MX ones? Whichever option you choose will inevitably raise a (probably nasty) discussion. Who will moderate that discussion (keeping in mind that it would be someone, who a)is qualified for that language and b)inevitably more familiar with one of the declinations)?

...including that fallback mechanism would allow me (and others) to provide a unique language file for all locales of that language...

Again, how will you provide that? Whose translation will it be? Will you be the one deciding which es_* is the generic one?

Look, I don't mean to sound irascible and unwilling to compromise, but you are approaching this from an angle that's completely irrelevant to translators or native speakers of another language. I may be wrong, but this ticket seems to aim to solve a problem of the developers, not of the translators or native speakers. We couldn't care less about fallback languages; from our point of view, meaning no offense, it's a solution looking for a problem to solve.

I would much rather see these efforts directed towards a procedure where a translator can easily and readily supply a current translation to a developer (instead of "oh, I'll include it in the next release", which is flawed), that we can trust that the code is properly gettexted and that .pot files are current with said code. Sadly, in 90% of the code I look at, this is most definitely not the case.

Also see ​http://make.wordpress.org/polyglots/2014/05/12/hi-folks-we-started-a-conversation-about-fallback/#comment-251797

[downstairsdev]

@cvedovini I generally agree with the criticisms of a generic language fallback. If the intention is to fallback to a specific locale (es_ES > es_MX for example), there is no need to create an additional language file or have translators change their workflow. WordPress would just load the fallback locale automatically.

This also appears to be too political and culturally sensitive to stack rank fallback locales based on native speaker population sizes (or other more arbitrary measures).

@grapplerulrich If the user needs to define the fallback rank order in wp-config.php, I'm not sure it's worth doing. The idea is that non-technical users and non-English speakers would have a better experience by providing a fallback automatically. A plugin would likely be easier for non-technical users to install and configure than editing code.

@vanillalounge This ticket is not aimed at solving a problem for translators nor developers- it's looking to provide a better experience for non-English speaking users.

That said, I think some exciting changes are on the way in terms of language packs and better tools for translators. I agree this is super important and would resolve some of the pain points in terms of getting plugin/theme authors to release translations and keep them up to date.

-

I appreciate all the discussion! I'm leaning towards closing the core ticket and releasing this idea as a plugin instead.

[cvedovini]

Replying to vanillalounge:

Replying to cvedovini:

...but eventually to consider one of the locale to be the default for one given language, like es_ES would be the default if a es_MX file is not present.

I'm sorry, but this is a recipe for disaster. Who gets to decide which one is the default? The es_ES translators? The es_MX ones? Whichever option you choose will inevitably raise a (probably nasty) discussion. Who will moderate that discussion (keeping in mind that it would be someone, who a)is qualified for that language and b)inevitably more familiar with one of the declinations)?

First I meant "may be" not "eventually" (typical French speaker error), sorry for that. Now, like I said several times before:

1. I know it's going to be a political issue (despite the fact it's detrimental to the end user experience and that other organisations actually manage to do it )
2. The core translators don't have to do anything if they don't want to.

...including that fallback mechanism would allow me (and others) to provide a unique language file for all locales of that language...

Again, how will you provide that? Whose translation will it be? Will you be the one deciding which es_* is the generic one?

I am the one who provide the Spanish translations for my plugins, and I am using the translations that have been done by an American guy who happens to speak Spanish. I don't know which locale this is and I don't care, that's the point, I am practical, those translations are good enough for the Spanish speakers who use my plugins

Look, I don't mean to sound irascible and unwilling to compromise, but you are approaching this from an angle that's completely irrelevant to translators or native speakers of another language. I may be wrong, but this ticket seems to aim to solve a problem of the developers, not of the translators or native speakers. We couldn't care less about fallback languages; from our point of view, meaning no offense, it's a solution looking for a problem to solve.

it may be irrelevant to a translator but not to a Chilean who will have the core translated in his own locale of Spanish but none of the plugins or themes he is using despite some of them having an es_ES or es_MX file. If you let plugin and theme developers provide an es file (instead of forcing them to choose a locale or provide all of them) then that Chilean user may have more of his Wordpress in his language (or close enough)

And you are right about one thing, that solution does not aim at solving any translator's problem. But you're blind if you can't see the problem (I just explained it again in the previous paragraph)

I would much rather see these efforts directed towards a procedure where a translator can easily and readily supply a current translation to a developer (instead of "oh, I'll include it in the next release", which is flawed), that we can trust that the code is properly gettexted and that .pot files are current with said code. Sadly, in 90% of the code I look at, this is most definitely not the case.

I don't think this is relevant here, my plugins are "properly gettexted" and I don't see translators rushing to offer their services. I have to chase after my friends who have a command in another language or accept translations from plugin users, any of those being hard enough to keep up-to-date so that I end up with a Mexican volunteer who will create the first Spanish version, have a Argentinian maintain that file for the next version and a Colombian for the one after that.

[cvedovini]

Replying to downstairsdev:

@cvedovini I generally agree with the criticisms of a generic language fallback. If the intention is to fallback to a specific locale (es_ES > es_MX for example), there is no need to create an additional language file or have translators change their workflow. WordPress would just load the fallback locale automatically.

That solution (fallback to es.mo when es_MX.mo is not there) is simple and just require that the chosen default locale file (say es_ES.mo) be provided without a locale (es.mo) that's how it works in Java for example and I guess many other platforms and I don't see an uprising. It doesn't require complex calculations or configuration, it's straightforward.
Now, choosing a default locale is the responsibility of those who provide the component, I get to choose what is the default Spanish in my plugins or themes (more like I take what's given to me actually) and if the Wordpress guys don't want to choose then they don't provide a default, your problem.

@vanillalounge This ticket is not aimed at solving a problem for translators nor developers- it's looking to provide a better experience for non-English speaking users.

exactly, if I wanted to solve the problem for myself I would tell my users to translate the plugin themselves and be done with it

I appreciate all the discussion! I'm leaning towards closing the core ticket and releasing this idea as a plugin instead.

I though about releasing a plugin already but that would be useless. To have my plugins in the right language my users would need to install another plugin. I am better off coding the hack in each and everyone of them, like I do now.

[nacin]

For the moment, this sounds like a good plugin.

[Kau-Boy]

As this whole topic is crucial for German websites using the new locale "de_DE_formal", we also needed a solution. I would agree with @nacin that it should start in a plugin and with @vanillalounge that the user should choose which fallback to use. I just released a plugin that will enable this functionality for one fallback language. You can find the plugin here: ​https://wordpress.org/plugins/language-fallback/

We also thought about a "fallback chain", but I think right now it's good enough to define one fallback, that fits best for your blog.

[p_enrique]

I found this ticket when my translations where not working as I expected -- I assumed that a fallback language would be included as a matter of course. I just finished translating my plugin to Spanish, and I was expecting that text-domain-es.mo would work for the entire Spanish-speaking world instead of needing text-domain-es_ES.mo, text-domain-es_AR.mo et cetera.

I have years of experience living in South America, and I believe firmly that a typical Chilean, Peruvian, Argentinian or Colombian user, to name but a few, would rather have anything in Spanish instead of getting an English-language UE. In the Spanish-speaking world, people are accustomed to the existence of regional variations and the appearance of technical terms from another culture in the IT context.

Now I am practically forced to copy the same translations over to es_AR, es_CL, es_CO, es_MX, es_PE, es_PR and es_ES to give my users at least something in their own language instead of the plugin's native English. I think this is just silly and bloats the size of the plugin. Additionally, copying translations this way is the real insult to cultural sensivity, as I am only pretending to offer a culture-specific custom translation for, let's say, Argentinians, but am actually only copy-pasting the generic Spanish content.

I want to add that speak Finnish, English and Spanish at a very fluent level and have a good command of French, German and Swedish, so I'm not totally unaware of the issues of inter-cultural communication.

[Kau-Boy]

@p_enrique I can totally understand, that you dislike the fact, that a locale es should be taken for all country specific variants. But even doing this would not be enough. Think about a locale such as ug_CN (Uighur - China). The best fallback in this case might be zh_CN (Simplified Chinese).

So simply takting the first part of a locale and falling back to any language starting with the same part, might not be enough.

I agree that it would be nice if the functionality of my plugin would enter core (sooner or later) and the user can define his prefered fallback. Although automatically picking a fallback, as the patch to this ticket proposes, would not solve the problem for all combinations.

[cvedovini]

@p_enrique you can use that trick in your plugin, that's what I do: ​http://vedovini.net/2013/12/smart-fallback-mechanism-for-loading-text-domains-in-wordpress/

[Kau-Boy]

@cvedovini this is not something every plugin developer should have to implement into it's own plugin. And as mentioned in many comments above, it is not an ideal solution to the issue.

The fix should be handled in core or in a plugin like the one I wrote for this issue (but preferably in core in the future).

[swissspidy]

A proper mapping of language codes that can be fallen back to could help a bit with both this and performance.

Why not create a new translatable string for a fallback chain. For example, in de_CH_formal we'd say we want to fall back to de_CH, de_DE_formal, de_DE — and of course to en_US in the end. This information could be translated on translate.wordpress.org by each locale.

The translation contributors know best which fallback works for their locale. We shouldn't force a specific fallback.

Summary:

• Core solution with minimal overhead (just 1 translatable string, no UI needed)
• Locales know best which fallbacks work for them
• Also works for cases like ug_CN -> zh_CN

[cvedovini]

@Kau-Boy Agreed, that's why I started a discussion about this on the "ideas" forum and @downstairsdev created that ticket, but since this is not going to end up in the core and your plugin doesn't solve the problem either that's all we are left with.

And the reason the solution in your plugin is not good is that it also ask the plugin developer to create multiple copies of the same file. Eg. if I want to cover Spanish in all locales with one translation I need to copy one file into all the locales.

And although I agree that there are other issues like the one you mentioned (ug_CN vs zn_CH) they are edge cases that shouldn't take precedence over the overwhelming number of people who speak Spanish, English or French (plus in that case I doubt Uyghur would like to see anything in Chinese, I have been there and most of them don't speak Chinese at all, let alone read it)

@swissspidy again, you are talking from the point of view of the core translators but this ticket in no way asked them to be implicated. I discussed this at length in my previous comments with others. Translators of the core don't need to do anything, if they don't want to take any decision about what language to fallback to then fair enough. If they want to leave one specific locale of a language without translation because they don't have a translator for that particular locale then that's their responsibility.

However, I as a plugin developer, I take my own responsibilities and decided that the Spanish translation I got from the person who volunteered to translate is good enough for every Spanish speaking person whatever their locale. This because I don't have the resource to have a translator for every locale, that duplicating the files is unpractical and that my Spanish users probably don't care that much. (you can replace Spanish with English also)

[cvedovini]

one last thing: this fallback solution I proposed, so far as been good enough for Microsoft, Sun and Apple (just to name a few) and I just think it should be good enough for Wordpress

[p_enrique]

I agree with @cvedovini. It's a sad situation for software development if you can't accept a new feature because it doesn't solve every situation, such as the case of ug_CN -> zh_CN. It should be understood that a language file with no sub-domain (regional variant) specification is to be used as the most neutral, generic translation. So that language-domain-es.mo is good for Colombian, Peninsular or Rioplatense Spanish. So that we don't need two copies of language-domain-sv for Swedish from Sweden and Swedish from Finland. So that people who are upset by an unfamiliar term in the generic translation are welcome to contribute their own regional variant. So that we can offer the users an understandable, communicable experience in a familiar language, instead of falling back to English when the user is Swiss-German and not German-German.

I don't see this as a feature to be added, but a shortcoming in the current implementation of gettext that needs to be corrected.

Additional fallback languages to be chosen by the user may be implemented as a plugin, I can see the added value of this even though I think it serves a minor purpose.

[Kau-Boy]

The problem is, that probably not a single theme or plugin has a mo-file for es or de but they more probably will have es_ES and de_DE. So falling back to a "generic locale" we have to define those generics.

So a solution like @swissspidy propesed would be a variant. Adding a filter on that which a plugin could use to alter the fallbacks (or a custom translation by the translation maintainers, as @swissspidy proposed originally) might work for core.

[downstairsdev]

Here's the plugin I was working on: ​https://github.com/devinsays/fallback-languages.

I got busy and never quite finished it, but the idea was that you could choose specific fallback locales to use. If none of those locales are available, you can also choose to fallback to the first locale found that matches the language. I'll finish that up when I have a free weekend and maybe see if @Kau-Boy wants to merge.

[ChantalC]

This should be in core. For both Dutch and German there are two different translations (normal and formal). In the Netherlands people are complaining that if they choose the formal translation, fallback language is English in stead of the normal Dutch translation. It takes a lot of time and efforts to translate the formal versions of core/plugins/themes.

[rianrietveld]

Replying to ChantalC:

This should be in core.

+1

[swissspidy]

This topic came up during this week's i18n bug scrub and I showed interest in it. I would really love to solve this issue in not so distant future.

Plenty of people showed interest in a solution for this, and ​the plugin mentioned earlier has over 1,000 installs.

Since the introduction of language packs for core, themes, and plugins, it's easier than ever to use the software in your desired language.

Now, we need to find a good way for having fallback languages. @ocean90 suggested tackling this in a feature project of some sorts, so if anyone's interested in joining me to do so, let me know.

Based on this thread and my experience with locale installs I want this project to solve this in a way that

• it's easy to implement
• we don't add any UI option
• the user doesn't need to worry about it
• developers don't need to worry about it

Perhaps it makes most sense if this could be handled on the WordPress.org side of things (the translation API), otherwise a translatable string for a fallback chain comes to mind again.

Anyway, we need to discuss the problem we aim to solve first before thinking about the implementation.

[Kau-Boy]

@swissspidy I already planed to suggest this as a feature project (I am the author of the plugin you mentioned). But after reading the post on the make blog, I thought I have to add some unit tests first.

If would love to see the language fallback implemented into core as soon as possible. Even though that would mean the "death" for my plugin as it is now. But that's a good thing :)

I also disliked the idea of additional UI elements. The first version only had a filter. But users demanded something they can easily use.

On the contributor day at WordCamp Cologne last year, we discussed some options. Implementing the fallback languages as a translatable string would make sense. Adding the option to filter it (as I have done in the plugin) would be a good solution.

So count me in for finding a solution that could go into core.

[cvedovini]

Just one remark: since this is supposed to address the problem of missing locale specific files I wonder how the "translatable string" solution is going to work?

[Kau-Boy]

@cvedovini something like this:

if ( 'off' !== _x( 'off', 'Fallback locales' ) ) {
// do the language fallback
}

A translation team could than translate this string e.g. for the locale "de_DE_formal" to the string "de_DE".

[cvedovini]

I see at least two problems here:

1. this is a new file that has to be maintain centrally by people who clearly expressed they were against the idea of fallback languagesat least because it's a political nightmare

2. if I don't have the resources to provide say all versions of Spanish, that the fallback for all Spanish locales is es_ES and have only a Mexican translator, I will have to provide a es_ES with mexican translations. Which is incorrect as far as labeling is concerned.

there is already a simple mechanism, built in the naming convention of language files, I don't understand why you wouldn't want to use that.

[Kau-Boy]

Using a translatable core string does not require any new files.

To whole point of this new core functionality is to have a generic fallback for all plugins and themes. You are right, that having a core translation file named with the locale name would be OK for the WordPress core. But all plugins and themes, that do not have a correct lanuguage variant file, would fall back to English.

As you can see from the active install of my plugin, there is a great nead for such a feature and having it in core would make it even easier for all users. As for German, using the formal variant has become more common, now that you can choose it from the backend settings or on installation. For those users, a default fallback should be in core.

[cvedovini]

don't get me wrong, I am 100% for a mechanism for fallback languages, I am the one who posted the idea in the "ideas" forum and then someone created that ticket based on that.

What I am discussing is the implementation of the solution and the fact that initially it's not a feature for core translators but for plugin and theme developers.

I believe the solution I proposed is the simplest and cover almost all cases without requiring the translators in the core to go into political warfare about what should be the fallback for such and such language in the core. I am not concerned about the core but I want a simple (for me) and transparent (for the users) way to provide translations to my plugin users. It's also a solution which is common everywhere else.

[cvedovini]

Replying to Kau-Boy:

Using a translatable core string does not require any new files.

still, it's a new data that needs to be managed (I thought of new file because I imagined a new domain, I see now you're using "_x" in your example, my bad)

To whole point of this new core functionality is to have a generic fallback for all plugins and themes. You are right, that having a core translation file named with the locale name would be OK for the WordPress core. But all plugins and themes, that do not have a correct lanuguage variant file, would fall back to English.

it's not working much for plugins and themes either, you will require that they actually provide the fallback (say "es_ES") which is the same as me asking them to provide "es", so in that front there is no difference

As you can see from the active install of my plugin, there is a great nead for such a feature and having it in core would make it even easier for all users. As for German, using the formal variant has become more common, now that you can choose it from the backend settings or on installation. For those users, a default fallback should be in core.

like I said before, I totally agree the mechanism should be in the core. the question is what mechanism? I prefer something simple that is conform to standards that are already used elsewhere, that's all.

[Kau-Boy]

I know that you are not against it :)

But how would a static fallback to a "generic" locale such as "es_ES" or "de_DE" be better than a translatable string? I don't think, that the translation teams would have to fight out a political warfare. If any user from a lanugage variant want's to change the fallback for all plugins and themes, he can easily change the translation of this one string.

If we would have a function with some generic fallback as in your example code, we must have a filter to allow users to alter the fallback anyways. But a translatable string is more flexible, as it easily provides a way to have a "fallback chain" and we don't have to add all those fallbacks statically into some core files.

For me as a plugin developer, I don't want to deal with those things myself. If I prepare it correctly, anyone can translate it on translate.wordpress.org into their own language.

[ocean90]

A translatable string doesn't seem to be the right way for this as it doesn't cover all use cases. Example: I speak English, Spanish and German. If German isn't available I'd like to have it fall back to Spanish. A translation team would probably never specify this. It needs to be a user choice IMO.

A few months ago I ​asked if anyone knows some software which supports language fallbacks and ​suggested to document the results. Looks like no one did that yet.

Using this Trac ticket to find a solution is a bad idea. Taking this as a feature project is probably the right way, regular meetings and documenting the findings in a make/core post is something which would keep the idea alive.

[cvedovini]

Replying to Kau-Boy:

But how would a static fallback to a "generic" locale such as "es_ES" or "de_DE" be better than a translatable string?

Not a static "es_ES" but "es" which is the naming convention for the locale-independent language file for Spanish (mydomain_es.pot)

I don't think, that the translation teams would have to fight out a political warfare.

It's been said before, not by me, and I can totally see it. What do you think will happen if the decision is officially made by the Wordpress.org entity that zh_CN should fallback to zh_TW (which is the logical choice from a language perspective since mainland Chinese can read traditional characters but Taiwanese cannot read simplified ones)

My solution doesn't require anybody in core to take that decision if they don't want to (there's no managing a central piece of data giving a fallback chain)

If any user from a language variant want's to change the fallback for all plugins and themes, he can easily change the translation of this one string.

It wouldn't work better than my solution because plugins and themes who don't provide all variants are only providing one or two, the chance that any set of plugins and themes would all provide one particular locale to fallback to is the same as the chance they would provide the actual locale you want (which is slim). Plus, such a mechanism can still be achieved with a plugin and you could recycle yours ;)

Now, if plugin and themes want to provide a locale-independent translation for Spanish (I use Spanish because there's a lot of locales) they can do so by providing an "es" file.

Same for the core translators. If they decide that for certain languages they deem non-politically dangerous (for example French) to provide a locale-independent version, they can do it by providing a new file (or renaming an old one). It works too is they start to cover a new language but do not want to bother covering locales for that language.

If we would have a function with some generic fallback as in your example code, we must have a filter to allow users to alter the fallback anyways. But a translatable string is more flexible, as it easily provides a way to have a "fallback chain" and we don't have to add all those fallbacks statically into some core files.

like I said just above, your plugin could provide that mechanism and it makes sense in that case that it's out of core.

For me as a plugin developer, I don't want to deal with those things myself. If I prepare it correctly, anyone can translate it on translate.wordpress.org into their own language.

Am afraid that if users who don't see the specific locale they want were concerned enough to actually translate our plugins we wouldn't have that discussion.

[cvedovini]

Replying to ocean90:

A translatable string doesn't seem to be the right way for this as it doesn't cover all use cases. Example: I speak English, Spanish and German. If German isn't available I'd like to have it fall back to Spanish. A translation team would probably never specify this. It needs to be a user choice IMO.

I agree and I think this particular use-case should be handled with a plugin.

[cvedovini]

Replying to ocean90:

A few months ago I ​asked if anyone knows some software which supports language fallbacks and ​suggested to document the results. Looks like no one did that yet.

Revisiting this thread I noticed I had missed that question before. So to answer it: all softwares developed for Windows, Apple iOS and MacOS or using Java as a programming language are using non-locale specific language files as fallback when a locale-specific language file is not present (for Microsoft and Apple it is baked in the OS itself, for Java it's the standard discovery mechanism for language files)

[kwadronaut]

And they all fall back to 'und'(efined)… @cvedovini is on the right track with fallback to 'generic.' What's used in the ​cldr are a distance between 2 pairs and a likelihood to calculate how good another match would be. ​Go takes a similar but simpler approach. It would avoid many frustrated users (my lang_reg_script isn't supported! Stupid imperialists!) and developers (what's this es-419? Do I also need an es_mx? What language symbol do they have there and there for their local currency?).

It must not be achieved by a separate file for each and every localization combination, a combo like: site in es_ec, plugin in es_mx, theme in es_ar could work fine and shouldn't display a single English word. Fallback means for me to be able to set preferences (nl_be, de_au, he, sr_latin, mo…) and that when there's a close enough match, to display that string (to continue the example, if a string isn't translated exactly check if it exists in: nl_nl, de_formal, de_informal, de_ch, il, hr, ro…).

That gives peace to:

• developers:
  • no need to continue es_ar if next time someone helps whose native tongue is es_mx, they translate only the new missing strings to es_mx and the fallback to es-419 makes most people believe it's well translated
  • no need to decide for a Galician website to fallback to Castillian or Portuguese (Iberia), the user will have set their preference
  • no need to copy paste 5 different files and claim you support 6 different German variants, with only formal and informal both the Swiss, Austrian and German users will enjoy your localized plugin!
  • focus on the engineering and designing, no fiddling with the sorting of Ä in Danish or understanding which scripts are used where for Azerbaijani.

• users:
  • in a great spot to know their content and (targeted) audience, see above
  • prefer Lingala and forgot to say that you prefer a plugin from a Belgian developer in fr_BE to the en version? No worries, we got your back.
  • I detest the dialect those people from across the river speak. Never ever fallback, give me the old behavior. Also the imperial approach of 1 language is faster ;-)

• Translators want their work used and appreciated. Users might prefer nl_be, but when a translation isn't updated for a new release, or some strings are missing, the translation will not get used and the translator will receive blame instead of praise. With fallback there's no blame. More time for quality control.

Things that make sense to me:

• stick to BCP-47 (aka convince @cvedovini I'm not crazy and accept en as English)
• Use CLDR: ​http://www.unicode.org/reports/tr35/#LanguageMatching it already has a ton of good data for currencies, cities, and localeMatcherTestfiles.
• Go's language matching: ​https://godoc.org/golang.org/x/text/language#Matcher
• how people pick their preferences is something that can be changed or ameliorated later on, a simple config file change can be a great start.
• those aiming for perfect localization can continue adding more and more specific files. Don't hold yourself back.

[joostdevalk]

I would like to revisit this ticket as this would really be helpful in many cases. @swissspidy @ocean90 @SergeyBiryukov what's needed to get this ball rolling?

[ChantalC]

@joostdevalk There is a feature project on ​https://github.com/swissspidy/preferred-languages

Don't know if it is still actively developed by @swissspidy though.

[swissspidy]

Hey!

So the Preferred Languages plugin has been available on WordPress.org for quite some time: ​https://wordpress.org/plugins/preferred-languages/. I don't know why I never mentioned it here in the ticket though.

I published the last update in October but have yet to announce its availability on make/core (​the last post is from May 2017). There are a few bugs I'd like to resolve / wrap my head around first though, but I don't have time until next month.

There are some great ideas being shared here, everything from manual to automatic fallbacks. I see the biggest potential for this going into core in the manual way though. That's why I developed this plugin as a proof-of-concept.

I'd be happy to lead another meeting in #core-i18n in February to discuss the status quo.

In the meantime, if you want to test the Preferred Languages plugin, please go ahead and report any issues on GitHub. Any help is highly appreciated!

[tobifjellner]

Replying to swissspidy:

I published the last update in October but have yet to announce its availability on make/core (​the last post is from May 2017). There are a few bugs I'd like to resolve / wrap my head around first though, but I don't have time until next month.

Have you had time yet to fix those issues you had? Would it be time soon to suggest this functionality for core?

[swissspidy]

@tobifjellner Not yet unfortunately. A lot has changed in the past 20 months in core JavaScript land (React, JS I18N), so the plugin needs some polishing in that regard. Perhaps I get to that by the end of the year. Please follow progress at ​https://github.com/swissspidy/preferred-languages. Contributions are of course welcome :-)