Bug 1400006 - Extend language negotiation in LocaleService to support looking for the best likelySubtag for the locale with region stripped. r=Pike

Add additional logic to our language negotation to do apply likelySubtags when a direct match is not available.

Currently, if the user specifies the locale with region, and we do not have a direct for that region, we pick all locales for the same language and other regions in no order.

The example of where it returns suboptimal results:

1) Requested locale "en-CA"
2) Available locales ["en-ZA", "en-GB", "en-US"]
3) Negotiated locales ["en-ZA", "en-GB", "en-US"]

This would not happen, if the user requested a generic "de", "en" etc.:

1) Requested locale "en"
2) Available locales ["en-ZA", "en-GB", "en-US"]
3) Negotiated locales ["en-US", "en-ZA", "en-GB"]

because after not finding a direct match, we would use likelySubtags to extend "en" to "en-Latn-US" and then find the priority match in "en-US".

This patch extends this logic to "en-US" or "de-LU" by adding a step which strips the region tag and then applies likelySubtag on the result.

This means that in absence of direct match the following fallbacks would happen:

"de-LU" -> "de-DE"
"es-CL" -> "es-ES"
"en-CA" -> "en-US"

This does not affect languages that use multiple scripts, so ar, sr and zh are not affected.

MozReview-Commit-ID: BR1WrgXSf6a

--HG--
extra : rebase_source : abc205c4f993680ab0cd0c8b8c016543d5462d01

intl/locale/LocaleService.cpp

@@ -350,7 +350,7 @@ LocaleService::OnLocalesChanged()
  * This is the raw algorithm for language negotiation based roughly
  * on RFC4647 language filtering, with changes from LDML language matching.
  *
- * The exact algorithm is custom, and consist of 5 level strategy:
+ * The exact algorithm is custom, and consists of a 6 level strategy:
  *
  * 1) Attempt to find an exact match for each requested locale in available
  *    locales.
@@ -373,7 +373,14 @@ LocaleService::OnLocalesChanged()
  *               ^^^^^^^^^
  *               |----------- replace variant with range: 'ja-JP-*'
  *
- * 5) Attempt to look up for a different region of the same locale.
+ * 5) Attempt to look up for a maximized version of the requested locale,
+ *    stripped of the region code.
+ *    Example: ['en-CA'] * ['en-ZA', 'en-US'] = ['en-US', 'en-ZA']
+ *               ^^^^^
+ *               |----------- look for likelySubtag of 'en': 'en-Latn-US'
+ *
+ *
+ * 6) Attempt to look up for a different region of the same locale.
  *    Example: ['en-GB'] * ['en-AU'] = ['en-AU']
  *               ^^^^^
  *               |----- replace region with range: 'en-*'
@@ -459,7 +466,15 @@ LocaleService::FilterMatches(const nsTArray<nsCString>& aRequested,
       HANDLE_STRATEGY;
     }
 
-    // 5) Try to match against a region as a range
+    // 5) Try to match against the likely subtag without region
+    if (requestedLocale.AddLikelySubtagsWithoutRegion()) {
+      if (findRangeMatches(requestedLocale)) {
+        HANDLE_STRATEGY;
+      }
+    }
+
+
+    // 6) Try to match against a region as a range
     requestedLocale.SetRegionRange();
     if (findRangeMatches(requestedLocale)) {
       HANDLE_STRATEGY;
@@ -821,13 +836,35 @@ LocaleService::Locale::SetRegionRange()
 
 bool
 LocaleService::Locale::AddLikelySubtags()
+{
+  return AddLikelySubtagsForLocale(mLocaleStr);
+}
+
+bool
+LocaleService::Locale::AddLikelySubtagsWithoutRegion()
+{
+  nsAutoCString locale(mLanguage);
+
+  if (!mScript.IsEmpty()) {
+    locale.Append("-");
+    locale.Append(mScript);
+  }
+
+  // We don't add variant here because likelySubtag doesn't care about it.
+
+  return AddLikelySubtagsForLocale(locale);
+}
+
+bool
+LocaleService::Locale::AddLikelySubtagsForLocale(const nsACString& aLocale)
 {
 #ifdef ENABLE_INTL_API
   const int32_t kLocaleMax = 160;
   char maxLocale[kLocaleMax];
+  nsAutoCString locale(aLocale);
 
   UErrorCode status = U_ZERO_ERROR;
-  uloc_addLikelySubtags(mLocaleStr.get(), maxLocale, kLocaleMax, &status);
+  uloc_addLikelySubtags(locale.get(), maxLocale, kLocaleMax, &status);
 
   if (U_FAILURE(status)) {
     return false;
@@ -843,7 +880,10 @@ LocaleService::Locale::AddLikelySubtags()
   mLanguage = loc.mLanguage;
   mScript = loc.mScript;
   mRegion = loc.mRegion;
-  mVariant = loc.mVariant;
+
+  // We don't update variant from likelySubtag since it's not going to
+  // provide it and we want to preserve the range
+
   return true;
 #else
   return false;

intl/locale/LocaleService.h

@@ -271,7 +271,9 @@ private:
     void SetVariantRange();
     void SetRegionRange();
 
-    bool AddLikelySubtags(); // returns false if nothing changed
+    // returns false if nothing changed
+    bool AddLikelySubtags();
+    bool AddLikelySubtagsWithoutRegion();
 
     const nsCString& AsString() const {
       return mLocaleStr;
@@ -291,6 +293,8 @@ private:
     nsCString mScript;
     nsCString mRegion;
     nsCString mVariant;
+
+    bool AddLikelySubtagsForLocale(const nsACString& aLocale);
   };
 
   void FilterMatches(const nsTArray<nsCString>& aRequested,

intl/locale/tests/unit/test_localeService_negotiateLanguages.js

@@ -35,6 +35,9 @@ const data = {
       [["sr", "ru"], ["sr-Latn", "ru"], ["ru"]],
       [["sr-RU"], ["sr-Latn-RO", "sr-Cyrl"], ["sr-Latn-RO"]],
     ],
+    "should match likelySubtag region over other regions": [
+      [["en-CA"], ["en-ZA", "en-GB", "en-US"], ["en-US", "en-ZA", "en-GB"]],
+    ],
     "should match on a requested locale as a range": [
       [["en-*-US"], ["en-US"], ["en-US"]],
       [["en-Latn-US-*"], ["en-Latn-US"], ["en-Latn-US"]],