Bug 1674157 - Expand TP6 list to include sites with higher numbers of origins r=perftest-reviewers,AlexandruIonescu

Differential Revision: https://phabricator.services.mozilla.com/D109709
2021-03-30 07:15:57 +00:00 · 2021-03-30 07:15:57 +00:00 · 04a818afa4
--- a/taskcluster/ci/test/browsertime-desktop.yml
+++ b/taskcluster/ci/test/browsertime-desktop.yml
@ -83,7 +83,11 @@ browsertime-tp6:
    raptor-subtests:
        - apple
        - [bing-search, bing]
+        - buzzfeed
+        - cnn
        - ebay
+        - espn
+        - expedia
        - [facebook, fb]
        - fandom
        - [google-docs, gdocs]
@ -94,11 +98,13 @@ browsertime-tp6:
        - linkedin
        - microsoft
        - netflix
+        - nytimes
        - office
        - outlook
        - paypal
        - pinterest
        - reddit
+        - wikia
        - wikipedia
        - yahoo-mail
        - yahoo-news
--- a/testing/perfdocs/generated/raptor.rst
+++ b/testing/perfdocs/generated/raptor.rst
@ -82,6 +82,52 @@ Tests for page-load performance. The links direct to the actual websites that ar
   * **use live sites**: false


+.. dropdown:: buzzfeed (BT, FF, CH, CU)
+
+   * **alert on**: fcp, loadtime
+   * **alert threshold**: 2.0
+   * **apps**: firefox, chrome, chromium
+   * **browser cycles**: 25
+   * **expected**: pass
+   * **gecko profile entries**: 14000000
+   * **gecko profile interval**: 1
+   * **lower is better**: true
+   * **measure**: fnbpaint, fcp, dcf, loadtime
+   * **page cycles**: 25
+   * **page timeout**: 60000
+   * **playback**: mitmproxy
+   * **playback pageset manifest**: mitm5-linux-firefox-{subtest}.manifest
+   * **playback recordings**: mitm5-linux-firefox-{subtest}.mp
+   * **playback version**: 5.1.1
+   * **test url**: `<https://www.buzzfeed.com/>`__
+   * **type**: pageload
+   * **unit**: ms
+   * **use live sites**: false
+
+
+.. dropdown:: cnn (BT, FF, CH, CU)
+
+   * **alert on**: fcp, loadtime
+   * **alert threshold**: 2.0
+   * **apps**: firefox, chrome, chromium
+   * **browser cycles**: 25
+   * **expected**: pass
+   * **gecko profile entries**: 14000000
+   * **gecko profile interval**: 1
+   * **lower is better**: true
+   * **measure**: fnbpaint, fcp, dcf, loadtime
+   * **page cycles**: 25
+   * **page timeout**: 60000
+   * **playback**: mitmproxy
+   * **playback pageset manifest**: mitm5-linux-firefox-{subtest}.manifest
+   * **playback recordings**: mitm5-linux-firefox-{subtest}.mp
+   * **playback version**: 5.1.1
+   * **test url**: `<https://www.cnn.com/2021/03/22/weather/climate-change-warm-waters-lake-michigan/index.html>`__
+   * **type**: pageload
+   * **unit**: ms
+   * **use live sites**: false
+
+
 .. dropdown:: ebay (BT, FF, CH, CU)

   * **alert on**: fcp, loadtime
@ -105,6 +151,52 @@ Tests for page-load performance. The links direct to the actual websites that ar
   * **use live sites**: false


+.. dropdown:: espn (BT, FF, CH, CU)
+
+   * **alert on**: fcp, loadtime
+   * **alert threshold**: 2.0
+   * **apps**: firefox, chrome, chromium
+   * **browser cycles**: 25
+   * **expected**: pass
+   * **gecko profile entries**: 14000000
+   * **gecko profile interval**: 1
+   * **lower is better**: true
+   * **measure**: fnbpaint, fcp, dcf, loadtime
+   * **page cycles**: 25
+   * **page timeout**: 60000
+   * **playback**: mitmproxy
+   * **playback pageset manifest**: mitm5-linux-firefox-{subtest}.manifest
+   * **playback recordings**: mitm5-linux-firefox-{subtest}.mp
+   * **playback version**: 5.1.1
+   * **test url**: `<http://www.espn.com/nba/story/_/page/allstarweekend25788027/the-comparison-lebron-james-michael-jordan-their-own-words>`__
+   * **type**: pageload
+   * **unit**: ms
+   * **use live sites**: false
+
+
+.. dropdown:: expedia (BT, FF, CH, CU)
+
+   * **alert on**: fcp, loadtime
+   * **alert threshold**: 2.0
+   * **apps**: firefox, chrome, chromium
+   * **browser cycles**: 25
+   * **expected**: pass
+   * **gecko profile entries**: 14000000
+   * **gecko profile interval**: 1
+   * **lower is better**: true
+   * **measure**: fnbpaint, fcp, dcf, loadtime
+   * **page cycles**: 25
+   * **page timeout**: 60000
+   * **playback**: mitmproxy
+   * **playback pageset manifest**: mitm5-linux-firefox-{subtest}.manifest
+   * **playback recordings**: mitm5-linux-firefox-{subtest}.mp
+   * **playback version**: 5.1.1
+   * **test url**: `<https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470&regionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2>`__
+   * **type**: pageload
+   * **unit**: ms
+   * **use live sites**: false
+
+
 .. dropdown:: facebook (BT, FF, CH, CU)

   * **alert on**: fcp, loadtime
@ -404,6 +496,29 @@ Tests for page-load performance. The links direct to the actual websites that ar
   * **use live sites**: false


+.. dropdown:: nytimes (BT, FF, CH, CU)
+
+   * **alert on**: fcp, loadtime
+   * **alert threshold**: 2.0
+   * **apps**: firefox, chrome, chromium
+   * **browser cycles**: 25
+   * **expected**: pass
+   * **gecko profile entries**: 14000000
+   * **gecko profile interval**: 1
+   * **lower is better**: true
+   * **measure**: fnbpaint, fcp, dcf, loadtime
+   * **page cycles**: 25
+   * **page timeout**: 60000
+   * **playback**: mitmproxy
+   * **playback pageset manifest**: mitm5-linux-firefox-{subtest}.manifest
+   * **playback recordings**: mitm5-linux-firefox-{subtest}.mp
+   * **playback version**: 5.1.1
+   * **test url**: `<https://www.nytimes.com/2020/02/19/opinion/surprise-medical-bill.html>`__
+   * **type**: pageload
+   * **unit**: ms
+   * **use live sites**: false
+
+
 .. dropdown:: office (BT, FF, CH, CU)

   * **alert on**: fcp, loadtime
@ -588,6 +703,29 @@ Tests for page-load performance. The links direct to the actual websites that ar
   * **use live sites**: false


+.. dropdown:: wikia (BT, FF, CH, CU)
+
+   * **alert on**: fcp, loadtime
+   * **alert threshold**: 2.0
+   * **apps**: firefox, chrome, chromium
+   * **browser cycles**: 25
+   * **expected**: pass
+   * **gecko profile entries**: 14000000
+   * **gecko profile interval**: 1
+   * **lower is better**: true
+   * **measure**: fnbpaint, fcp, dcf, loadtime
+   * **page cycles**: 25
+   * **page timeout**: 60000
+   * **playback**: mitmproxy
+   * **playback pageset manifest**: mitm5-linux-firefox-{subtest}.manifest
+   * **playback recordings**: mitm5-linux-firefox-{subtest}.mp
+   * **playback version**: 5.1.1
+   * **test url**: `<https://marvel.fandom.com/wiki/Black_Panther>`__
+   * **type**: pageload
+   * **unit**: ms
+   * **use live sites**: false
+
+
 .. dropdown:: wikipedia (BT, FF, CH, CU)

   * **alert on**: fcp, loadtime
--- a/testing/raptor/raptor/perfdocs/config.yml
+++ b/testing/raptor/raptor/perfdocs/config.yml
@ -12,7 +12,11 @@ suites:
            amazon: "BT, FF, CH, CU"
            apple: "BT, FF, CH, CU"
            bing-search: "BT, FF, CH, CU"
+            buzzfeed: "BT, FF, CH, CU"
+            cnn: "BT, FF, CH, CU"
            ebay: "BT, FF, CH, CU"
+            espn: "BT, FF, CH, CU"
+            expedia: "BT, FF, CH, CU"
            facebook: "BT, FF, CH, CU"
            fandom: "BT, FF, CH, CU"
            google-docs: "BT, FF, CH, CU"
@ -26,6 +30,7 @@ suites:
            linkedin: "BT, FF, CH, CU"
            microsoft: "BT, FF, CH, CU"
            netflix: "BT, FF, CH, CU"
+            nytimes: "BT, FF, CH, CU"
            office: "BT, FF, CH, CU"
            outlook: "BT, FF, CH, CU"
            paypal: "BT, FF, CH, CU"
@ -34,6 +39,7 @@ suites:
            tumblr: "BT, FF, CH, CU"
            twitch: "BT, FF, CH, CU"
            twitter: "BT, FF, CH, CU"
+            wikia: "BT, FF, CH, CU"
            wikipedia: "BT, FF, CH, CU"
            yahoo-mail: "BT, FF, CH, CU"
            yahoo-news: "BT, FF, CH, CU"
--- a/testing/raptor/raptor/perftest.py
+++ b/testing/raptor/raptor/perftest.py
@ -6,7 +6,6 @@

 from __future__ import absolute_import

-import six
 import json
 import os
 import re
@ -19,12 +18,12 @@ from abc import ABCMeta, abstractmethod

 import mozinfo
 import mozprocess
-import mozversion
 import mozproxy.utils as mpu
+import mozversion
+import six
 from mozprofile import create_profile
 from mozproxy import get_playback

-
 # need this so raptor imports work both from /raptor and via mach
 here = os.path.abspath(os.path.dirname(__file__))
 paths = [here]
@ -37,7 +36,6 @@ for path in paths:
        raise IOError("%s does not exist. " % path)
    sys.path.insert(0, path)

-
 from cmdline import FIREFOX_ANDROID_APPS
 from condprof.client import get_profile, ProfileNotFoundError
 from condprof.util import get_current_platform
--- a/testing/raptor/raptor/tests/tp6/desktop/browsertime-tp6.ini
+++ b/testing/raptor/raptor/tests/tp6/desktop/browsertime-tp6.ini
@ -34,9 +34,21 @@ test_url = https://www.apple.com/macbook-pro/
 [bing-search]
 test_url = https://www.bing.com/search?q=barack+obama

+[buzzfeed]
+test_url = https://www.buzzfeed.com/
+
+[cnn]
+test_url = https://www.cnn.com/2021/03/22/weather/climate-change-warm-waters-lake-michigan/index.html
+
 [ebay]
 test_url = https://www.ebay.com/

+[espn]
+test_url = http://www.espn.com/nba/story/_/page/allstarweekend25788027/the-comparison-lebron-james-michael-jordan-their-own-words
+
+[expedia]
+test_url = https://expedia.com/Hotel-Search?destination=New+York%2C+New+York&latLong=40.756680%2C-73.986470&regionId=178293&startDate=&endDate=&rooms=1&_xpid=11905%7C1&adults=2
+
 [facebook]
 test_url = https://www.facebook.com

@ -76,6 +88,9 @@ test_url = https://www.microsoft.com/en-us/
 [netflix]
 test_url = https://www.netflix.com/title/80117263

+[nytimes]
+test_url = https://www.nytimes.com/2020/02/19/opinion/surprise-medical-bill.html
+
 [office]
 test_url = https://office.live.com/start/Word.aspx?omkt=en-US
 playback_recordings = mitm5-linux-firefox-live-office.mp
@ -104,6 +119,9 @@ test_url = https://www.twitch.tv/videos/326804629
 [twitter]
 test_url = https://twitter.com/BarackObama

+[wikia]
+test_url = https://marvel.fandom.com/wiki/Black_Panther
+
 [wikipedia]
 test_url = https://en.wikipedia.org/wiki/Barack_Obama

--- a/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-buzzfeed.manifest
+++ b/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-buzzfeed.manifest
@ -0,0 +1,9 @@
+[
+ {
+    "filename": "mitm5-linux-firefox-buzzfeed.zip",
+    "size": 9558243,
+    "algorithm": "sha512",
+    "digest": "f73b07ae7df78a54cb660ea6e1e6375ee49c24bcef687b90d5356cf3bd6e74635bfb8a26160459ac326466443a62b41daecf5edb31dfefff2f0adc7b9fe0768f",
+    "visibility": "public"
+  }
+]
--- a/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-cnn.manifest
+++ b/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-cnn.manifest
@ -0,0 +1,9 @@
+[
+  {
+    "filename": "mitm5-linux-firefox-cnn.zip",
+    "size": 39806225,
+    "algorithm": "sha512",
+    "digest": "033d5b87d18a299dacc441f007febb5d104f5da39a8004a116904dea309a8351efedf604014196839107ad93abd264b02a70b8bea06a46ff9ae467d312b2efb2",
+    "visibility": "public"
+  }
+]
--- a/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-espn.manifest
+++ b/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-espn.manifest
@ -0,0 +1,9 @@
+[
+  {
+    "filename": "mitm5-linux-firefox-espn.zip",
+    "size": 7665306,
+    "algorithm": "sha512",
+    "digest": "ca0a6d9561bbac193162f797ac3e6226446408ca0d8268313956414db9170be6d7c0d7dc316df4de15a3ed4e0e8c948b60472511339313e58cfc331b556db522",
+    "visibility": "public"
+  }
+]
--- a/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-expedia.manifest
+++ b/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-expedia.manifest
@ -0,0 +1,9 @@
+[
+  {
+    "filename": "mitm5-linux-firefox-expedia.zip",
+    "size": 9707769,
+    "algorithm": "sha512",
+    "digest": "55c8d2d1e1c4a7417e80cd9f3ae250fd871b1ba583feca0a2bb29861bba4d39b9d75c2ca5b1d32c133a64b528cd8787c1af37c6308df8941b12494ddc63bcd17",
+    "visibility": "public"
+  }
+]
--- a/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-nytimes.manifest
+++ b/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-nytimes.manifest
@ -0,0 +1,9 @@
+[
+  {
+    "filename": "mitm5-linux-firefox-nytimes.zip",
+    "size": 6117949,
+    "algorithm": "sha512",
+    "digest": "037dafd334046bdf9837ad5a321eb03e7c043f0c9f145c2086d88ca345d9385ea2239109e0a3119976fe8cf44bdbf8ac2ade2411d0ff92ded8dba300ce2f9e81",
+    "visibility": "public"
+  }
+]
--- a/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-wikia.manifest
+++ b/testing/raptor/raptor/tooltool-manifests/playback/mitm5-linux-firefox-wikia.manifest
@ -0,0 +1,9 @@
+[
+  {
+    "filename": "mitm5-linux-firefox-wikia.zip",
+    "size": 4847101,
+    "algorithm": "sha512",
+    "digest": "c53c45c374cd924d03972f56b6a7416451861147f0a14b8c51c2c722a82582ae7ce8927a500981152aa6f4d3617b5ce455b85910981a9aabbd672e1ea47c0ff7",
+    "visibility": "public"
+  }
+]