12 KiB

Исходник Ответственный История

Treeherder Test Cases

These tests should be run prior to deploying code changes to the Treeherder production environment.

Initial page load

Load Treeherder. eg.

Depending on your test requirement.

Expected: Page loads displaying pushes pushed to mozilla-inbound.

Treeherder logo > Perfherder

Expected: Perfherder loads displaying its initial Graph page.

Perfherder logo > Intermittent Failures View

Expected: IFV loads (this may timeout and show an error instead of failure data)

Intermittent Failures View logo > Treeherder

Expected: Treeherder loads again, displaying pushes per step 1.

Check Job details Tab selection

Load Treeherder and select a completed/success job.

Expected: The Job details tab should load by default.

Select a completed/failed job.

Expected: The Failure summary tab should load.

Select a completed-success Talos job.

Expected: The Performance tab should load.

Select a completed-failed Talos job.

Expected: The Failure summary tab should load.

Select a Running job.

Expected: The (empty) Failure summary tab should load.

Pin a job

Select a job, and click the 'pin' button in the lower navbar.

Expected: Selected job pinned

Select another job, and hit [spacebar]

Expected: Selected job pinned

Pinboard > Right hand menu dropdown > Clear all

Expected: Both jobs are removed from the pinboard.

Failure summary tab

Select a classified or unclassified failed job.

Expected: Ensure the Failure summary tab loads by default.

If a Bug suggestion is present in the failure summary:

Click on the bug description link
Click on the bug pin icon

Expected:

Bug description link should load the correct BMO bug in a new tab
Pin should pin the job and add the bug to the bug classification field

Pinboard > Right hand dropdown menu > Clear all

Expected:

The job should be removed from the Pinboard
The bug number should be cleared from the bug classification field

Similar jobs tab

Select a job, select the Similar jobs tab, wait several seconds.

Expected: Recent jobs with matching symbols should load.

Select a Similar job row.

Expected: The adjacent panel should update with its job information.

Scroll to the bottom of the Similar jobs tab, click 'Show previous jobs'.

Expected: Additional, older jobs with matching symbols should load.

Job details pane

Select any job and confirm the following loads in the bottom left pane:

Job:
Task: (test this link)
Machine name: (test this link) (this may not exist for taskcluster jobs)
Build: (test this link) (this may not be a link for taskcluster jobs)
Job name:
Requested:
Started: (only present for jobs in either running or completed states)
Ended: (only present for jobs in a completed state)
Duration:
Log parsing status:

Expected: Values load, are visible and correct, and links are valid.

Classify a job with associated bugs

Select and pin 3 jobs to the pinboard, select a classification type, add a classification comment and add bug 1164485. Select 'Save' in the pinboard.

Expected: The jobs show with an asterisk in the job table, green notification banners appear confirming successful classification for each job.

Click Annotations tab.

Expected: Ensure the same data appears in the panel.

Annotations tab > delete the bug and classification for that job. Select the other two jobs and repeat.

Expected: The jobs should be unclassified, annotations removed.

Reload the page.

Expected: The job should still be unclassified.

Switch repos

Click on the Repos menu, select a different repo.

Expected: The new repo and its pushes should load.

Reverse the process, and switch back.

Expected: The original repo and pushes should load.

Toggle unclassified failures

Load Treeherder and click on the "(n) unclassified" button in the top navbar.

Expected: Only unclassified failures should be visible in the job table.

Filters panel

Click and open the 'Filters' menu panel in the top navbar, and turn off several job types in the panel.

Expected: Job types turned off are suppressed in the job table.

Click on 'Reset' in the Filters panel.

Expected: Filters UI should revert and suppressed jobs should reappear in the job table.

Filters panel > Field Filters > click new. Add a new filter eg. Platform, Linux.

Expected: Only Linux platforms should be visible in the job table.

Filter by Job details name and signature

Select any job and in the lower left panel, click on the Job: keywords eg. "Linux x64 asan Mochitest Chrome"

Expected: Ensure only jobs containing those keywords are visible.

Select any job and click on the adjacent "(sig)" signature link.

Expected: Ensure only jobs using that unique signature SHA are visible.

Pin all visible jobs in push

Click on the Pin 'all' pin-icon in the right hand side of any push bar.

Expected: Up to a maximum of 500 jobs should be pinned, and a matching notification warning should appear if that limit is reached.

Click in the pinboard on the extreme right hand drop down menu, and select 'Clear all'.

Expected: All jobs should be removed from the pinboard.

Expected: The login button should switch to a generic "Person" avatar, and the user email should appear on hover.

Logout

Expected: The login button should switch back to "Login / Register".

View the Logviewer

Select any failed job and click the 'Log' icon in the lower navbar.

Expected: The Logviewer loads in a new tab, and it contains correct job and revision information in the top left corner, and it preloads and scrolls to the first failure line if one exists.

Click on another failure line.

Expected: The log should scroll to that failure line. Thumbwheel/scroll/swipe downwards or upwards.

Expected: The log should quickly load new chunks when encountering a log boundary.

Click on the Raw Log link.

Expected: The raw log for the same job should load in a new tab.

Click all the available links in the result header, eg. "Inspect Task".

Expected: Each should load correctly for that job.

Select Treeherder from the nav menu.

Expected: Treeherder should load in the same window.

View the raw log

Select any completed job and click the raw log button in the lower navbar.

Expected: The raw log for that job should load in a new tab.

View pushes by Author

Click on the Author email (eg. ryanvm@gmail.com) in a push bar.

Expected: Only pushes pushed by that Author should appear.

Get next 10| pushes via the main page footer.

Expected: Only pushes from that Author should be added.

View a single push

Load Treeherder and click on the 'Date' on the left side of any push.

Expected: Only that push should load, with an accompanying URL param "&revision=(SHA)"

(optional) Wait a minute or two for ingestion updates.

Expected: Only newly started jobs for that same push (if any have occurred) should appear. No new pushes should load.

Quick Filter input field

Click the 'Filter platforms & jobs' input field in the top navbar, aka. Quick Filter.

Expected: Input field should expand in width for long input.

Enter any text (eg. 'Android') and hit Enter

Expected: Filter should be applied against the visible jobs and platform rows.

Click the grey (x) 'Clear this filter' icon the right hand side of the input field, and hit Enter.

Expected: Filter should be cleared and input should shrink to original width.

From any push bar, select each entry in the far right dropdown that doesn't involve retriggers. eg:

Mark with Bugherder

Expected: Each should open without error or hanging.

Get next 10|20|50 pushes

Click on Get next 10| pushes.

Expected: Ensure exactly 10 additional pushes were loaded.

Click on Get next 50| pushes.

Expected: Ensure the page has a reasonable load time of ~10 seconds.

View a single push via its Date link. Click Get next 10| pushes.

Expected: Ensure the page loads the 10 prior pushes and the "tochange" and "fromchange" in the url appear correct.

Filter pushes by URL fromchange, tochange

See also the Treeherder userguide for URL Query String Parameters. Please test variants and perform exploratory testing as top/bottom of range is new functionality (Jun 3, 15') Navigate to the 2nd push loaded, from the push action menu select 'Set as top of range'.

Expected: Ensure: (1) 1st push is omitted (2) url contains &tochange=SHA and (3) ten pushes are loaded from that new top

Navigate to the 3rd push loaded and select 'Set as bottom of range'

Expected: Ensure (1) only the 3 ranged pushes are loaded (2) url contains &tochange=[top-SHA]&fromchange=[bottom-SHA]

Click Get Next | 10 in the page footer.

Expected: Ensure 10 additional pages load for a total of 13 pushes.

(optional) wait a minute or two for job and push updates

Expected: Updates should only occur for the visible pushes. No new pushes should appear.

Filter pushes by URL date range

See also the Treeherder userguide for URL Query String Parameters Add a revision range to the URL in the format, eg:

&startdate=2015-09-28&enddate=2015-09-28

Warning: With the latest volume of jobs and pushes, anything greater than a single day window risks loading too much data for the browser with Treeherder default filter settings.

Expected: pushes loaded should honor that range.

(Optional) Wait for new pushes to that repo.

Expected: pushes loaded should continue to honor that range.

Perfherder Graphs

Load Perfherder at eg. https://treeherder.allizom.org/perf.html

Expected: Landing page should appear.

Click the blue 'Add test data' button, select a platform, enter a test series, and click Add+.

Expected: Performance series should load with scatter graph and line graph.

Click Add more test data, and add a 2nd series.

Expected: The second series is drawn in an alternate color, and both series can have their displays disabled/enabled via Show/Hide series tick UI.

Change display range dropdown to 90 days (or other value)

Expected: Ensure both series expand to that date range. Confirm the data which has expired beyond the 6 week data cycle still appears, but the SHA just will instead display "loading revision".

No console errors throughout test run

Ensure the browser console is error free during and after the test run. Open the console during the test run.

Expected: No errors should appear in the console.

Perfherder Compare

Load Perfherder Compare at eg. https://treeherder.allizom.org/perf.html#/comparechooser

Expected: Landing page should appear.

Select two push revisions from the 'Recent' dropdowns, and click 'Compare revisions'.

Expected: Some kind of result should appear (likely a warning "tests with no results: " table).

Click on the 'Substests' link for a row.

Expected: Sub-compare results should appear.

Click on the 'Graph' link for a sub-compare row if it exists.

Expected: The plotted graph for that series should appear.

All keyboard shortcuts

Check all keyboard shortcut functionality as listed in the userguide.

Expected: Each shortcut should work as expected.

Job counts

In any push with job counts, click on the group button eg. B( ) to expand the count.

Expected: Jobs should appear.

Select an expanded job, and click again on the group button B() to collapse the count back down.

Expected: The count should appear as a highlighted large button. eg. pending gray "+14"

Click in empty space to deselect the collapsed job.

Expected: The count "+14" should be deselected.

Click on the ( + ) global Expand/Collapse icon in the navbar to toggle all +n counts.

Expected: Counts should expand and collapse on all visible pushes.

Navigate via the n,p and left/right keys.

Expected: +n counts should be skipped during navigation.

expand all the groups, (the url querystring will reflect this) then reload the page

Expected: groups should still be expanded for all pushes

Optional: There are other variants that can be tested: classification of expanded job count members, Filters, and any other workflow integration testing.

12 KiB Исходник Ответственный История