Bug 1820699: Make nsCocoaWindow choose a fallback normal size if it can't unmaximize when requested. r=mstange

Without this change, it is possible for a macOS window in maximized state
to be unable to return to normal state via a SetSizeMode call. This means
that a JS call to "window.restore()" has no effect. This patch fixes that
case by detecting the failure and resizing to  a slightly smaller size
that qualifies for the normal state.

This is very difficult to test intentionally with a window that is
maximized and has forgotten its Cocoa-tracked "user state". But that state
is reached in our test harness and this will get us out of it. It has the
desirable(?) side effect of making the macOS menu bar action Window->Zoom
do something with a window that is already maximized to the screen.

Differential Revision: https://phabricator.services.mozilla.com/D171831

widget/cocoa/nsCocoaWindow.mm

@@ -1809,6 +1809,19 @@ void nsCocoaWindow::ProcessTransitions() {
           }
         } else if (mWindow.zoomed) {
           [mWindow zoom:nil];
+
+          // Check if we're still zoomed. If we are, we need to do *something* to make the
+          // window smaller than the zoom size so Cocoa will treat us as being out of the
+          // zoomed state. Otherwise, we could stay zoomed and never be able to be "normal"
+          // from calls to SetSizeMode.
+          if (mWindow.zoomed) {
+            NSRect maximumFrame = mWindow.frame;
+            const CGFloat INSET_OUT_OF_ZOOM = 20.0f;
+            [mWindow setFrame:NSInsetRect(maximumFrame, INSET_OUT_OF_ZOOM, INSET_OUT_OF_ZOOM)
+                      display:YES];
+            MOZ_ASSERT(!mWindow.zoomed,
+                       "We should be able to unzoom by shrinking the frame a bit.");
+          }
         }
         break;
       }