Bug 1417310 - Don't attempt to create buffers of unreasonable sizes. r=bas

Currently ContentClient will attempt to create a buffer of any size
that Layers requires it to. If that fails it will then warn if the
requested buffer was an unreasonable size. This change makes it so
that we check whether the size is reasonable first, and do not attempt
to create the buffer if it is unreasonable.

Previously the operating system would have managed to allocate some
buffers that we deem unreasonable, but at the cost of the system
becoming unresponsive. Now, some large buffers which would previously
have been created won't be, but the chances of the system becoming
unresponsive should decrease.

Ideally Layers wouldn't request buffers of these sizes in the first
place - there are measures in place to try to prevent it, and more work
is certainly required. However, this will act as a last ditch defence.

MozReview-Commit-ID: 7WCqEwkmViy

gfx/layers/client/ContentClient.cpp

@@ -243,18 +243,20 @@ ContentClient::BeginPaint(PaintedLayer* aLayer,
       bufferFlags |= BUFFER_COMPONENT_ALPHA;
     }
 
-    RefPtr<RotatedBuffer> newBuffer = CreateBuffer(result.mContentType,
-                                                   dest.mBufferRect,
-                                                   bufferFlags);
+    RefPtr<RotatedBuffer> newBuffer;
+    if (Factory::ReasonableSurfaceSize(IntSize(dest.mBufferRect.Width(), dest.mBufferRect.Height()))) {
+      newBuffer = CreateBuffer(result.mContentType, dest.mBufferRect, bufferFlags);
 
-    if (!newBuffer) {
-      if (Factory::ReasonableSurfaceSize(IntSize(dest.mBufferRect.Width(), dest.mBufferRect.Height()))) {
+      if (!newBuffer) {
         gfxCriticalNote << "Failed buffer for "
                         << dest.mBufferRect.x << ", "
                         << dest.mBufferRect.y << ", "
                         << dest.mBufferRect.Width() << ", "
                         << dest.mBufferRect.Height();
       }
+    }
+
+    if (!newBuffer) {
       Clear();
       return result;
     }