Bug 1568119 - Setup OOM simulation testing for MarkPagesInUseHard r=jonco

There are three cases where this function could OOM:

 * Zeal mode - we want to crash if this OOMs
 * Growing the Nursery - We've added a new test to exercise this code path
 * Disabling the Nursery - Plenty of existing tests seem to cover this
   with OOM testing.

Differential Revision: https://phabricator.services.mozilla.com/D39457

--HG--
extra : moz-landing-system : lando

js/src/gc/Memory.cpp

@@ -801,6 +801,10 @@ void MarkPagesInUseSoft(void* region, size_t length) {
 }
 
 bool MarkPagesInUseHard(void* region, size_t length) {
+  if (js::oom::ShouldFailWithOOM()) {
+    return false;
+  }
+
   CheckDecommit(region, length);
 
   MOZ_MAKE_MEM_UNDEFINED(region, length);

js/src/gc/Nursery.cpp

@@ -379,8 +379,11 @@ void js::Nursery::enterZealMode() {
       // which could be attempting to decommit the currently-unused part of this
       // chunk.
       decommitTask.join();
-      if (!chunk(0).markPagesInUseHard(ChunkSize - ArenaSize)) {
-        MOZ_CRASH("Out of memory trying to extend chunk for zeal mode");
+      {
+        AutoEnterOOMUnsafeRegion oomUnsafe;
+        if (!chunk(0).markPagesInUseHard(ChunkSize - ArenaSize)) {
+          oomUnsafe.crash("Out of memory trying to extend chunk for zeal mode");
+        }
       }
 
       // It'd be simpler to poison the whole chunk, but we can't do that

js/src/jit-test/tests/gc/bug-1568119.js

@@ -0,0 +1,21 @@
+// |jit-test| skip-if: !('oomTest' in this)
+
+function allocateSomeStuff() {
+  return {a: "a fish", b: [1, 2, 3]};
+}
+
+oomTest(() => {
+  // Run a minor GC with a small nursery.
+  gcparam('minNurseryBytes', 256 * 1024);
+  gcparam('maxNurseryBytes', 256 * 1024);
+  allocateSomeStuff();
+  gc();
+
+  // Run a minor GC with a larger nursery to get it to attempt to grow and
+  // fail the allocation there.
+  gcparam('maxNurseryBytes', 1024 * 1024);
+  gcparam('minNurseryBytes', 1024 * 1024);
+  allocateSomeStuff();
+  gc();
+});
+