Bug 1388935 - Avoid overflowing the chunk recycling limit with very large chunks. r=njn

When recycling chunks, mozjemalloc tries to avoid keeping around more
than 128MB worth of chunks around, but it doesn't actually look at the
size of the chunks that are recycled, so if chunk larger than 128MB is
recycled, it is kept as a whole, going well over the limit.

The chunks are still properly reused, and further recycling doesn't
occur, but that can limit other mmap users from getting enough address
space.

With this change, mozjemalloc now doesn't keep more than 128MB, by
splitting the chunks it recycles if they are too large.

Note this was not a problem on Windows, where chunks larger than 1MB are
never recycled (per CAN_RECYCLE).

--HG--
extra : rebase_source : 6765fd30b78ca5ddc7d55aac861355d960e47828

memory/mozjemalloc/mozjemalloc.cpp

@@ -140,6 +140,7 @@
 #include <stdarg.h>
 #include <stdio.h>
 #include <string.h>
+#include <algorithm>
 
 #ifdef MOZ_MEMORY_WINDOWS
 
@@ -148,7 +149,6 @@
 #include <io.h>
 #include <windows.h>
 #include <intrin.h>
-#include <algorithm>
 
 #define	SIZE_T_MAX SIZE_MAX
 #define	STDERR_FILENO 2
@@ -2181,9 +2181,22 @@ chunk_dealloc(void *chunk, size_t size, ChunkType type)
 
 	malloc_rtree_set(chunk_rtree, (uintptr_t)chunk, nullptr);
 
-	if (CAN_RECYCLE(size) && load_acquire_z(&recycled_size) < recycle_limit) {
-		chunk_record(&chunks_szad_mmap, &chunks_ad_mmap, chunk, size, type);
-		return;
+	if (CAN_RECYCLE(size)) {
+		size_t recycled_so_far = load_acquire_z(&recycled_size);
+		// In case some race condition put us above the limit.
+		if (recycled_so_far < recycle_limit) {
+			size_t recycle_remaining = recycle_limit - recycled_so_far;
+			size_t to_recycle;
+			if (size > recycle_remaining) {
+				to_recycle = recycle_remaining;
+				// Drop pages that would overflow the recycle limit
+				pages_trim(chunk, size, 0, to_recycle);
+			} else {
+				to_recycle = size;
+			}
+			chunk_record(&chunks_szad_mmap, &chunks_ad_mmap, chunk, to_recycle, type);
+			return;
+		}
 	}
 
 	pages_unmap(chunk, size);