Invoking cpu intensive python processes more than machine cores has some overhead on Mac(crbug.com/695864) and Win.
This change introduces pool mainly for python generator to restrict the number of running process when we specify large parallelism with goma.
I took 3 time build stats using target generate_bindings_modules_v8_interfaces and generate_bindings_core_v8_interfaces which have 1148 python tasks.
With this CL on z840 windows10
TotalSeconds : 18.2953436
TotalSeconds : 18.6283626
TotalSeconds : 19.2731436
Without this CL on Z840 windows10
TotalSeconds : 23.8277797
TotalSeconds : 23.6952018
TotalSeconds : 23.0853999
Linux looks to have good task scheduler.
With this CL on z840 linux
0m9.067s
0m8.771s
0m8.953s
Without this CL on Z840 linux
0m8.998s
0m9.022s
0m8.958s
Also this improves UI's responsiveness when we are building chrome on windows.
Stats of clean chrome build in each major OS is like below.
5 time clean build of chrome on Z840 windows 10 with -j1000 and warm goma backend cache is like below.
With this CL
333.3425057
317.4724857
305.0217898
317.8907203
305.1031952
Avg: 315.76613934
Without this CL
369.9731363
331.296758
329.0041556
329.1472297
333.3883952
Avg: 338.56193496
5 time clean build of chrome on Z840 linux with -j1000 and warm goma backend cache is like below.
With this CL
90.42
87.91
90.45
90.50
89.02
avg: 89.66
Without this CL
89.52
86.34
86.08
85.67
85.89
avg: 86.7
3 time clean build of chrome on 24 thread Mac Pro with -j500 and warm goma backend cache is like below.
With this CL
638.28
627.28
624.69
avg: 630.083
Without this CL
667.52
663.83
655.95
avg: 662.433
Bug: 695864
Change-Id: I6838c0f71b8d8030e6eab58b2990810aaa997dfa
Reviewed-on: https://chromium-review.googlesource.com/882581
Reviewed-by: Dirk Pranke <dpranke@chromium.org>
Commit-Queue: Takuto Ikuta <tikuta@chromium.org>
Cr-Original-Commit-Position: refs/heads/master@{#535589}
Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src
Cr-Mirrored-Commit: 2c098797567226115653c2567dbcfc72ee5af5ae
Define an explicit pool //build/toolchain:link_pool with a depth
of concurrent_links and convert all the toolchain link targets to
use it instead of setting concurrent_links property of the tool.
BUG=612786
Review-Url: https://codereview.chromium.org/2201363004
Cr-Original-Commit-Position: refs/heads/master@{#409846}
Cr-Mirrored-From: https://chromium.googlesource.com/chromium/src
Cr-Mirrored-Commit: 96e7407d139b8a8e43f6db7d4af3118078aba7de
Takuto Ikuta
sdefresne