We have data showing that the Places SQLite database can consume
gigabytes of I/O during Firefox test automation jobs. This is
because a number of tests load pages as rapdily as possible,
effectively stress testing Places and SQLite. As the SQLite
database is committed to, we incur I/O for the WAL journal
and when flushing/synchronizing commits. This can add up to a
lot of overhead, especially on spinning disks.
It is important for Places to run during many tests. But it
isn't necessarily important to run with robust I/O guarantees:
SQLite itself has tests that ensure different journal and
synchronizing modes work as advertised.
This commit introduces a preference that changes the SQLite
journal and synchronization modes to be less robust. We use
an in-memory journal so no I/O is incurred for journal writing.
We disable synchronization during commit so no expensive
file(system) flushing is performed. Because setting this
preference would be dangerous for end users, we only honor the
pref if a scary sounding environment variable is set. Hopefully
that's enough of an obstacle to prevent people from footgunning
themselves.
A preliminary Try run reveals this has the potential to shave
hundreds of megabytes of I/O from various jobs. Although this
commit stops short of changing the configuration in automation
to use the new volatile storage preference.
MozReview-Commit-ID: KCoDVzwkSbg
--HG--
extra : rebase_source : 4e630f4341fc8c07e16383480356bd1bff4d4ba2
This aims at speeding up DELETE FROM moz_places like queries.
The primary reason of slowness is the FOR EACH ROW trigger that takes care of updating the moz_hosts table when places are removed.
Unfortunately Sqlite doesn't support FOR EACH STATEMENT triggers, that means the trigger will hit multiple times for pages in the same host.
The patch introduces an additional temp table to accumulate hosts during a delete, then a trigger takes care of updating moz_hosts only once per touched host, rather than once per removed place.
MozReview-Commit-ID: BlJRLQZoC07
--HG--
extra : rebase_source : 5715efe580348b3810000d67a0a692dece36d306
The problem is due to sanitization happening too late in the shutdown cycle.
The Sanitizer depends on Places shutdown, that recently moved to async shutdown.
That change caused shutdown to happen completely at profile-before-change, unfortunately
during that phase it's impossible to predict which services are already shutdown.
The patch restores the previous Places shutdown procedure, thus clients are notified
earlier, during profile-change-teardown.
Additional meaningful changes:
* Fixes FX_SANITIZE_TOTAL telemetry to properly count total time taken by sanitize.
* Makes each cleanup operation isolated from other errors to try cleaning up as most as possible.
* In case of multiple sanitization sub steps, each step is isolated by a try/catch, the last seen exception is reported upstream.
* Makes FX_SANITIZE_HISTORY actually measure history, not other random stuff.
* Removes TOPIC_SIMULATE_PLACES_MUST_CLOSE_1 since we can now just use profile-change-teardown for shutdown phase 1.
MozReview-Commit-ID: HroLvbi25IC
--HG--
extra : rebase_source : 5cd01a803fbc1f72dc5174dd0c5a5b5aea473fc2
The bulk of this commit was generated with a script, executed at the top
level of a typical source code checkout. The only non-machine-generated
part was modifying MFBT's moz.build to reflect the new naming.
CLOSED TREE makes big refactorings like this a piece of cake.
# The main substitution.
find . -name '*.cpp' -o -name '*.cc' -o -name '*.h' -o -name '*.mm' -o -name '*.idl'| \
xargs perl -p -i -e '
s/nsRefPtr\.h/RefPtr\.h/g; # handle includes
s/nsRefPtr ?</RefPtr</g; # handle declarations and variables
'
# Handle a special friend declaration in gfx/layers/AtomicRefCountedWithFinalize.h.
perl -p -i -e 's/::nsRefPtr;/::RefPtr;/' gfx/layers/AtomicRefCountedWithFinalize.h
# Handle nsRefPtr.h itself, a couple places that define constructors
# from nsRefPtr, and code generators specially. We do this here, rather
# than indiscriminantly s/nsRefPtr/RefPtr/, because that would rename
# things like nsRefPtrHashtable.
perl -p -i -e 's/nsRefPtr/RefPtr/g' \
mfbt/nsRefPtr.h \
xpcom/glue/nsCOMPtr.h \
xpcom/base/OwningNonNull.h \
ipc/ipdl/ipdl/lower.py \
ipc/ipdl/ipdl/builtin.py \
dom/bindings/Codegen.py \
python/lldbutils/lldbutils/utils.py
# In our indiscriminate substitution above, we renamed
# nsRefPtrGetterAddRefs, the class behind getter_AddRefs. Fix that up.
find . -name '*.cpp' -o -name '*.h' -o -name '*.idl' | \
xargs perl -p -i -e 's/nsRefPtrGetterAddRefs/RefPtrGetterAddRefs/g'
if [ -d .git ]; then
git mv mfbt/nsRefPtr.h mfbt/RefPtr.h
else
hg mv mfbt/nsRefPtr.h mfbt/RefPtr.h
fi
--HG--
rename : mfbt/nsRefPtr.h => mfbt/RefPtr.h
The goal here is to leave creation stuff mostly for JS, so we can
convert it entirely over to a non-threadsafe cycle-collected version
without breaking any existing C++ users.
I didn't do this for a remaining use in nsGlobalWindow.h to avoid
including nsVariant.h all over the place.
The bulk of this commit was generated by running:
run-clang-tidy.py \
-checks='-*,llvm-namespace-comment' \
-header-filter=^/.../mozilla-central/.* \
-fix
Marco Bonardo
Gregory Szorc
Katie Broida
Cedric Desgranges
Aidin Gharibnavaz
Carsten "Tomcat" Book
Chris Peterson
Nathan Froyd
Andrew McCreight
Birunthan Mohanathas
Dragana Damjanovic
Phil Ringnalda
Juan Gomez
Timur Valeev
Ryan VanderMeulen
David Rajchenbach-Teller
Ehsan Akhgari
Asaf Romano
Althaf Hameez
Benoit Jacob
Drew Willcoxon
Kyle Huey
Marco Castelluccio
Trevor Saunders