Bug 1831680 - Document how to locally symbolicate TreeHerder stacks. r=gsvelto DONTBUILD

Also, change the refcount logging section to say that you probably
need to run fix_stacks.py on all platforms.

Differential Revision: https://phabricator.services.mozilla.com/D177331

docs/contributing/debugging/local_symbols.rst

@@ -0,0 +1,64 @@
+Symbolicating TreeHerder stacks locally
+=======================================
+
+When using tools like the :ref:`Dark Matter Detector (DMD)` or
+:ref:`refcount logging<Refcount Tracing and Balancing>` to
+investigate issues occurring on TreeHerder that you can't reproduce locally, you
+can often end up with unsymbolicated stacks. Fortunately, there is a way to
+symbolicate these stacks on your own machine.
+
+These instructions are for a Linux TreeHerder build for MacOS, so they might
+require some modifications for other combinations of platforms.
+
+Download ``target.tar.bz2`` and ``target.crashreporter-symbols.zip`` from the
+Build job. **Note that these files are very large so you'll want to delete
+them and the extracted files when you are done.**
+
+These files each contain a large number of files, so I'd recommend creating
+a directory for each of them. Call these ``<TARGET_DIR>`` and ``<SYMB_DIR>``,
+and move the prior two files into these two directories, respectively.
+
+Go to ``<TARGET_DIR>`` and run something like
+
+.. code-block:: shell
+
+    tar xf target.tar.bz2
+
+then go to ``<SYMB_DIR>`` and run something like
+
+.. code-block:: shell
+
+    unzip target.crashreporter-symbols.zip
+
+You should be able to delete the two original files now.
+
+Next we need to ensure that the locations of binaries are rewritten from
+where they are on TreeHerder to where we have them locally. We'll do this by
+editing ``fix_stacks.py``. This file is located in the ``tools/rb/`` directory of
+the Firefox source directory. You need to add these two lines to the function
+``fixSymbols``, after ``line_str`` is defined and before it is written to
+``fix_stacks.stdin``. I've done this right before the definition of
+``is_missing_newline``.
+
+.. code-block:: python
+
+    line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/firefox",
+                                "<TARGET_DIR>/firefox/firefox-bin")
+    line_str = line_str.replace("/builds/worker/workspace/build/application/firefox/libxul.so",
+                                "<TARGET_DIR>/firefox/libxul.so")
+
+The initial locations should appear verbatim in the stack you are trying to
+symbolicate, so double check that they match. Also, ``<TARGET_DIR>`` of course
+needs to be replaced with the actual local directories where those files are
+located. Note that the ``firefox`` executable is changed to ``firefox-bin``.
+I don't know why that is necessary, but only the latter existed for me.
+
+Finally, we need to make it so that the stack fixer can find the location of
+the breakpad symbols we downloaded. If you are running ``fix_stacks.py`` via
+``dmd.py`` or directly (in a recent version), you can do this by running with the
+environment variable ``BREAKPAD_SYMBOLS_PATH`` set to the ``<SYMB_DIR>`` from above.
+If that doesn't work, you'll have to edit ``initFixStacks`` in ``fix_stacks.py`` to
+set ``breakpadSymsDir`` to ``<SYMB_DIR>``.
+
+With all of that done, you should now be able to run ``dmd.py`` or ``fix_stacks.py``
+to fix the stacks. Note that the stack fixing process can take a minute or two.

docs/performance/memory/heap_scan_mode.md

@@ -206,6 +206,10 @@ be invoked like this:
 This can take a few minutes due to symbolification, but you only need to
 run it once on a log file.
 
+You can also locally symbolicate stacks from DMD logs generated on TreeHerder,
+but it will [take a few extra steps](/contributing/debugging/local_symbols.rst)
+that you need to do before running `dmd.py`.
+
 After that is done, we can finally find out which objects (possibly)
 point to other objects, using the block_analyzer script:
 

docs/performance/memory/refcount_tracing_and_balancing.md

@@ -121,13 +121,16 @@ and also improves the performance.
 
     perl -w tools/rb/filter-log.pl --object 0x00253ab0 < ./refcounts.log > my-leak.log
 
-### Linux Users
+### Symbolicating stacks
 
-The log file generated on Linux system often lack function names, file
+The log files often lack function names, file
-names and line numbers. Linux users need to run a script to fix the call
+names and line numbers. You'll need to run a script to fix the call
 stack.
 
-    python tools/rb/fix_stacks.py < ./refcounts.log > fixed_stack.log
+    python3 tools/rb/fix_stacks.py < ./refcounts.log > fixed_stack.log
+
+Also, it is possible to [locally symbolicate](/contributing/debugging/local_symbols.rst)
+logs generated on TreeHerder.
 
 ## **Post-processing step 3: building the balance tree**