Bug 964646 part 10 - Fix floating point precision issues when comparing matrices; r=dbaron

This patch addresses and issue where the OMTA style and computed style were not
comparing equal in one particular case.

In this case AddTransformTranslate in nsStyleAnimation would give us
a translate-y value of 94.331673 in both cases (i.e. when calculating the
animated value on the compositor thread or when fetching computed style).

For the OMTA case, however, after we apply additional layer transformations and
then reverse them (so we can query the CSS value) we'd end up with 94.331642,
a difference of 0.000031. The reversing procedure is only used for testing so
the actual error introduced here by the additional layer transformations is
probably less.

Unfortunately, when we pass 94.331642 this along to MatrixToCSSValue we get back
matrix(1, 0, 0, 1, 94.3316) since it only outputs 6 digits of precision.

On the other hand, on the computed style end we'd pass 94.331673 to
MatrixToCSSValue which gives us matrix(1, 0, 0, 1, 94.3317), so the error swells
from 0.000031 to 0.0001.

Then when we subtract 94.3316 from 94.3317 in Javascript we get
0.00010000000000331966 due to floating-point precision issues which compares
greater than the default tolerance of 0.0001.

This patch simply adjusts the default tolerance to 0.00011 to accommodate
these floating-point differences.

layout/style/test/test_animations_omta.html

@@ -914,12 +914,9 @@ addAsyncTest(function *() {
                  RunningOn.Compositor, 0.01,
                  "animation-iteration-count test 3 at 20.2s");
   advance_clock(3600);
-  // XXX OMTA style and computed style fail here -- fixed in follow-up patch
-  /*
   omta_is_approx("transform", { ty: 100 * gTF.ease_out(0.81) },
                  RunningOn.Compositor, 0.01,
                  "animation-iteration-count test 3 at 23.8s");
-                 */
   advance_clock(200);
   omta_is_approx("transform", { ty: 100 * gTF.ease_out(0.8) },
                  RunningOn.Compositor, 0.01,
@@ -1155,7 +1152,7 @@ function omta_is_approx(property, expected, runningOn, tolerance, desc) {
 }
 
 function matricesRoughlyEqual(a, b, tolerance) {
-  tolerance = tolerance || 0.0001;
+  tolerance = tolerance || 0.00011;
   for (var i = 0; i < 4; i++) {
     for (var j = 0; j < 4; j++) {
       var diff = Math.abs(a[i][j] - b[i][j]);