mozilla
/
gecko-dev
зеркало из https://github.com/mozilla/gecko-dev.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Bug 1676771 - Add a TouchResampler class. r=kats 

This is similar to GeckoTouchDispatcher from the B2G days:
https://hg.mozilla.org/mozilla-central/file/49bbfe7887d5739df62d6b8d05bc41cfe3161f08/widget/gonk/GeckoTouchDispatcher.cpp

The values for the various kTouchResample* constants were taken from the
original pref values:
https://hg.mozilla.org/mozilla-central/file/49bbfe7887d5739df62d6b8d05bc41cfe3161f08/gfx/thebes/gfxPrefs.h#l225

There are some extra sources of complexity:
 - TouchResampler tries hard to generate one outgoing event per incoming event,
   so that the result code tracking to the Java front-end code works properly.
 - TouchResampler tries hard to never lose any historicalData information, so
   that the velocity tracker has a maximum amount of information to work with.
 - TouchResampler has a "reset to non-resampled state" functionality so that
   overpredictions are corrected when the finger pauses or when a touch non-move
   event fires.

Differential Revision: https://phabricator.services.mozilla.com/D96795
This commit is contained in:
Markus Stange 2020-11-17 20:20:32 +00:00
Родитель 1e8bac4c18
Коммит 231f0ad6be
5 изменённых файлов: 1514 добавлений и 0 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

380
widget/TouchResampler.cpp Normal file
Просмотреть файл

@ -0,0 +1,380 @@
/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim: set ts=8 sts=2 et sw=2 tw=80: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#include "TouchResampler.h"
#include "nsAlgorithm.h"
/**
* TouchResampler implementation
*/
namespace mozilla {
namespace widget {
// The values below have been tested and found to be acceptable on a device
// with a display refresh rate of 60Hz and touch sampling rate of 100Hz.
// While their "ideal" values are dependent on the exact rates of each device,
// the values we've picked below should be somewhat robust across a variation of
// different rates. They mostly aim to avoid making predictions that are too far
// away (in terms of distance) from the finger, and to detect pauses in the
// finger motion without too much delay.
// Maximum time between two consecutive data points to consider resampling
// between them.
// Values between 1x and 5x of the touch sampling interval are reasonable.
static const double kTouchResampleWindowSize = 40.0;
// These next two values constrain the sampling timestamp.
// Our caller will usually adjust frame timestamps to be slightly in the past,
// for example by 5ms. This means that, during normal operation, we will
// maximally need to predict by [touch sampling rate] minus 5ms.
// So we would like kTouchResampleMaxPredictMs to satisfy the following:
// kTouchResampleMaxPredictMs + [frame time adjust] > [touch sampling rate]
static const double kTouchResampleMaxPredictMs = 8.0;
// This one is a protection against very outdated frame timestamps.
// Values larger than the touch sampling interval and less than 3x of the vsync
// interval are reasonable.
static const double kTouchResampleMaxBacksampleMs = 20.0;
// The maximum age of the most recent data point to consider resampling.
// Should be between 1x and 3x of the touch sampling interval.
static const double kTouchResampleOldTouchThresholdMs = 17.0;
uint64_t TouchResampler::ProcessEvent(MultiTouchInput&& aInput) {
mCurrentTouches.UpdateFromEvent(aInput);
uint64_t eventId = mNextEventId;
mNextEventId++;
if (aInput.mType == MultiTouchInput::MULTITOUCH_MOVE) {
// Touch move events are deferred until NotifyFrame.
mDeferredTouchMoveEvents.push({std::move(aInput), eventId});
} else {
// Non-move events are transferred to the outgoing queue unmodified.
// If there are pending touch move events, flush those out first, so that
// events are emitted in the right order.
FlushDeferredTouchMoveEventsUnresampled();
if (mInResampledState) {
// Return to a non-resampled state before emitting a non-move event.
ReturnToNonResampledState();
}
EmitEvent(std::move(aInput), eventId);
}
return eventId;
}
void TouchResampler::NotifyFrame(const TimeStamp& aTimeStamp) {
TimeStamp lastTouchTime = mCurrentTouches.LatestDataPointTime();
if (mDeferredTouchMoveEvents.empty() ||
(lastTouchTime &&
lastTouchTime < aTimeStamp - TimeDuration::FromMilliseconds(
kTouchResampleOldTouchThresholdMs))) {
// We haven't received a touch move event in a while, so the fingers must
// have stopped moving. Flush any old touch move events.
FlushDeferredTouchMoveEventsUnresampled();
if (mInResampledState) {
// Make sure we pause at the resting position that we actually observed,
// and not at a resampled position.
ReturnToNonResampledState();
}
// Clear touch location history so that we don't resample across a pause.
mCurrentTouches.ClearDataPoints();
return;
}
MOZ_RELEASE_ASSERT(lastTouchTime);
TimeStamp lowerBound = lastTouchTime - TimeDuration::FromMilliseconds(
kTouchResampleMaxBacksampleMs);
TimeStamp upperBound = lastTouchTime + TimeDuration::FromMilliseconds(
kTouchResampleMaxPredictMs);
TimeStamp sampleTime = clamped(aTimeStamp, lowerBound, upperBound);
if (mLastEmittedEventTime && sampleTime < mLastEmittedEventTime) {
// Keep emitted timestamps in order.
sampleTime = mLastEmittedEventTime;
}
// We have at least one pending touch move event. Pick one of the events from
// mDeferredTouchMoveEvents as the base event for the resampling adjustment.
// We want to produce an event stream whose timestamps are in the right order.
// As the base event, use the first event that's at or after sampleTime,
// unless there is no such event, in that case use the last one we have. We
// will set the timestamp on the resampled event to sampleTime later.
// Flush out any older events so that everything remains in the right order.
MultiTouchInput input;
uint64_t eventId;
while (true) {
MOZ_RELEASE_ASSERT(!mDeferredTouchMoveEvents.empty());
std::tie(input, eventId) = std::move(mDeferredTouchMoveEvents.front());
mDeferredTouchMoveEvents.pop();
if (mDeferredTouchMoveEvents.empty() || input.mTimeStamp >= sampleTime) {
break;
}
// Flush this event to the outgoing queue without resampling. What ends up
// on the screen will still be smooth because we will proceed to emit a
// resampled event before the paint for this frame starts.
PrependLeftoverHistoricalData(&input);
MOZ_RELEASE_ASSERT(input.mTimeStamp < sampleTime);
EmitEvent(std::move(input), eventId);
}
mOriginalOfResampledTouchMove = Nothing();
// Compute the resampled touch positions.
nsTArray<ScreenIntPoint> resampledPositions;
bool anyPositionDifferentFromOriginal = false;
for (const auto& touch : input.mTouches) {
ScreenIntPoint resampledPosition =
mCurrentTouches.ResampleTouchPositionAtTime(
touch.mIdentifier, touch.mScreenPoint, sampleTime);
if (resampledPosition != touch.mScreenPoint) {
anyPositionDifferentFromOriginal = true;
}
resampledPositions.AppendElement(resampledPosition);
}
if (anyPositionDifferentFromOriginal) {
// Store a copy of the original event, so that we can return to an
// non-resampled position later, if necessary.
mOriginalOfResampledTouchMove = Some(input);
// Add the original observed position to the historical data, as well as any
// leftover historical positions from the previous touch move event, and
// store the resampled values in the "final" position of the event.
PrependLeftoverHistoricalData(&input);
for (size_t i = 0; i < input.mTouches.Length(); i++) {
auto& touch = input.mTouches[i];
touch.mHistoricalData.AppendElement(SingleTouchData::HistoricalTouchData{
input.mTimeStamp,
touch.mScreenPoint,
touch.mLocalScreenPoint,
touch.mRadius,
touch.mRotationAngle,
touch.mForce,
});
// Remove any historical touch data that's in the future, compared to
// sampleTime. This data will be included by upcoming touch move
// events. This only happens if the frame timestamp can be older than the
// event timestamp, i.e. if interpolation occurs (rather than
// extrapolation).
auto futureDataStart = std::find_if(
touch.mHistoricalData.begin(), touch.mHistoricalData.end(),
[sampleTime](
const SingleTouchData::HistoricalTouchData& aHistoricalData) {
return aHistoricalData.mTimeStamp > sampleTime;
});
if (futureDataStart != touch.mHistoricalData.end()) {
nsTArray<SingleTouchData::HistoricalTouchData> futureData(
Span(touch.mHistoricalData).From(futureDataStart.GetIndex()));
touch.mHistoricalData.TruncateLength(futureDataStart.GetIndex());
mRemainingTouchData.insert({touch.mIdentifier, std::move(futureData)});
}
touch.mScreenPoint = resampledPositions[i];
}
input.mTimeStamp = sampleTime;
}
EmitEvent(std::move(input), eventId);
mInResampledState = anyPositionDifferentFromOriginal;
}
void TouchResampler::PrependLeftoverHistoricalData(MultiTouchInput* aInput) {
for (auto& touch : aInput->mTouches) {
auto leftoverData = mRemainingTouchData.find(touch.mIdentifier);
if (leftoverData != mRemainingTouchData.end()) {
nsTArray<SingleTouchData::HistoricalTouchData> data =
std::move(leftoverData->second);
mRemainingTouchData.erase(leftoverData);
touch.mHistoricalData.InsertElementsAt(0, data);
}
if (TimeStamp cutoffTime = mLastEmittedEventTime) {
// If we received historical touch data that was further in the past than
// the last resampled event, discard that data so that the touch data
// points are emitted in order.
touch.mHistoricalData.RemoveElementsBy(
[cutoffTime](const SingleTouchData::HistoricalTouchData& aTouchData) {
return aTouchData.mTimeStamp < cutoffTime;
});
}
}
mRemainingTouchData.clear();
}
void TouchResampler::FlushDeferredTouchMoveEventsUnresampled() {
while (!mDeferredTouchMoveEvents.empty()) {
MultiTouchInput input;
uint64_t eventId;
std::tie(input, eventId) = std::move(mDeferredTouchMoveEvents.front());
mDeferredTouchMoveEvents.pop();
PrependLeftoverHistoricalData(&input);
EmitEvent(std::move(input), eventId);
mInResampledState = false;
mOriginalOfResampledTouchMove = Nothing();
}
}
void TouchResampler::ReturnToNonResampledState() {
MOZ_RELEASE_ASSERT(mInResampledState);
MOZ_RELEASE_ASSERT(mDeferredTouchMoveEvents.empty(),
"Don't call this if there is a deferred touch move event. "
"We can return to the non-resampled state by sending that "
"event, rather than a copy of a previous event.");
// The last outgoing event was a resampled touch move event.
// Return to the non-resampled state, by sending a touch move event to
// "overwrite" any resampled positions with the original observed positions.
MultiTouchInput input = std::move(*mOriginalOfResampledTouchMove);
mOriginalOfResampledTouchMove = Nothing();
// For the event's timestamp, we want to backdate the correction as far as we
// can, while still preserving timestamp ordering. But we also don't want to
// backdate it to be older than it was originally.
if (mLastEmittedEventTime > input.mTimeStamp) {
input.mTimeStamp = mLastEmittedEventTime;
}
// Assemble the correct historical touch data for this event.
// We don't want to include data points that we've already sent out with the
// resampled event. And from the leftover data points, we only want those that
// don't duplicate the final time + position of this event.
for (auto& touch : input.mTouches) {
touch.mHistoricalData.Clear();
}
PrependLeftoverHistoricalData(&input);
for (auto& touch : input.mTouches) {
touch.mHistoricalData.RemoveElementsBy([&](const auto& histData) {
return histData.mTimeStamp >= input.mTimeStamp;
});
}
EmitExtraEvent(std::move(input));
mInResampledState = false;
}
void TouchResampler::TouchInfo::Update(const SingleTouchData& aTouch,
const TimeStamp& aEventTime) {
for (const auto& historicalData : aTouch.mHistoricalData) {
mBaseDataPoint = mLatestDataPoint;
mLatestDataPoint =
Some(DataPoint{historicalData.mTimeStamp, historicalData.mScreenPoint});
}
mBaseDataPoint = mLatestDataPoint;
mLatestDataPoint = Some(DataPoint{aEventTime, aTouch.mScreenPoint});
}
ScreenIntPoint TouchResampler::TouchInfo::ResampleAtTime(
const ScreenIntPoint& aLastObservedPosition, const TimeStamp& aTimeStamp) {
TimeStamp cutoff =
aTimeStamp - TimeDuration::FromMilliseconds(kTouchResampleWindowSize);
if (!mBaseDataPoint || !mLatestDataPoint ||
!(mBaseDataPoint->mTimeStamp < mLatestDataPoint->mTimeStamp) ||
mBaseDataPoint->mTimeStamp < cutoff) {
return aLastObservedPosition;
}
// For the actual resampling, connect the last two data points with a line and
// sample along that line.
TimeStamp t1 = mBaseDataPoint->mTimeStamp;
TimeStamp t2 = mLatestDataPoint->mTimeStamp;
double t = (aTimeStamp - t1) / (t2 - t1);
double x1 = mBaseDataPoint->mPosition.x;
double x2 = mLatestDataPoint->mPosition.x;
double y1 = mBaseDataPoint->mPosition.y;
double y2 = mLatestDataPoint->mPosition.y;
int32_t resampledX = round(x1 + t * (x2 - x1));
int32_t resampledY = round(y1 + t * (y2 - y1));
return ScreenIntPoint(resampledX, resampledY);
}
void TouchResampler::CurrentTouches::UpdateFromEvent(
const MultiTouchInput& aInput) {
switch (aInput.mType) {
case MultiTouchInput::MULTITOUCH_START: {
// A new touch has been added; make sure mTouches reflects the current
// touches in the event.
nsTArray<TouchInfo> newTouches;
for (const auto& touch : aInput.mTouches) {
const auto touchInfo = TouchByIdentifier(touch.mIdentifier);
if (touchInfo != mTouches.end()) {
// This is one of the existing touches.
newTouches.AppendElement(std::move(*touchInfo));
mTouches.RemoveElementAt(touchInfo);
} else {
// This is the new touch.
newTouches.AppendElement(TouchInfo{
touch.mIdentifier, Nothing(),
Some(DataPoint{aInput.mTimeStamp, touch.mScreenPoint})});
}
}
MOZ_ASSERT(mTouches.IsEmpty(), "Missing touch end before touch start?");
mTouches = std::move(newTouches);
break;
}
case MultiTouchInput::MULTITOUCH_MOVE: {
// The touches have moved.
// Add position information to the history data points.
for (const auto& touch : aInput.mTouches) {
const auto touchInfo = TouchByIdentifier(touch.mIdentifier);
MOZ_ASSERT(touchInfo != mTouches.end());
if (touchInfo != mTouches.end()) {
touchInfo->Update(touch, aInput.mTimeStamp);
}
}
mLatestDataPointTime = aInput.mTimeStamp;
break;
}
case MultiTouchInput::MULTITOUCH_END: {
// A touch has been removed.
MOZ_RELEASE_ASSERT(aInput.mTouches.Length() == 1);
const auto touchInfo = TouchByIdentifier(aInput.mTouches[0].mIdentifier);
MOZ_ASSERT(touchInfo != mTouches.end());
if (touchInfo != mTouches.end()) {
mTouches.RemoveElementAt(touchInfo);
}
break;
}
case MultiTouchInput::MULTITOUCH_CANCEL:
// All touches are canceled.
mTouches.Clear();
break;
}
}
nsTArray<TouchResampler::TouchInfo>::iterator
TouchResampler::CurrentTouches::TouchByIdentifier(int32_t aIdentifier) {
return std::find_if(mTouches.begin(), mTouches.end(),
[aIdentifier](const TouchInfo& info) {
return info.mIdentifier == aIdentifier;
});
}
ScreenIntPoint TouchResampler::CurrentTouches::ResampleTouchPositionAtTime(
int32_t aIdentifier, const ScreenIntPoint& aLastObservedPosition,
const TimeStamp& aTimeStamp) {
const auto touchInfo = TouchByIdentifier(aIdentifier);
MOZ_ASSERT(touchInfo != mTouches.end());
if (touchInfo != mTouches.end()) {
return touchInfo->ResampleAtTime(aLastObservedPosition, aTimeStamp);
}
return aLastObservedPosition;
}
} // namespace widget
} // namespace mozilla

191
widget/TouchResampler.h Normal file
Просмотреть файл

@ -0,0 +1,191 @@
/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim: set ts=8 sts=2 et sw=2 tw=80: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#ifndef mozilla_widget_TouchResampler_h
#define mozilla_widget_TouchResampler_h
#include <queue>
#include <unordered_map>
#include "mozilla/Maybe.h"
#include "mozilla/TimeStamp.h"
#include "InputData.h"
namespace mozilla {
namespace widget {
/**
* De-jitters touch motions by resampling (interpolating or extrapolating) touch
* positions for the vsync timestamp.
*
* Touch resampling improves the touch panning experience on devices where touch
* positions are sampled at a rate that's not an integer multiple of the display
* refresh rate, for example 100Hz touch sampling on a 60Hz display: Without
* resampling, we would alternate between taking one touch sample or two touch
* samples into account each frame, creating a jittery motion ("small step, big
* step, small step, big step").
* Intended for use on Android, where both touch events and vsync notifications
* arrive on the same thread, the Java UI thread.
* This class is not thread safe.
*
* TouchResampler operates in the following way:
*
* Original events are fed into ProcessEvent().
* Outgoing events (potentially resampled for resampling) are added to a queue
* and can be consumed by calling ConsumeOutgoingEvents(). Touch events which
* are not touch move events are forwarded instantly and not resampled. Only
* touch move events are resampled. Whenever a touch move event is received, it
* gets delayed until NotifyFrame() is called, at which point it is resampled
* into a resampled version for the given frame timestamp, and added to the
* outgoing queue. If no touch move event is received between two consecutive
* frames, this is treated as a stop in the touch motion. If the last outgoing
* event was an resampled touch move event, we return back to the non-resampled
* state by emitting a copy of the last original touch move event, which has
* unmodified position data. Touch events which are not touch move events also
* force a return to the non-resampled state before they are moved to the
* outgoing queue.
*/
class TouchResampler final {
public:
// Feed a touch event into the interpolater. Returns an ID that can be used to
// match outgoing events to this incoming event, to track data associated with
// this event.
uint64_t ProcessEvent(MultiTouchInput&& aInput);
// Emit events, potentially resampled, for this timestamp. The events are put
// into the outgoing queue. May not emit any events if there's no update.
void NotifyFrame(const TimeStamp& aTimeStamp);
// Returns true between the start and the end of a touch gesture. During this
// time, the caller should keep itself registered with the system frame
// callback mechanism, so that NotifyFrame() can be called on every frame.
// (Otherwise, if we only registered the callback after receiving a touch move
// event, the frame callback might be delayed by a full frame.)
bool InTouchingState() const { return mCurrentTouches.HasTouch(); }
struct OutgoingEvent {
// The event, potentially modified from the original for resampling.
MultiTouchInput mEvent;
// Some(eventId) if this event is a modified version of an original event,
// Nothing() if this is an extra event.
Maybe<uint64_t> mEventId;
};
// Returns the outgoing events that were produced since the last call.
// No event IDs will be skipped. Returns at least one outgoing event for each
// incoming event (possibly after a delay), and potential extra events with
// no originating event ID.
// Outgoing events should be consumed after every call to ProcessEvent() and
// after every call to NotifyFrame().
std::queue<OutgoingEvent> ConsumeOutgoingEvents() {
return std::move(mOutgoingEvents);
}
private:
// Add the event to the outgoing queue.
void EmitEvent(MultiTouchInput&& aInput, uint64_t aEventId) {
mLastEmittedEventTime = aInput.mTimeStamp;
mOutgoingEvents.push(OutgoingEvent{std::move(aInput), Some(aEventId)});
}
// Emit an event that does not correspond to an incoming event.
void EmitExtraEvent(MultiTouchInput&& aInput) {
mLastEmittedEventTime = aInput.mTimeStamp;
mOutgoingEvents.push(OutgoingEvent{std::move(aInput), Nothing()});
}
// Move any touch move events that we deferred for resampling to the outgoing
// queue unmodified, leaving mDeferredTouchMoveEvents empty.
void FlushDeferredTouchMoveEventsUnresampled();
// Must only be called if mInResampledState is true and
// mDeferredTouchMoveEvents is empty. Emits mOriginalOfResampledTouchMove,
// with a potentially adjusted timestamp for correct ordering.
void ReturnToNonResampledState();
// Takes historical touch data from mRemainingTouchData and prepends it to the
// data in aInput.
void PrependLeftoverHistoricalData(MultiTouchInput* aInput);
struct DataPoint {
TimeStamp mTimeStamp;
ScreenIntPoint mPosition;
};
struct TouchInfo {
void Update(const SingleTouchData& aTouch, const TimeStamp& aEventTime);
ScreenIntPoint ResampleAtTime(const ScreenIntPoint& aLastObservedPosition,
const TimeStamp& aTimeStamp);
int32_t mIdentifier = 0;
Maybe<DataPoint> mBaseDataPoint;
Maybe<DataPoint> mLatestDataPoint;
};
struct CurrentTouches {
void UpdateFromEvent(const MultiTouchInput& aInput);
bool HasTouch() const { return !mTouches.IsEmpty(); }
TimeStamp LatestDataPointTime() { return mLatestDataPointTime; }
ScreenIntPoint ResampleTouchPositionAtTime(
int32_t aIdentifier, const ScreenIntPoint& aLastObservedPosition,
const TimeStamp& aTimeStamp);
void ClearDataPoints() {
for (auto& touch : mTouches) {
touch.mBaseDataPoint = Nothing();
touch.mLatestDataPoint = Nothing();
}
}
private:
nsTArray<TouchInfo>::iterator TouchByIdentifier(int32_t aIdentifier);
nsTArray<TouchInfo> mTouches;
TimeStamp mLatestDataPointTime;
};
// The current touch positions with historical data points. This data only
// contains original non-resampled positions from the incoming touch events.
CurrentTouches mCurrentTouches;
// Incoming touch move events are stored here until NotifyFrame is called.
std::queue<std::pair<MultiTouchInput, uint64_t>> mDeferredTouchMoveEvents;
// Stores any touch samples that were not included in the last emitted touch
// move event because they were in the future compared to the emitted event's
// timestamp. These data points should be prepended to the historical data of
// the next emitted touch move evnt.
// Can only be non-empty if mInResampledState is true.
std::unordered_map<int32_t, nsTArray<SingleTouchData::HistoricalTouchData>>
mRemainingTouchData;
// If we're in an resampled state, because the last outgoing event was a
// resampled touch move event, then this contains a copy of the unresampled,
// original touch move event.
// Some() iff mInResampledState is true.
Maybe<MultiTouchInput> mOriginalOfResampledTouchMove;
// The stream of outgoing events that can be consumed by our caller.
std::queue<OutgoingEvent> mOutgoingEvents;
// The timestamp of the event that was emitted most recently, or the null
// timestamp if no event has been emitted yet.
TimeStamp mLastEmittedEventTime;
uint64_t mNextEventId = 0;
// True if the last outgoing event was a touch move event with an resampled
// position. We only want to stay in this state as long as a continuous stream
// of touch move events is coming in.
bool mInResampledState = false;
};
} // namespace widget
} // namespace mozilla
#endif // mozilla_widget_TouchResampler_h

1
widget/moz.build
Просмотреть файл

@ -219,6 +219,7 @@ UNIFIED_SOURCES += [
"ScrollbarDrawingMac.cpp",
"SharedWidgetUtils.cpp",
"TextEventDispatcher.cpp",
"TouchResampler.cpp",
"VsyncDispatcher.cpp",
"WidgetEventImpl.cpp",
"WidgetUtils.cpp",

941
widget/tests/gtest/TestTouchResampler.cpp Normal file
Просмотреть файл

@ -0,0 +1,941 @@
/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim: set ts=8 sts=2 et sw=2 tw=80: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this file,
* You can obtain one at http://mozilla.org/MPL/2.0/. */
#include <initializer_list>
#include "InputData.h"
#include "Units.h"
#include "gtest/gtest.h"
#include "mozilla/Maybe.h"
#include "mozilla/TimeStamp.h"
#include "TouchResampler.h"
using namespace mozilla;
using widget::TouchResampler;
class TouchResamplerTest : public ::testing::Test {
protected:
virtual void SetUp() { baseTimeStamp = TimeStamp::Now(); }
TimeStamp Time(double aMilliseconds) {
return baseTimeStamp + TimeDuration::FromMilliseconds(aMilliseconds);
}
uint64_t ProcessEvent(
MultiTouchInput::MultiTouchType aType,
std::initializer_list<std::pair<TimeStamp, ScreenIntPoint>>
aHistoricalData,
const TimeStamp& aTimeStamp, const ScreenIntPoint& aPosition) {
MultiTouchInput input(aType, 0, aTimeStamp, 0);
input.mTouches.AppendElement(SingleTouchData(1, aPosition, {}, 0.0f, 0.0f));
for (const auto& histData : aHistoricalData) {
input.mTouches[0].mHistoricalData.AppendElement(
SingleTouchData::HistoricalTouchData{
histData.first, histData.second, {}, {}, 0.0f, 0.0f});
}
return resampler.ProcessEvent(std::move(input));
}
void CheckTime(const TimeStamp& aTimeStamp,
const TimeStamp& aExpectedTimeStamp) {
EXPECT_EQ((aTimeStamp - baseTimeStamp).ToMilliseconds(),
(aExpectedTimeStamp - baseTimeStamp).ToMilliseconds());
}
void CheckEvent(const MultiTouchInput& aEvent,
MultiTouchInput::MultiTouchType aExpectedType,
std::initializer_list<std::pair<TimeStamp, ScreenIntPoint>>
aExpectedHistoricalData,
const TimeStamp& aExpectedTimeStamp,
const ScreenIntPoint& aExpectedPosition) {
EXPECT_EQ(aEvent.mType, aExpectedType);
EXPECT_EQ(aEvent.mTouches.Length(), size_t(1));
EXPECT_EQ(aEvent.mTouches[0].mHistoricalData.Length(),
aExpectedHistoricalData.size());
for (size_t i = 0; i < aExpectedHistoricalData.size(); i++) {
CheckTime(aEvent.mTouches[0].mHistoricalData[i].mTimeStamp,
aExpectedHistoricalData.begin()[i].first);
EXPECT_EQ(aEvent.mTouches[0].mHistoricalData[i].mScreenPoint,
aExpectedHistoricalData.begin()[i].second);
}
CheckTime(aEvent.mTimeStamp, aExpectedTimeStamp);
EXPECT_EQ(aEvent.mTouches[0].mScreenPoint, aExpectedPosition);
}
struct ExpectedOutgoingEvent {
Maybe<uint64_t> mEventId;
MultiTouchInput::MultiTouchType mType = MultiTouchInput::MULTITOUCH_START;
std::initializer_list<std::pair<TimeStamp, ScreenIntPoint>> mHistoricalData;
TimeStamp mTimeStamp;
ScreenIntPoint mPosition;
};
void CheckOutgoingEvents(
std::initializer_list<ExpectedOutgoingEvent> aExpectedEvents) {
auto outgoing = resampler.ConsumeOutgoingEvents();
EXPECT_EQ(outgoing.size(), aExpectedEvents.size());
for (const auto& expectedEvent : aExpectedEvents) {
auto outgoingEvent = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingEvent.mEventId, expectedEvent.mEventId);
CheckEvent(outgoingEvent.mEvent, expectedEvent.mType,
expectedEvent.mHistoricalData, expectedEvent.mTimeStamp,
expectedEvent.mPosition);
}
}
TimeStamp baseTimeStamp;
TouchResampler resampler;
};
TEST_F(TouchResamplerTest, BasicExtrapolation) {
// Execute the following sequence:
//
// 0----------10-------16-----20---------------32------------
// * touchstart at (10, 10)
// * touchmove at (20, 20)
// * frame
// * touchend at (20, 20)
// * frame
//
// And expect the following output:
//
// 0----------10-------16-----20---------------32------------
// * touchstart at (10, 10)
// * touchmove at (26, 26)
// * touchmove at (20, 20)
// * touchend at (20, 20)
//
// The first frame should emit an extrapolated touchmove from the position
// data in the touchstart and touchmove events.
// The touchend should force a synthesized touchmove that returns back to a
// non-resampled position.
EXPECT_FALSE(resampler.InTouchingState());
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(10, 10));
EXPECT_TRUE(resampler.InTouchingState());
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(10.0),
ScreenIntPoint(20, 20));
resampler.NotifyFrame(Time(16.0));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(10, 10)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(20, 20)}},
Time(16.0),
ScreenIntPoint(26, 26)},
});
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(20.0),
ScreenIntPoint(20, 20));
EXPECT_FALSE(resampler.InTouchingState());
CheckOutgoingEvents({
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(16.0),
ScreenIntPoint(20, 20)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(20.0),
ScreenIntPoint(20, 20)},
});
// No more events should be produced from here on out.
resampler.NotifyFrame(Time(32.0));
auto outgoing = resampler.ConsumeOutgoingEvents();
EXPECT_TRUE(outgoing.empty());
}
TEST_F(TouchResamplerTest, BasicInterpolation) {
// Same test as BasicExtrapolation, but with a frame time that's 10ms earlier.
//
// Execute the following sequence:
//
// 0------6---10-----------20--22------------30-------------
// * touchstart at (10, 10)
// * touchmove at (20, 20)
// * frame
// * touchend at (20, 20)
// * frame
//
// And expect the following output:
//
// 0------6---10-----------20--22------------30-------------
// * touchstart at (10, 10)
// * touchmove (16, 16)
// * touchmove (20, 20)
// * touchend at (20, 20)
//
// The first frame should emit an interpolated touchmove from the position
// data in the touchstart and touchmove events.
// The touchend should create a touchmove that returns back to a non-resampled
// position.
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(10, 10));
EXPECT_TRUE(resampler.InTouchingState());
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(10.0),
ScreenIntPoint(20, 20));
resampler.NotifyFrame(Time(6.0));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(10, 10)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(6.0),
ScreenIntPoint(16, 16)},
});
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(20.0),
ScreenIntPoint(20, 20));
EXPECT_FALSE(resampler.InTouchingState());
CheckOutgoingEvents({
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(10.0),
ScreenIntPoint(20, 20)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(20.0),
ScreenIntPoint(20, 20)},
});
// No more events should be produced from here on out.
resampler.NotifyFrame(Time(22.0));
auto outgoing = resampler.ConsumeOutgoingEvents();
EXPECT_TRUE(outgoing.empty());
}
TEST_F(TouchResamplerTest, InterpolationFromHistoricalData) {
// Interpolate from the historical data in a touch move event.
//
// Execute the following sequence:
//
// 0----------10-------16-----20-----------30--32------------
// * touchstart at (10, 10)
// * [hist] at (20, 25) for
// `---------------* touchmove at (30, 30)
// * frame
// * touchend at (30, 30)
// * frame
//
// And expect the following output:
//
// 0----------10-------16-----20-----------30--32------------
// * touchstart at (10, 10)
// * [hist] at (20, 25) for
// `--------* touchmove at (26, 28)
// * touchmove at (30, 30)
// * touchend at (30, 30)
//
// The first frame should emit an interpolated touchmove from the position
// data in the touchmove event, and integrate the historical data point into
// the resampled event.
// The touchend should force a synthesized touchmove that returns back to a
// non-resampled position.
// This also tests that interpolation works for both x and y, by giving the
// historical datapoint different values for x and y.
// (26, 28) is 60% of the way from (20, 25) to (30, 30).
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(10, 10));
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(20, 25)}},
Time(20.0), ScreenIntPoint(30, 30));
resampler.NotifyFrame(Time(16.0));
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(30.0),
ScreenIntPoint(30, 30));
resampler.NotifyFrame(Time(32.0));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(10, 10)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(20, 25)}},
Time(16.0),
ScreenIntPoint(26, 28)},
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(20.0),
ScreenIntPoint(30, 30)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(30.0),
ScreenIntPoint(30, 30)},
});
}
TEST_F(TouchResamplerTest, MultipleTouches) {
EXPECT_FALSE(resampler.InTouchingState());
// Touch start
MultiTouchInput inputStart0(MultiTouchInput::MULTITOUCH_START, 0, Time(0.0),
0);
inputStart0.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(10, 10), {}, 0.0f, 0.0f));
auto idStart0 = resampler.ProcessEvent(std::move(inputStart0));
EXPECT_TRUE(resampler.InTouchingState());
// Touch move
MultiTouchInput inputMove1(MultiTouchInput::MULTITOUCH_MOVE, 0, Time(20.0),
0);
inputMove1.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(30, 30), {}, 0.0f, 0.0f));
inputMove1.mTouches[0].mHistoricalData.AppendElement(
SingleTouchData::HistoricalTouchData{
Time(10.0), ScreenIntPoint(20, 25), {}, {}, 0.0f, 0.0f});
auto idMove1 = resampler.ProcessEvent(std::move(inputMove1));
EXPECT_TRUE(resampler.InTouchingState());
// Frame
resampler.NotifyFrame(Time(16.0));
// Touch move
MultiTouchInput inputMove2(MultiTouchInput::MULTITOUCH_MOVE, 0, Time(30.0),
0);
inputMove2.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(30, 40), {}, 0.0f, 0.0f));
auto idMove2 = resampler.ProcessEvent(std::move(inputMove2));
EXPECT_TRUE(resampler.InTouchingState());
// Touch start
MultiTouchInput inputStart3(MultiTouchInput::MULTITOUCH_START, 0, Time(30.0),
0);
inputStart3.mTouches.AppendElement(
SingleTouchData(2, ScreenIntPoint(100, 10), {}, 0.0f, 0.0f));
inputStart3.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(30, 40), {}, 0.0f, 0.0f));
auto idStart3 = resampler.ProcessEvent(std::move(inputStart3));
EXPECT_TRUE(resampler.InTouchingState());
// Touch move
MultiTouchInput inputMove4(MultiTouchInput::MULTITOUCH_MOVE, 0, Time(40.0),
0);
inputMove4.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(30, 50), {}, 0.0f, 0.0f));
inputMove4.mTouches.AppendElement(
SingleTouchData(2, ScreenIntPoint(100, 30), {}, 0.0f, 0.0f));
auto idMove4 = resampler.ProcessEvent(std::move(inputMove4));
EXPECT_TRUE(resampler.InTouchingState());
// Frame
resampler.NotifyFrame(Time(32.0));
// Touch move
MultiTouchInput inputMove5(MultiTouchInput::MULTITOUCH_MOVE, 0, Time(50.0),
0);
inputMove5.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(30, 60), {}, 0.0f, 0.0f));
inputMove5.mTouches.AppendElement(
SingleTouchData(2, ScreenIntPoint(100, 40), {}, 0.0f, 0.0f));
auto idMove5 = resampler.ProcessEvent(std::move(inputMove5));
EXPECT_TRUE(resampler.InTouchingState());
// Touch end
MultiTouchInput inputEnd6(MultiTouchInput::MULTITOUCH_END, 0, Time(50.0), 0);
// Touch point with identifier 1 is lifted
inputEnd6.mTouches.AppendElement(
SingleTouchData(1, ScreenIntPoint(30, 60), {}, 0.0f, 0.0f));
auto idEnd6 = resampler.ProcessEvent(std::move(inputEnd6));
EXPECT_TRUE(resampler.InTouchingState());
// Frame
resampler.NotifyFrame(Time(48.0));
// Touch move
MultiTouchInput inputMove7(MultiTouchInput::MULTITOUCH_MOVE, 0, Time(60.0),
0);
inputMove7.mTouches.AppendElement(
SingleTouchData(2, ScreenIntPoint(100, 60), {}, 0.0f, 0.0f));
auto idMove7 = resampler.ProcessEvent(std::move(inputMove7));
EXPECT_TRUE(resampler.InTouchingState());
// Frame
resampler.NotifyFrame(Time(64.0));
// Touch end
MultiTouchInput inputEnd8(MultiTouchInput::MULTITOUCH_END, 0, Time(70.0), 0);
// Touch point with identifier 2 is lifted
inputEnd8.mTouches.AppendElement(
SingleTouchData(2, ScreenIntPoint(100, 60), {}, 0.0f, 0.0f));
auto idEnd8 = resampler.ProcessEvent(std::move(inputEnd8));
EXPECT_FALSE(resampler.InTouchingState());
// Check outgoing events
auto outgoing = resampler.ConsumeOutgoingEvents();
EXPECT_EQ(outgoing.size(), size_t(9));
auto outgoingStart0 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingStart0.mEventId, Some(idStart0));
CheckEvent(outgoingStart0.mEvent, MultiTouchInput::MULTITOUCH_START, {},
Time(0.0), ScreenIntPoint(10, 10));
auto outgoingMove1 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingMove1.mEventId, Some(idMove1));
// (26, 28) is 60% of the way from (20, 25) to (30, 30).
CheckEvent(outgoingMove1.mEvent, MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(20, 25)}}, Time(16.0),
ScreenIntPoint(26, 28));
auto outgoingMove2 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingMove2.mEventId, Some(idMove2));
CheckEvent(outgoingMove2.mEvent, MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(30, 30)}}, Time(30.0),
ScreenIntPoint(30, 40));
auto outgoingStart3 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingStart3.mEventId, Some(idStart3));
EXPECT_EQ(outgoingStart3.mEvent.mType, MultiTouchInput::MULTITOUCH_START);
CheckTime(outgoingStart3.mEvent.mTimeStamp, Time(30.0));
EXPECT_EQ(outgoingStart3.mEvent.mTouches.Length(), size_t(2));
// touch order should be taken from the original touch start event
EXPECT_EQ(outgoingStart3.mEvent.mTouches[0].mIdentifier, 2);
EXPECT_EQ(outgoingStart3.mEvent.mTouches[0].mScreenPoint,
ScreenIntPoint(100, 10));
EXPECT_EQ(outgoingStart3.mEvent.mTouches[1].mIdentifier, 1);
EXPECT_EQ(outgoingStart3.mEvent.mTouches[1].mScreenPoint,
ScreenIntPoint(30, 40));
auto outgoingMove4 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingMove4.mEventId, Some(idMove4));
EXPECT_EQ(outgoingMove4.mEvent.mType, MultiTouchInput::MULTITOUCH_MOVE);
CheckTime(outgoingMove4.mEvent.mTimeStamp, Time(32.0));
EXPECT_EQ(outgoingMove4.mEvent.mTouches.Length(), size_t(2));
// Touch order should be taken from the original touch move event.
// Both touches should be resampled.
EXPECT_EQ(outgoingMove4.mEvent.mTouches[0].mIdentifier, 1);
EXPECT_EQ(outgoingMove4.mEvent.mTouches[0].mScreenPoint,
ScreenIntPoint(30, 42));
EXPECT_EQ(outgoingMove4.mEvent.mTouches[1].mIdentifier, 2);
EXPECT_EQ(outgoingMove4.mEvent.mTouches[1].mScreenPoint,
ScreenIntPoint(100, 14));
auto outgoingMove5 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingMove5.mEventId, Some(idMove5));
EXPECT_EQ(outgoingMove5.mEvent.mType, MultiTouchInput::MULTITOUCH_MOVE);
CheckTime(outgoingMove5.mEvent.mTimeStamp, Time(50.0));
EXPECT_EQ(outgoingMove5.mEvent.mTouches.Length(), size_t(2));
// touch order should be taken from the original touch move event
EXPECT_EQ(outgoingMove5.mEvent.mTouches[0].mIdentifier, 1);
EXPECT_EQ(outgoingMove5.mEvent.mTouches[0].mScreenPoint,
ScreenIntPoint(30, 60));
EXPECT_EQ(outgoingMove5.mEvent.mTouches[0].mHistoricalData.Length(),
size_t(1));
CheckTime(outgoingMove5.mEvent.mTouches[0].mHistoricalData[0].mTimeStamp,
Time(40.0));
EXPECT_EQ(outgoingMove5.mEvent.mTouches[0].mHistoricalData[0].mScreenPoint,
ScreenIntPoint(30, 50));
EXPECT_EQ(outgoingMove5.mEvent.mTouches[1].mIdentifier, 2);
EXPECT_EQ(outgoingMove5.mEvent.mTouches[1].mScreenPoint,
ScreenIntPoint(100, 40));
EXPECT_EQ(outgoingMove5.mEvent.mTouches[1].mHistoricalData.Length(),
size_t(1));
CheckTime(outgoingMove5.mEvent.mTouches[1].mHistoricalData[0].mTimeStamp,
Time(40.0));
EXPECT_EQ(outgoingMove5.mEvent.mTouches[1].mHistoricalData[0].mScreenPoint,
ScreenIntPoint(100, 30));
auto outgoingEnd6 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingEnd6.mEventId, Some(idEnd6));
CheckEvent(outgoingEnd6.mEvent, MultiTouchInput::MULTITOUCH_END, {},
Time(50.0), ScreenIntPoint(30, 60));
auto outgoingMove7 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingMove7.mEventId, Some(idMove7));
// No extrapolation because the frame at 64.0 cleared the data points because
// there was no pending touch move event at that point
CheckEvent(outgoingMove7.mEvent, MultiTouchInput::MULTITOUCH_MOVE, {},
Time(60.0), ScreenIntPoint(100, 60));
EXPECT_EQ(outgoingMove7.mEvent.mTouches[0].mIdentifier, 2);
auto outgoingEnd8 = std::move(outgoing.front());
outgoing.pop();
EXPECT_EQ(outgoingEnd8.mEventId, Some(idEnd8));
CheckEvent(outgoingEnd8.mEvent, MultiTouchInput::MULTITOUCH_END, {},
Time(70.0), ScreenIntPoint(100, 60));
}
TEST_F(TouchResamplerTest, MovingPauses) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(10, 10));
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(10.0),
ScreenIntPoint(20, 20));
resampler.NotifyFrame(Time(16.0));
auto idMove2 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(30.0),
ScreenIntPoint(40, 40));
resampler.NotifyFrame(Time(32.0));
auto idMove3 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(40.0),
ScreenIntPoint(50, 40));
resampler.NotifyFrame(Time(48.0));
resampler.NotifyFrame(Time(64.0));
auto idEnd4 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(50, 40));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(10, 10)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(20, 20)}},
Time(16.0),
ScreenIntPoint(26, 26)},
{Some(idMove2),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(30.0), ScreenIntPoint(40, 40)}},
Time(32.0),
ScreenIntPoint(42, 42)},
{Some(idMove3),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(40.0), ScreenIntPoint(50, 40)}},
Time(48.0),
ScreenIntPoint(58, 40)},
// There was no event between two frames here, so we expect a reset event,
// so that we pause at a non-resampled position.
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(48.0),
ScreenIntPoint(50, 40)},
{Some(idEnd4),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(50, 40)},
});
}
TEST_F(TouchResamplerTest, MixedInterAndExtrapolation) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(10.0),
ScreenIntPoint(0, 10));
resampler.NotifyFrame(Time(11.0)); // 16 - 5
auto idMove2 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)}}, Time(30.0),
ScreenIntPoint(0, 30));
resampler.NotifyFrame(Time(27.0)); // 32 - 5
auto idMove3 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(40.0),
ScreenIntPoint(0, 40));
resampler.NotifyFrame(Time(43.0)); // 48 - 5
auto idMove4 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(50.0), ScreenIntPoint(0, 50)}}, Time(60.0),
ScreenIntPoint(0, 60));
resampler.NotifyFrame(Time(59.0)); // 64 - 5
auto idEnd5 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 60));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}},
Time(11.0),
ScreenIntPoint(0, 11)},
{Some(idMove2),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)}},
Time(27.0),
ScreenIntPoint(0, 27)},
{Some(idMove3),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(30.0), ScreenIntPoint(0, 30)},
{Time(40.0), ScreenIntPoint(0, 40)}},
Time(43.0),
ScreenIntPoint(0, 43)},
{Some(idMove4),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(50.0), ScreenIntPoint(0, 50)}},
Time(59.0),
ScreenIntPoint(0, 59)},
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(60.0),
ScreenIntPoint(0, 60)},
{Some(idEnd5),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 60)},
});
}
TEST_F(TouchResamplerTest, MultipleMoveEvents) {
// Test what happens if multiple touch move events appear between two frames.
// This scenario shouldn't occur on Android but we should be able to deal with
// it anyway. Check that we don't discard any event IDs.
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(10.0),
ScreenIntPoint(0, 10));
resampler.NotifyFrame(Time(11.0)); // 16 - 5
auto idMove2 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)}}, Time(30.0),
ScreenIntPoint(0, 30));
auto idMove3 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(40.0),
ScreenIntPoint(0, 40));
auto idMove4 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(45.0), ScreenIntPoint(0, 45)}}, Time(50.0),
ScreenIntPoint(0, 50));
auto idMove5 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE, {}, Time(55.0),
ScreenIntPoint(0, 55));
resampler.NotifyFrame(Time(43.0)); // 48 - 5
resampler.NotifyFrame(Time(59.0)); // 64 - 5
auto idEnd5 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 60));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}},
Time(11.0),
ScreenIntPoint(0, 11)},
{Some(idMove2),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)}},
Time(30.0),
ScreenIntPoint(0, 30)},
{Some(idMove3),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(40.0),
ScreenIntPoint(0, 40)},
{Some(idMove4),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(43.0),
ScreenIntPoint(0, 43)},
{Some(idMove5),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(45.0), ScreenIntPoint(0, 45)},
{Time(50.0), ScreenIntPoint(0, 50)},
{Time(55.0), ScreenIntPoint(0, 55)}},
Time(59.0),
ScreenIntPoint(0, 59)},
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(59.0),
ScreenIntPoint(0, 55)},
{Some(idEnd5),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 60)},
});
}
TEST_F(TouchResamplerTest, LimitFuturePrediction) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
// Fingers move until time 44, then pause. UI thread is occupied until 64.
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)},
{Time(32.0), ScreenIntPoint(0, 32)}},
Time(44.0), ScreenIntPoint(0, 44));
resampler.NotifyFrame(Time(59.0)); // 64 - 5
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 44));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
// kTouchResampleMaxPredictMs == 8
// Refuse to predict more than 8ms into the future, the fingers might have
// paused. Make an event for time 52 (= 44 + 8) instead of 59.
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)},
{Time(32.0), ScreenIntPoint(0, 32)},
{Time(44.0), ScreenIntPoint(0, 44)}},
Time(52.0),
ScreenIntPoint(0, 52)},
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(52.0),
ScreenIntPoint(0, 44)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 44)},
});
}
TEST_F(TouchResamplerTest, LimitBacksampling) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
// Fingers move until time 44, then pause. UI thread is occupied until 64.
// Then we get a frame callback with a very outdated frametime.
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)},
{Time(32.0), ScreenIntPoint(0, 32)}},
Time(44.0), ScreenIntPoint(0, 44));
resampler.NotifyFrame(Time(11.0)); // 16 - 5
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 44));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
// kTouchResampleMaxBacksampleMs == 20
// Refuse to sample further back than 20ms before the last data point.
// Make an event for time 24 (= 44 - 20) instead of time 11.
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)}},
Time(24.0),
ScreenIntPoint(0, 24)},
{Nothing(),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(32.0), ScreenIntPoint(0, 32)}},
Time(44.0),
ScreenIntPoint(0, 44)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 44)},
});
}
TEST_F(TouchResamplerTest, DontExtrapolateFromOldTouch) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
// Fingers move until time 40, then pause. UI thread is occupied until 64.
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)},
{Time(30.0), ScreenIntPoint(0, 30)}},
Time(40.0), ScreenIntPoint(0, 40));
resampler.NotifyFrame(Time(59.0)); // 64 - 5
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 44));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
// kTouchResampleOldTouchThresholdMs == 17
// Refuse to extrapolate from a data point that's more than 17ms older
// than the frame time.
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(20.0), ScreenIntPoint(0, 20)},
{Time(30.0), ScreenIntPoint(0, 30)}},
Time(40.0),
ScreenIntPoint(0, 40)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 44)},
});
}
TEST_F(TouchResamplerTest, DontExtrapolateIfTooOld) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
// Fingers move until time 10, pause, and move again at 55.
// UI thread is occupied until 64.
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}}, Time(55.0),
ScreenIntPoint(0, 55));
resampler.NotifyFrame(Time(59.0)); // 64 - 5
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 60));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
// kTouchResampleWindowSize == 40
// Refuse to resample between two data points that are more than 40ms
// apart.
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}},
Time(55.0),
ScreenIntPoint(0, 55)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 60)},
});
}
TEST_F(TouchResamplerTest, DontInterpolateIfTooOld) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
// Fingers move until time 10, pause, and move again at 60.
// UI thread is occupied until 64.
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}}, Time(60.0),
ScreenIntPoint(0, 60));
resampler.NotifyFrame(Time(59.0)); // 64 - 5
auto idEnd2 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(70.0),
ScreenIntPoint(0, 60));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
// kTouchResampleWindowSize == 40
// Refuse to resample between two data points that are more than 40ms
// apart.
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}},
Time(60.0),
ScreenIntPoint(0, 60)},
{Some(idEnd2),
MultiTouchInput::MULTITOUCH_END,
{},
Time(70.0),
ScreenIntPoint(0, 60)},
});
}
TEST_F(TouchResamplerTest, DiscardOutdatedHistoricalData) {
auto idStart0 = ProcessEvent(MultiTouchInput::MULTITOUCH_START, {}, Time(0.0),
ScreenIntPoint(0, 0));
auto idMove1 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)}}, Time(16.0),
ScreenIntPoint(0, 16));
resampler.NotifyFrame(Time(20.0));
auto idMove2 = ProcessEvent(MultiTouchInput::MULTITOUCH_MOVE,
{{Time(18.0), ScreenIntPoint(0, 18)}}, Time(25.0),
ScreenIntPoint(0, 25));
auto idEnd3 = ProcessEvent(MultiTouchInput::MULTITOUCH_END, {}, Time(35.0),
ScreenIntPoint(0, 25));
CheckOutgoingEvents({
{Some(idStart0),
MultiTouchInput::MULTITOUCH_START,
{},
Time(0.0),
ScreenIntPoint(0, 0)},
{Some(idMove1),
MultiTouchInput::MULTITOUCH_MOVE,
{{Time(10.0), ScreenIntPoint(0, 10)},
{Time(16.0), ScreenIntPoint(0, 16)}},
Time(20.0),
ScreenIntPoint(0, 20)},
// Discard the historical data point from time 18, because we've already
// sent out an event with time 20 and don't want to go back before that.
{Some(idMove2),
MultiTouchInput::MULTITOUCH_MOVE,
{},
Time(25.0),
ScreenIntPoint(0, 25)},
{Some(idEnd3),
MultiTouchInput::MULTITOUCH_END,
{},
Time(35.0),
ScreenIntPoint(0, 25)},
});
}

1
widget/tests/gtest/moz.build
Просмотреть файл

@ -6,6 +6,7 @@
UNIFIED_SOURCES = [
"TestTimeConverter.cpp",
"TestTouchResampler.cpp",
]
FINAL_LIBRARY = "xul-gtest"