nextcloud
/
desktop
зеркало из https://github.com/nextcloud/desktop.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
desktop/test/testsyncengine.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

467 строки
21 KiB
C++
Исходник Обычный вид История

Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
/*
* This software is in the public domain, furnished "as is", without technical
* support, and with no warranty, express or implied, as to its usefulness for
* any purpose.
*
*/
#include <QtTest>
#include "syncenginetestutils.h"
SyncEngine: Fix renaming a folder should keep the selective sync state Issue #5224 Two problems: - In the discovery phase, we need to check the selective sync entries of the source path in case of renames. - When the rename is done, we need to actually update the black list in the database.
2016-10-10 17:55:31 +03:00
#include <syncengine.h>
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
using namespace OCC;
bool itemDidComplete(const QSignalSpy &spy, const QString &path)
{
for(const QList<QVariant> &args : spy) {
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
auto item = args[0].value<SyncFileItemPtr>();
if (item->destination() == path)
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
return true;
}
return false;
}
bool itemDidCompleteSuccessfully(const QSignalSpy &spy, const QString &path)
{
for(const QList<QVariant> &args : spy) {
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
auto item = args[0].value<SyncFileItemPtr>();
if (item->destination() == path)
return item->_status == SyncFileItem::Success;
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
}
return false;
}
class TestSyncEngine : public QObject
{
Q_OBJECT
private slots:
void testFileDownload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.remoteModifier().insert("A/a0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testFileUpload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.localModifier().insert("A/a0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testDirDownload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.remoteModifier().mkdir("Y");
fakeFolder.remoteModifier().mkdir("Z");
fakeFolder.remoteModifier().insert("Z/d0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Y"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z/d0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testDirUpload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.localModifier().mkdir("Y");
fakeFolder.localModifier().mkdir("Z");
fakeFolder.localModifier().insert("Z/d0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Y"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z/d0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testLocalDelete() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.remoteModifier().remove("A/a1");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a1"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testRemoteDelete() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.localModifier().remove("A/a1");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a1"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testEmlLocalChecksum() {
FakeFolder fakeFolder{FileInfo{}};
fakeFolder.localModifier().insert("a1.eml", 64, 'A');
fakeFolder.localModifier().insert("a2.eml", 64, 'A');
fakeFolder.localModifier().insert("a3.eml", 64, 'A');
// Upload and calculate the checksums
// fakeFolder.syncOnce();
fakeFolder.syncOnce();
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
// Touch the file without changing the content, shouldn't upload
fakeFolder.localModifier().setContents("a1.eml", 'A');
// Change the content/size
fakeFolder.localModifier().setContents("a2.eml", 'B');
fakeFolder.localModifier().appendByte("a3.eml");
fakeFolder.syncOnce();
QVERIFY(!itemDidComplete(completeSpy, "a1.eml"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "a2.eml"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "a3.eml"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
SyncEngine: Fix renaming of folder when file are changed (#5195) Two bugs: - The change filed are not considered as move, they are re-downloaded but the old file was not removed from the database. The change in owncloudpropagator.cpp takes care of removing the old entries. - Next sync would then remove the file in the server in the old folder This was not a problem until we start reusing the sync engine, and that the _renamedFolders map is not cleared. We were before deleting a non-existing file. But now we delete the actual file. Also improve the tests to be able to do move on the server. This include support for file id. Issue #5192
2016-09-22 10:02:47 +03:00
void testRemoteChangeInMovedFolder() {
// issue #5192
FakeFolder fakeFolder{FileInfo{ QString(), {
FileInfo { QStringLiteral("folder"), {
FileInfo{ QStringLiteral("folderA"), { { QStringLiteral("file.txt"), 400 } } },
QStringLiteral("folderB")
}
}}}};
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// Edit a file in a moved directory.
fakeFolder.remoteModifier().setContents("folder/folderA/file.txt", 'a');
fakeFolder.remoteModifier().rename("folder/folderA", "folder/folderB/folderA");
fakeFolder.syncOnce();
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
auto oldState = fakeFolder.currentLocalState();
SyncEngine: Fix renaming a folder should keep the selective sync state Issue #5224 Two problems: - In the discovery phase, we need to check the selective sync entries of the source path in case of renames. - When the rename is done, we need to actually update the black list in the database.
2016-10-10 17:55:31 +03:00
QVERIFY(oldState.find("folder/folderB/folderA/file.txt"));
QVERIFY(!oldState.find("folder/folderA/file.txt"));
SyncEngine: Fix renaming of folder when file are changed (#5195) Two bugs: - The change filed are not considered as move, they are re-downloaded but the old file was not removed from the database. The change in owncloudpropagator.cpp takes care of removing the old entries. - Next sync would then remove the file in the server in the old folder This was not a problem until we start reusing the sync engine, and that the _renamedFolders map is not cleared. We were before deleting a non-existing file. But now we delete the actual file. Also improve the tests to be able to do move on the server. This include support for file id. Issue #5192
2016-09-22 10:02:47 +03:00
// This sync should not remove the file
fakeFolder.syncOnce();
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
QCOMPARE(fakeFolder.currentLocalState(), oldState);
}
SyncEngine: Fix renaming a folder should keep the selective sync state Issue #5224 Two problems: - In the discovery phase, we need to check the selective sync entries of the source path in case of renames. - When the rename is done, we need to actually update the black list in the database.
2016-10-10 17:55:31 +03:00
void testSelectiveSyncModevFolder() {
// issue #5224
FakeFolder fakeFolder{FileInfo{ QString(), {
FileInfo { QStringLiteral("parentFolder"), {
FileInfo{ QStringLiteral("subFolderA"), { { QStringLiteral("fileA.txt"), 400 } } },
FileInfo{ QStringLiteral("subFolderB"), { { QStringLiteral("fileB.txt"), 400 } } }
}
}}}};
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
auto expectedServerState = fakeFolder.currentRemoteState();
// Remove subFolderA with selectiveSync:
fakeFolder.syncEngine().journal()->setSelectiveSyncList(SyncJournalDb::SelectiveSyncBlackList,
{"parentFolder/subFolderA/"});
fakeFolder.syncEngine().journal()->avoidReadFromDbOnNextSync("parentFolder/subFolderA/");
fakeFolder.syncOnce();
{
// Nothing changed on the server
QCOMPARE(fakeFolder.currentRemoteState(), expectedServerState);
// The local state should not have subFolderA
auto remoteState = fakeFolder.currentRemoteState();
remoteState.remove("parentFolder/subFolderA");
QCOMPARE(fakeFolder.currentLocalState(), remoteState);
}
// Rename parentFolder on the server
fakeFolder.remoteModifier().rename("parentFolder", "parentFolderRenamed");
expectedServerState = fakeFolder.currentRemoteState();
fakeFolder.syncOnce();
{
QCOMPARE(fakeFolder.currentRemoteState(), expectedServerState);
auto remoteState = fakeFolder.currentRemoteState();
// The subFolderA should still be there on the server.
QVERIFY(remoteState.find("parentFolderRenamed/subFolderA/fileA.txt"));
// But not on the client because of the selective sync
remoteState.remove("parentFolderRenamed/subFolderA");
QCOMPARE(fakeFolder.currentLocalState(), remoteState);
}
// Rename it again, locally this time.
fakeFolder.localModifier().rename("parentFolderRenamed", "parentThirdName");
fakeFolder.syncOnce();
{
auto remoteState = fakeFolder.currentRemoteState();
// The subFolderA should still be there on the server.
QVERIFY(remoteState.find("parentThirdName/subFolderA/fileA.txt"));
// But not on the client because of the selective sync
remoteState.remove("parentThirdName/subFolderA");
QCOMPARE(fakeFolder.currentLocalState(), remoteState);
expectedServerState = fakeFolder.currentRemoteState();
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
SyncEngine: Fix renaming a folder should keep the selective sync state Issue #5224 Two problems: - In the discovery phase, we need to check the selective sync entries of the source path in case of renames. - When the rename is done, we need to actually update the black list in the database.
2016-10-10 17:55:31 +03:00
fakeFolder.syncOnce(); // This sync should do nothing
QCOMPARE(completeSpy.count(), 0);
QCOMPARE(fakeFolder.currentRemoteState(), expectedServerState);
QCOMPARE(fakeFolder.currentLocalState(), remoteState);
}
}
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
void testSelectiveSyncBug() {
// issue owncloud/enterprise#1965: files from selective-sync ignored
// folders are uploaded anyway is some circumstances.
FakeFolder fakeFolder{FileInfo{ QString(), {
FileInfo { QStringLiteral("parentFolder"), {
FileInfo{ QStringLiteral("subFolder"), {
{ QStringLiteral("fileA.txt"), 400 },
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
{ QStringLiteral("fileB.txt"), 400, 'o' },
FileInfo { QStringLiteral("subsubFolder"), {
{ QStringLiteral("fileC.txt"), 400 },
{ QStringLiteral("fileD.txt"), 400, 'o' }
}},
FileInfo{ QStringLiteral("anotherFolder"), {
FileInfo { QStringLiteral("emptyFolder"), { } },
FileInfo { QStringLiteral("subsubFolder"), {
{ QStringLiteral("fileE.txt"), 400 },
{ QStringLiteral("fileF.txt"), 400, 'o' }
}}
}}
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
}}
}}
}}};
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
auto expectedServerState = fakeFolder.currentRemoteState();
// Remove subFolder with selectiveSync:
fakeFolder.syncEngine().journal()->setSelectiveSyncList(SyncJournalDb::SelectiveSyncBlackList,
{"parentFolder/subFolder/"});
fakeFolder.syncEngine().journal()->avoidReadFromDbOnNextSync("parentFolder/subFolder/");
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
// But touch local file before the next sync, such that the local folder
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
// can't be removed
fakeFolder.localModifier().setContents("parentFolder/subFolder/fileB.txt", 'n');
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
fakeFolder.localModifier().setContents("parentFolder/subFolder/subsubFolder/fileD.txt", 'n');
fakeFolder.localModifier().setContents("parentFolder/subFolder/anotherFolder/subsubFolder/fileF.txt", 'n');
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
// Several follow-up syncs don't change the state at all,
// in particular the remote state doesn't change and fileB.txt
// isn't uploaded.
for (int i = 0; i < 3; ++i) {
fakeFolder.syncOnce();
{
// Nothing changed on the server
QCOMPARE(fakeFolder.currentRemoteState(), expectedServerState);
// The local state should still have subFolderA
auto local = fakeFolder.currentLocalState();
QVERIFY(local.find("parentFolder/subFolder"));
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
QVERIFY(!local.find("parentFolder/subFolder/fileA.txt"));
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
QVERIFY(local.find("parentFolder/subFolder/fileB.txt"));
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
QVERIFY(!local.find("parentFolder/subFolder/subsubFolder/fileC.txt"));
QVERIFY(local.find("parentFolder/subFolder/subsubFolder/fileD.txt"));
QVERIFY(!local.find("parentFolder/subFolder/anotherFolder/subsubFolder/fileE.txt"));
QVERIFY(local.find("parentFolder/subFolder/anotherFolder/subsubFolder/fileF.txt"));
QVERIFY(!local.find("parentFolder/subFolder/anotherFolder/emptyFolder"));
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
}
}
}
Delete finished propagation jobs in PropagateDirectory #5269 (#5400)
2017-01-26 12:03:22 +03:00
void abortAfterFailedMkdir() {
FakeFolder fakeFolder{FileInfo{}};
QSignalSpy finishedSpy(&fakeFolder.syncEngine(), SIGNAL(finished(bool)));
fakeFolder.serverErrorPaths().append("NewFolder");
fakeFolder.localModifier().mkdir("NewFolder");
// This should be aborted and would otherwise fail in FileInfo::create.
fakeFolder.localModifier().insert("NewFolder/NewFile");
fakeFolder.syncOnce();
QCOMPARE(finishedSpy.size(), 1);
QCOMPARE(finishedSpy.first().first().toBool(), false);
}
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
void testDirDownloadWithError() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
fakeFolder.remoteModifier().mkdir("Y");
fakeFolder.remoteModifier().mkdir("Y/Z");
fakeFolder.remoteModifier().insert("Y/Z/d0");
fakeFolder.remoteModifier().insert("Y/Z/d1");
fakeFolder.remoteModifier().insert("Y/Z/d2");
fakeFolder.remoteModifier().insert("Y/Z/d3");
fakeFolder.remoteModifier().insert("Y/Z/d4");
fakeFolder.remoteModifier().insert("Y/Z/d5");
fakeFolder.remoteModifier().insert("Y/Z/d6");
fakeFolder.remoteModifier().insert("Y/Z/d7");
fakeFolder.remoteModifier().insert("Y/Z/d8");
fakeFolder.remoteModifier().insert("Y/Z/d9");
fakeFolder.serverErrorPaths().append("Y/Z/d2", 503); // 503 is a fatal error
fakeFolder.serverErrorPaths().append("Y/Z/d3", 503); // 503 is a fatal error
QVERIFY(!fakeFolder.syncOnce());
QCoreApplication::processEvents(); // should not crash
QSet<QString> seen;
for(const QList<QVariant> &args : completeSpy) {
auto item = args[0].value<SyncFileItemPtr>();
qDebug() << item->_file << item->_isDirectory << item->_status;
QVERIFY(!seen.contains(item->_file)); // signal only sent once per item
seen.insert(item->_file);
Fix TestSyncEngine::testDirDownloadWithError On master, the scheduling is different and the tasks abort in another order
2017-03-09 12:35:58 +03:00
if (item->_file == "Y/Z/d2") {
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
QVERIFY(item->_status == SyncFileItem::FatalError);
Fix TestSyncEngine::testDirDownloadWithError On master, the scheduling is different and the tasks abort in another order
2017-03-09 12:35:58 +03:00
} else if(item->_file == "Y/Z/d3") {
QVERIFY(item->_status != SyncFileItem::Success);
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
}
QVERIFY(item->_file != "Y/Z/d9"); // we should have aborted the sync before d9 starts
}
}
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
void testFakeConflict()
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
int nGET = 0;
fakeFolder.setServerOverride([&](QNetworkAccessManager::Operation op, const QNetworkRequest &) {
if (op == QNetworkAccessManager::GetOperation)
++nGET;
return nullptr;
});
// For directly editing the remote checksum
FileInfo &remoteInfo = dynamic_cast<FileInfo &>(fakeFolder.remoteModifier());
// Base mtime with no ms content (filesystem is seconds only)
auto mtime = QDateTime::currentDateTime().addDays(-4);
mtime.setMSecsSinceEpoch(mtime.toMSecsSinceEpoch() / 1000 * 1000);
// Conflict: Same content, mtime, but no server checksum
// -> ignored in reconcile
fakeFolder.localModifier().setContents("A/a1", 'C');
fakeFolder.localModifier().setModTime("A/a1", mtime);
fakeFolder.remoteModifier().setContents("A/a1", 'C');
fakeFolder.remoteModifier().setModTime("A/a1", mtime);
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(nGET, 0);
// Conflict: Same content, mtime, but weak server checksum
// -> ignored in reconcile
mtime = mtime.addDays(1);
fakeFolder.localModifier().setContents("A/a1", 'D');
fakeFolder.localModifier().setModTime("A/a1", mtime);
fakeFolder.remoteModifier().setContents("A/a1", 'D');
fakeFolder.remoteModifier().setModTime("A/a1", mtime);
remoteInfo.find("A/a1")->checksums = "Adler32:bad";
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(nGET, 0);
// Conflict: Same content, mtime, but server checksum differs
// -> downloaded
mtime = mtime.addDays(1);
fakeFolder.localModifier().setContents("A/a1", 'W');
fakeFolder.localModifier().setModTime("A/a1", mtime);
fakeFolder.remoteModifier().setContents("A/a1", 'W');
fakeFolder.remoteModifier().setModTime("A/a1", mtime);
remoteInfo.find("A/a1")->checksums = "SHA1:bad";
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(nGET, 1);
// Conflict: Same content, mtime, matching checksums
// -> PropagateDownload, but it skips the download
mtime = mtime.addDays(1);
fakeFolder.localModifier().setContents("A/a1", 'C');
fakeFolder.localModifier().setModTime("A/a1", mtime);
fakeFolder.remoteModifier().setContents("A/a1", 'C');
fakeFolder.remoteModifier().setModTime("A/a1", mtime);
remoteInfo.find("A/a1")->checksums = "SHA1:56900fb1d337cf7237ff766276b9c1e8ce507427";
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(nGET, 1);
// Extra sync reads from db, no difference
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(nGET, 1);
}
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
/**
* Checks whether SyncFileItems have the expected properties before start
* of propagation.
*/
void testSyncFileItemProperties()
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
auto initialMtime = fakeFolder.currentLocalState().find("A/a1")->lastModified;
auto changedMtime = QDateTime::currentDateTime().addDays(-4);
auto changedMtime2 = QDateTime::currentDateTime().addDays(-3);
// Base mtime with no ms content (filesystem is seconds only)
initialMtime.setMSecsSinceEpoch(initialMtime.toMSecsSinceEpoch() / 1000 * 1000);
changedMtime.setMSecsSinceEpoch(changedMtime.toMSecsSinceEpoch() / 1000 * 1000);
changedMtime2.setMSecsSinceEpoch(changedMtime2.toMSecsSinceEpoch() / 1000 * 1000);
// upload a
fakeFolder.localModifier().appendByte("A/a1");
fakeFolder.localModifier().setModTime("A/a1", changedMtime);
// download b
fakeFolder.remoteModifier().appendByte("B/b1");
fakeFolder.remoteModifier().setModTime("B/b1", changedMtime);
// conflict c
fakeFolder.localModifier().appendByte("C/c1");
fakeFolder.localModifier().appendByte("C/c1");
fakeFolder.localModifier().setModTime("C/c1", changedMtime);
fakeFolder.remoteModifier().appendByte("C/c1");
fakeFolder.remoteModifier().setModTime("C/c1", changedMtime2);
connect(&fakeFolder.syncEngine(), &SyncEngine::aboutToPropagate, [&](SyncFileItemVector &items) {
SyncFileItemPtr a1, b1, c1;
for (auto &item : items) {
if (item->_file == "A/a1")
a1 = item;
if (item->_file == "B/b1")
b1 = item;
if (item->_file == "C/c1")
c1 = item;
}
// a1: should have local size and modtime
QVERIFY(a1);
QCOMPARE(a1->_instruction, CSYNC_INSTRUCTION_SYNC);
QCOMPARE(a1->_direction, SyncFileItem::Up);
// NOTE: This is currently a bug! #5855
//QCOMPARE(a1->_size, quint64(5));
QCOMPARE(Utility::qDateTimeFromTime_t(a1->_modtime), changedMtime);
QCOMPARE(a1->log._other_size, quint64(4));
QCOMPARE(Utility::qDateTimeFromTime_t(a1->log._other_modtime), initialMtime);
// b2: should have remote size and modtime
QVERIFY(b1);
QCOMPARE(b1->_instruction, CSYNC_INSTRUCTION_SYNC);
QCOMPARE(b1->_direction, SyncFileItem::Down);
QCOMPARE(b1->_size, quint64(17));
QCOMPARE(Utility::qDateTimeFromTime_t(b1->_modtime), changedMtime);
QCOMPARE(b1->log._other_size, quint64(16));
QCOMPARE(Utility::qDateTimeFromTime_t(b1->log._other_modtime), initialMtime);
// c1: conflicts are downloads, so remote size and modtime
QVERIFY(c1);
QCOMPARE(c1->_instruction, CSYNC_INSTRUCTION_CONFLICT);
QCOMPARE(c1->_direction, SyncFileItem::None);
QCOMPARE(c1->_size, quint64(25));
QCOMPARE(Utility::qDateTimeFromTime_t(c1->_modtime), changedMtime2);
QCOMPARE(c1->log._other_size, quint64(26));
QCOMPARE(Utility::qDateTimeFromTime_t(c1->log._other_modtime), changedMtime);
});
QVERIFY(fakeFolder.syncOnce());
}
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
};
QTEST_GUILESS_MAIN(TestSyncEngine)
#include "testsyncengine.moc"