Move the core implementation of profile dumping into a `dump_profile`
function, which can be used both internally when closing or restarting a
crashed browser and from the `execute()` method of `DumpProfileCommand`.
Also, make compression the default in `DumpProfileCommand`. Finally, do
not compress the tar archive of the crashed browser's profile when
restarting from a crash. We should avoid the extra compression/
decompression step as this is a short-lived tar file.
1. In `deploy_firefox` do not use `driver.capabilities["moz:profile"]`
to get the profile location. Custom profiles, unlike profiles created
via `FirefoxProfile`, are used in-place, so we already know the
location.
2. In `launch_browser_manager`, `spawned_profile_path` and
`driver_profile_path` point to the same location now that we are using a
custom profile. Replace them with a single `browser_profile_path`
variable.
3. Rename `prof_folder` and `browser_profile_folder` to
`browser_profile_path` for consistency.
4. Improve naming of the temporary Firefox profile.
Geckodriver has a bug that makes it write the browser preferences we
set, as well as its own default browser preferences, to a user.js file
in the wrong profile directory when using a custom profile:
https://github.com/mozilla/geckodriver/issues/1844. As a temporary
workaround until this issue gets fixed, we create the user.js file
ourselves. In order to do this, we keep a copy of geckodriver's default
preferences in our code.
Closes#423
Fix a bug in PatchedGeckoDriverService that caused geckodriver not to
receive the service_args passed when starting the browser.
PatchedGeckoDriverService is a modified version of Selenium's Service
class and this bug has been fixed in the original version.
Use a custom profile by setting it as an argument via the Options class,
instead of using the FirefoxProfile class. This way geckodriver does not
delete it when crashing or closing. Also, remove some unused arguments
from the function that configures privacy settings in Firefox. Finally,
remove the code that clears driver.profile before calling driver.quit(),
as driver.profile is always None when using a custom profile.
Reenable stateful crawling and profile tests. Also, update the docs now
that stateful crawling is supported. Currently, stateful crawling is
broken, as geckodriver deletes the browser profile when closing or
crashing before we can archive it.
* First steps in the rewrite
* Fixed import paths
* One giant refactor
* Fixing tests
* Adding mypy
* Removed mypy from pre-commit workflow
* First draft on DataAggregator
* Wrote a DataAggregator that starts and shuts down
* Created tests and added more empty types
* Got demo.py working
* Created sql_provider
* Cleaned up imports in TaskManager
* Added async
* Fixed minor bugs
* First steps at porting arrow
* Introduced TableName and different Task handling
* Added more failing tests
* First first completes others don't
* It works
* Started working on arrow_provider
* Implemented ArrowProvider
* Added logger fixture
* Fixed test_storage_controller
* Fixing OpenWPMTest.visit()
* Moved test/storage_providers to test/storage
* Fixing up tests
* Moved automation to openwpm
* Readded datadir to .gitignore
* Ran repin.sh
* Fixed formatting
* Let's see if this works
* Fixed imports
* Got arrow_memory_provider working
* Starting to rewrite tests
* Setting up fixtures
* Attempting to fix all the tests
* Still fixing tests
* Broken content saving
* Added node
* Fixed screenshot tests
* Fixing more tests
* Fixed tests
* Implemented local_storage.py
* Cleaned up flush_cache
* Fixing more tests
* Wrote test for LocalArrowProvider
* Introduced tests for local_storage_provider.py
* Asserting test dir is empty
* Creating subfolder for different aggregators
* New depencies and init()
* Everything is terribly broken
* Figured out finalize_visit_id
* Running two event loops kinda works???
* Rearming the event
* Introduced mypy
* Downgraded black in pre-commit
* Modifying the database directly
* Fixed formatting
* Made mypy a lil stricter
* Fixing docs and config printing
* Realising I've been using the wrong with
* Trying to figure arrow_storage
* Moving lock initialization in in_memory_storage
* Fixing tests
* Fixing up tests and adding more typechecking
* Fixed num_browsers in test_cache_hits_recorded
* Parametrized unstructured
* String fix
* Added failing test
* New test
* Review changes with Steven
* Fixed repin.sh and test_arrow_cache
* Minor change
* Fixed prune-environment.py
* Removing references to DataAggregator
* Fixed test_seed_persistance
* More paths
* Fixed test display shutdown
* Made cache test more robust
* Update crawler.py
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Slimming down ManagerParams
* Fixing more tests
* Update test/storage/test_storage_controller.py
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Purging references to DataAggregator
* Reverted changes to .travis.yml
* Demo.py saves locally again
* Readjusting test paths
* Expanded comment on initialize to reference #846
* Made token optional in finalize_visit_id
* Simplified test paramtetrization
* Fixed callback semantics change
* Removed test_parse_http_stack_trace_str
* Added DataSocket
* WIP need to fix path encoding
* Fixed path encoding
* Added task and crawl to schema
* Fixed paths in GitHub actions
* Refactored completion handling
* Fix tests
* Trying to fix tests on CI
* Removed redundant setting of tag
* Removing references to S3
* Purging more DataAggregator references
* Craking up logging to figure out test failure
* Moved test_values into a fixture
* Fixing GcpUnstructuredProvider
* Fixed paths for future crawls
* Renamed sqllite to official sqlite
* Restored demo.py
* Update openwpm/commands/profile_commands.py
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Restored previous behaviour of DumpProfileCommand
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Removed leftovers
* Cleaned up comments
* Expanded lock check
* Fixed more stuff
* More comment updates
* Update openwpm/socket_interface.py
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Removed outdated comment
* Using config_encoder
* Renamed tar_location to tar_path
* Removed references to database_name in docs
* Cleanup
* Moved screenshot_path and source_dump_path to ManagerParamsInternal
* Fixed imports
* Fixing up comments
* Fixing up comments
* More docs
* updated dependencies
* Fixed test_task_manager
* Reupgraded to python 3.9.1
* Restoring crawl_reference in mp_logger
* Removed unused imports
* Apply suggestions from code review
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Cleaned up socket handling
* Fixed TaskManager.__exit__
* Moved validation code into config.py
* Removed comment
* Removed comment
* Removed comment
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Set failure_limit default in config.py
Set the default value for failure_limit in config.py instead of setting
it in the TaskManager. Also, accept 0 as a valid value for
failure_limit.
* Rename failurecount to failure_count
* Clear failure_count on successful command sequence
Reset failure_count to 0 only at the end of each successfully completed
command sequence. Before it was reset after each successful command.
This would result in failures of subsequent commands that belonged to
different command sequences not triggering a CommandExecutionError
because failure_count was reset upon every InitializeCommand. Also,
update the docs to reflect the current behavior of failure_limit.
Closes#851
* Enable test_crash
* Fix some minor typos
* Test failure limit behavior
Move `test_crash` out of test_profile.py, as it does not depend on
profile saving support and rename it to `test_failure_limit_exceeded`,
which is more descriptive. Also, add two more tests to cover more
aspects of failure limit behavior.
* Use local test server in TaskManager tests
* Clarify failure_limit behavior in docs
Mention that the CommandExecutionError gets raised by the next command
sequence after failure_limit has been exceeded.
* Add type annotations for failure_limit property
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* Ported SaveScreenshotFullPage #763
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove custom function command and format code
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove duplicate append_command
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* generate new xpi
* Fixing tests
* Fixing tests
* Fixing up more tests
* Removed type annotations
* Fixing tests
* Fixing tests
* Removed command_executor
* Moved Commands to commands
* Fixing imports
* Fixed skipped test
* Removed duplicate append_command
* manual test breaking post-#807
* replacing with standard json function for single string
* modifying test to see if stops running for over two hours:
* docs: update adding command in usingOpenWPM
* Forgot to save
* Removed datadir
* Reverting error, updating function to current name
* Attempted fix treats display as object
* changing display; pytest unchanged
* added a test case to ensure xvfb runs
* simplifying test
* manual test breaking post-#807
* replacing with standard json function for single string
* modifying test to see if stops running for over two hours:
* Deleted redundant files and references to default_browser_params.json and default_manager_params.json (#822)
* Deleted redundant files and references to -
default_browser_params.json and default_manager_params.json
* added validate_crawl_configs to docs
* updated the docs on how to instantiate TaskManager
* Updated Using_OpenWPM.md
* Updated Platform-Architecture.md
* Attempted fix treats display as object
* changing display; pytest unchanged
* added a test case to ensure xvfb runs
* simplifying test
* fixing requested issues
* runs; modified test to include exception
* fix import
* Queue not optional; removed
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* updated test to check for lockfile
* Update test/test_xvfb_browser.py
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* sticky key in the last commit, same change
* Command refactoring (#750)
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* Ported SaveScreenshotFullPage #763
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove custom function command and format code
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove duplicate append_command
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* generate new xpi
* Fixing tests
* Fixing tests
* Fixing up more tests
* Removed type annotations
* Fixing tests
* Fixing tests
* Removed command_executor
* Moved Commands to commands
* Fixing imports
* Fixed skipped test
* Removed duplicate append_command
* docs: update adding command in usingOpenWPM
* Forgot to save
* Removed datadir
* Cleaning up imports
* Implemented simple command
* Added documentation to simple_command.py
* Renamed to custom_command.py
* Moved docs around
* Referencing BaseCommand.execute
* Update docs/Using_OpenWPM.md
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Changed finally condition
* Restoring manual_test.py
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: Mollie Bakal <bakalm@umich.edu>
Co-authored-by: MollieBakal <molliebakal@gmail.com>
Co-authored-by: Ankush Dua <61025943+ankushduacodes@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* Ported SaveScreenshotFullPage #763
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove custom function command and format code
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove duplicate append_command
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* generate new xpi
* Fixing tests
* Fixing tests
* Fixing up more tests
* Removed type annotations
* Fixing tests
* Fixing tests
* Removed command_executor
* Moved Commands to commands
* Fixing imports
* Fixed skipped test
* Removed duplicate append_command
* docs: update adding command in usingOpenWPM
* Forgot to save
* Removed datadir
* removed use of depricated methods
* Command refactoring (#750)
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* Ported SaveScreenshotFullPage #763
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove custom function command and format code
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove duplicate append_command
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* generate new xpi
* Fixing tests
* Fixing tests
* Fixing up more tests
* Removed type annotations
* Fixing tests
* Fixing tests
* Removed command_executor
* Moved Commands to commands
* Fixing imports
* Fixed skipped test
* Removed duplicate append_command
* docs: update adding command in usingOpenWPM
* Forgot to save
* Removed datadir
* Cleaning up imports
* Implemented simple command
* Added documentation to simple_command.py
* Renamed to custom_command.py
* Moved docs around
* Referencing BaseCommand.execute
* Update docs/Using_OpenWPM.md
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: ankushduacodes <61025943+ankushduacodes@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* Ported SaveScreenshotFullPage #763
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove custom function command and format code
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* remove duplicate append_command
* Refactored GetCommand, BrowseCommand to have execute method
* Fixed type name format issues in __issue_command
* Fixed everything I broke
* Changed import style so tests can run
* Added BrowseCommad to imports
* Added some more self
* Added logging to explain failing test
* Added one more self
* Ported SaveScreenshotCommand
It now uses the new command.execute(...) syntax
* Ported SaveScreenshotFullPage #763
* Ported DumpPageSource and RecursiveDumpPageSource (#767)
* Command refactoring (#770)
* attempt at refactoring save_screenshot
* fixed indentation, attempt at refactoring save_screenshot
* refactored SaveScreenshot command to have execute method
* reformatted code using black
* refactored savefullscreenshot command to follow command sequence
* formatted files with black
* removed extraneous commands
* refactored dump page source and formatted code with black
* reformatted recursive dump page source command and formatted code w black
* formatted files using isort
* formatted all files with isort
* refactor finalize command
* refactored initalize command and formatted with black and isort
* missed a conflict
* Ran isort
* Added append_command
* generate new xpi
* Fixing tests
* Fixing tests
* Fixing up more tests
* Removed type annotations
* Fixing tests
* Fixing tests
* Removed command_executor
* Moved Commands to commands
* Fixing imports
* Fixed skipped test
* Removed duplicate append_command
* docs: update adding command in usingOpenWPM
* Forgot to save
* Removed datadir
* Cleaning up imports
* Implemented simple command
* Added documentation to simple_command.py
* Renamed to custom_command.py
* Moved docs around
* Referencing BaseCommand.execute
* Update docs/Using_OpenWPM.md
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Deleted redundant files and references to -
default_browser_params.json and default_manager_params.json
* added validate_crawl_configs to docs
* updated the docs on how to instantiate TaskManager
* Updated Using_OpenWPM.md
* Updated Platform-Architecture.md
* initial file commit
* add new dependency for dataclasses
* implemeted basic BrowserParams dataclass
* dependencies update
* file reformat
* implemented basic ManagerParams dataclass
* Update environment dependencies
* Added new error class to validate
browser and manager params
* file reformat
* Update scripts/environment-unpinned.yaml
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* added validations for BrowserParams dataclass
* Update openwpm/config.py
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Removed unnecessary checks
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Changed error string formatting
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Changed filenamea and necessary imports to resolve conflicts with new master branch(refering to PEP-8 reformatting)
* Revert "Changed filenamea and necessary imports to resolve conflicts with new master branch(refering to PEP-8 reformatting)"
This reverts commit e550c3bd60.
* Revert "Merge branch 'master' into turn_browser_and_manager_params_into_dataclasses"
This reverts commit aff5a384e7, reversing
changes made to 6ecaf5d0a9.
* Revert "Update environment dependencies"
This reverts commit 385825b10a.
* Revert "Merge branch 'turn_browser_and_manager_params_into_dataclasses' of https://github.com/ankushduacodes/OpenWPM into turn_browser_and_manager_params_into_dataclasses"
This reverts commit 6ecaf5d0a9, reversing
changes made to e550c3bd60.
* file reformat
* finalized validate_browser_params function
* fixed typo in error string
* added validations for manager_params
* Explanation for using list for supported browser
* Revert "Revert "Merge branch 'master' into turn_browser_and_manager_params_into_dataclasses""
This reverts commit 6c3e98e57b.
* Revert "Revert "Changed filenamea and necessary imports to resolve conflicts with new master branch(refering to PEP-8 reformatting)""
This reverts commit fc8f48f187.
* import name change from .Error to .error
* moved call_instrument check to config.py
* fixed accidental use of dict syntax in a class
* moved save_content check from deploy_firefox.py
* deleting redundent file
* deleted more redundent files
* removed redundant imports
* added new save_content check
* property name changevariables can not have '-'
* added new attribute to ManagerParams
* adapted files to validate manager & broswer params
- also added logic to convert the objects(BrowserParams and ManagerParams) to dictionaries to not break the functionality
- also updated demo.py to work with new file names on this branch
* removed obsolete documentaion
* Dependency Update
* Revert "Dependency Update"
This reverts commit 8ee3a02b17.
* Dependencies Update
* unset memory and process watchdogs
* add new output_format and failure_limit checks
* inheriting dataclasses and added type hints to fn
* added todo
* fixed inheritance of dataclasses acc. to plan
* refactor use of dict to use dataclasses(pending)
* more refactoring use of dict to dataclasses -
Also changed some type hints related to new refactoring
* fixed screenshot directory issue -
because of which some of the tests were failing
* added try-except clause for unexpected errors
* added tests to cover dataclasses
* added some new and edited some old docs
* refactor use of __dict__ to dataclass.to_dict()
* Revert "refactor use of __dict__ to dataclass.to_dict()"
This reverts commit a4f35513fa.
* fixed some tests
* refactor use of __dict__ in favor of
dataclass.to_dict() method
* removed some TODOS
* fixed dataclases validation tests
* Update docs/Configuration.md
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Update docs/Configuration.md
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Update docs/Configuration.md
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Update openwpm/config.py
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Update openwpm/config.py
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Update openwpm/task_manager.py
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* minor fixed wrt polishing the PR
* added new check and test for crawl configs
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Module & Imports conformed to PEP8
* Conformed tests to PEP8
* Conformed tests to PEP8 (2)
* Updated webdriver test for PEP8
* Updated test_timer for PEP8
* Deleting Workspace file
* renamed files to match PEP8
* Update docs/Using_OpenWPM.md
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Changed serversocket to ServerSocket
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* Preparing v0.13.0 release
* Made links work
* Rebuild extension
* Fixed extension build script
* Removed prepare step as it breaks fresh installs
* Changed openwpm import in tests to absolute
* Fixed test_timer.py
* Reverted changes to webext-instrumentation/package.json
* Pinning to node<15.0.0
* Rebuilding package-lock.json
* Fixed test
* Also mentioned LordReigns
* Fixed links to CONTRIBUTING.md
* Moved to FF83
* Added new duration column to crawl_history table
* adding timer to populate new duration column
* replaced .time() with .time_ns() to store duration
in nanoseconds
* Update Schema-Documenation with new duration col.
* documenting unit of duration being stored
* Update automation/TaskManager.py
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
* reformat file in accordance with pre-commit config
* add test -
to assert that new column for command duration is storing valid values
* making the test run locally without internet
Co-authored-by: Stefan Zabka <zabkaste@informatik.hu-berlin.de>
Georgia Kokkinou
Stefan Zabka
jhabarsingh
dependabot[bot]
Ankush Dua
MollieBakal
Fukurou Makoto
Nisan Abeywickrama
7brokenmirrors