* Save full Firefox profile
Save the whole Firefox profile directory instead of only saving a few of
its subcomponents. Remove an unused import of shutil from
profile_commands.py.
Additionally, remove the `extension_port.txt` file after reading the
port from it, to prevent reading stale port information when a browser
is restarted after a crash.
Finally, remove a part of the documentation that references the old way
of dumping the profile and update a leftover reference to the
`log_directory` config option.
Closes#62.
* Test saving full profile
Add a test that checks that attempting to save an incomplete profile
raises an error. Also, extend `test_saving` to check that a few basic
files and directories of the Firefox profile are present in the archived
profile.
* Combined log_directory and log_file to log_path
* Updated documentation
* Fixed tests
* Implemented test, need to change CSP
* Extension logging restored and tested
* Renamed extra to custom_params
* Reverting stackdump changes
* Re-enable test_profile_saved_when_launch_crashes
Update `test_profile_saved_when_launch_crashes` so that it does not
depend on the no longer supported proxy to make browser restarts fail.
Instead, set the `FIREFOX_BINARY` environment variable to point to a
wrong path.
Also, fix a bug in `kill_browser_manager()`, which would cause OpenWPM
to crash because of a `psutil.NoSuchProcess` exception and without
archiving the browser profile, whenever a browser restart failed to
launch geckodriver.
Finally, make `kill_process_and_children()` use the timeout set via its
arguments, which it previously ignored.
* Update docstring of dump_profile
Add a note for callers that they should make sure Firefox is closed, in
order to prevent ending up with a corrupted database in the archived
profile.
* Update test_browser_profile_coverage
Remove the buggy and outdated for loop that determined whether a url is
expected to be missing from the places.sqlite database of the browser
profile, as we have not observed any missing urls when running this
test.
* Fix documentation module index
Populate the module index by setting up Sphinx to automatically run
sphinx-apidoc for every build. Also, move readthedocs dependencies under
docs/ and make prune-environment.py automatically generate the
environment-rtd.yaml file whenever we run repin.sh.
* Fix black and mypy errors
We can now generate documentation to a variety of display formats including HTML by using sphinx.
With this new infrastructure we are now also able to generate documentation on readthedocs.io.
Co-authored-by: jhabarsingh <jhabarsinghbhati23@gmail.com>
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: ankushduacodes <61025943+ankushduacodes@users.noreply.github.com>
Co-authored-by: Mollie Bakal <bakalm@umich.edu>
Co-authored-by: MollieBakal <molliebakal@gmail.com>
Co-authored-by: jhabarsingh <43932986+jhabarsingh@users.noreply.github.com>
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Removing localtest.me
As it has been highly unreliable when running
local tests (returning DnsNotFound errors)
* Fixing tests
* Switched to localhost
* Localtest.me to localhost
* Renamed Browser to BrowserManagerHandler
* Renamed TaskManager._issue_command to BrowserManagerHandle.execute_command_sequence
* Fixing stuff
* Apply suggestions from code review
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* tm to task_manager
* Found and renamed only mention of in the docs
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
Introduced `cleaned_js_instrument_settings` in BrowserParamsInternal to hold the expanded config dict.
Propagating the `js_instrument_settings` through the extension as an object for as long as possible.
webdriver.switch_to.alert unlike most other variants of the switch_to API is not a function but a property.
This led to TypeError:'Alert' object is not callable when there was actually an Alert to switch to.
This PR fixes that behaviour.
Make `PatchedGeckoDriverService` class subclass
selenium.webdriver.firefox.service.Service instead of
selenium.webdriver.common.service.Service, so that we only have to keep
track of the changes in the `__init__()` method of the former class.
Use the public suffix + 1 instead of the public suffix when comparing
the domains in the crawl database with those in the profile history.
Also, update an incorrectly formed query to the crawl database.
Move the core implementation of profile dumping into a `dump_profile`
function, which can be used both internally when closing or restarting a
crashed browser and from the `execute()` method of `DumpProfileCommand`.
Also, make compression the default in `DumpProfileCommand`. Finally, do
not compress the tar archive of the crashed browser's profile when
restarting from a crash. We should avoid the extra compression/
decompression step as this is a short-lived tar file.
* Release preparations
* Apply suggestions from code review
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Bumped typedoc version number
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
1. In `deploy_firefox` do not use `driver.capabilities["moz:profile"]`
to get the profile location. Custom profiles, unlike profiles created
via `FirefoxProfile`, are used in-place, so we already know the
location.
2. In `launch_browser_manager`, `spawned_profile_path` and
`driver_profile_path` point to the same location now that we are using a
custom profile. Replace them with a single `browser_profile_path`
variable.
3. Rename `prof_folder` and `browser_profile_folder` to
`browser_profile_path` for consistency.
4. Improve naming of the temporary Firefox profile.
Geckodriver has a bug that makes it write the browser preferences we
set, as well as its own default browser preferences, to a user.js file
in the wrong profile directory when using a custom profile:
https://github.com/mozilla/geckodriver/issues/1844. As a temporary
workaround until this issue gets fixed, we create the user.js file
ourselves. In order to do this, we keep a copy of geckodriver's default
preferences in our code.
Closes#423
Fix a bug in PatchedGeckoDriverService that caused geckodriver not to
receive the service_args passed when starting the browser.
PatchedGeckoDriverService is a modified version of Selenium's Service
class and this bug has been fixed in the original version.
Use a custom profile by setting it as an argument via the Options class,
instead of using the FirefoxProfile class. This way geckodriver does not
delete it when crashing or closing. Also, remove some unused arguments
from the function that configures privacy settings in Firefox. Finally,
remove the code that clears driver.profile before calling driver.quit(),
as driver.profile is always None when using a custom profile.
Reenable stateful crawling and profile tests. Also, update the docs now
that stateful crawling is supported. Currently, stateful crawling is
broken, as geckodriver deletes the browser profile when closing or
crashing before we can archive it.
* First steps in the rewrite
* Fixed import paths
* One giant refactor
* Fixing tests
* Adding mypy
* Removed mypy from pre-commit workflow
* First draft on DataAggregator
* Wrote a DataAggregator that starts and shuts down
* Created tests and added more empty types
* Got demo.py working
* Created sql_provider
* Cleaned up imports in TaskManager
* Added async
* Fixed minor bugs
* First steps at porting arrow
* Introduced TableName and different Task handling
* Added more failing tests
* First first completes others don't
* It works
* Started working on arrow_provider
* Implemented ArrowProvider
* Added logger fixture
* Fixed test_storage_controller
* Fixing OpenWPMTest.visit()
* Moved test/storage_providers to test/storage
* Fixing up tests
* Moved automation to openwpm
* Readded datadir to .gitignore
* Ran repin.sh
* Fixed formatting
* Let's see if this works
* Fixed imports
* Got arrow_memory_provider working
* Starting to rewrite tests
* Setting up fixtures
* Attempting to fix all the tests
* Still fixing tests
* Broken content saving
* Added node
* Fixed screenshot tests
* Fixing more tests
* Fixed tests
* Implemented local_storage.py
* Cleaned up flush_cache
* Fixing more tests
* Wrote test for LocalArrowProvider
* Introduced tests for local_storage_provider.py
* Asserting test dir is empty
* Creating subfolder for different aggregators
* New depencies and init()
* Everything is terribly broken
* Figured out finalize_visit_id
* Running two event loops kinda works???
* Rearming the event
* Introduced mypy
* Downgraded black in pre-commit
* Modifying the database directly
* Fixed formatting
* Made mypy a lil stricter
* Fixing docs and config printing
* Realising I've been using the wrong with
* Trying to figure arrow_storage
* Moving lock initialization in in_memory_storage
* Fixing tests
* Fixing up tests and adding more typechecking
* Fixed num_browsers in test_cache_hits_recorded
* Parametrized unstructured
* String fix
* Added failing test
* New test
* Review changes with Steven
* Fixed repin.sh and test_arrow_cache
* Minor change
* Fixed prune-environment.py
* Removing references to DataAggregator
* Fixed test_seed_persistance
* More paths
* Fixed test display shutdown
* Made cache test more robust
* Update crawler.py
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Slimming down ManagerParams
* Fixing more tests
* Update test/storage/test_storage_controller.py
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Purging references to DataAggregator
* Reverted changes to .travis.yml
* Demo.py saves locally again
* Readjusting test paths
* Expanded comment on initialize to reference #846
* Made token optional in finalize_visit_id
* Simplified test paramtetrization
* Fixed callback semantics change
* Removed test_parse_http_stack_trace_str
* Added DataSocket
* WIP need to fix path encoding
* Fixed path encoding
* Added task and crawl to schema
* Fixed paths in GitHub actions
* Refactored completion handling
* Fix tests
* Trying to fix tests on CI
* Removed redundant setting of tag
* Removing references to S3
* Purging more DataAggregator references
* Craking up logging to figure out test failure
* Moved test_values into a fixture
* Fixing GcpUnstructuredProvider
* Fixed paths for future crawls
* Renamed sqllite to official sqlite
* Restored demo.py
* Update openwpm/commands/profile_commands.py
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Restored previous behaviour of DumpProfileCommand
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Removed leftovers
* Cleaned up comments
* Expanded lock check
* Fixed more stuff
* More comment updates
* Update openwpm/socket_interface.py
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
* Removed outdated comment
* Using config_encoder
* Renamed tar_location to tar_path
* Removed references to database_name in docs
* Cleanup
* Moved screenshot_path and source_dump_path to ManagerParamsInternal
* Fixed imports
* Fixing up comments
* Fixing up comments
* More docs
* updated dependencies
* Fixed test_task_manager
* Reupgraded to python 3.9.1
* Restoring crawl_reference in mp_logger
* Removed unused imports
* Apply suggestions from code review
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
* Cleaned up socket handling
* Fixed TaskManager.__exit__
* Moved validation code into config.py
* Removed comment
* Removed comment
* Removed comment
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
Georgia Kokkinou
Stefan Zabka
Ayush Anand
Steven Englehardt