Граф коммитов

69 Коммитов

Автор SHA1 Сообщение Дата
vringar 3005dce08e refactor(Extension): move from Extension/firefox to extension 2022-09-21 21:07:42 +02:00
Stefan Zabka 45e86424a8
Chore/release prep (#984)
* chore(updates): update all dependencies

* fix(Dockerfile): install Firefox instead of its dependencies

This change protects us against Firefox changing its dependencies

* fix(typing): minor typing enhancements

* chore(release): update VERSION and write CHANGELOG
2022-03-31 14:59:42 +02:00
Stefan Zabka 4966dae212
chore(release): v0.19.0 release prep (#976)
* chore(firefox-bin): update to FF98

* chore(pre-commit): update and run pre-commit hooks

* chore(conda): update environment.yaml

* chore(javascript): update all package.jsons

* chore(Release): update VERSION and CHANGELOG

* chore(conda): remove mamba

* chore(conda): added easyprocess dependency
2022-03-23 18:32:18 +01:00
Stefan Zabka 2cb663718f
chore(selenium): upgrade to selenium 4.1 (#977) 2022-03-22 09:54:42 +01:00
Georgia Kokkinou 46131a180d
Revert workaround of manually creating user.js (#968)
* Update extension path in manual_test.py

* Revert workaround of manually creating user.js

Go back to letting geckodriver handle the creation of the user.js file
now that https://github.com/mozilla/geckodriver/issues/1844 has been
fixed and the fix has been included in geckodriver v0.30.0. Remove the
step about updating geckodriver default preferences from the Release
Checklist.

Closes #867
2022-01-16 17:17:04 +01:00
Stefan Zabka 771b6db416
chore: removed invalid resource type (#966)
Closes #702
2021-12-22 08:36:51 +01:00
Steven Englehardt 8b62f18749
Remove references to Mozilla (#965) 2021-12-20 10:47:32 -05:00
Stefan Zabka b95e9d2975
Stop AssertionErrors crashing production crawls (#945)
* Stop AssertionErrors crashing production crawls

Fixes #166

* Wrote tests for propagating exceptions

* Logs were too noisy

* Test_crawl should run like a real crawl
2021-07-16 18:32:51 +02:00
Stefan Zabka 9b4d3582bc
Moved Extension folder to the top-level (#939)
* Moved Extension folder to the top-level

Also removed useless number tests

* Fixed conftests xpi fixture

* Updated docs
2021-07-16 16:15:22 +02:00
dependabot[bot] 96f94c7756
Bump css-what from 5.0.0 to 5.0.1 in /openwpm/Extension/firefox (#935)
Bumps [css-what](https://github.com/fb55/css-what) from 5.0.0 to 5.0.1.
- [Release notes](https://github.com/fb55/css-what/releases)
- [Commits](https://github.com/fb55/css-what/compare/v5.0.0...v5.0.1)

---
updated-dependencies:
- dependency-name: css-what
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-10 12:22:36 +02:00
Stefan Zabka ddd6f0be0a
V0.16.0 release preparations (#934)
* V0.16.0 release preparations

* Added changelog
2021-06-10 10:13:26 +02:00
Stefan Zabka 3ff9bf6c4e
Introduced update.sh (#931) 2021-05-25 17:55:02 +02:00
Stefan Zabka 2cb831fac5
Reenabling extension tests (#929)
* Fixed and enabled extension tests

* Narrowed types for display_mode

* Updated package-lock.json

* Removed storage.rst
2021-05-24 13:05:41 +02:00
Stefan Zabka 613e7d6633
Fixed default extension config (#927)
* Fixed default extension config

* Updated dependencies
2021-05-24 12:05:40 +02:00
Georgia Kokkinou c7849bca2f
Expand profile tests (#924)
* Test browser profile recovery in various scenarios

Add a test that implements the following test matrix and checks that the
profile behavior is the expected in each case:

  * Crash types: normal operation, crash during page visit,
                 crash during launch, timeout during page visit
  * Operational mode: stateful, stateless
  * Seed profile: exists, doesn't exist

* Move profile tests to a separate CI job

* Test saving the browser profile using a command

* Remove unused manager_params arg from load_profile

* Test that load_profile does not modify tar file

* Test crashing during Task Manager initialization

* Remove some redundant testcases

Adjust the parameterization of test_profile_recovery in order to remove
the `stateless-without_seed_tar` testcases, as they are not adding any
value.
2021-05-19 12:17:19 +02:00
Stefan Zabka 6a40b9460b
Preparations for v0.15.0 release (#913)
* Preparations for v0.15.0 release

* removed warning about v0.14.0 and v0.14.1

* Comment formatting

* Fixed date in Changelog

As well as formatting and spelling

Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>

* Update openwpm/task_manager.py

Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>

* Updated VERSION file

* Getting GCSFS to work again

* Set black to the version

* Updated JS dependecies

Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
2021-05-10 18:48:47 +02:00
Ayush Anand 7e126af0b0
Move config documentation into docstrings (#920) 2021-05-10 18:01:23 +02:00
Georgia Kokkinou 889b54d429
Save full Firefox profile (#917)
* Save full Firefox profile

Save the whole Firefox profile directory instead of only saving a few of
its subcomponents. Remove an unused import of shutil from
profile_commands.py.

Additionally, remove the `extension_port.txt` file after reading the
port from it, to prevent reading stale port information when a browser
is restarted after a crash.

Finally, remove a part of the documentation that references the old way
of dumping the profile and update a leftover reference to the
`log_directory` config option.

Closes #62.

* Test saving full profile

Add a test that checks that attempting to save an incomplete profile
raises an error. Also, extend `test_saving` to check that a few basic
files and directories of the Firefox profile are present in the archived
profile.
2021-05-07 13:33:17 +02:00
Stefan Zabka dafc26cb56
Blocking on startup (#902)
* Refactored BrowserManager into class

* Completing refactor

* Improved type annotations

* Blocking on OPENWPM_STARTUP_SUCCESS.txt

* Cherrypicked changes from logging_extension_test

* Intermediate

* Implemented test

* Refactoring shutdown control flow

* Refactored status_queue check in close_browser_manager

* Updated log message

* Update openwpm/browser_manager.py

Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>

* Guarded property access

* Fixed timeout code

* Fixing up docstrings

Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
2021-05-03 12:57:56 +02:00
Stefan Zabka 9943a218ca
Fixing extension logging (#912)
* Combined log_directory and log_file to log_path

* Updated documentation

* Fixed tests

* Implemented test, need to change CSP

* Extension logging restored and tested

* Renamed extra to custom_params

* Reverting stackdump changes
2021-04-30 19:38:08 +02:00
Stefan Zabka 05e5dcb0a5
Combined log_directory and log_file to log_path (#911)
* Combined log_directory and log_file to log_path

* Updated documentation

* Fixed tests
2021-04-30 16:52:22 +02:00
Stefan Zabka 262e4f2847
Browser manager as class (#901)
* Refactored BrowserManager into class

* Completing refactor

* Improved type annotations
2021-04-28 14:45:36 +02:00
Georgia Kokkinou 3619b55682
Update profile tests (#893)
* Re-enable test_profile_saved_when_launch_crashes

Update `test_profile_saved_when_launch_crashes` so that it does not
depend on the no longer supported proxy to make browser restarts fail.
Instead, set the `FIREFOX_BINARY` environment variable to point to a
wrong path.

Also, fix a bug in `kill_browser_manager()`, which would cause OpenWPM
to crash because of a `psutil.NoSuchProcess` exception and without
archiving the browser profile, whenever a browser restart failed to
launch geckodriver.

Finally, make `kill_process_and_children()` use the timeout set via its
arguments, which it previously ignored.

* Update docstring of dump_profile

Add a note for callers that they should make sure Firefox is closed, in
order to prevent ending up with a corrupted database in the archived
profile.

* Update test_browser_profile_coverage

Remove the buggy and outdated for loop that determined whether a url is
expected to be missing from the places.sqlite database of the browser
profile, as we have not observed any missing urls when running this
test.
2021-04-28 13:04:10 +02:00
Stefan Zabka a159496e22
Fixed default directories in ManagerParams (#903)
Our default was to create a literal folder called ~ instead of writing to the homedir
2021-04-27 16:29:15 +03:00
Stefan Zabka 6efeade359
Updated NPM dependencies (#888) 2021-04-20 15:30:42 +02:00
Stefan Zabka 625d81460b
Introducing Sphinx (#863)
We can now generate documentation to a variety of display formats including HTML by using sphinx.
With this new infrastructure we are now also able to generate documentation on readthedocs.io.

Co-authored-by: jhabarsingh <jhabarsinghbhati23@gmail.com>
Co-authored-by: Cyrus <cyruskarsan@gmail.com>
Co-authored-by: cyruskarsan <55566678+cyruskarsan@users.noreply.github.com>
Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
Co-authored-by: ankushduacodes <61025943+ankushduacodes@users.noreply.github.com>
Co-authored-by: Mollie Bakal <bakalm@umich.edu>
Co-authored-by: MollieBakal <molliebakal@gmail.com>
Co-authored-by: jhabarsingh <43932986+jhabarsingh@users.noreply.github.com>
Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
2021-04-15 12:04:55 +02:00
Stefan Zabka e72ed2339c
Removing localtest.me (#886)
* Removing localtest.me

As it has been highly unreliable when running
local tests (returning DnsNotFound errors)

* Fixing tests

* Switched to localhost

* Localtest.me to localhost
2021-04-13 17:39:03 +02:00
Stefan Zabka e20fc6a29e
Moved _issue_command to BrowserManagerHandle (#882)
* Renamed Browser to BrowserManagerHandler

* Renamed TaskManager._issue_command to BrowserManagerHandle.execute_command_sequence

* Fixing stuff

* Apply suggestions from code review

Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>

* tm to task_manager

* Found and renamed only mention of  in the docs

Co-authored-by: Georgia Kokkinou <geor5ko@gmail.com>
2021-04-12 16:22:59 +03:00
Georgia Kokkinou b8f6262cb2
Clean up some unused files (#881)
* Remove some unused files

* Fix minor typos
2021-04-06 14:26:49 +02:00
Stefan Zabka 8030c2c470
Remove python to js string (#879)
Introduced `cleaned_js_instrument_settings` in BrowserParamsInternal to hold the expanded config dict.
Propagating the `js_instrument_settings` through the extension as an object for as long as possible.
2021-03-30 14:22:02 +02:00
Steven Englehardt bfc4644a71
Merge pull request #864 from boolean5/restore-stateful-crawls
Restore stateful crawling support
2021-03-29 11:34:05 -04:00
Stefan Zabka 3241482608
Adjusted check for debug mode in loggingdb.js (#878)
This way the default config in index.js and the check in loggingdb.js
match up
2021-03-29 12:30:39 +03:00
Stefan Zabka 9a8dea34d9
Hotfixed GetCommand (#875)
webdriver.switch_to.alert unlike most other variants of the switch_to API is not a function but a property.
This led to TypeError:'Alert' object is not callable when there was actually an Alert to switch to.
This PR fixes that behaviour.
2021-03-23 22:38:09 +01:00
Georgia Kokkinou 37271ba62d
Remove unnecessary import 2021-03-22 19:22:17 +02:00
Georgia Kokkinou 9a21d86e8f
Simplify PatchedGeckoDriverService class
Make `PatchedGeckoDriverService` class subclass
selenium.webdriver.firefox.service.Service instead of
selenium.webdriver.common.service.Service, so that we only have to keep
track of the changes in the `__init__()` method of the former class.
2021-03-22 15:57:01 +02:00
Georgia Kokkinou 06b83596ab
Do not copy tar before extracting in load_profile 2021-03-21 23:59:51 +02:00
Georgia Kokkinou e536c630cc
Add comment for Marionette port race condition 2021-03-21 23:59:51 +02:00
Georgia Kokkinou c51f9e56bf
Fix minor typos 2021-03-21 23:59:18 +02:00
Georgia Kokkinou 3b4219d0f9
Add some type annotations 2021-03-19 21:09:20 +02:00
Steven Englehardt ab01a2f6bd
Remove commented out code; monitor speculative connections (#872) 2021-03-19 11:50:21 +01:00
Georgia Kokkinou 1e16513370
Improve profile dumping logic
Move the core implementation of profile dumping into a `dump_profile`
function, which can be used both internally when closing or restarting a
crashed browser and from the `execute()` method of `DumpProfileCommand`.
Also, make compression the default in `DumpProfileCommand`. Finally, do
not compress the tar archive of the crashed browser's profile when
restarting from a crash. We should avoid the extra compression/
decompression step as this is a short-lived tar file.
2021-03-16 17:10:50 +02:00
Stefan Zabka b6849c71f5
Release preparations (#866)
* Release preparations

* Apply suggestions from code review

Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>

* Bumped typedoc version number

Co-authored-by: Steven Englehardt <senglehardt@mozilla.com>
2021-03-16 15:23:05 +01:00
Georgia Kokkinou 9ea8e8a051
Rename temp dir of crashed browser's profile tar 2021-03-15 15:37:12 +02:00
Georgia Kokkinou 1d3de72292
Reference our own issue instead of geckodriver's 2021-03-15 12:23:06 +02:00
Georgia Kokkinou d2aff836f4
Remove unused status string "Proxy Ready" 2021-03-15 11:09:44 +02:00
Georgia Kokkinou 403185a38a
Simplify profile location handling
1. In `deploy_firefox` do not use `driver.capabilities["moz:profile"]`
to get the profile location. Custom profiles, unlike profiles created
via `FirefoxProfile`, are used in-place, so we already know the
location.

2. In `launch_browser_manager`, `spawned_profile_path` and
`driver_profile_path` point to the same location now that we are using a
custom profile. Replace them with a single `browser_profile_path`
variable.

3. Rename `prof_folder` and `browser_profile_folder` to
`browser_profile_path` for consistency.

4. Improve naming of the temporary Firefox profile.
2021-03-15 11:03:10 +02:00
Georgia Kokkinou 2237822eab
Do not intercept profile location from logs 2021-03-08 09:58:48 +02:00
Georgia Kokkinou a355dc840d
Create user.js manually in custom profile
Geckodriver has a bug that makes it write the browser preferences we
set, as well as its own default browser preferences, to a user.js file
in the wrong profile directory when using a custom profile:
https://github.com/mozilla/geckodriver/issues/1844. As a temporary
workaround until this issue gets fixed, we create the user.js file
ourselves. In order to do this, we keep a copy of geckodriver's default
preferences in our code.

Closes #423
2021-03-03 13:16:15 +02:00
Georgia Kokkinou 7f51e50f44
Pass service_args to geckodriver
Fix a bug in PatchedGeckoDriverService that caused geckodriver not to
receive the service_args passed when starting the browser.
PatchedGeckoDriverService is a modified version of Selenium's Service
class and this bug has been fixed in the original version.
2021-03-03 13:16:15 +02:00
Georgia Kokkinou 51c0849cbd
Use custom browser profile
Use a custom profile by setting it as an argument via the Options class,
instead of using the FirefoxProfile class. This way geckodriver does not
delete it when crashing or closing. Also, remove some unused arguments
from the function that configures privacy settings in Firefox. Finally,
remove the code that clears driver.profile before calling driver.quit(),
as driver.profile is always None when using a custom profile.
2021-03-03 13:15:56 +02:00