OpenWPM/docs/Configuration.md

Browser and Platform Configuration

The browser and platform can be configured by two separate dictionaries. The platform configuration options can be set in manager_params, while the browser configuration options can be set in browser_params. The default settings are given in automation/default_manager_params.json and automation/default_browser_params.json.

To load the default configuration parameter dictionaries we provide a helper function TaskManager::load_default_params. For example:

from automation import TaskManager
manager_params, browser_params = TaskManager.load_default_params(num_browsers=5)

where manager_params is a dictionary and browser_params is a length 5 list of configuration dictionaries.

• Platform Configuration Options
• Browser Configuration Options
• Instruments
  • http_instrument
  • js_instrument
  • navigation_instrument
  • callstack_instrument
  • dns_instrument
  • cookie_instrument
• Browser Profile Support
  • Stateful vs Stateless crawls
  • Loading and saving a browser profile
    • Save a profile
    • Load a profile
• Non instrument data gathering
  • Log Files
  • Browser Profile
  • Rendered Page Source
  • Screenshots
  • save_content

Platform Configuration Options

• data_directory
  • The directory in which to output the crawl database and related files. The directory given will be created if it does not exist.
• log_directory
  • The directory in which to output platform logs. The directory given will be created if it does not exist.
• log_file
  • The name of the log file to be written to log_directory.
• database_name
  • The name of the database file to be written to data_directory
• failure_limit
  • The number of successive command failures the platform will tolerate before raising a CommandExecutionError exception. Otherwise the default is set to 2 x the number of browsers plus 10.
• testing
  • A platform wide flag that can be used to only run certain functionality while testing. For example, the Javascript instrumentation exposes its instrumentation function on the page script global to allow test scripts to instrument objects on-the-fly. Depending on where you would like to add test functionality, you may need to propagate the flag.
  • This is not something you should enable during normal crawls.
• memory_watchdog
  • A watchdog that tries to ensure that no Firefox instance takes up to much memory. It is set to false by default
  • It is mostly useful for long running cloud crawls

Browser Configuration Options

Note: Instrumentation configuration options are described in the Instrumentation and Data Access section and profile configuration options are described in the Browser Profile Support section. As such, these options are left out of this section.

• bot_mitigation
  • Performs some actions to prevent the platform from being detected as a bot.
  • Note, these aren't comprehensive and automated interaction with the site will still appear very bot-like.
• display_mode:
  • native:
    • Launch the browser normally - GUI will be visible
  • headless:
    • Launch the browser in headless mode (supported as of Firefox 56), no GUI will be visible.
    • Use this when running browsers on a remote machine or to run crawls in the background on a local machine.
  • xvfb:
    • Launch the browser using the X virtual frame buffer. In this mode, Firefox is not running in it's own headless mode, but no GUI will be displayed.
    • This mode requires Xvfb to be on your path. On Ubuntu that is achieved by running sudo apt-get install xvfb. For other platforms check www.X.org.
  • headless mode and xvfb are not equivalent. xvfb is a full browser, but you get "headless" browsing because you do not need to be in a full X environment e.g. on a server. headless mode is supported on all platforms and is implemented by the browser but has some differences. For example webGL is not supported in headless mode. https://github.com/mozilla/OpenWPM/issues/448 discusses additional factors to consider when picking a display_mode.
• browser
  • Used to specify which browser to launch. Currently only firefox is supported.
  • Other browsers may be added in the future.
• tp_cookies
  • Specifies the third-party cookie policy to set in Firefox.
  • The following options are supported:
    • always: Accept all third-party cookies
    • never: Never accept any third-party cookies
    • from_visited: Only accept third-party cookies from sites that have been visited as a first party.
• donottrack
  • Set to True to enable Do Not Track in the browser.
• tracking-protection
  • NOT SUPPORTED. See #101.
  • Set to True to enable Firefox's built-in Tracking Protection.

Instruments

Instruments are the core of the data collection infrastructure that OpenWPM provides. They allow to collect various types of data that is labeled per visit and aim to capture as much of a websites behaviour as we can.

If you feel that we are missing a fundamental instrument and are willing to implement it, please file an issue and we'll try to assist you in writing that instrument.

Below you'll find a description for every single instrument, however if you want to just look at the output schema look here

To activate a given instrument set browser_params[i][instrument_name] = True

http_instrument

• HTTP Request and Response Headers, redirects, and POST request bodies
• Data is saved to the http_requests, http_responses, and http_redirects tables.
  • http_requests schema documentation
  • channel_id can be used to link a request saved in the http_requests table to its corresponding response in the http_responses table.
  • channel_id can also be used to link a request to the subsequent request that results after an HTTP redirect (3XX response). Use the http_redirects table, which includes a mapping between old_channel_id, the channel_id of the HTTP request that resulted in a 3XX response, and new_channel_id, the HTTP request that resulted from that redirect. TODO: channel_ids now persist across redirects
• OCSP POST request bodies are not recorded
• Note: request and response headers for cached content are also saved, with the exception of images. See: Bug 634073.

js_instrument

• Records all method calls (with arguments) and property accesses for configured APIs
• Configure browser_params['js_instrument_settings'] to desired settings.
• Data is saved to the javascript table.
• The full specification for js_instrument_settings is defined by a JSON schema. Details of that schema are available in docs/schemas/README.md. In summary, a list is passed with JS objects to be instrumented and details about how that object should be instrumented. The js_instrument_settings you pass to browser_params will be validated python side against the JSON schema before the crawl starts running.
• A number of shortcuts are available to make writing js_instrument_settings less cumbersome than spelling out the full schema. These shortcuts are converted to a full specification by the clean_js_instrumentation_settings method in automation/js_instrumentation.py.
• The first shortcut is the fingerprinting collection, specified by collection_fingerprinting. This was the default prior to v0.11.0. It contains a collection of APIs of potential fingerprinting interest:
  • HTML5 Canvas
  • HTML5 WebRTC
  • HTML5 Audio
  • Plugin access (via navigator.plugins)
  • MIMEType access (via navigator.mimeTypes)
  • window.Storage, window.localStorage, window.sessionStorage, and window.name access.
  • Navigator properties (e.g. appCodeName, oscpu, userAgent, ...)
  • Window properties (via window.screen)
• collection_fingerprinting is the default if js_instrument is True.
• The fingerprinting collection is specified by the json file fingerprinting.json. This file is also a nice reference example for specifying your own APIs using the other shortcuts.
• Shortcuts:
  • Specifying just a string will instrument the whole API with the default log settings
  • For just strings you can specify a Web API such as XMLHttpRequest. Or you can specify instances on window e.g. window.document.
  • Alternatively, you can specify a single-key dictionary that maps an API name to the properties / settings you'd like to use. The key of this dictionary can be an instance on window or a Web API. The value of this dictionary can be:
    • A list - this is a shortcut for propertiesToInstrument (see log settings)
    • A dictionary - with non default log settings. Items missing from this dictionary will be filled in with the default log settings.
  • Here are some examples:

    // Collections
    "collection_fingerprinting",
    // APIs, with or without settings details
    "Storage",
    "XMLHttpRequest",
    {"XMLHttpRequest": {"excludedProperties": ["send"]}},
    // APIs with shortcut to includedProperties
    {"Prop1": ["hi"], "Prop2": ["hi2"]},
    {"XMLHttpRequest": ["send"]},
    // Specific instances on window
    {"window.document": ["cookie", "referrer"]},
    {"window": ["name", "localStorage", "sessionStorage"]}
    

  • Note, the key / string will only have it's properties instrumented. That is, if you want to instrument window.fetch function you must specify {"window": ["fetch",]}. If you specify just window.fetch the instrumentation will try to instrument sub properties of window.fetch (which won't work as fetch is a function). As another example, to instrument window.document.cookie, you must use {"window.document": ["cookie"]}. In instances, such as fetch, where you do not need to specify window.fetch, but can use the alias fetch, in JavaScript code. The instrumentation {"window": ["fetch",]} will pick up calls to both fetch() and window.fetch().

navigation_instrument

TODO

callstack_instrument

TODO

dns_instrument

TODO

cookie_instrument

• Data is saved to the javascript_cookies table.
• Will record cookies set both by Javascript and via HTTP Responses

Browser Profile Support

WARNING: Stateful crawls are currently not supported. Attempts to run stateful crawls will throw NotImplementedErrors. The work required to restore support is tracked in this project.

Stateful vs Stateless crawls

By default OpenWPM performs a "stateful" crawl, in that it keeps a consistent browser profile between page visits in the same browser. If the browser freezes or crashes during the crawl, the profile is saved to disk and restored before the next page visit.

It's also possible to run "stateless" crawls, in which each new page visit uses a fresh browser profile. To perform a stateless crawl you can restart the browser after each command sequence by setting the reset initialization argument to True when creating the command sequence. As an example:

manager = TaskManager.TaskManager(manager_params, browser_params)

for site in sites:
    command_sequence = CommandSequence.CommandSequence(site, reset=True)
    command_sequence.get(sleep=30, timeout=60)
    manager.execute_command_sequence(command_sequence)

In this example, the browser will get the requested site, sleep for 30 seconds, dump the profile cookies to the crawl database, and then restart the browser before visiting the next site in sites.

Loading and saving a browser profile

It's possible to load and save profiles during stateful crawls. Profile dumps currently consist of the following browser storage items:

• cookies
• localStorage
• IndexedDB
• browser history

Other browser state, such as the browser cache, is not saved. In Issue #62 we plan to expand profiles to include all browser storage.

Save a profile

A browser's profile can be saved to disk for use in later crawls. This can be done using a browser command or by setting a browser configuration parameter. For long running crawls we recommend saving the profile using the browser configuration parameter as the platform will take steps to save the profile in the event of a platform-level crash, whereas there is no guarantee the browser command will run before a crash.

Browser configuration parameter: Set the profile_archive_dir browser parameter to a directory where the browser profile should be saved. The profile will be automatically saved when TaskManager::close is called or when a platform-level crash occurs.

Load a profile

To load a profile, specify the seed_tar browser parameter in the browser configuration dictionary. This should point to the location of the profile.tar or (profile.tar.gz if compressed) file produced by OpenWPM or by manually tarring a firefox profile directory. The profile will be automatically extracted and loaded into the browser instance for which the configuration parameter was set.

The profile specified by seed_tar will be loaded anytime the browser is deliberately reset (i.e., using the reset=True CommandSequence argument), but will not be used during crash recovery. Specifically:

• For stateful crawls the initial load of Firefox will use the profile specified by seed_tar. If OpenWPM determines that Firefox needs to restart for some reason during the crawl, it will use the profile from the most recent page visit (pre-crash) rather than the seed_tar profile. Note that stateful crawl are currently unsupported).
• For stateless crawls, the initial seed_tar will be loaded during each new page visit. Note that this means the profile will very likely be incomplete, as cookies or storage may have been set or changed during the page load that are not reflected back into the seed profile.

Non instrument data gathering

Log Files

• Stored in the directory specified by manager_params['data_directory'].
• Name specified by manager_params['log_file'].

Browser Profile

• Contains cookies, Flash objects, and so on that are dumped after a crawl is finished
• Automatically saved when the platform closes or crashes by specifying browser_params['profile_archive_dir'].
• Save on-demand with the CommandSequence::dump_profile command.

Rendered Page Source

• Save the top-level frame's rendered source with the CommandSequence::dump_page_source command.
• Save the full rendered source (including all nested iframes) with the CommandSequence::recursive_dump_page_source command.
  • The page source is saved in the following nested json structure:

    {
        'doc_url': "http://example.com",
        'source': "<html> ... </html>",
        'iframes': {
            'frame_1': {'doc_url': ...,
                        'source': ...,
                        'iframes: { ... }},
            'frame_2': {'doc_url': ...,
                        'source': ...,
                        'iframes: { ... }},
            'frame_3': { ... }
        }
    }
    

Screenshots

• Selenium 3 can be used to screenshot an individual element. None of the built-in commands offer this functionality, but you can use it when writing your own. See the Selenium documentation.
• Viewport screenshots (i.e. a screenshot of the portion of the website visible in the browser's window) are available with the CommandSequence::save_screenshot command.
• Full-page screenshots (i.e. a screenshot of the entire rendered DOM) are available with the CommandSequence::screenshot_full_page command.
  • This functionality is not yet supported by Selenium/geckodriver, though it is planned. We produce screenshots by using JS to scroll the page and take a viewport screenshot at each location. This method will save the parts and a stitched version in the screenshot_path.
  • Since the screenshots are stitched they have some limitations:
    • On the area of the page present when the command is called will be captured. Sites which dynamically expand when scrolled (i.e., infinite scroll) will only go as far as the original height.
    • We only scroll vertically, so pages that are wider than the viewport will be clipped.
    • In geckodriver v0.15 doing any scrolling (or having devtools open) seems to break element-only screenshots. So using this command will cause any future element-only screenshots to be misaligned.

save_content

Response body content

• Saves all files encountered during the crawl to a LevelDB database de-duplicated by the md5 hash of the content.
• The content_hash column of the http_responses table contains the md5 hash for each script, and can be used to do content lookups in the LevelDB content database.
• NOTE: this instrumentation may lead to performance issues when a large number of browsers are in use.
• Set browser_params['save_content'] to a comma-separated list of resource_types to save only specific types of files, for instance browser_params['save_content'] = "script" to save only Javascript files. This will lessen the performance impact of this instrumentation when a large number of browsers are used in parallel.