This document is designed to help developers and interested research parties to experiment with OpenWPM code customizations directly on their Mac workstations. The guide is not relevant unless you plan to actually modify some of the OpenWPM source code.
Date/relevance: This document was originally written and tested 2019-04-18, just after the release of OpenWPM v0.9.0, using a fork of the master branch (where development of v0.10.0 is performed) at the time.
- OSX: Mojave 10.14.4
- XCode: 10.2
- Python: Python 3.7 via Anaconda + venv
Before you begin
- Read Running a simple crawl to get familiar with how OpenWPM crawls are run in general
- Preferably, fire up a Ubuntu installation (via Docker is fine) and verify that you can get OpenWPM running using the official getting started instructions
Ways to run OpenWPM on Mac
There are two ways of running OpenWPM on Mac: 1. Via Docker, 2. Natively.
The advantage of running it via Docker is that the installation of all requirements will run smoothly since an Ubuntu installation is used as a base. Running in headless mode will also run smoothly without any special setup. However, since troubleshooting OpenWPM many times requires one to fire up Firefox GUI, you will need to set up X Forwarding between Mac and the Docker environment. When set up properly, will allow the Ubuntu Firefox installation to pop up as a window on your Mac workstation. If this meets your requirements, you don't need to read further. Instead, go ahead and try it out using the instructions at https://github.com/mozilla/OpenWPM#docker-deployment-for-openwpm.
Advantages of running OpenWPM natively on your Mac includes:
- Not having to be familiar with Docker
- Not having to set up shared volumes in order to run your locally modified development OpenWPM code, as well as access OpenWPM logs and collected data
- Be able to use your preferred version of Firefox for Mac (rather than Firefox for Ubuntu)
Getting started - running OpenWPM natively on your Mac
- Check out OpenWPM
git clone https://github.com/mozilla/OpenWPM.git
cd OpenWPM
- Install requirements (Note: Python packages are installed in a local Python 3 venv):
./install-mac-dev.sh
source venv/bin/activate
- Run a sample crawl:
python demo.py
- Open the results of the sample crawl:
open ~/Desktop/crawl-data.sqlite
Building and installing the Firefox web extension used by OpenWPM
After changing code in automation/Extension/firefox/:
./build-extension.sh
cp automation/Extension/firefox/dist/openwpm-1.0.zip automation/Extension/firefox/openwpm.xpi
Running future crawls
After having set up the requirements as per above, all that is necessary to run a crawl in a new terminal window is:
cd OpenWPM
source venv/bin/activate
python demo.py
Troubleshooting
In the case of any error, report it as an issue against OpenWPM together with steps to reproduce and the relevant log output (~/Desktop/openwpm.log).
Some known errors and workarounds:
sqlite3.ProgrammingError: Cannot operate on a closed database.
(Or sqlite3.OperationalError: database is locked)
For this, remove the sqlite database before running python demo.py:
rm ~/Desktop/crawl-data.sqlite*