1.2 KiB
Worker
The worker populates the database with information about new modules.
For additional information on functionality of the worker, see the design document.
Development
The main program lives in cmd/worker.
Running
You can run the worker locally like so:
go run ./cmd/worker
See experiment.md for instructions how to enable experiments.
Populating data locally using the worker
When run locally, the worker uses an in-memory queue. This implementation has
bounded parallelism (configurable via the -workers flag) but does not
automatically retry failures.
In order to populate local versions, you can either fetch the version explicitly
(via http://localhost:8000/fetch/path/to/package/@v/v1.2.3), or you can visit the
Worker dashboard, and click 'Enqueue from module index'. This will enqueue the
next N versions from the index for processing.
Bypassing license checks
By default, the worker does not insert readme contents or documentation into the
database if we determine that the module or package is not redistributable,
based on the licenses it finds in the module zip. To bypass the license check,
pass the flag -bypass_license_check.