CONTRIBUTING.md,docs: clean up for external use

Change-Id: I1ab3bcb5eed7000ab104fa126ddefedcde0f653c
Reviewed-on: https://team-review.git.corp.google.com/c/golang/discovery/+/692510
Reviewed-by: Julie Qiu <julieqiu@google.com>

CONTRIBUTING.md

@@ -14,7 +14,10 @@ before sending patches.
 
 ## Finding issues
 
-All issues are labeled with the [`go.dev` label](https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Ago.dev). Issues that are suitable for contributors are additionally tagged with the [`help-wanted` label](https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Ago.dev+label%3A%22help+wanted%22+).
+All issues are labeled with the [`go.dev`
+label](https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Ago.dev).
+Issues that are suitable for contributors are additionally tagged with the
+[`help wanted` label](https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+label%3Ago.dev+label%3A%22help+wanted%22+).
 
 Before you begin working on an issue, please leave a comment that you are claiming it.
 
@@ -29,13 +32,15 @@ Before you begin working on an issue, please leave a comment that you are claimi
 
 2. Review the [design document](design.md).
 
-3. Download and install Cloud SDK at https://cloud.google.com/sdk/docs/
+3. We deploy to the [Google Cloud Platform](https://cloud.google.com). If you
+wish to set up a similar environment, you will want to
+download and install the Google Cloud SDK at https://cloud.google.com/sdk/docs/.
 
 4. Depending on the feature you are working on, review the contributing guides for:
 
-- [Frontend development](frontend.md)
-- [ETL development](etl.md)
-- [Database setup](database.md)
+- [Frontend development](doc/frontend.md)
+- [Worker development](doc/worker.md)
+- [Database setup](doc/postgres.md)
 
 ## Questions
 

doc/design.md

@@ -22,7 +22,7 @@ The frontend has the following dependencies:
 - The Postgres DB, to retrieve information about modules and packages.
 - A Redis instance, as a cache.
 
-See [The Frontend](doc/frontend.md) for details.
+See [The Frontend](frontend.md) for details.
 
 ## The Worker
 
@@ -53,4 +53,4 @@ The worker has the following dependencies:
 - The proxy (proxy.golang.org by default) to fetch the module zip files.
 - The Postgres database.
 
-See [The Worker](doc/worker.md) for details.
+See [The Worker](worker.md) for details.

doc/etl.md

@@ -1,16 +0,0 @@
-# ETL
-
-### Populating data locally using the ETL
-
-When deployed on AppEngine, the discovery ETL uses cloud tasks to manage new
-version processing. Cloud tasks offers some nice features, such as built-in
-rate limiting and retries with exponential backoff.
-
-When run locally, the ETL uses an in-memory queue. This implementation has
-bounded parallelism (configurable via the `-workers` flag) but does not
-automatically retry failures.
-
-In order to populate local versions, you can either fetch the version explicitly
-(via `http://localhost:8000/fetch/path/to/package/@v/v1.2.3`), or you can visit the
-ETL dashboard, and click 'Enqueue from module index'.  This will enqueue the
-next N versions from the index for processing.

doc/postgres.md

@@ -1,23 +1,7 @@
 ### Databases
 
-#### Cloud SQL databases
-
 We use [PostgreSQL](https://www.postgresql.org).
 
-We currently have two databases in Cloud SQL:
-
-- Production: [discovery-db]()
-- Dev: [dev-discovery-db]()
-
-Run
-
-```
-go run devtools/cmd/dbadmin/dbadmin.go newuser <your-ldap> dev-discovery-db discovery-db
-```
-to create read-only user accounts for yourself in both databases.
-
-If the need ever arises to create a new database, use  `devtools/cmd/cloudsql/setupdb.go`.
-
 #### Local development database
 
 You will want to run PostgreSQL on your machine for local development.
@@ -85,9 +69,10 @@ local server, and running `CREATE DATABASE "discovery-database"` (or whatever
 name you choose).
 
 Once this database exists and these variables are correctly configured, run
-`./devtools/migrate_db.sh local up` once to initialize your local database, as
-described in 'Migrations' below. You will need to do this each time a new
-migration is added, to keep your local schema up-to-date.
+`scripts/create_local_db.sh` once to initialize your local database.
+
+Then apply migrations, as described in 'Migrations' below. You will need to do
+this each time a new migration is added, to keep your local schema up-to-date.
 
 
 ### Migrations
@@ -103,7 +88,7 @@ Migrations are managed with the [golang-migrate/migrate][] [CLI tool][].
 To create a new migration:
 
 ```
-migrate create -ext sql -dir migrations -seq <title>
+scripts/create_migration.sh <title>
 ```
 
 This creates two empty files in `/migrations`:
@@ -120,18 +105,10 @@ for details.
 
 #### Applying migrations for local development
 
-Do not use the `migrate` CLI directly to apply migrations. Use the
-`migrate_db.sh` script instead.
-
-
-To apply migrations to your local development database, run
+Use the `migrate` CLI:
 
 ```
-./devtools/migrate_db.sh local up
+migrate -source file:migrations \
+        -database "postgres://postgres@$GO_DISCOVERY_DATABASE_HOST:$GO_DISCOVERY_DATABASE_PORT/$GO_DISCOVERY_DATABASE_NAME}?sslmode=disable&password=${GO_DISCOVERY_DATABASE_PASSWORD}" \
+        up
 ```
-That assumes that your local database is running on port 5432. It will
-read the password from the `GO_DISCOVERY_DATABASE_PASSWORD` environment variable.
-
-#### Applying migrations for Cloud SQL
-
-See [the playbook page]().