It seems it's just simpler all around if the zk topology plugin creates
the higher level directories when needed. The deletion still stops at
the root though, seems safer.
This change adds full support for a Consul-based topology service.
See https://www.consul.io/ for more information.
- Adding consul server to bootstrap.sh for download.
- Adding client library to vendor.json.
- In unit tests, we start a dev Consul instance.
- Adding consul support in end-to-end tests.
Also running the test in travis.
- Updating topo doc.
- Adding Consul to 2.1 features.
The new etcd2 topology service is the new more consistent version of the
etcd topology. It has the following improvements:
- Uses the same file structure as zk2topo. It then also uses the generic
vtctld.Explorer as well.
- Uses the new etcd clientv3 library, with new key-value API and leases.
It is not backwards compatible with the old etcd topology.
- Uses a configurable toplevel directory for topology data in global and
local cells.
Vitess 2.1 will still include the old etcd topology client, but it will
be deprecated, and removed from Vitess 2.2. Use topo2topo binary to
upgrade, see documentation.
Implementation details:
- bootstrap.sh now downloads etcd.
- etcd is used in unit tests of the library, instead of using a mock.
- Fixing wrangler.InitTablet error return.
And adding -allow_update to test calls, so they work anyway.
- Running tabletmanager.py for zookeeper, zk2, etcd, etcd2 flavors.
- Ignoring the 'test' tag for vendored libraries. See:
https://github.com/kardianos/govendor#ignoring-build-tags-and-excluding-packages
- in end-to-end tests, restarting etcd to wipe it.
- publish web site.
- Introducing topodata.CellInfo. Stored in the global topology server, it
describes a connection to a topology server for a cell. It has both a
server address and a root directory to use. Also adding vtctl commands
to deal with it. Existing zk and etcd implementations ignore these flags
for now.
- Now passing toplevel server address and root as generic topology
parameters. And using a topo server factory method.
- Changing the topo server registration to use a Factory method. It
creates a topology server implementation with a server address and a
root path. Existing zk and etcd implementations ignore these flags for
now.
- adding a zk2 topology implementation. It doesn't use any of the go/zk
code, and allows the specification of a root directory for both the
global cell, and each individual cell. It also is using a different
directory structure, consistent with what we want all new topo
implementations to use. And it stores the data as protobuf, not json.
- deprecating the legacy 'zookeeper' implementation. Adding instructions
to migrate from old server style to new server style.
- removing old janitor code, been replaced by topo validator workflow.
- Using a more generic vtctld topo explorer, will use it in all servers
soon.
- Fixing the ZK command line to decode protos and use new connection
library.
- topo2topo now also copies VSchema.
For example, http://vitess.io/doc/TopologyService/ was published but
never reachable from any navigation entry.
Stopping to publish /doc/ pages will also avoid duplication e.g.
- http://vitess.io/doc/Reparenting/ vs.
- http://vitess.io/user-guide/reparenting.html
Instead, to publish a page, an explicit *.md file must be created in /vitess.io and it must be linked in the navigation file vitess.io/_includes/left-nav-menu.html.
I did this for TopologyService.md and created vitess.io/user-guide/topology-service.md because it is a relevant page which is linked from several other pages.
This change also allows us to remove the script replace_doc_link.py. It
replaced github.com links (e.g.
https://github.com/youtube/vitess/blob/master/doc/TopologyService.md)
with a vitess.io link. Since we're always using absolute links anyway
(e.g. /user-guide/topology-service.html), this is no longer necessary.
By getting rid of the tool and the extra doc files (published under
_post), we can also avoid creating a copy when
running preview-site.sh. This way, Jekyll can watch the original folder
and automatically generate pages when something changes.
To make this work, I had to add the following symlinks:
vitess.io/_includes/doc
vitess.io/_includes/index.md
vitess.io/_includes/README.md
After this change, the following files in doc/ are not published on vitess.io:
- LifeOfAQuery.md
- Monitoring.md
- Production.md
- ReplicationStreamTiming.md
- TestingOnARamDisk.md
- TwoPhaseCommit.md
- V3HighLevelDesign.md
- V3VindexDesign.md
- Vision.md
- VTGateV3Features.md
Except for TestingOnARamDisk.md they also haven't been visible on vitess.io before. That's because no other page did link to them.
huynq0911
Alain Jobart
Michael Berlin
Anthony Yeh
Alain Jobart