vitess-gh/doc/ReferenceTables.md

Reference Tables

This document describes a proposed design and implementation guidelines for the Reference Tables Vitess feature.

The idea is to have a reference keyspace that contains a small number of reference tables, and replicate these tables to every shard of another keyspace, the destination keyspace. Any update to the reference tables will be propagated to the destination keyspace. The reference tables in the destination keyspace can then be used directly, in read-only mode (in JOIN queries for instance). This provides for much better performance than cross-keyspace joins.

Since the data is replicated to every shard on the destination keyspace, the write QPS on the reference keyspace is also applied to every shard on the destination keyspace. So the change rate in the reference keyspace cannot be very high, and so let's also assume it is not sharded.

Vitess already has all the right components to support this scenario, it's just a matter of plumbing it the right way. Let's explore the required changes.

Replication Setup

We can copy all the data and then setup Filtered Replication between the reference keyspace and each shard of the destination keyspace. This is really just a corner case of the vertical splits Vitess already supports.

Action items:

• First, this setup probably needs to be explicitely mentioned somewhere in the topology, not just as SourceShard objects in the destination keyspace, so Vitess can know about this setup at a higher level. Let's add a repeated ReferenceKeyspace field to the Keyspace object. Each ReferenceKeyspace object contains the name of the reference keyspace, the list of tables to copy, and the UID of the SourceShard entry (the same UID in all shards). By making this a repeated field, the destination keyspace should be able to support multiple reference keyspaces to copy data from, if necessary.

• vtctl CopySchemaShard can already be used to copy the schema from the reference keyspace to each destination shard.

• A new vtworker data copy job needs to be added. vtworker VerticalSplitClone would be a good start, but the new copy has a few special requirements: the destination keyspace needs the data in all its shards, and the write rate cannot cause the destination shards to be overloaded (or lag behind on replication). This job would also populate an entry in the \_vt/blp\_checkpoint table in the destination shards.

• Setting up Filtered Replication after the copy is easy, each destination Shard just needs to have a SourceShard with the proper data, and after a RefreshTablet, the destination masters will start the replication.

All these steps can be supported by a vtctld workflow.

Supporting Horizontal Resharding in the Destination Keyspace

We still need to support horizontal resharding in the Destination Keyspace while the Reference Tables feature is enabled.

Action items:

• Each step of the process would know what to do because of the ReferenceKeyspace entries in the destinatin keyspace.

• vtctl CopySchemaShard needs to also copy the schema of the reference tables.

• vtworker SplitClone needs to also copy all of the reference table data, and the \_vt/blp\_checkpoint entry for the reference keyspace. It needs to do that copy from the first source shard to each destination shard only once. So in case of a split, the source shard data is copied to each destination shard. In case of a merge, only the first source shard data is copied to the destination shard.

• Enabling filtered replication on the destination shards needs to not use the same UID for replication as the reference keyspace entries. Right now, their UID are hardcoded to start at 0. But since the reserved UIDs are documented in the ReferenceKeyspace entries, it's easy.

• At this point, the destination shards will also replicate from the reference keyspace. When the vtctl MigrateServedType master command is issued, it needs to just remove the horizontal resharding Filtered Replication entries, not the ReferenceKeyspace entries entries.

Other Use Cases

Other scenarios might also need to be supported, or explicitely disabled:

• Simple schema changes, or complicated Schema Swap in the reference keyspace: They would also need to be applied to the destination keyspace, the same way.

• Vertical Split of the reference keyspace: Since it is replicated, splitting it will be more complicated.

Query Routing

This would be handled by the vtgate and the VSchema. Once the reference tables are documented in the VSchema, vtgate will know to do the following:

• DMLs on the reference tables are routed to the reference keyspace.

• Select queries on the reference tables only are also routed to the reference keyspace.

• JOIN queries between reference tables and destination keyspace tables can be routed only to the right destination keyspace (based on that keyspace sharding situation).

Note this introduces some corner cases: for instance, if the client is asking for a JOIN between reference tables and destination keyspace tables, with tablet type master. Routing this to the destination keyspace would satisfy the critical read for the destination tables, but not for the reference tables. vtgate may need to perform the JOIN to both masters at this point.

Action Items:

• Find the right way to represent reference tables in the VSchema.

• Implement corresponding query routing.

Notes

Vitess Keyspace vs MySQL Database

This may force us to revisit the use of databases in our tablets. The current assumption is that a keyspace only has one MySQL database (with a name usually derived from the keyspace name with a vt_ prefix, but that can also be changed):

• When vttablet connects to MySQL for data queries, it uses that database name by default.

• The VSchema also maps tables to keyspaces, so it can just send queries that have no keyspace to the right shard (which in turns is configured properly for that database).

• Vitess' Filtered Replication only replicates data related to that single database. The database name has to be the same when we horizontally split a keyspace, so statements from the source shards can be applied on the destination shards.

• Vitess' Query Service only loads the schema for that single database.

Maybe it's time to change this assumption:

• A keyspace could be defined as a group of databases, each having a group of tables.

• When addressing a table, we could support the keyspace.database.table syntax.

• We could support moving databases from one keyspace to another.

But maybe this is too many indirections for nothing? Saying one keyspace is one database may be just the complexity we need.