vtgate now gets the groups and sends them in the immediate caller id.
we store the time of the ldap query and refresh (in another goroutine) if
older than configurable threshold. this happens upon ComQuery. the simpleacl
implementation is now group aware, and the ldapacl plugin has been removed.
Created the Code enum to match grpc error codes, and renamed
all previous error codes to not have a name clash. Also renamed
The older field to LegacyCode.
The new message feature in vttablet has to generate its own DMLs.
In the current design, vttablet does not access vindex. So, it may
not be able to generate keyspace id info for those DMLs.
OTOH, RBR also has similar limitations because the replication
event will not have keyspace id info.
This PR sprays such non-annotated DMLs to all target shards. This
is safe because vttablet rewrites all queries to be PK-based. So,
the replay will only affect the shards where the rows are present.
This is a backward compatible change that should not break existing
behavior.
That's what the actual protocol size for the field is. Also documenting
the other fields, as they are uint16 or uint8 in the protocol.
And rebuilding all proto docs too.
Removes exclude_field_name from ExecuteOptions, replacing with an
IncludedFields enum. Enum values allow specifying whether to return
all fields, just type and name (default), or just type (perf optimization)
Adds and updates tests
- Introducing topodata.CellInfo. Stored in the global topology server, it
describes a connection to a topology server for a cell. It has both a
server address and a root directory to use. Also adding vtctl commands
to deal with it. Existing zk and etcd implementations ignore these flags
for now.
- Now passing toplevel server address and root as generic topology
parameters. And using a topo server factory method.
- Changing the topo server registration to use a Factory method. It
creates a topology server implementation with a server address and a
root path. Existing zk and etcd implementations ignore these flags for
now.
- adding a zk2 topology implementation. It doesn't use any of the go/zk
code, and allows the specification of a root directory for both the
global cell, and each individual cell. It also is using a different
directory structure, consistent with what we want all new topo
implementations to use. And it stores the data as protobuf, not json.
- deprecating the legacy 'zookeeper' implementation. Adding instructions
to migrate from old server style to new server style.
- removing old janitor code, been replaced by topo validator workflow.
- Using a more generic vtctld topo explorer, will use it in all servers
soon.
- Fixing the ZK command line to decode protos and use new connection
library.
- topo2topo now also copies VSchema.
* 2pc: refactored metadata tables
Changing the table names: This allows me to start with a clean slate.
Since the feature is not yet in use, I don't have to migrate the old
tables.
enums->ints: Enums looked like a good idea in the beginning. But it's
a pain if you have to add a new state, because it requires a schema
change. Switching to numbers in db and enums in code.
Dropping deprecated columns & indexes: During implementation, I made
some improvements to the prepare flow (compared to the design). That
allows me to simplify that table.
Summary:
redo_log_transaction -> redo_state, drop resolution, drop
state_time_idx.
redo_log_statement ->redo_statement.
transaction -> dt_state.
participant -> dt_participant, drop time_updated
* code review comments
A new SingleDb flag has been added to Begin, which would survive
through the Session. If set, any transaction that exceeds a single
shard will be failed.
VTGate acquires a command line flag: transaction_mode, which can be
single, multi or 2pc. In single mode, it will fail Begins that request
multi-shard. In multi mode, it will fail commits that request 2pc.
In 2pc mode, everything will be allowed.
The per-request flags specify what the app wants, and the vtgate flags
specify what it allows.
* 2pc: Go and python clients
For Go, the transaction mode settings are supported through
the context. This means that it will only work with go1.8.
For Python, it's a cursor constructor parameter.
* 2pc: end-to-end test
The 'Resolve' name was misleading. It was implying that
some kind of resolution work was being performed. Conclude
is a more accurate description of this action.
This will make the type look more generic, so that it could be used for other
purposes as well, not only for a worker during resharding. In particular I plan
to use the same tablet type for schema swap reserving a seed vttablet to execute
the schema change.
I also add a "drain_reason" tag that will be set by anyone setting the DRAINED
tablet type, so that it's easy to understand why it is drained.
This adds support for an 'allprivs' user which is supposed to have more
privileges than 'app' user, but unlike 'dba' user 'allprivs' won't have SUPER
privilege. New method ExecuteFetchAsAllPrivs() is added to TabletManager
service to be able to execute queries as this new 'allprivs' user.
'allprivs' user will be used for administrative tasks done by Vitess, such as
changes in metadata tables that schema swap process will be doing. 'allprivs'
user shouldn't have SUPER privilege so that schema swap could safely change
replicated metadata on the master without risk to commit the changes when
master has been already demoted to be a slave.
ResultExtras is part of Query Result. It contains:
- an optional EventToken (returned if ExecuteOptions.include_event_token
is set)
- an optional 'fresher' flag (set if ExecuteOptions.compare_event_token
is set, and that token is older than the current TabletServer event
token.)
The flags are set by TabletServer, and merged by VtGate.ScatterConn.
They are added to both to queryservice and vtgateservice.
Right now, they only contain the exclude_field_names option.
exclude_field_names is used by vttablet to strip the name from the
fields record. Note the clients don't use that yet, I want the ability
to do it first, then I'll add support for it in the clients.
Allowing ExecuteOptions in vtctl commands. That's how they're tested
now.
Side-changes:
- Removing wantFields from dbclient interface. We never want them.
- Small optimization on query fields for sequences.
The new setting is similar, but independent to the existing option "ignore_n_slowest_rdonlys".
For example, if both are set to 1, you can have one 1 slow REPLICA and 1 slow RDONLY tablet.
It is enabled with the -watch_replication_stream command line
argument. It does two things:
- remember the last applied Event Token (and exposes it in vars).
- if the event is a DDL, it forces a schema reload.
Also ignoring local binlog connections in FindSlaves, as they have no
impact on reparenting.
min_duration_between_changes_sec was split into separate "increases" and "decreases" fields.
This is required because an insert test needs to wait longer than a decrease test.
Additionally, there is a new field "spread_backlog_across_sec". Before this change, the value of "min_duration_between_changes_sec" was used for this.
The vtgate API takes a starting timestamp, or a starting EventToken. It
will only use the starting EventToken if it's relevant. This is mostly
for tests, but could be used by real clients too to avoid the timestamp
search on the servers.
The only restriction in the vtgate routing implementation is that a
query can only end up on one shard. The stream aggregation code inside
vtgate will be added later.
This change includes:
- proto changes.
- implementing the server side interface.
- implementing the client side interface.
- adding a vtctl VtTabletUpdateStream command to stream from a given
tablet. This is used in end-to-end tests.
- using the python vtgate_client update_stream API in all end-to-end
tests.
- removing the python vttablet direct stream_update API.
- vtgate now better preserves remote errors through its API now, as
withSuffix and withPrefix will preserve the error codes of all
VtError, not just *VitessError.
- Also adding callerid vtgateclienttest tests for all API calls.
The UpdateStream call makes a lot more sense in the QueryService part of
the API. It is meant to be used by vtgate. The Binlog side of the API is
then only used by filtered replication.
Adding vttablet support to start streaming from a timestamp, not just a
replication position. It finds the starting position by loking at the
binary logs.
In the process, change the following things:
- UpdateStream is now added to gateway.Gateway, so it flows through
l2vtgate if needed.
- the python client now uses the new service.
- updated the tests as required.
- using Context instead of sync2.ServiceManager. Removing
sync2.ServiceManager as it is now unused.
- tabletserver.TabletServer now remembers the ongoing UpdateStream
connections, and closes them on state change.
- Adding an EventToken structure in query.proto.
- Using it in Binlog Streamer and filtered replication.
- Using it in Update Stream in POS events as well.
Note the main change there is that an EventToken has a replication
position (GTIDSet), not a transaction ID (GTID). Both server and clients
were computing the position individually anyway by accumulating
transaction IDs, might as well just send the position. And it will make
more sense for later use of EventToken.
Also, we don't set the 'Shard' field of the EventToken just yet. I'm
still not sure vttablet should do it, as opposed to vtgate.
In order to do that, I had to:
- Adding File/Dir APIs to topo.Backend.
- Implementing them for all flavors: memorytopo, zk, etcd, tee.
None of the backward compatible code is there yet,
so these can only be used for new objects.
- Adding corresponding unit tests.
- Fixing a few fake etcd-specific bugs.
Also now using CancelFunc for all topo watches.
Alex Charis
Michael Berlin
Sugu Sougoumarane
wangyipei01
Erez Louidor
Alain Jobart
Bryan Beaudreault
harshit-gangal
Pavel Ivanov