16174fc5e2
v3: fix []byte vs string bug
...
The v3 router used to convert all []byte to string to
check for uniqueness (a futile exercise). The uniqueness
check was removed, but the conversion remained. Now, I've
removed it, and fixed tests and other dependencies accordingly.
2016-04-26 22:53:06 -07:00
169207ebc0
v3: rename binary & char vindexes
...
The new vindexes now have more precise names. This is because
their behavior is not supposed to change over time.
2016-04-26 21:48:57 -07:00
fec6433e28
Merge pull request #1651 from erzel/fix_equal_splits_default
...
Made equal_splits use the first split_column if more than one is give rather than complain
2016-04-26 20:36:26 -07:00
79423e9f6e
v3: fix binary & char vindexes
...
Fix 1: Use proper unicode methods to generate
reliable unique keys for unicode strings. The reference
is here: http://www.unicode.org/reports/tr10/#Multi_Level_Comparison .
We're using level 1 comparison, which is the most
permissive.
Fix 2: Use MD5 hash instead of the weaker 3DES algorithm.
2016-04-26 18:23:27 -07:00
57812424fd
Made equal_splits use the first split_column if more than one is given, rather than complain.
2016-04-26 16:27:34 -07:00
734d20a78e
Merge pull request #1652 from erzel/adding_support_for_random_population_of_vttest_to_go_client
...
Allow initialization of vttest DB with random data from go.
2016-04-26 13:52:03 -07:00
5c5b269b79
Allow initialization of vttest DB with random data from go.
2016-04-26 11:46:14 -07:00
39980a6184
Fixing a few tests that were using GetSessionId.
...
It was all implicit, and lazy, really.
2016-04-26 10:36:16 -07:00
dbc71f23e5
Removing sessionID in one more place in tests.
2016-04-26 10:15:41 -07:00
7eda94d924
Clean up of tabletconntest for no more Session.
2016-04-26 10:02:39 -07:00
b440b602d3
Removing client-side use of GetSessionId.
...
Now the vtgate gateways all use Target instead.
2016-04-26 09:58:54 -07:00
d66231aca6
Refactor tabletconntest.
...
It is now checking all error types. Lots of clean-up too.
2016-04-26 09:09:27 -07:00
a4d70c7370
Adding unit tests for new methods.
...
The BeginExecute and BeginExecuteBatch RPC methods now have tests, that
also make sure the transactionID is returned in case of an exec error.
2016-04-26 09:09:27 -07:00
8aa12343ac
Adding combo methods to gateway interface.
...
Now we have BeginExecute and BeginExecuteBatch in gateway. They call the
corresponding tabletconn methods.
Also using them in scatter_conn.go. And in there, fixing a bug where the
transactionID would be forgotten after an ExecuteBatch error (but a
successful Begin).
2016-04-26 09:09:27 -07:00
ed80827889
Adding combo methods to tabletconn.
...
These are BeginExecute and BeginExecuteBatch. In most places for now,
they still do the two RPCs. Will add tests first before adding a single
RPC in the gRPC proto.
2016-04-26 09:09:27 -07:00
2c3b2dec7c
Removing unused test code.
...
(good thing it's unused, sleeps are not a good idea).
2016-04-26 09:09:27 -07:00
7a0c90d15e
Fixing the test fixtures to use separate counts.
...
In vtgate tests, we were counting a Begin, Commit or Rollback twice.
With the upcoming merge, it didn't make much sense any more.
2016-04-26 09:09:27 -07:00
41e59f03b9
Moving begin() calls around.
...
This way Begin() and Execute() or ExecuteBatch() calls are right next to
eachother. It is now very easy to merge these into a single call.
2016-04-26 09:09:27 -07:00
78d657a986
Refactoring scatter_conn.
...
To separate the transaction paths and the non-transaction paths.
2016-04-26 09:09:27 -07:00
ac81c132ca
Incorporating Review Comments
2016-04-26 03:28:47 +05:30
33880c5613
Removed the bytes.ToLower in block3DES.Encrypt func, already taken care of in GetBytes()
2016-04-25 15:44:29 +05:30
d0265aa713
Incorporating Review Comments
2016-04-25 15:39:08 +05:30
1c773b1cd2
Removing Commented Code
2016-04-25 02:06:11 +05:30
03c10ffc22
Adding tests for varchar and varbinary Vindexes
2016-04-25 02:03:58 +05:30
347cf7199b
VarChar/VarBinary Column Support Changes in Vindex
2016-04-24 00:26:24 +05:30
bd52c0814b
worker: Refactored tests to use Run()+WaitForCommand() instead of calling the command's method.
...
The previous version was very low level. The new version is much closer to what the vtworker binary and the RPC servers do.
2016-04-22 20:16:05 -07:00
b6efc506b2
Adding a flag to run_local_database.py to allow populating the database with initial random data.
2016-04-21 13:26:37 -07:00
e1082849d3
Removing transactionID from client side interface.
...
transaction_id is not present on the server side for streaming queries
(and won't be in the foreseeable future). So removing it from the
client side interfaces as well. gRPC protos don't even have it.
2016-04-21 08:59:58 -07:00
ce210b356c
Update generated grpc Go code after breaking change in
...
https://github.com/grpc/grpc-go/pull/642 .
2016-04-20 11:53:32 -07:00
75c4fd89f0
Exporting internal changes back to open-source.
...
NOTE: This is an automated export. Changes were already LGTM'd internally.
2016-04-19 21:59:49 -07:00
f7087cf57a
vtctl: Fix vschema commands for new keyspace argument.
2016-04-15 17:59:34 -07:00
9c51b62d09
Fix healthcheck ipv6 bug
2016-04-15 13:33:26 -07:00
05bc6655d6
Merge pull request #1635 from alainjobart/merge
...
Adding a db merge end to end test.
2016-04-15 08:23:23 -07:00
5bf08455d2
Export internal changes.
...
Note: Changes were already LGTM'd internally.
2016-04-14 23:27:55 -07:00
417519bbe8
Merge pull request #1629 from youtube/ceph1
...
Ceph Backup Storage Plugin
2016-04-14 19:13:11 -07:00
446087180a
changed name for flag to take ceph configuration/credentials
2016-04-15 06:50:36 +05:30
f07a42457c
Removing a source of flakiness in initial_sharding.
...
In the process, re-factoring utils.wait_for_vars so it can also take a
key and value, and therefore can be used by a
VtGate.wait_for_endpoints() method. Using that method in a few tests
instead of custom cut&paste code.
2016-04-14 13:48:54 -07:00
898e9e225b
Factoring out common sharding test code.
...
Creating a new base class for sharding tests.
Using it in initial_sharding, merge_sharding, resharding and
vertical_split. Adding consistent checks in all of these.
Refactoring them to all match (all_tablets list at beginning, ...).
Also fixing a reporting bug in binlog server, the wrong var was used
for update stream. Actually found this with the tests!
2016-04-14 09:26:17 -07:00
04e32a2e2f
removing the log statement and returning the error
2016-04-14 14:13:57 +05:30
3c53dc9d02
corrections from Anthony's review
2016-04-14 13:58:40 +05:30
02fde77842
Merge pull request #1607 from michael-berlin/vtworker_retry_when_busy
...
automation: Retry vtworker commands when busy until cancelled.
2016-04-13 11:39:16 -07:00
1152cc675b
Adding a db merge end to end test.
...
And fix all the bugs it un-covers.
2016-04-13 09:29:45 -07:00
d730a5f5a9
quick fix for bucket pointer usage
2016-04-13 17:20:09 +05:30
d88f6c49fc
some more corrections from Anthony's review
2016-04-13 13:24:02 +05:30
7190584172
corrections from Anthony's review
2016-04-13 13:10:27 +05:30
93dd8db97f
deletion working
2016-04-13 13:00:13 +05:30
6519d598aa
Merge pull request #1634 from alainjobart/v3resharding
...
Adding code to properly split_clone a v3 table.
2016-04-12 15:27:06 -07:00
6576771e1f
Simplifying split diff worker.
...
We used to think we should run one diff per destination shard. Now we
assume we want one diff for each (source, destination) tuple. That way
splits and merges will work the same way, and we never need merge-sort.
In the process, fix the keyrange overlap usage to work for both splits
and merges.
2016-04-12 12:29:46 -07:00
8a78ec96b9
Adding v3 support in split_diff.
...
The test was returning all kinds of rows, had to fix that, because now
the filtering can be done at the app layer for v3.
Also, the test was describing views in the test schema, which is not
right, as the code doesn't ask for them.
2016-04-12 11:39:16 -07:00
85e8949cbc
Making SplitQueryV2 return TabletError
...
Removing timeout for SplitQuery
2016-04-12 10:55:38 -07:00
e0bda920a1
Factoring out some vschema code.
...
Building the formal vschema for a keyspace was shared in a couple
places.
2016-04-12 10:03:44 -07:00
c2462e938b
Now using the primary vindex only for splitclone.
...
It turns out it's the most likely one, no need to scan them all.
2016-04-12 09:50:22 -07:00
9f39741cf5
Factoring out a schema method.
2016-04-12 08:21:26 -07:00
d6b64cd553
Adding code to properly split_clone a v3 table.
...
We need a vindex of cost < 2 on a table (that is numeric or hash).
Then we use it to split the rows.
2016-04-12 08:05:08 -07:00
083512e1cc
Merge pull request #1595 from michael-berlin/min_rdonly_global_to_local
...
worker: Remove global flag --min_healthy_rdonly_endpoints and make it a flag in each vtworker command.
2016-04-11 14:10:04 -07:00
cf3ff453e3
mysql: better error on unrecognized types
...
Addresses issue #1516
2016-04-09 13:04:38 -07:00
f169f259ab
parser: recognize all MySQL reserved words
...
We need to recognize all MySQL reserved words because they
have to be back-quoted correctly if they're used as sql ids.
2016-04-08 20:06:58 -07:00
e6d689ae48
parser: support union distinct
...
I've also removed minus, except and intersect because
they're not supported by MySQL.
2016-04-08 14:33:57 -07:00
673907f084
parser: allow straight_join as hint
...
MySQL allows you to specify straight_join as a hint
in the select list instead of the FROM clause. We can
support this constrcut as pass-through.
2016-04-08 14:23:55 -07:00
8108abc9be
vtworkerclient: Cancel the context when the user presses Ctrl+C (SIGINT).
2016-04-08 11:03:24 -07:00
9c513ce7fb
automation: Retry vtworker RPCs if we encounter a retryable error.
...
Log the fact that we're retrying and for how long we did.
Use vterrors.WithPrefix() instead of fmt.Errorf() to preserve the error code.
2016-04-08 11:03:24 -07:00
3f50bda928
automation: Log executed vtctl and vtworker commands to the output.
...
This will, automation users can better follow which command (including all arguments) was executed.
2016-04-08 11:03:24 -07:00
553607b87e
vtworker: Fix bug that we reacted only to the first SIGINT signal.
...
Before this, it was not possible to shutdown vtworker when it was running the "Block" command: the first SIGINT would cancel the job. However, a second SIGINT is required to gracefully terminate the now cancelled vtworker process.
2016-04-08 11:03:24 -07:00
e9119fa020
vtworker: Do not respond with a retryable error when the job was cancelled but not reset.
...
Since Cancel (or the completion of the current command) and Reset are no atomic operation, it's possible that a client might observe the following:
1. job running => client retries
2. job stopped, but not reset yet => client should retry
3. job is reset => client will succeed
To make sure that the client retries in 2. vtworker will respect a grace period of a minute between cancel/completion and reset. If the cancel/completion is more than one minute ago, the client in step 2 will see a non-retryable error.
The new filed, which preserves the time the job was cancelled/completed, will be shown in the status as well.
2016-04-08 11:03:24 -07:00
d195029b19
vtworker: Remove unused worker.Instace parameter from Go test.
2016-04-08 11:03:24 -07:00
da892ecb39
vtworker: Return retryable error TRANSIENT_ERROR when a command is already being executed.
...
- Used vterrors.WithPrefix() instead of fmt.Errorf() where necessary to preserve the error code.
- Added unit test.
2016-04-08 11:03:24 -07:00
a96b6d876a
vtworker: Add internal command "Block" which blocks until cancelled.
...
We'll use this command for Go integration tests to verify that vtworker correctly responds with a retryable error when a command is already being executed.
2016-04-08 11:02:02 -07:00
1a5e15ec62
vtworker: Expose cancel mechanism as "Cancel" RPC.
...
This comes in handy when we have to cancel a running job.
2016-04-08 11:02:02 -07:00
1744ed28b0
vterrors: Do not wrap io.EOF when converting from a GRPCError to a VitessError.
2016-04-08 11:02:02 -07:00
5280e835d8
vterrors: Added an example to FromError().
...
Reversed order of FromError() and NewVitessError() because the first function should be used more often.
2016-04-08 11:02:02 -07:00
e7cc432515
vterrors: Fix spelling issues.
2016-04-08 11:02:02 -07:00
31f60bc49d
cephStorage
2016-04-08 19:44:09 +05:30
773d224c9d
Remove deprecated vtgateconn *2 methods
...
* Remove deprecated VTGate *2 methods from client code
* Use the correct verification for StreamExecute errors
2016-04-08 00:00:21 -07:00
7eb2ac2230
Also clearing the plan cache after vschema reload.
2016-04-06 12:23:13 -07:00
50ed5a4b6f
Factoring out vschema verification.
...
That way each implementation doesn't have to do it.
2016-04-06 12:23:13 -07:00
c8ee7590f3
vtgate's planner now uses WatchVSchema.
...
One caveat: if the list of keyspaces changes, vtgate won't notice.
2016-04-06 12:23:13 -07:00
bbe2ab1fd8
Adding topo.Impl.WatchVSchema.
...
And unit tests, implementing it for etcd and zk.
2016-04-06 12:23:13 -07:00
fc21924156
Merge branch 'master' into aaijazi_buffer_all_master
2016-04-05 11:53:38 -07:00
813b1a8fc7
Various fixes after the gRPC secure changes.
...
vtgateclienttest has to register services in 'OnRun',
instead of right away, as the gRPC server is not created yet.
Various fixes related to socket file. We were wrongly assuming
we needed the gRPC server when a port is specified, forgetting
the unix socket case. Refactoring the unix socket case to be
better aligned with gRPC code.
2016-04-05 07:43:22 -07:00
f66772fa05
Wiring in immediate callerid for gRPC.
...
Now extract the gRPC TLS certificate name the client used,
and use that as immediate caller id. When table ACLs are used,
this in effect propagates the certificate name as the client name.
Adding a simple unit test case in secure_transport.py: one vtgate
client cert is allowed to access the test table, another in not.
(also fixing doc for format of table acl config file).
2016-04-05 07:43:22 -07:00
a946c6ad7f
Adding client -> vtgate encryption, with test.
2016-04-05 07:43:22 -07:00
c8a6430478
Adding preliminary support for TLS for gRPC.
...
The server side takes a cert and key, and a ca to verify clients.
The client side takes a cert and key, a ca to verify the server,
and an optional server name in case the server host name does not
match the certificate name.
Renaming secure.py to be more descriptive.
2016-04-05 07:43:22 -07:00
02a0dde7e4
Rename file path from txbuffer to masterbuffer
2016-04-04 23:46:30 -07:00
84275f623c
All comments mention that we buffer master requests now, not just transactions
2016-04-04 23:46:30 -07:00
b051d5778e
Buffer master reads, not just transactions
2016-04-04 23:46:30 -07:00
1af5e91442
keyspaceID resolver errors if not a base table
2016-04-04 17:27:39 -07:00
196e3feb1f
Fix RowSplitter tests
2016-04-04 17:13:27 -07:00
cd28aa0e74
Make key resolvers per table instead of per keyspace
2016-04-04 17:13:27 -07:00
72c917d114
Rename sharding_key.go to key_resolver.go
2016-04-04 17:13:27 -07:00
96accf801e
Add a sharding key interface for vtworker, in preparation for supporting V3 sharding keys
2016-04-04 17:13:26 -07:00
1db5148a3b
v3: comment fix
2016-04-04 16:24:21 -07:00
b9b3fbe319
v3: misc fixes
...
* Plumnb initialization context from main function.
* Verify GetShardNames in vschema test.
2016-04-04 16:06:14 -07:00
4e9fc99ef9
v3: name params of SaveVSchema
2016-04-04 12:04:43 -07:00
6da03785c8
v3: vschema broken out by keyspace
2016-04-01 17:23:26 -07:00
d855f13571
v3: mockgen fix
2016-03-30 23:28:13 -07:00
a5e601481e
Merge branch 'master' into v3
2016-03-30 22:20:42 -07:00
aa06883f87
v3: Classes are history
2016-03-30 22:17:54 -07:00
3d2fb0c4c2
v3: optional keyspace in request
...
If a keyspace is in the request, tables will only be searched
in that keyspace unless the query contains explicit references.
2016-03-30 18:03:52 -07:00
008ea01308
v3: vschema implict & explicit table refs
2016-03-30 16:50:45 -07:00
a7cf5f32b0
Return an error when tx buffer is full and another request tries to buffer
2016-03-30 16:04:15 -07:00
Sugu Sougoumarane
Erez Louidor
Erez Louidor
Alain Jobart
Ashudeep Sharma
Michael Berlin
Anthony Yeh
Matt Harden
Alain Jobart
raunak-agarwal-flipkart
“raunak-agarwal-flipkart”
Ammar Aijazi
Ammar Aijazi