As Vitess developers, our goal is to have great unit test coverage, and complete that with whatever integration or end-to-end test makes sense. In the past, we have been relying too much on end-to-end tests, so we are in the process on scaling down our tests to lower levels when appropriate, and increaing our coverage.
However, since we use other components extensively, and multiple languages, it is not sufficient. Sometimes, we just gotta have a real MySQL process, if we're testing interactions with MySQL.
## Client Library Tests and vtgateclienttest
All client libraries should be tested with vtgateclienttest. It is meant to validate all client inputs, and make sure all corner cases are handled properly in the client. For instance, to make sure all errors the server can return are translated into the right form on the client.
It is a single stand-alone process, light-weight enough that it runs very quickly and doesn't take many resources at all, and just requires a port to listen on.
It is expected that Vitess users may change py/vttest/environment.py to match their setup. For instance, YouTube internally does this to use different scripts to bring up a MySQL instance.
These tests need to be as light weight as possible, so the developers who use Vitess can run as many unit tests as they want with minimal resources. We are running a single MySQL and a single vtcombo instance (that encapsulates one vtgate and multiple vttablets). The MySQL process has multiple databases, one per keyspace / shard, and replication is not enabled. It is the smallest setup with a real MySQL.
This framework supports an initial schema to apply to all keyspaces / shards, per keyspace. This is also the base for testing schema changes before rolling them out to production.
Due to its constant nature, this is not an appropriate framework to test cluster events, like re-sharding, re-parenting, advanced query routing, ... No replication is setup on the single MySQL instance, obviously.
These tests run more complicated setups, and take a lot more resources. They are meant to test end-to-end behaviors of the Vitess ecosystem, and complement the unit tests.
For instance, we test each RPC interaction independently (client to vtgate, vtgate to vttablet, vttablet to MySQL, see previous sections). But is is also good to have an end-to-end test that validates everything works together.
These tests almost always launch a topology service, a few mysqld instances, a few vttablets, a vtctld process, a few vtgates, ... They use the real production processes, and real replication. This setup is mandatory for properly testing re-sharding, cluster operations, ... They all however run on the same machine, so they might be limited by the environment.
The vitest sandbox is a multi-server framework that runs production processes exactly as they would be running in production, in a cloud environment. It is very mature and powerful. It is also used for benchmarking, not just workflow tests.
The following action items exist to make it all consistent:
* Switch java tests to use vttest if they need a full cluster, and not java\_vtgate\_test\_helper.py, retire java\_vtgate\_test\_helper.py. Client unit tests should already use vtgateclienttest.
* the python client tests (using vtgateclienttest) are in test/python\_client\_test.py. They need to be finished. Note these are hard to differentiate from the tests in vtgatev2\_test.py. The rule of thumb is if you are testing a client-library feature, tests should be in test/python\_client\_test.py. If you are testing an end-to-end behavior, tests should be in vtgatev2\_test.py.
