github
/
vitess-gh
зеркало из https://github.com/github/vitess-gh.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Delete all legacy sharding related code (#10278) 

* Delete all legacy sharding related code

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Move used until functions from initialsharding to cluster

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove vtctl commands

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Kill vtworker and SetKeyspaceServedFrom cmd

Signed-off-by: Matt Lord <mattalord@gmail.com>

* WaitForDrain related stragglers

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Legacy local straggler workflow

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Get rid of SetKeyspaceShardingInfo & wait for drain stragglers

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove vtworker stragglers

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Update throttlerservice protobuf

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Rename test 24, add hashicorp vault test to it (now mysql_server_vault)

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Remove last mentiones of legacy sharding in vtctl

Signed-off-by: Matt Lord <mattalord@gmail.com>

* remove binlog_use_v3_resharding_mode

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Address review comments

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Address review comments

Signed-off-by: Matt Lord <mattalord@gmail.com>

* Correct vtgate help output

Signed-off-by: Matt Lord <mattalord@gmail.com>
This commit is contained in:
Matt Lord 2022-07-07 13:52:32 -04:00 коммит произвёл GitHub
Родитель 3f79a0ea50
Коммит a1f1f12be8
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
250 изменённых файлов: 519 добавлений и 29666 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

8
.github/workflows/cluster_endtoend_24.yml → .github/workflows/cluster_endtoend_mysql_server_vault.yml поставляемый
Просмотреть файл

@ -1,9 +1,9 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
name: Cluster (24)
name: Cluster (mysql_server_vault)
on: [push, pull_request]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (24)')
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (mysql_server_vault)')
cancel-in-progress: true
env:
@ -13,7 +13,7 @@ env:
jobs:
build:
name: Run endtoend tests on Cluster (24)
name: Run endtoend tests on Cluster (mysql_server_vault)
runs-on: ubuntu-18.04
steps:
@ -99,7 +99,7 @@ jobs:
set -x
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard 24 | tee -a output.txt | go-junit-report -set-exit-code > report.xml
eatmydata -- go run test.go -docker=false -follow -shard mysql_server_vault | tee -a output.txt | go-junit-report -set-exit-code > report.xml
- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()

106
.github/workflows/cluster_endtoend_resharding.yml поставляемый
Просмотреть файл

@ -1,106 +0,0 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
name: Cluster (resharding)
on: [push, pull_request]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (resharding)')
cancel-in-progress: true
env:
LAUNCHABLE_ORGANIZATION: "vitess"
LAUNCHABLE_WORKSPACE: "vitess-app"
GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"
jobs:
build:
name: Run endtoend tests on Cluster (resharding)
runs-on: ubuntu-18.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/**'
- name: Set up Go
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.3
- name: Set up python
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-python@v2
- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
# install JUnit report formatter
go install github.com/jstemmer/go-junit-report@latest
- name: Setup launchable dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
pip3 install --user launchable~=1.0 > /dev/null
# verify that launchable setup is all correct.
launchable verify || true
# Tell Launchable about the build you are producing and testing
launchable record build --name "$GITHUB_RUN_ID" --source .
- name: Run cluster endtoend test
if: steps.changes.outputs.end_to_end == 'true'
timeout-minutes: 30
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard resharding | tee -a output.txt | go-junit-report -set-exit-code > report.xml
- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()
run: |
# send recorded tests to launchable
launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
# print test output
cat output.txt

106
.github/workflows/cluster_endtoend_resharding_bytes.yml поставляемый
Просмотреть файл

@ -1,106 +0,0 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
name: Cluster (resharding_bytes)
on: [push, pull_request]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (resharding_bytes)')
cancel-in-progress: true
env:
LAUNCHABLE_ORGANIZATION: "vitess"
LAUNCHABLE_WORKSPACE: "vitess-app"
GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"
jobs:
build:
name: Run endtoend tests on Cluster (resharding_bytes)
runs-on: ubuntu-18.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/**'
- name: Set up Go
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.3
- name: Set up python
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-python@v2
- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
# install JUnit report formatter
go install github.com/jstemmer/go-junit-report@latest
- name: Setup launchable dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
pip3 install --user launchable~=1.0 > /dev/null
# verify that launchable setup is all correct.
launchable verify || true
# Tell Launchable about the build you are producing and testing
launchable record build --name "$GITHUB_RUN_ID" --source .
- name: Run cluster endtoend test
if: steps.changes.outputs.end_to_end == 'true'
timeout-minutes: 30
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard resharding_bytes | tee -a output.txt | go-junit-report -set-exit-code > report.xml
- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()
run: |
# send recorded tests to launchable
launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
# print test output
cat output.txt

125
.github/workflows/cluster_endtoend_shardedrecovery_stress_verticalsplit_heavy.yml поставляемый
Просмотреть файл

@ -1,125 +0,0 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
name: Cluster (shardedrecovery_stress_verticalsplit_heavy)
on: [push, pull_request]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (shardedrecovery_stress_verticalsplit_heavy)')
cancel-in-progress: true
env:
LAUNCHABLE_ORGANIZATION: "vitess"
LAUNCHABLE_WORKSPACE: "vitess-app"
GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"
jobs:
build:
name: Run endtoend tests on Cluster (shardedrecovery_stress_verticalsplit_heavy)
runs-on: ubuntu-18.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/**'
- name: Set up Go
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.3
- name: Set up python
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-python@v2
- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
# install JUnit report formatter
go install github.com/jstemmer/go-junit-report@latest
- name: Setup launchable dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
pip3 install --user launchable~=1.0 > /dev/null
# verify that launchable setup is all correct.
launchable verify || true
# Tell Launchable about the build you are producing and testing
launchable record build --name "$GITHUB_RUN_ID" --source .
- name: Run cluster endtoend test
if: steps.changes.outputs.end_to_end == 'true'
timeout-minutes: 30
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# Increase our local ephemeral port range as we could exhaust this
sudo sysctl -w net.ipv4.ip_local_port_range="22768 61999"
# Increase our open file descriptor limit as we could hit this
ulimit -n 65536
cat <<-EOF>>./config/mycnf/mysql57.cnf
innodb_buffer_pool_dump_at_shutdown=OFF
innodb_buffer_pool_load_at_startup=OFF
innodb_buffer_pool_size=64M
innodb_doublewrite=OFF
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_numa_interleave=ON
innodb_adaptive_hash_index=OFF
sync_binlog=0
sync_relay_log=0
performance_schema=OFF
slow-query-log=OFF
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard shardedrecovery_stress_verticalsplit_heavy | tee -a output.txt | go-junit-report -set-exit-code > report.xml
- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()
run: |
# send recorded tests to launchable
launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
# print test output
cat output.txt

125
.github/workflows/cluster_endtoend_worker_vault_heavy.yml поставляемый
Просмотреть файл

@ -1,125 +0,0 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"
name: Cluster (worker_vault_heavy)
on: [push, pull_request]
concurrency:
group: format('{0}-{1}', ${{ github.ref }}, 'Cluster (worker_vault_heavy)')
cancel-in-progress: true
env:
LAUNCHABLE_ORGANIZATION: "vitess"
LAUNCHABLE_WORKSPACE: "vitess-app"
GITHUB_PR_HEAD_SHA: "${{ github.event.pull_request.head.sha }}"
jobs:
build:
name: Run endtoend tests on Cluster (worker_vault_heavy)
runs-on: ubuntu-18.04
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- '.github/workflows/**'
- name: Set up Go
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.3
- name: Set up python
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-python@v2
- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
# Increase the asynchronous non-blocking I/O. More information at https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_use_native_aio
echo "fs.aio-max-nr = 1048576" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
- name: Get dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download
# install JUnit report formatter
go install github.com/jstemmer/go-junit-report@latest
- name: Setup launchable dependencies
if: steps.changes.outputs.end_to_end == 'true'
run: |
# Get Launchable CLI installed. If you can, make it a part of the builder image to speed things up
pip3 install --user launchable~=1.0 > /dev/null
# verify that launchable setup is all correct.
launchable verify || true
# Tell Launchable about the build you are producing and testing
launchable record build --name "$GITHUB_RUN_ID" --source .
- name: Run cluster endtoend test
if: steps.changes.outputs.end_to_end == 'true'
timeout-minutes: 30
run: |
# We set the VTDATAROOT to the /tmp folder to reduce the file path of mysql.sock file
# which musn't be more than 107 characters long.
export VTDATAROOT="/tmp/"
source build.env
set -x
# Increase our local ephemeral port range as we could exhaust this
sudo sysctl -w net.ipv4.ip_local_port_range="22768 61999"
# Increase our open file descriptor limit as we could hit this
ulimit -n 65536
cat <<-EOF>>./config/mycnf/mysql57.cnf
innodb_buffer_pool_dump_at_shutdown=OFF
innodb_buffer_pool_load_at_startup=OFF
innodb_buffer_pool_size=64M
innodb_doublewrite=OFF
innodb_flush_log_at_trx_commit=0
innodb_flush_method=O_DIRECT
innodb_numa_interleave=ON
innodb_adaptive_hash_index=OFF
sync_binlog=0
sync_relay_log=0
performance_schema=OFF
slow-query-log=OFF
EOF
# run the tests however you normally do, then produce a JUnit XML file
eatmydata -- go run test.go -docker=false -follow -shard worker_vault_heavy | tee -a output.txt | go-junit-report -set-exit-code > report.xml
- name: Print test output and Record test result in launchable
if: steps.changes.outputs.end_to_end == 'true' && always()
run: |
# send recorded tests to launchable
launchable record tests --build "$GITHUB_RUN_ID" go-test . || true
# print test output
cat output.txt

50
.github/workflows/cluster_initial_sharding_multi.yml поставляемый
Просмотреть файл

@ -1,50 +0,0 @@
name: cluster_initial_sharding_multi
on: [push, pull_request]
jobs:
build:
name: cluster initial sharding multi
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
end_to_end:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- name: Set up Go
if: steps.changes.outputs.end_to_end == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.3
- name: Tune the OS
if: steps.changes.outputs.end_to_end == 'true'
run: |
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
# TEMPORARY WHILE GITHUB FIXES THIS https://github.com/actions/virtual-environments/issues/3185
- name: Add the current IP address, long hostname and short hostname record to /etc/hosts file
if: steps.changes.outputs.end_to_end == 'true'
run: |
echo -e "$(ip addr show eth0 | grep "inet\b" | awk '{print $2}' | cut -d/ -f1)\t$(hostname -f) $(hostname -s)" | sudo tee -a /etc/hosts
# DON'T FORGET TO REMOVE CODE ABOVE WHEN ISSUE IS ADRESSED!
- name: Run initial sharding multi
if: steps.changes.outputs.end_to_end == 'true'
run: |
go run test.go -print-log initial_sharding_multi

88
.github/workflows/legacy_local_example.yml поставляемый
Просмотреть файл

@ -1,88 +0,0 @@
name: local_example
on: [push, pull_request]
jobs:
build:
name: Legacy local example using ${{ matrix.topo }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest]
topo: [etcd,k8s]
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Check for changes in relevant files
uses: frouioui/paths-filter@main
id: changes
with:
token: ''
filters: |
examples:
- 'go/**/*.go'
- 'test.go'
- 'Makefile'
- 'build.env'
- 'go.[sumod]'
- 'proto/*.proto'
- 'tools/**'
- 'config/**'
- 'bootstrap.sh'
- 'examples/**'
- name: Set up Go
if: steps.changes.outputs.examples == 'true'
uses: actions/setup-go@v2
with:
go-version: 1.18.3
- name: Tune the OS
if: steps.changes.outputs.examples == 'true'
run: |
echo '1024 65535' | sudo tee -a /proc/sys/net/ipv4/ip_local_port_range
- name: Get dependencies
if: steps.changes.outputs.examples == 'true'
run: |
if [ ${{matrix.os}} = "ubuntu-latest" ]; then
# Setup MySQL 8.0
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 467B942D3A79BD29
wget -c https://dev.mysql.com/get/mysql-apt-config_0.8.20-1_all.deb
echo mysql-apt-config mysql-apt-config/select-server select mysql-8.0 | sudo debconf-set-selections
sudo DEBIAN_FRONTEND="noninteractive" dpkg -i mysql-apt-config*
sudo apt-get update
# Install everything else we need, and configure
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
elif [ ${{matrix.os}} = "macos-latest" ]; then
brew install mysql@5.7 make unzip etcd curl git wget
fi
go mod download
- name: Run make minimaltools
if: steps.changes.outputs.examples == 'true'
run: |
make minimaltools
- name: Build
if: steps.changes.outputs.examples == 'true'
run: |
make build
- name: local_example
if: steps.changes.outputs.examples == 'true'
timeout-minutes: 30
run: |
export TOPO=${{matrix.topo}}
if [ ${{matrix.os}} = "macos-latest" ]; then
export PATH="/usr/local/opt/mysql@5.7/bin:$PATH"
fi
# Make sure that testing is entirely non-reliant on config
mv config config-moved
eatmydata -- go run test.go -print-log -follow -retry=1 legacy_local_example

6
Makefile
Просмотреть файл

@ -107,21 +107,21 @@ endif
install: build
# binaries
mkdir -p "$${PREFIX}/bin"
cp "$${VTROOT}/bin/"{mysqlctl,mysqlctld,vtorc,vtadmin,vtctld,vtctlclient,vtctldclient,vtgate,vttablet,vtworker,vtbackup} "$${PREFIX}/bin/"
cp "$${VTROOT}/bin/"{mysqlctl,mysqlctld,vtorc,vtadmin,vtctld,vtctlclient,vtctldclient,vtgate,vttablet,vtbackup} "$${PREFIX}/bin/"
# Will only work inside the docker bootstrap for now
cross-install: cross-build
# binaries
mkdir -p "$${PREFIX}/bin"
# Still no vtorc for cross-compile
cp "/go/bin/${GOOS}_${GOARCH}/"{mysqlctl,mysqlctld,vtadmin,vtctld,vtctlclient,vtctldclient,vtgate,vttablet,vtworker,vtbackup} "$${PREFIX}/bin/"
cp "/go/bin/${GOOS}_${GOARCH}/"{mysqlctl,mysqlctld,vtadmin,vtctld,vtctlclient,vtctldclient,vtgate,vttablet,vtbackup} "$${PREFIX}/bin/"
# Install local install the binaries needed to run vitess locally
# Usage: make install-local PREFIX=/path/to/install/root
install-local: build
# binaries
mkdir -p "$${PREFIX}/bin"
cp "$${VTROOT}/bin/"{mysqlctl,mysqlctld,vtorc,vtadmin,vtctl,vtctld,vtctlclient,vtctldclient,vtgate,vttablet,vtworker,vtbackup} "$${PREFIX}/bin/"
cp "$${VTROOT}/bin/"{mysqlctl,mysqlctld,vtorc,vtadmin,vtctl,vtctld,vtctlclient,vtctldclient,vtgate,vttablet,vtbackup} "$${PREFIX}/bin/"
# install copies the files needed to run test Vitess using vtcombo into the given directory tree.

1
docker/k8s/Dockerfile
Просмотреть файл

@ -49,7 +49,6 @@ COPY --from=base /vt/bin/vtctl /vt/bin/
COPY --from=base /vt/bin/vtctlclient /vt/bin/
COPY --from=base /vt/bin/vtgate /vt/bin/
COPY --from=base /vt/bin/vttablet /vt/bin/
COPY --from=base /vt/bin/vtworker /vt/bin/
COPY --from=base /vt/bin/vtbackup /vt/bin/
COPY --from=base /vt/bin/vtadmin /vt/bin/

38
docker/k8s/vtworker/Dockerfile
Просмотреть файл

@ -1,38 +0,0 @@
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ARG VT_BASE_VER=latest
ARG DEBIAN_VER=stable-slim
FROM vitess/k8s:${VT_BASE_VER} AS k8s
FROM debian:${DEBIAN_VER}
# Set up Vitess environment (just enough to run pre-built Go binaries)
ENV VTROOT /vt
# Prepare directory structure.
RUN mkdir -p /vt/bin && mkdir -p /vtdataroot
# Copy certs to allow https calls
COPY --from=k8s /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
# Copy binaries
COPY --from=k8s /vt/bin/vtworker /vt/bin/
# add vitess user/group and add permissions
RUN groupadd -r --gid 2000 vitess && \
useradd -r -g vitess --uid 1000 vitess && \
chown -R vitess:vitess /vt && \
chown -R vitess:vitess /vtdataroot

5
docker/release.sh
Просмотреть файл

@ -56,11 +56,6 @@ do
docker push vitess/vtctld:$vt_base_version-$debian_version
if [[ $debian_version == $default_debian_version ]]; then docker push vitess/vtctld:$vt_base_version; fi
docker build --platform linux/amd64 --build-arg VT_BASE_VER=$vt_base_version --build-arg DEBIAN_VER=$debian_version-slim -t vitess/vtworker:$vt_base_version-$debian_version k8s/vtworker
docker tag vitess/vtworker:$vt_base_version-$debian_version vitess/vtworker:$vt_base_version
docker push vitess/vtworker:$vt_base_version-$debian_version
if [[ $debian_version == $default_debian_version ]]; then docker push vitess/vtworker:$vt_base_version; fi
docker build --platform linux/amd64 --build-arg VT_BASE_VER=$vt_base_version --build-arg DEBIAN_VER=$debian_version-slim -t vitess/logrotate:$vt_base_version-$debian_version k8s/logrotate
docker tag vitess/logrotate:$vt_base_version-$debian_version vitess/logrotate:$vt_base_version
docker push vitess/logrotate:$vt_base_version-$debian_version

13
examples/compose/docker-compose.yml
Просмотреть файл

@ -408,17 +408,4 @@ services:
- "3306"
volumes:
- .:/script
vtwork:
command:
- sh
- -c
- '/vt/bin/vtworker -topo_implementation consul -topo_global_server_address consul1:8500
-topo_global_root vitess/global -cell test -logtostderr=true -service_map ''grpc-vtworker''
-port 8080 -grpc_port 15999 -use_v3_resharding_mode=true '
depends_on:
- vtctld
image: vitess/lite:${VITESS_TAG:-latest}
ports:
- "8080"
- "15999"
version: "2.1"

14
examples/compose/vtcompose/docker-compose.test.yml
Просмотреть файл

@ -301,18 +301,4 @@ services:
- "3306"
volumes:
- .:/script
vtwork:
command:
- sh
- -c
- '$$VTROOT/bin/vtworker -topo_implementation consul -topo_global_server_address
consul1:8500 -topo_global_root vitess/global -cell test -logtostderr=true -service_map
''grpc-vtworker'' -port 8080 -grpc_port 15999 -use_v3_resharding_mode=true -pid_file
$$VTDATAROOT/tmp/vtwork.pid '
depends_on:
- vtctld
image: vitess/base
ports:
- 15100:8080
- "15999"
version: "2.1"

24
examples/compose/vtcompose/vtcompose.go
Просмотреть файл

@ -481,7 +481,6 @@ func applyDefaultDockerPatches(
dockerComposeFile = applyInMemoryPatch(dockerComposeFile, generateVtctld(opts))
dockerComposeFile = applyInMemoryPatch(dockerComposeFile, generateVtgate(opts))
dockerComposeFile = applyInMemoryPatch(dockerComposeFile, generateVtwork(opts))
dockerComposeFile = applyInMemoryPatch(dockerComposeFile, generateVreplication(dbInfo, opts))
dockerComposeFile = applyInMemoryPatch(dockerComposeFile, generateVtorc(dbInfo, keyspaceInfoMap, opts))
return dockerComposeFile
@ -724,29 +723,6 @@ func generateVtgate(opts vtOptions) string {
`, opts.webPort, opts.gRpcPort, opts.mySqlPort, opts.topologyFlags, opts.cell)
}
func generateVtwork(opts vtOptions) string {
return fmt.Sprintf(`
- op: add
path: /services/vtwork
value:
image: vitess/lite:${VITESS_TAG:-latest}
ports:
- "%[1]d"
- "%[2]d"
command: ["sh", "-c", "/vt/bin/vtworker \
%[3]s \
-cell %[4]s \
-logtostderr=true \
-service_map 'grpc-vtworker' \
-port %[1]d \
-grpc_port %[2]d \
-use_v3_resharding_mode=true \
"]
depends_on:
- vtctld
`, opts.webPort, opts.gRpcPort, opts.topologyFlags, opts.cell)
}
func generateVtorc(dbInfo externalDbInfo, keyspaceInfoMap map[string]keyspaceInfo, opts vtOptions) string {
externalDb := "0"
if dbInfo.dbName != "" {

53
examples/legacy_local/101_initial_cluster.sh
Просмотреть файл

@ -1,53 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script brings up zookeeper and all the vitess components
# required for a single shard deployment.
source ./env.sh
# start topo server
if [ "${TOPO}" = "zk2" ]; then
CELL=zone1 ./scripts/zk-up.sh
elif [ "${TOPO}" = "k8s" ]; then
CELL=zone1 ./scripts/k3s-up.sh
else
CELL=zone1 ./scripts/etcd-up.sh
fi
# start vtctld
CELL=zone1 ./scripts/vtctld-up.sh
# start vttablets for keyspace commerce
for i in 100 101 102; do
CELL=zone1 TABLET_UID=$i ./scripts/mysqlctl-up.sh
CELL=zone1 KEYSPACE=commerce TABLET_UID=$i ./scripts/vttablet-up.sh
done
# set the correct durability policy for the keyspace
vtctldclient --server localhost:15999 SetKeyspaceDurabilityPolicy --durability-policy=semi_sync commerce
# set one of the replicas to primary
vtctlclient --server localhost:15999 InitShardPrimary -- --force commerce/0 zone1-100
# create the schema
vtctlclient --server localhost:15999 ApplySchema -- --sql-file create_commerce_schema.sql commerce
# create the vschema
vtctlclient --server localhost:15999 ApplyVSchema -- --vschema_file vschema_commerce_initial.json commerce
# start vtgate
CELL=zone1 ./scripts/vtgate-up.sh

20
examples/legacy_local/201_customer_keyspace.sh
Просмотреть файл

@ -1,20 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script creates a new keyspace in preparation for vertical resharding
vtctldclient --server localhost:15999 CreateKeyspace --served-from='primary:commerce,replica:commerce,rdonly:commerce' --durability-policy=semi_sync customer

32
examples/legacy_local/202_customer_tablets.sh
Просмотреть файл

@ -1,32 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script creates the tablets and initializes them for vertical
# resharding it also splits the vschema between the two keyspaces
# old (commerce) and new (customer)
source ./env.sh
for i in 200 201 202; do
CELL=zone1 TABLET_UID=$i ./scripts/mysqlctl-up.sh
CELL=zone1 KEYSPACE=customer TABLET_UID=$i ./scripts/vttablet-up.sh
done
vtctlclient --server localhost:15999 InitShardPrimary -- --force customer/0 zone1-200
vtctlclient --server localhost:15999 CopySchemaShard -- --tables customer,corder commerce/0 customer/0
vtctlclient --server localhost:15999 ApplyVSchema -- --vschema_file vschema_commerce_vsplit.json commerce
vtctlclient --server localhost:15999 ApplyVSchema -- --vschema_file vschema_customer_vsplit.json customer

29
examples/legacy_local/203_vertical_split.sh
Просмотреть файл

@ -1,29 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script copies over all the data from commerce keyspace to
# customer keyspace for the customer and corder tables
source ./env.sh
vtworker \
$TOPOLOGY_FLAGS \
--cell zone1 \
--log_dir "$VTDATAROOT"/tmp \
--alsologtostderr \
--use_v3_resharding_mode \
VerticalSplitClone -- --min_healthy_tablets=1 --tables=customer,corder customer/0

22
examples/legacy_local/204_vertical_migrate_replicas.sh
Просмотреть файл

@ -1,22 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script migrates traffic for the new customer keyspace to the new
# tablets of types rdonly and replica
vtctlclient --server localhost:15999 MigrateServedFrom customer/0 rdonly
vtctlclient --server localhost:15999 MigrateServedFrom customer/0 replica

20
examples/legacy_local/205_vertical_migrate_master.sh
Просмотреть файл

@ -1,20 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script migrates primary traffic for the customer keyspace to the
# new primary tablet
vtctlclient --server localhost:15999 MigrateServedFrom customer/0 primary

23
examples/legacy_local/206_clean_commerce.sh
Просмотреть файл

@ -1,23 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script removes the customer and corder tables from the commerce
# keyspace
vtctlclient --server localhost:15999 ApplySchema -- --sql-file drop_commerce_tables.sql commerce
vtctlclient --server localhost:15999 SetShardTabletControl -- --denied_tables=customer,corder --remove commerce/0 rdonly
vtctlclient --server localhost:15999 SetShardTabletControl -- --denied_tables=customer,corder --remove commerce/0 replica
vtctlclient --server localhost:15999 SetShardTabletControl -- --denied_tables=customer,corder --remove commerce/0 primary

26
examples/legacy_local/301_customer_sharded.sh
Просмотреть файл

@ -1,26 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script creates vitess sequences for the auto_increment fields
# and alters the fields to no longer be auto_increment in preparation
# for horizontal sharding
# it also changes the customer vschema from unsharded to sharded and
# sets up the necessary vindexes
vtctlclient --server localhost:15999 ApplySchema -- --sql-file create_commerce_seq.sql commerce
vtctlclient --server localhost:15999 ApplyVSchema -- --vschema_file vschema_commerce_seq.json commerce
vtctlclient --server localhost:15999 ApplySchema -- --sql-file create_customer_sharded.sql customer
vtctlclient --server localhost:15999 ApplyVSchema -- --vschema_file vschema_customer_sharded.json customer

35
examples/legacy_local/302_new_shards.sh
Просмотреть файл

@ -1,35 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script brings up new tablets for the two new shards that we will
# be creating in the customer keyspace and copies the schema
source ./env.sh
for i in 300 301 302; do
CELL=zone1 TABLET_UID=$i ./scripts/mysqlctl-up.sh
SHARD=-80 CELL=zone1 KEYSPACE=customer TABLET_UID=$i ./scripts/vttablet-up.sh
done
for i in 400 401 402; do
CELL=zone1 TABLET_UID=$i ./scripts/mysqlctl-up.sh
SHARD=80- CELL=zone1 KEYSPACE=customer TABLET_UID=$i ./scripts/vttablet-up.sh
done
vtctlclient --server localhost:15999 InitShardPrimary -- --force customer/-80 zone1-300
vtctlclient --server localhost:15999 InitShardPrimary -- --force customer/80- zone1-400
vtctlclient --server localhost:15999 CopySchemaShard customer/0 customer/-80
vtctlclient --server localhost:15999 CopySchemaShard customer/0 customer/80-

28
examples/legacy_local/303_horizontal_split.sh
Просмотреть файл

@ -1,28 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script copies the data from customer/0 to customer/-80 and customer/80-
# each row will be copied to exactly one shard based on the vindex value
source ./env.sh
vtworker \
$TOPOLOGY_FLAGS \
--cell zone1 \
--log_dir "$VTDATAROOT"/tmp \
--alsologtostderr \
--use_v3_resharding_mode \
SplitClone -- --min_healthy_rdonly_tablets=1 customer/0

20
examples/legacy_local/304_migrate_replicas.sh
Просмотреть файл

@ -1,20 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script migrates traffic for the rdonly and replica tablets
vtctlclient --server localhost:15999 MigrateServedTypes customer/0 rdonly
vtctlclient --server localhost:15999 MigrateServedTypes customer/0 replica

21
examples/legacy_local/305_migrate_master.sh
Просмотреть файл

@ -1,21 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script migrates traffic for the primary tablet
vtctlclient --server localhost:15999 MigrateServedTypes customer/0 primary
# data has been copied over to shards, and databases for the new shards are now available

22
examples/legacy_local/306_down_shard_0.sh
Просмотреть файл

@ -1,22 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script brings down the tablets for customer/0 keyspace
for i in 200 201 202; do
CELL=zone1 TABLET_UID=$i ./scripts/vttablet-down.sh
CELL=zone1 TABLET_UID=$i ./scripts/mysqlctl-down.sh
done

19
examples/legacy_local/307_delete_shard_0.sh
Просмотреть файл

@ -1,19 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# this script deletes the old shard 0 which has been replaced by 2 shards
vtctlclient --server localhost:15999 DeleteShard -- --recursive customer/0

61
examples/legacy_local/401_teardown.sh
Просмотреть файл

@ -1,61 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# We should not assume that any of the steps have been executed.
# This makes it possible for a user to cleanup at any point.
source ./env.sh
./scripts/vtgate-down.sh
for tablet in 100 200 300 400; do
if vtctlclient --server localhost:15999 GetTablet zone1-$tablet >/dev/null 2>&1 ; then
# The zero tablet is up. Try to shutdown 0-2 tablet + mysqlctl
for i in 0 1 2; do
uid=$[$tablet + $i]
CELL=zone1 TABLET_UID=$uid ./scripts/vttablet-down.sh
CELL=zone1 TABLET_UID=$uid ./scripts/mysqlctl-down.sh
done
fi
done
./scripts/vtctld-down.sh
if [ "${TOPO}" = "zk2" ]; then
CELL=zone1 ./scripts/zk-down.sh
elif [ "${TOPO}" = "k8s" ]; then
CELL=zone1 ./scripts/k3s-down.sh
else
CELL=zone1 ./scripts/etcd-down.sh
fi
# pedantic check: grep for any remaining processes
if [ ! -z "$VTDATAROOT" ]; then
if pgrep -f -l "$VTDATAROOT" > /dev/null; then
echo "ERROR: Stale processes detected! It is recommended to manuallly kill them:"
pgrep -f -l "$VTDATAROOT"
else
echo "All good! It looks like every process has shut down"
fi
# shellcheck disable=SC2086
rm -r ${VTDATAROOT:?}/*
fi
disown -a

10
examples/legacy_local/README.md
Просмотреть файл

@ -1,10 +0,0 @@
# Local Vitess Cluster
This directory contains example scripts to bring up a Vitess cluster on your
local machine, which may be useful for experimentation. These scripts can
also serve as a starting point for configuring Vitess into your preferred
deployment strategy or toolset.
See the [Run Vitess Locally](https://vitess.io/docs/tutorials/local/)
tutorial ("Start a Vitess cluster" section) for instructions on using these scripts.

6
examples/legacy_local/ceph_backup_config.json
Просмотреть файл

@ -1,6 +0,0 @@
{
"accessKey" : "AccessKey",
"secretKey" : "SecretKey",
"endPoint" : "URL",
"useSSL" : true
}

18
examples/legacy_local/create_commerce_schema.sql
Просмотреть файл

@ -1,18 +0,0 @@
create table product(
sku varbinary(128),
description varbinary(128),
price bigint,
primary key(sku)
) ENGINE=InnoDB;
create table customer(
customer_id bigint not null auto_increment,
email varbinary(128),
primary key(customer_id)
) ENGINE=InnoDB;
create table corder(
order_id bigint not null auto_increment,
customer_id bigint,
sku varbinary(128),
price bigint,
primary key(order_id)
) ENGINE=InnoDB;

4
examples/legacy_local/create_commerce_seq.sql
Просмотреть файл

@ -1,4 +0,0 @@
create table customer_seq(id int, next_id bigint, cache bigint, primary key(id)) comment 'vitess_sequence';
insert into customer_seq(id, next_id, cache) values(0, 1000, 100);
create table order_seq(id int, next_id bigint, cache bigint, primary key(id)) comment 'vitess_sequence';
insert into order_seq(id, next_id, cache) values(0, 1000, 100);

2
examples/legacy_local/create_customer_sharded.sql
Просмотреть файл

@ -1,2 +0,0 @@
alter table customer change customer_id customer_id bigint not null;
alter table corder change order_id order_id bigint not null;

7
examples/legacy_local/create_test_table.sql
Просмотреть файл

@ -1,7 +0,0 @@
CREATE TABLE messages (
page BIGINT(20) UNSIGNED,
time_created_ns BIGINT(20) UNSIGNED,
message VARCHAR(10000),
PRIMARY KEY (page, time_created_ns)
) ENGINE=InnoDB

2
examples/legacy_local/drop_commerce_tables.sql
Просмотреть файл

@ -1,2 +0,0 @@
drop table customer;
drop table corder;

72
examples/legacy_local/env.sh
Просмотреть файл

@ -1,72 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -e
hostname=`hostname -f`
vtctld_web_port=15000
export VTDATAROOT="${VTDATAROOT:-${PWD}/vtdataroot}"
function fail() {
echo "ERROR: $1"
exit 1
}
if [[ $EUID -eq 0 ]]; then
fail "This script refuses to be run as root. Please switch to a regular user."
fi
# mysqld might be in /usr/sbin which will not be in the default PATH
PATH="/usr/sbin:$PATH"
for binary in mysqld etcd etcdctl curl vtctlclient vttablet vtgate vtctld vtctl mysqlctl; do
command -v "$binary" > /dev/null || fail "${binary} is not installed in PATH. See https://vitess.io/docs/get-started/local/ for install instructions."
done;
if [ "${TOPO}" = "zk2" ]; then
# Each ZooKeeper server needs a list of all servers in the quorum.
# Since we're running them all locally, we need to give them unique ports.
# In a real deployment, these should be on different machines, and their
# respective hostnames should be given.
zkcfg=(\
"1@$hostname:28881:38881:21811" \
"2@$hostname:28882:38882:21812" \
"3@$hostname:28883:38883:21813" \
)
printf -v zkcfg ",%s" "${zkcfg[@]}"
zkcfg=${zkcfg:1}
zkids='1 2 3'
# Set topology environment parameters.
ZK_SERVER="localhost:21811,localhost:21812,localhost:21813"
# shellcheck disable=SC2034
TOPOLOGY_FLAGS="--topo_implementation zk2 --topo_global_server_address ${ZK_SERVER} --topo_global_root /vitess/global"
elif [ "${TOPO}" = "k8s" ]; then
# Set topology environment parameters.
K8S_ADDR="localhost"
K8S_PORT="8443"
K8S_KUBECONFIG=$VTDATAROOT/tmp/k8s.kubeconfig
# shellcheck disable=SC2034
TOPOLOGY_FLAGS="--topo_implementation k8s --topo_k8s_kubeconfig ${K8S_KUBECONFIG} --topo_global_server_address ${K8S_ADDR}:${K8S_PORT} --topo_global_root /vitess/global"
else
ETCD_SERVER="localhost:2379"
TOPOLOGY_FLAGS="--topo_implementation etcd2 --topo_global_server_address $ETCD_SERVER --topo_global_root /vitess/global"
mkdir -p "${VTDATAROOT}/etcd"
fi
# Create a tmp dir
mkdir -p "${VTDATAROOT}/tmp"

6
examples/legacy_local/grpc_static_auth.json
Просмотреть файл

@ -1,6 +0,0 @@
[
{
"Username": "vitess",
"Password": "vitess_password"
}
]

4
examples/legacy_local/grpc_static_client_auth.json
Просмотреть файл

@ -1,4 +0,0 @@
{
"Username": "vitess",
"Password": "vitess_password"
}

33
examples/legacy_local/mysql_auth_server_static_creds.json
Просмотреть файл

@ -1,33 +0,0 @@
{
"mysql_user": [
{
"MysqlNativePassword": "*9E128DA0C64A6FCCCDCFBDD0FC0A2C967C6DB36F",
"Password": "mysql_password",
"UserData": "mysql_user"
}
],
"mysql_user2": [
{
"Password": "mysql_password",
"UserData": "mysql_user"
}
],
"mysql_user3": [
{
"MysqlNativePassword": "*9E128DA0C64A6FCCCDCFBDD0FC0A2C967C6DB36F",
"UserData": "mysql_user"
}
],
"vt_appdebug": [
{
"Password": "vtappdebug_password",
"UserData": "vt_appdebug"
},
{
"SourceHost": "localhost",
"Password": "",
"UserData": "vt_appdebug"
}
]
}

22
examples/legacy_local/scripts/etcd-down.sh
Просмотреть файл

@ -1,22 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops the etcd servers started by etcd-up.sh.
source ./env.sh
echo "Stopping etcd..."
kill -9 `cat $VTDATAROOT/tmp/etcd.pid`

51
examples/legacy_local/scripts/etcd-up.sh
Просмотреть файл

@ -1,51 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a quorum of Etcd servers.
source ./env.sh
cell=${CELL:-'test'}
export ETCDCTL_API=2
# Check that etcd is not already running
curl "http://${ETCD_SERVER}" > /dev/null 2>&1 && fail "etcd is already running. Exiting."
etcd --enable-v2=true --data-dir "${VTDATAROOT}/etcd/" --listen-client-urls "http://${ETCD_SERVER}" --advertise-client-urls "http://${ETCD_SERVER}" > "${VTDATAROOT}"/tmp/etcd.out 2>&1 &
PID=$!
echo $PID > "${VTDATAROOT}/tmp/etcd.pid"
sleep 5
echo "add /vitess/global"
etcdctl --endpoints "http://${ETCD_SERVER}" mkdir /vitess/global &
echo "add /vitess/$cell"
etcdctl --endpoints "http://${ETCD_SERVER}" mkdir /vitess/$cell &
# And also add the CellInfo description for the cell.
# If the node already exists, it's fine, means we used existing data.
echo "add $cell CellInfo"
set +e
# shellcheck disable=SC2086
vtctl $TOPOLOGY_FLAGS AddCellInfo -- \
--root /vitess/$cell \
--server_address "${ETCD_SERVER}" \
$cell
set -e
echo "etcd start done..."

30
examples/legacy_local/scripts/k3s-down.sh
Просмотреть файл

@ -1,30 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops the k3s server started by k3s-up.sh.
set -e
# shellcheck source=./env.sh
# shellcheck disable=SC1091
source ./env.sh
# Stop K3s server.
echo "Stopping k3s server..."
pid=`cat $VTDATAROOT/tmp/k3s.pid`
echo "Stopping k3s..."
kill -9 $pid

49
examples/legacy_local/scripts/k3s-up.sh
Просмотреть файл

@ -1,49 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a Kubernetes api for topo use by running k3s
set -e
cell=${CELL:-'test'}
script_root=$(dirname "${BASH_SOURCE[0]}")
# shellcheck source=./env.sh
# shellcheck disable=SC1091
source ./env.sh
k3s server --disable-agent --data-dir "${VTDATAROOT}/k3s/" --https-listen-port "${K8S_PORT}" --write-kubeconfig "${K8S_KUBECONFIG}" > "${VTDATAROOT}"/tmp/k3s.out 2>&1 &
PID=$!
echo $PID > "${VTDATAROOT}/tmp/k3s.pid"
disown -a
echo "Waiting for k3s server to start"
sleep 15
# Use k3s built-in kubectl with custom config
KUBECTL="k3s kubectl --kubeconfig=${K8S_KUBECONFIG}"
# Create the CRD for vitesstopologynodes
$KUBECTL create -f ../../go/vt/topo/k8stopo/VitessTopoNodes-crd.yaml
# Add the CellInfo description for the cell
set +e
echo "add $cell CellInfo"
vtctl $TOPOLOGY_FLAGS AddCellInfo -- \
--root /vitess/$cell \
$cell
set -e
echo "k3s start done..."

28
examples/legacy_local/scripts/mysqlctl-down.sh
Просмотреть файл

@ -1,28 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops the mysqld and vttablet instances
# created by vttablet-up.sh
source ./env.sh
cell=${CELL:-'test'}
uid=$TABLET_UID
printf -v alias '%s-%010d' $cell $uid
echo "Shutting down MySQL for tablet $alias..."
mysqlctl --tablet_uid $TABLET_UID shutdown

42
examples/legacy_local/scripts/mysqlctl-up.sh
Просмотреть файл

@ -1,42 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a single shard vttablet deployment.
source ./env.sh
cell=${CELL:-'test'}
uid=$TABLET_UID
mysql_port=$[17000 + $uid]
printf -v alias '%s-%010d' $cell $uid
printf -v tablet_dir 'vt_%010d' $uid
mkdir -p $VTDATAROOT/backups
echo "Starting MySQL for tablet $alias..."
action="init"
if [ -d $VTDATAROOT/$tablet_dir ]; then
echo "Resuming from existing vttablet dir:"
echo " $VTDATAROOT/$tablet_dir"
action='start'
fi
mysqlctl \
--log_dir $VTDATAROOT/tmp \
--tablet_uid $uid \
--mysql_port $mysql_port \
$action

22
examples/legacy_local/scripts/vtctld-down.sh
Просмотреть файл

@ -1,22 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops vtctld.
source ./env.sh
echo "Stopping vtctld..."
kill -9 `cat $VTDATAROOT/tmp/vtctld.pid`

38
examples/legacy_local/scripts/vtctld-up.sh
Просмотреть файл

@ -1,38 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that starts vtctld.
source ./env.sh
cell=${CELL:-'test'}
grpc_port=15999
echo "Starting vtctld..."
# shellcheck disable=SC2086
vtctld \
$TOPOLOGY_FLAGS \
--cell $cell \
--workflow_manager_init \
--workflow_manager_use_election \
--service_map 'grpc-vtctl,grpc-vtctld' \
--backup_storage_implementation file \
--file_backup_storage_root $VTDATAROOT/backups \
--log_dir $VTDATAROOT/tmp \
--port $vtctld_web_port \
--grpc_port $grpc_port \
--pid_file $VTDATAROOT/tmp/vtctld.pid \
> $VTDATAROOT/tmp/vtctld.out 2>&1 &

23
examples/legacy_local/scripts/vtgate-down.sh
Просмотреть файл

@ -1,23 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops the instance started by vtgate-up.sh.
source ./env.sh
# Stop vtgate.
echo "Stopping vtgate..."
kill `cat $VTDATAROOT/tmp/vtgate.pid`

57
examples/legacy_local/scripts/vtgate-up.sh
Просмотреть файл

@ -1,57 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that starts a single vtgate.
source ./env.sh
cell=${CELL:-'test'}
web_port=15001
grpc_port=15991
mysql_server_port=15306
mysql_server_socket_path="/tmp/mysql.sock"
# Start vtgate.
# shellcheck disable=SC2086
vtgate \
$TOPOLOGY_FLAGS \
--log_dir $VTDATAROOT/tmp \
--log_queries_to_file $VTDATAROOT/tmp/vtgate_querylog.txt \
--port $web_port \
--grpc_port $grpc_port \
--mysql_server_port $mysql_server_port \
--mysql_server_socket_path $mysql_server_socket_path \
--cell $cell \
--cells_to_watch $cell \
--tablet_types_to_wait PRIMARY,REPLICA \
--service_map 'grpc-vtgateservice' \
--pid_file $VTDATAROOT/tmp/vtgate.pid \
--mysql_auth_server_impl none \
> $VTDATAROOT/tmp/vtgate.out 2>&1 &
# Block waiting for vtgate to be listening
# Not the same as healthy
echo "Waiting for vtgate to be up..."
while true; do
curl -I "http://$hostname:$web_port/debug/status" >/dev/null 2>&1 && break
sleep 0.1
done;
echo "vtgate is up!"
echo "Access vtgate at http://$hostname:$web_port/debug/status"
disown -a

30
examples/legacy_local/scripts/vttablet-down.sh
Просмотреть файл

@ -1,30 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops the mysqld and vttablet instances
# created by vttablet-up.sh
source ./env.sh
printf -v tablet_dir 'vt_%010d' $TABLET_UID
pid=`cat $VTDATAROOT/$tablet_dir/vttablet.pid`
kill $pid
# Wait for vttablet to die.
while ps -p $pid > /dev/null; do sleep 1; done

69
examples/legacy_local/scripts/vttablet-up.sh
Просмотреть файл

@ -1,69 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
source ./env.sh
cell=${CELL:-'test'}
keyspace=${KEYSPACE:-'test_keyspace'}
shard=${SHARD:-'0'}
uid=$TABLET_UID
mysql_port=$[17000 + $uid]
port=$[15000 + $uid]
grpc_port=$[16000 + $uid]
printf -v alias '%s-%010d' $cell $uid
printf -v tablet_dir 'vt_%010d' $uid
tablet_hostname=''
printf -v tablet_logfile 'vttablet_%010d_querylog.txt' $uid
tablet_type=replica
if [[ "${uid: -1}" -gt 1 ]]; then
tablet_type=rdonly
fi
echo "Starting vttablet for $alias..."
# shellcheck disable=SC2086
vttablet \
$TOPOLOGY_FLAGS \
--log_dir $VTDATAROOT/tmp \
--log_queries_to_file $VTDATAROOT/tmp/$tablet_logfile \
--tablet-path $alias \
--tablet_hostname "$tablet_hostname" \
--init_keyspace $keyspace \
--init_shard $shard \
--init_tablet_type $tablet_type \
--health_check_interval 5s \
--enable_semi_sync \
--enable_replication_reporter \
--backup_storage_implementation file \
--file_backup_storage_root $VTDATAROOT/backups \
--restore_from_backup \
--port $port \
--grpc_port $grpc_port \
--service_map 'grpc-queryservice,grpc-tabletmanager,grpc-updatestream' \
--pid_file $VTDATAROOT/$tablet_dir/vttablet.pid \
--vtctld_addr http://$hostname:$vtctld_web_port/ \
> $VTDATAROOT/$tablet_dir/vttablet.out 2>&1 &
# Block waiting for the tablet to be listening
# Not the same as healthy
for i in $(seq 0 300); do
curl -I "http://$hostname:$port/debug/status" >/dev/null 2>&1 && break
sleep 0.1
done
# check one last time
curl -I "http://$hostname:$port/debug/status" || fail "tablet could not be started!"

26
examples/legacy_local/scripts/zk-down.sh
Просмотреть файл

@ -1,26 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that stops the ZooKeeper servers started by zk-up.sh.
source ./env.sh
# Stop ZooKeeper servers.
echo "Stopping zk servers..."
for zkid in $zkids; do
zkctl -zk.myid $zkid -zk.cfg $zkcfg -log_dir $VTDATAROOT/tmp shutdown
done

62
examples/legacy_local/scripts/zk-up.sh
Просмотреть файл

@ -1,62 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a quorum of ZooKeeper servers.
source ./env.sh
cell=${CELL:-'test'}
# Start ZooKeeper servers.
# The "zkctl init" command won't return until the server is able to contact its
# peers, so we need to start them all in the background and then wait for them.
echo "Starting zk servers..."
for zkid in $zkids; do
action='init'
printf -v zkdir 'zk_%03d' $zkid
if [ -f $VTDATAROOT/$zkdir/myid ]; then
echo "Resuming from existing ZK data dir:"
echo " $VTDATAROOT/$zkdir"
action='start'
fi
zkctl -zk.myid $zkid -zk.cfg $zkcfg -log_dir $VTDATAROOT/tmp $action \
> $VTDATAROOT/tmp/zkctl_$zkid.out 2>&1 &
pids[$zkid]=$!
done
# Wait for all the zkctl commands to return.
echo "Waiting for zk servers to be ready..."
for zkid in $zkids; do
if ! wait ${pids[$zkid]}; then
echo "ZK server number $zkid failed to start. See log:"
echo " $VTDATAROOT/tmp/zkctl_$zkid.out"
fi
done
echo "Started zk servers."
# Add the CellInfo description for the $CELL cell.
# If the node already exists, it's fine, means we used existing data.
set +e
# shellcheck disable=SC2086
vtctl $TOPOLOGY_FLAGS AddCellInfo -- \
--root /vitess/$cell \
--server_address $ZK_SERVER \
$cell
set -e
echo "Configured zk servers."

21
examples/legacy_local/topo-etcd2.sh
Просмотреть файл

@ -1,21 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a single shard vttablet deployment.
export TOPO='etcd2'

21
examples/legacy_local/topo-k8s.sh
Просмотреть файл

@ -1,21 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a single shard vttablet deployment.
export TOPO='k8s'

21
examples/legacy_local/topo-zk2.sh
Просмотреть файл

@ -1,21 +0,0 @@
#!/bin/bash
# Copyright 2019 The Vitess Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# This is an example script that creates a single shard vttablet deployment.
export TOPO='zk2'

18
examples/legacy_local/vschema.json
Просмотреть файл

@ -1,18 +0,0 @@
{
"sharded": true,
"vindexes": {
"hash": {
"type": "hash"
}
},
"tables": {
"messages": {
"column_vindexes": [
{
"column": "page",
"name": "hash"
}
]
}
}
}

7
examples/legacy_local/vschema_commerce_initial.json
Просмотреть файл

@ -1,7 +0,0 @@
{
"tables": {
"product": {},
"customer": {},
"corder": {}
}
}

11
examples/legacy_local/vschema_commerce_seq.json
Просмотреть файл

@ -1,11 +0,0 @@
{
"tables": {
"customer_seq": {
"type": "sequence"
},
"order_seq": {
"type": "sequence"
},
"product": {}
}
}

5
examples/legacy_local/vschema_commerce_vsplit.json
Просмотреть файл

@ -1,5 +0,0 @@
{
"tables": {
"product": {}
}
}

34
examples/legacy_local/vschema_customer_sharded.json
Просмотреть файл

@ -1,34 +0,0 @@
{
"sharded": true,
"vindexes": {
"hash": {
"type": "hash"
}
},
"tables": {
"customer": {
"column_vindexes": [
{
"column": "customer_id",
"name": "hash"
}
],
"auto_increment": {
"column": "customer_id",
"sequence": "customer_seq"
}
},
"corder": {
"column_vindexes": [
{
"column": "customer_id",
"name": "hash"
}
],
"auto_increment": {
"column": "order_id",
"sequence": "order_seq"
}
}
}
}

6
examples/legacy_local/vschema_customer_vsplit.json
Просмотреть файл

@ -1,6 +0,0 @@
{
"tables": {
"customer": {},
"corder": {}
}
}

23
go/cmd/vtctld/plugin_grpcvtworkerclient.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// Imports and registers the gRPC vtworker client.
import (
_ "vitess.io/vitess/go/vt/worker/grpcvtworkerclient"
)

70
go/cmd/vtctldclient/command/keyspaces.go
Просмотреть файл

@ -26,7 +26,6 @@ import (
"vitess.io/vitess/go/cmd/vtctldclient/cli"
"vitess.io/vitess/go/vt/logutil"
"vitess.io/vitess/go/vt/topo"
"vitess.io/vitess/go/vt/topo/topoproto"
topodatapb "vitess.io/vitess/go/vt/proto/topodata"
vtctldatapb "vitess.io/vitess/go/vt/proto/vtctldata"
@ -107,24 +106,6 @@ SetKeyspaceDurabilityPolicy --durability-policy='semi_sync' customer`,
Args: cobra.ExactArgs(1),
RunE: commandSetKeyspaceDurabilityPolicy,
}
// SetKeyspaceServedFrom makes a SetKeyspaceServedFrom gRPC call to a vtcltd.
SetKeyspaceServedFrom = &cobra.Command{
Use: "SetKeyspaceServedFrom [--source <keyspace>] [--remove] [--cells=<cells>] <keyspace> <tablet_type>",
Short: "Updates the ServedFromMap for a keyspace manually. This command is intended for emergency fixes. This command does not rebuild the serving graph.",
DisableFlagsInUseLine: true,
Args: cobra.ExactArgs(2),
RunE: commandSetKeyspaceServedFrom,
Deprecated: "and will soon be removed! Please use VReplication instead: https://vitess.io/docs/reference/vreplication",
}
// SetKeyspaceShardingInfo makes a SetKeyspaceShardingInfo gRPC call to a vtcltd.
SetKeyspaceShardingInfo = &cobra.Command{
Use: "SetKeyspaceShardingInfo [--force] <keyspace> [<column name> [<column type>]]",
Short: "Updates the sharding information for a keyspace.",
DisableFlagsInUseLine: true,
Args: cobra.RangeArgs(1, 3),
RunE: commandSetKeyspaceShardingInfo,
Deprecated: "and will soon be removed! Please use VReplication instead: https://vitess.io/docs/reference/vreplication",
}
// ValidateSchemaKeyspace makes a ValidateSchemaKeyspace gRPC call to a vtctld.
ValidateSchemaKeyspace = &cobra.Command{
Use: "ValidateSchemaKeyspace [--exclude-tables=<exclude_tables>] [--include-views] [--skip-no-primary] [--include-vschema] <keyspace>",
@ -365,49 +346,6 @@ func commandSetKeyspaceDurabilityPolicy(cmd *cobra.Command, args []string) error
return nil
}
var setKeyspaceServedFromOptions = struct {
Cells []string
SourceKeyspace string
Remove bool
}{}
func commandSetKeyspaceServedFrom(cmd *cobra.Command, args []string) error {
keyspace := cmd.Flags().Arg(0)
tabletType, err := topoproto.ParseTabletType(cmd.Flags().Arg(1))
if err != nil {
return err
}
cli.FinishedParsing(cmd)
resp, err := client.SetKeyspaceServedFrom(commandCtx, &vtctldatapb.SetKeyspaceServedFromRequest{
Keyspace: keyspace,
TabletType: tabletType,
Cells: setKeyspaceServedFromOptions.Cells,
SourceKeyspace: setKeyspaceServedFromOptions.SourceKeyspace,
Remove: setKeyspaceServedFromOptions.Remove,
})
if err != nil {
return err
}
data, err := cli.MarshalJSON(resp)
if err != nil {
return err
}
fmt.Printf("%s\n", data)
return nil
}
var setKeyspaceShardingInfoOptions = struct {
Force bool
}{}
func commandSetKeyspaceShardingInfo(cmd *cobra.Command, args []string) error {
return nil
}
var validateSchemaKeyspaceOptions = struct {
ExcludeTables []string
IncludeViews bool
@ -483,17 +421,9 @@ func init() {
RemoveKeyspaceCell.Flags().BoolVarP(&removeKeyspaceCellOptions.Recursive, "recursive", "r", false, "Also delete all tablets in that cell beloning to the specified keyspace.")
Root.AddCommand(RemoveKeyspaceCell)
SetKeyspaceServedFrom.Flags().StringSliceVarP(&setKeyspaceServedFromOptions.Cells, "cells", "c", nil, "Cells to affect (comma-separated).")
SetKeyspaceServedFrom.Flags().BoolVarP(&setKeyspaceServedFromOptions.Remove, "remove", "r", false, "If set, remove the ServedFrom record.")
SetKeyspaceServedFrom.Flags().StringVar(&setKeyspaceServedFromOptions.SourceKeyspace, "source", "", "Specifies the source keyspace name.")
Root.AddCommand(SetKeyspaceServedFrom)
SetKeyspaceDurabilityPolicy.Flags().StringVar(&setKeyspaceDurabilityPolicyOptions.DurabilityPolicy, "durability-policy", "none", "Type of durability to enforce for this keyspace. Default is none. Other values include 'semi_sync' and others as dictated by registered plugins.")
Root.AddCommand(SetKeyspaceDurabilityPolicy)
SetKeyspaceShardingInfo.Flags().BoolVarP(&setKeyspaceShardingInfoOptions.Force, "force", "f", false, "Updates fields even if they are already set. Use caution before passing force to this command.")
Root.AddCommand(SetKeyspaceShardingInfo)
ValidateSchemaKeyspace.Flags().BoolVar(&validateSchemaKeyspaceOptions.IncludeViews, "include-views", false, "Includes views in compared schemas.")
ValidateSchemaKeyspace.Flags().BoolVar(&validateSchemaKeyspaceOptions.IncludeVSchema, "include-vschema", false, "Includes VSchema validation in validation results.")
ValidateSchemaKeyspace.Flags().BoolVar(&validateSchemaKeyspaceOptions.SkipNoPrimary, "skip-no-primary", false, "Skips validation on whether or not a primary exists in shards.")

5
go/cmd/vtctldclient/command/shards.go
Просмотреть файл

@ -80,11 +80,10 @@ that shard.`,
// SetShardTabletControl makes a SetShardTabletControl gRPC call to a vtctld.
SetShardTabletControl = &cobra.Command{
Use: "SetShardTabletControl [--cells=c1,c2...] [--denied-tables=t1,t2,...] [--remove] [--disable-query-service[=0|false]] <keyspace/shard> <tablet_type>",
Short: "Sets the TabletControl record for a shard and tablet type. Only use this for an emergency fix or after a finished MoveTables. The MigrateServedFrom and MigrateServedType commands set this record appropriately already.",
Short: "Sets the TabletControl record for a shard and tablet type. Only use this for an emergency fix or after a finished MoveTables.",
Long: `Sets the TabletControl record for a shard and tablet type.
Only use this for an emergency fix or after a finished MoveTables. The MigrateServedFrom
and MigrateServedType commands set this record appropriately already.
Only use this for an emergency fix or after a finished MoveTables.
Always specify the denied-tables flag for MoveTables, but never for Reshard operations.

23
go/cmd/vtworker/plugin_consultopo.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreedto in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// This plugin imports consultopo to register the consul implementation of TopoServer.
import (
_ "vitess.io/vitess/go/vt/topo/consultopo"
)

23
go/cmd/vtworker/plugin_etcd2topo.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// This plugin imports etcd2topo to register the etcd2 implementation of TopoServer.
import (
_ "vitess.io/vitess/go/vt/topo/etcd2topo"
)

23
go/cmd/vtworker/plugin_grpctabletconn.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// Imports and register the gRPC tabletconn client
import (
_ "vitess.io/vitess/go/vt/vttablet/grpctabletconn"
)

23
go/cmd/vtworker/plugin_grpcthrottlerserver.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// Imports and register the gRPC throttler server.
import (
_ "vitess.io/vitess/go/vt/throttler/grpcthrottlerserver"
)

23
go/cmd/vtworker/plugin_grpctmclient.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// Imports and register the gRPC tabletmanager client
import (
_ "vitess.io/vitess/go/vt/vttablet/grpctmclient"
)

30
go/cmd/vtworker/plugin_grpcvtworkerserver.go
Просмотреть файл

@ -1,30 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import (
"vitess.io/vitess/go/vt/servenv"
"vitess.io/vitess/go/vt/worker/grpcvtworkerserver"
)
func init() {
servenv.OnRun(func() {
if servenv.GRPCCheckServiceMap("vtworker") {
grpcvtworkerserver.StartServer(servenv.GRPCServer, wi)
}
})
}

23
go/cmd/vtworker/plugin_kubernetestopo.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2020 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// This plugin imports k8stopo to register the kubernetes implementation of TopoServer.
import (
_ "vitess.io/vitess/go/vt/topo/k8stopo"
)

30
go/cmd/vtworker/plugin_opentracing.go
Просмотреть файл

@ -1,30 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import (
"vitess.io/vitess/go/trace"
"vitess.io/vitess/go/vt/servenv"
)
func init() {
servenv.OnInit(func() {
closer := trace.StartTracing("vtworker")
servenv.OnClose(trace.LogErrorsWhenClosing(closer))
})
}

27
go/cmd/vtworker/plugin_opentsdb.go
Просмотреть файл

@ -1,27 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// This plugin imports opentsdb to register the opentsdb stats backend.
import (
"vitess.io/vitess/go/stats/opentsdb"
)
func init() {
opentsdb.Init("vtworker")
}

31
go/cmd/vtworker/plugin_prometheusbackend.go
Просмотреть файл

@ -1,31 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// This plugin imports Prometheus to allow for instrumentation
// with the Prometheus client library
import (
"vitess.io/vitess/go/stats/prometheusbackend"
"vitess.io/vitess/go/vt/servenv"
)
func init() {
servenv.OnRun(func() {
prometheusbackend.Init("vtworker")
})
}

23
go/cmd/vtworker/plugin_zk2topo.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreedto in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// Imports and register the zk2 TopologyServer
import (
_ "vitess.io/vitess/go/vt/topo/zk2topo"
)

128
go/cmd/vtworker/vtworker.go
Просмотреть файл

@ -1,128 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
vtworker is the main program to run a worker job.
It has two modes: single command or interactive.
- in single command, it will start the job passed in from the command line,
and exit.
- in interactive mode, use a web browser to start an action.
*/
package main
import (
"context"
"flag"
"io"
"os"
"time"
"vitess.io/vitess/go/exit"
"vitess.io/vitess/go/vt/callerid"
"vitess.io/vitess/go/vt/log"
"vitess.io/vitess/go/vt/logutil"
"vitess.io/vitess/go/vt/servenv"
"vitess.io/vitess/go/vt/topo"
"vitess.io/vitess/go/vt/worker"
// Include deprecation warnings for soon-to-be-unsupported flag invocations.
_flag "vitess.io/vitess/go/internal/flag"
)
var (
cell = flag.String("cell", "", "cell to pick servers from")
commandDisplayInterval = flag.Duration("command_display_interval", time.Second, "Interval between each status update when vtworker is executing a single command from the command line")
username = flag.String("username", "", "If set, value is set as immediate caller id in the request and used by vttablet for TableACL check")
_ = flag.String("durability_policy", "none", "type of durability to enforce. Default is none. Other values are dictated by registered plugins")
)
func init() {
servenv.RegisterDefaultFlags()
logger := logutil.NewConsoleLogger()
logger.Printf("*** This is a legacy sharding tool that will soon be removed! Please use the MoveTables or Reshard commands instead: https://vitess.io/docs/reference/vreplication/ ***\n")
flag.CommandLine.SetOutput(logutil.NewLoggerWriter(logger))
_flag.SetUsage(flag.CommandLine, _flag.UsageOptions{
Preface: func(w io.Writer) {
logger.Printf("Usage: %s [global parameters] command [command parameters]\n", os.Args[0])
logger.Printf("\nThe global optional parameters are:\n")
},
Epilogue: func(w io.Writer) {
logger.Printf("\nThe commands are listed below, sorted by group. Use '%s <command> -h' for more help.\n\n", os.Args[0])
worker.PrintAllCommands(logger)
},
})
}
var (
wi *worker.Instance
)
func main() {
defer exit.Recover()
_flag.Parse()
args := _flag.Args()
servenv.Init()
defer servenv.Close()
if *servenv.Version {
servenv.AppVersion.Print()
os.Exit(0)
}
ts := topo.Open()
defer ts.Close()
wi = worker.NewInstance(ts, *cell, *commandDisplayInterval)
wi.InstallSignalHandlers()
wi.InitStatusHandling()
if len(args) == 0 {
// In interactive mode, initialize the web UI to choose a command.
wi.InitInteractiveMode()
} else {
// In single command mode, just run it.
ctx := context.Background()
if *username != "" {
ctx = callerid.NewContext(ctx,
callerid.NewEffectiveCallerID("vtworker", "" /* component */, "" /* subComponent */),
callerid.NewImmediateCallerID(*username))
}
worker, done, err := wi.RunCommand(ctx, args, nil /*custom wrangler*/, true /*runFromCli*/)
if err != nil {
log.Error(err)
exit.Return(1)
}
// Run the subsequent, blocking wait asynchronously.
go func() {
if err := wi.WaitForCommand(worker, done); err != nil {
log.Error(err)
logutil.Flush()
// We cannot use exit.Return() here because we are in a different go routine now.
os.Exit(1)
}
logutil.Flush()
os.Exit(0)
}()
}
servenv.RunDefault()
}

23
go/cmd/vtworkerclient/plugin_grpcvtworkerclient.go
Просмотреть файл

@ -1,23 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
// Imports and registers the gRPC vtworker client.
import (
_ "vitess.io/vitess/go/vt/worker/grpcvtworkerclient"
)

64
go/cmd/vtworkerclient/vtworkerclient.go
Просмотреть файл

@ -1,64 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package main
import (
"context"
"flag"
"os"
"os/signal"
"syscall"
"vitess.io/vitess/go/vt/log"
"vitess.io/vitess/go/vt/logutil"
"vitess.io/vitess/go/vt/worker/vtworkerclient"
logutilpb "vitess.io/vitess/go/vt/proto/logutil"
// Include deprecation warnings for soon-to-be-unsupported flag invocations.
_flag "vitess.io/vitess/go/internal/flag"
)
var (
server = flag.String("server", "", "server to use for connection")
)
func main() {
_flag.Parse()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT)
go func() {
s := <-sigChan
log.Errorf("Trying to cancel current command after receiving signal: %v", s)
cancel()
}()
logger := logutil.NewConsoleLogger()
err := vtworkerclient.RunCommandAndWait(
ctx, *server, _flag.Args(),
func(e *logutilpb.Event) {
logutil.LogEvent(logger, e)
})
if err != nil {
log.Error(err)
os.Exit(1)
}
}

2
go/flags/endtoend/vtgate.txt
Просмотреть файл

@ -278,7 +278,7 @@ Usage of vtgate:
--security_policy string
the name of a registered security policy to use for controlling access to URLs - empty means allow all for anyone (built-in policies: deny-all, read-only)
--service_map value
comma separated list of services to enable (or disable if prefixed with '-') Example: grpc-vtworker
comma separated list of services to enable (or disable if prefixed with '-') Example: grpc-queryservice
--sql-max-length-errors int
truncate queries in error logs to the given length (default unlimited)
--sql-max-length-ui int

4
go/flags/endtoend/vttablet.txt
Просмотреть файл

@ -55,8 +55,6 @@ Usage of vttablet:
PITR restore parameter: Filename containing mTLS client private key for use in binlog server authentication.
--binlog_ssl_server_name string
PITR restore parameter: TLS server name (common name) to verify against for the binlog server we are connecting to (If not set: use the hostname or IP supplied in -binlog_host).
--binlog_use_v3_resharding_mode
(DEPRECATED) True if and only if the binlog streamer should use V3-style sharding, which doesn't require a preset sharding key column. (default true)
--binlog_user string
PITR restore parameter: username of binlog server.
--builtinbackup_mysqld_timeout duration
@ -734,7 +732,7 @@ Usage of vttablet:
--security_policy string
the name of a registered security policy to use for controlling access to URLs - empty means allow all for anyone (built-in policies: deny-all, read-only)
--service_map value
comma separated list of services to enable (or disable if prefixed with '-') Example: grpc-vtworker
comma separated list of services to enable (or disable if prefixed with '-') Example: grpc-queryservice
--serving_state_grace_period duration
how long to pause after broadcasting health to vtgate, before enforcing a new serving state
--shard_sync_retry_delay duration

5
go/test/endtoend/backup/transform/backup_transform_utils.go
Просмотреть файл

@ -30,7 +30,6 @@ import (
"github.com/stretchr/testify/require"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
"vitess.io/vitess/go/vt/log"
)
@ -87,11 +86,11 @@ func TestMainSetup(m *testing.M, useMysqlctld bool) {
}
shard := &localCluster.Keyspaces[0].Shards[0]
// changing password for mysql user
dbCredentialFile = initialsharding.WriteDbCredentialToTmp(localCluster.TmpDirectory)
dbCredentialFile = cluster.WriteDbCredentialToTmp(localCluster.TmpDirectory)
initDb, _ := os.ReadFile(path.Join(os.Getenv("VTROOT"), "/config/init_db.sql"))
sql := string(initDb)
newInitDBFile = path.Join(localCluster.TmpDirectory, "init_db_with_passwords.sql")
sql = sql + initialsharding.GetPasswordUpdateSQL(localCluster)
sql = sql + cluster.GetPasswordUpdateSQL(localCluster)
os.WriteFile(newInitDBFile, []byte(sql), 0666)
extraArgs := []string{"--db-credentials-file", dbCredentialFile}

5
go/test/endtoend/backup/vtbackup/main_test.go
Просмотреть файл

@ -25,7 +25,6 @@ import (
"testing"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
"vitess.io/vitess/go/vt/log"
)
@ -87,11 +86,11 @@ func TestMain(m *testing.M) {
// Create a new init_db.sql file that sets up passwords for all users.
// Then we use a db-credentials-file with the passwords.
dbCredentialFile = initialsharding.WriteDbCredentialToTmp(localCluster.TmpDirectory)
dbCredentialFile = cluster.WriteDbCredentialToTmp(localCluster.TmpDirectory)
initDb, _ := os.ReadFile(path.Join(os.Getenv("VTROOT"), "/config/init_db.sql"))
sql := string(initDb)
newInitDBFile = path.Join(localCluster.TmpDirectory, "init_db_with_passwords.sql")
sql = sql + initialsharding.GetPasswordUpdateSQL(localCluster)
sql = sql + cluster.GetPasswordUpdateSQL(localCluster)
err = os.WriteFile(newInitDBFile, []byte(sql), 0666)
if err != nil {
return 1, err

6
go/test/endtoend/backup/vtctlbackup/backup_utils.go
Просмотреть файл

@ -28,8 +28,6 @@ import (
"testing"
"time"
"vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
"vitess.io/vitess/go/vt/mysqlctl"
"vitess.io/vitess/go/vt/proto/topodata"
@ -103,11 +101,11 @@ func LaunchCluster(setupType int, streamMode string, stripes int) (int, error) {
}
shard := &localCluster.Keyspaces[0].Shards[0]
dbCredentialFile = initialsharding.WriteDbCredentialToTmp(localCluster.TmpDirectory)
dbCredentialFile = cluster.WriteDbCredentialToTmp(localCluster.TmpDirectory)
initDb, _ := os.ReadFile(path.Join(os.Getenv("VTROOT"), "/config/init_db.sql"))
sql := string(initDb)
newInitDBFile = path.Join(localCluster.TmpDirectory, "init_db_with_passwords.sql")
sql = sql + initialsharding.GetPasswordUpdateSQL(localCluster)
sql = sql + cluster.GetPasswordUpdateSQL(localCluster)
err = os.WriteFile(newInitDBFile, []byte(sql), 0666)
if err != nil {
return 1, err

27
go/test/endtoend/cellalias/cell_alias_test.go
Просмотреть файл

@ -34,7 +34,6 @@ import (
"vitess.io/vitess/go/sqltypes"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/sharding"
"vitess.io/vitess/go/vt/proto/topodata"
)
@ -241,8 +240,8 @@ func TestAlias(t *testing.T) {
expectedPartitions[topodata.TabletType_PRIMARY] = []string{shard1.Name, shard2.Name}
expectedPartitions[topodata.TabletType_REPLICA] = []string{shard1.Name, shard2.Name}
expectedPartitions[topodata.TabletType_RDONLY] = []string{shard1.Name, shard2.Name}
sharding.CheckSrvKeyspace(t, cell1, keyspaceName, expectedPartitions, *localCluster)
sharding.CheckSrvKeyspace(t, cell2, keyspaceName, expectedPartitions, *localCluster)
cluster.CheckSrvKeyspace(t, cell1, keyspaceName, expectedPartitions, *localCluster)
cluster.CheckSrvKeyspace(t, cell2, keyspaceName, expectedPartitions, *localCluster)
// Adds alias so vtgate can route to replica/rdonly tablets that are not in the same cell, but same alias
err = localCluster.VtctlclientProcess.ExecuteCommand("AddCellsAlias", "--",
@ -305,8 +304,8 @@ func TestAddAliasWhileVtgateUp(t *testing.T) {
expectedPartitions[topodata.TabletType_PRIMARY] = []string{shard1.Name, shard2.Name}
expectedPartitions[topodata.TabletType_REPLICA] = []string{shard1.Name, shard2.Name}
expectedPartitions[topodata.TabletType_RDONLY] = []string{shard1.Name, shard2.Name}
sharding.CheckSrvKeyspace(t, cell1, keyspaceName, expectedPartitions, *localCluster)
sharding.CheckSrvKeyspace(t, cell2, keyspaceName, expectedPartitions, *localCluster)
cluster.CheckSrvKeyspace(t, cell1, keyspaceName, expectedPartitions, *localCluster)
cluster.CheckSrvKeyspace(t, cell2, keyspaceName, expectedPartitions, *localCluster)
vtgateInstance := localCluster.NewVtgateInstance()
vtgateInstance.CellsToWatch = allCells
@ -363,39 +362,39 @@ func testQueriesOnTabletType(t *testing.T, tabletType string, vtgateGrpcPort int
}
func insertInitialValues(t *testing.T) {
sharding.ExecuteOnTablet(t,
fmt.Sprintf(sharding.InsertTabletTemplateKsID, tableName, 1, "msg1", 1),
cluster.ExecuteOnTablet(t,
fmt.Sprintf(cluster.InsertTabletTemplateKsID, tableName, 1, "msg1", 1),
*shard1Primary,
keyspaceName,
false)
sharding.ExecuteOnTablet(t,
fmt.Sprintf(sharding.InsertTabletTemplateKsID, tableName, 2, "msg2", 2),
cluster.ExecuteOnTablet(t,
fmt.Sprintf(cluster.InsertTabletTemplateKsID, tableName, 2, "msg2", 2),
*shard1Primary,
keyspaceName,
false)
sharding.ExecuteOnTablet(t,
fmt.Sprintf(sharding.InsertTabletTemplateKsID, tableName, 4, "msg4", 4),
cluster.ExecuteOnTablet(t,
fmt.Sprintf(cluster.InsertTabletTemplateKsID, tableName, 4, "msg4", 4),
*shard2Primary,
keyspaceName,
false)
}
func deleteInitialValues(t *testing.T) {
sharding.ExecuteOnTablet(t,
cluster.ExecuteOnTablet(t,
fmt.Sprintf("delete from %s where id = %v", tableName, 1),
*shard1Primary,
keyspaceName,
false)
sharding.ExecuteOnTablet(t,
cluster.ExecuteOnTablet(t,
fmt.Sprintf("delete from %s where id = %v", tableName, 2),
*shard1Primary,
keyspaceName,
false)
sharding.ExecuteOnTablet(t,
cluster.ExecuteOnTablet(t,
fmt.Sprintf("delete from %s where id = %v", tableName, 4),
*shard2Primary,
keyspaceName,

17
go/test/endtoend/cluster/cluster_process.go
Просмотреть файл

@ -87,7 +87,6 @@ type LocalProcessCluster struct {
TopoProcess TopoProcess
VtctldProcess VtctldProcess
VtgateProcess VtgateProcess
VtworkerProcess VtworkerProcess
VtbackupProcess VtbackupProcess
VtorcProcesses []*VtorcProcess
@ -828,22 +827,6 @@ func (cluster *LocalProcessCluster) waitForMySQLProcessToExit(mysqlctlProcessLis
wg.Wait()
}
// StartVtworker starts a vtworker
func (cluster *LocalProcessCluster) StartVtworker(cell string, extraArgs ...string) error {
httpPort := cluster.GetAndReservePort()
grpcPort := cluster.GetAndReservePort()
log.Infof("Starting vtworker with http_port=%d, grpc_port=%d", httpPort, grpcPort)
cluster.VtworkerProcess = *VtworkerProcessInstance(
httpPort,
grpcPort,
cluster.TopoPort,
cluster.Hostname,
cluster.TmpDirectory)
cluster.VtworkerProcess.ExtraArgs = extraArgs
return cluster.VtworkerProcess.Setup(cell)
}
// StartVtbackup starts a vtbackup
func (cluster *LocalProcessCluster) StartVtbackup(newInitDBFile string, initalBackup bool,
keyspace string, shard string, cell string, extraArgs ...string) error {

119
go/test/endtoend/cluster/cluster_util.go
Просмотреть файл

@ -20,24 +20,28 @@ import (
"context"
"fmt"
"os"
"path"
"reflect"
"strings"
"testing"
"time"
replicationdatapb "vitess.io/vitess/go/vt/proto/replicationdata"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"vitess.io/vitess/go/json2"
"vitess.io/vitess/go/mysql"
tabletpb "vitess.io/vitess/go/vt/proto/topodata"
"vitess.io/vitess/go/vt/vtgate/vtgateconn"
replicationdatapb "vitess.io/vitess/go/vt/proto/replicationdata"
topodatapb "vitess.io/vitess/go/vt/proto/topodata"
tmc "vitess.io/vitess/go/vt/vttablet/grpctmclient"
)
var (
tmClient = tmc.NewClient()
tmClient = tmc.NewClient()
dbCredentialFile string
InsertTabletTemplateKsID = `insert into %s (id, msg) values (%d, '%s') /* id:%d */`
)
// Restart restarts vttablet and mysql.
@ -183,10 +187,10 @@ func ResetTabletDirectory(tablet Vttablet) error {
return tablet.MysqlctlProcess.Start()
}
func getTablet(tabletGrpcPort int, hostname string) *tabletpb.Tablet {
func getTablet(tabletGrpcPort int, hostname string) *topodatapb.Tablet {
portMap := make(map[string]int32)
portMap["grpc"] = int32(tabletGrpcPort)
return &tabletpb.Tablet{Hostname: hostname, PortMap: portMap}
return &topodatapb.Tablet{Hostname: hostname, PortMap: portMap}
}
func filterResultForWarning(input string) string {
@ -292,3 +296,104 @@ func filterDoubleDashArgs(args []string, version int) (filtered []string) {
return filtered
}
// WriteDbCredentialToTmp writes JSON formatted db credentials to the
// specified tmp directory.
func WriteDbCredentialToTmp(tmpDir string) string {
data := []byte(`{
"vt_dba": ["VtDbaPass"],
"vt_app": ["VtAppPass"],
"vt_allprivs": ["VtAllprivsPass"],
"vt_repl": ["VtReplPass"],
"vt_filtered": ["VtFilteredPass"]
}`)
dbCredentialFile = path.Join(tmpDir, "db_credentials.json")
os.WriteFile(dbCredentialFile, data, 0666)
return dbCredentialFile
}
// GetPasswordUpdateSQL returns the SQL for updating the users' passwords
// to the static creds used throughout tests.
func GetPasswordUpdateSQL(localCluster *LocalProcessCluster) string {
pwdChangeCmd := `
# Set real passwords for all users.
UPDATE mysql.user SET %s = PASSWORD('RootPass')
WHERE User = 'root' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtDbaPass')
WHERE User = 'vt_dba' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtAppPass')
WHERE User = 'vt_app' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtAllprivsPass')
WHERE User = 'vt_allprivs' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtReplPass')
WHERE User = 'vt_repl' AND Host = '%%';
UPDATE mysql.user SET %s = PASSWORD('VtFilteredPass')
WHERE User = 'vt_filtered' AND Host = 'localhost';
FLUSH PRIVILEGES;
`
pwdCol, _ := getPasswordField(localCluster)
return fmt.Sprintf(pwdChangeCmd, pwdCol, pwdCol, pwdCol, pwdCol, pwdCol, pwdCol)
}
// getPasswordField determines which column is used for user passwords in this MySQL version.
func getPasswordField(localCluster *LocalProcessCluster) (pwdCol string, err error) {
tablet := &Vttablet{
Type: "relpica",
TabletUID: 100,
MySQLPort: 15000,
MysqlctlProcess: *MysqlCtlProcessInstance(100, 15000, localCluster.TmpDirectory),
}
if err = tablet.MysqlctlProcess.Start(); err != nil {
return "", err
}
tablet.VttabletProcess = VttabletProcessInstance(tablet.HTTPPort, tablet.GrpcPort, tablet.TabletUID, "", "", "", 0, tablet.Type, localCluster.TopoPort, "", "", nil, false, localCluster.DefaultCharset)
result, err := tablet.VttabletProcess.QueryTablet("select password from mysql.user limit 0", "", false)
if err == nil && len(result.Rows) > 0 {
return "password", nil
}
tablet.MysqlctlProcess.Stop()
os.RemoveAll(path.Join(tablet.VttabletProcess.Directory))
return "authentication_string", nil
}
// CheckSrvKeyspace confirms that the cell and keyspace contain the expected
// shard mappings.
func CheckSrvKeyspace(t *testing.T, cell string, ksname string, expectedPartition map[topodatapb.TabletType][]string, ci LocalProcessCluster) {
srvKeyspace := GetSrvKeyspace(t, cell, ksname, ci)
currentPartition := map[topodatapb.TabletType][]string{}
for _, partition := range srvKeyspace.Partitions {
currentPartition[partition.ServedType] = []string{}
for _, shardRef := range partition.ShardReferences {
currentPartition[partition.ServedType] = append(currentPartition[partition.ServedType], shardRef.Name)
}
}
assert.True(t, reflect.DeepEqual(currentPartition, expectedPartition))
}
// GetSrvKeyspace returns the SrvKeyspace structure for the cell and keyspace.
func GetSrvKeyspace(t *testing.T, cell string, ksname string, ci LocalProcessCluster) *topodatapb.SrvKeyspace {
output, err := ci.VtctlclientProcess.ExecuteCommandWithOutput("GetSrvKeyspace", cell, ksname)
require.Nil(t, err)
var srvKeyspace topodatapb.SrvKeyspace
err = json2.Unmarshal([]byte(output), &srvKeyspace)
require.Nil(t, err)
return &srvKeyspace
}
// ExecuteOnTablet executes a query on the specified vttablet.
// It should always be called with a primary tablet for a keyspace/shard.
func ExecuteOnTablet(t *testing.T, query string, vttablet Vttablet, ks string, expectFail bool) {
_, _ = vttablet.VttabletProcess.QueryTablet("begin", ks, true)
_, err := vttablet.VttabletProcess.QueryTablet(query, ks, true)
if expectFail {
require.Error(t, err)
} else {
require.Nil(t, err)
}
_, _ = vttablet.VttabletProcess.QueryTablet("commit", ks, true)
}

3
go/test/endtoend/cluster/vtctld_process.go
Просмотреть файл

@ -64,9 +64,6 @@ func (vtctld *VtctldProcess) Setup(cell string, extraArgs ...string) (err error)
"--service_map", vtctld.ServiceMap,
"--backup_storage_implementation", vtctld.BackupStorageImplementation,
"--file_backup_storage_root", vtctld.FileBackupStorageRoot,
// hard-code these two soon-to-be deprecated drain values.
"--wait_for_drain_sleep_rdonly", "1s",
"--wait_for_drain_sleep_replica", "1s",
"--log_dir", vtctld.LogDir,
"--port", fmt.Sprintf("%d", vtctld.Port),
"--grpc_port", fmt.Sprintf("%d", vtctld.GrpcPort),

2
go/test/endtoend/cluster/vtgate_process.go
Просмотреть файл

@ -236,7 +236,7 @@ func VtgateProcessInstance(
Binary: "vtgate",
FileToLogQueries: path.Join(tmpDirectory, "/vtgate_querylog.txt"),
Directory: os.Getenv("VTDATAROOT"),
ServiceMap: "grpc-tabletmanager,grpc-throttler,grpc-queryservice,grpc-updatestream,grpc-vtctl,grpc-vtworker,grpc-vtgateservice",
ServiceMap: "grpc-tabletmanager,grpc-throttler,grpc-queryservice,grpc-updatestream,grpc-vtctl,grpc-vtgateservice",
LogDir: tmpDirectory,
Port: port,
GrpcPort: grpcPort,

233
go/test/endtoend/cluster/vtworker_process.go
Просмотреть файл

@ -1,233 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cluster
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"os/exec"
"strings"
"syscall"
"time"
"vitess.io/vitess/go/vt/log"
)
// VtworkerProcess is a generic handle for a running vtworker .
// It can be spawned manually
type VtworkerProcess struct {
Name string
Binary string
CommonArg VtctlProcess
ServiceMap string
LogDir string
Port int
GrpcPort int
VerifyURL string
Directory string
ExecuteRetryTime string
Cell string
Server string
CommandDisplayInterval string
ExtraArgs []string
proc *exec.Cmd
exit chan error
}
// Setup starts vtworker process with required arguements
func (vtworker *VtworkerProcess) Setup(cell string) (err error) {
vtworker.proc = exec.Command(
vtworker.Binary,
"--log_dir", vtworker.LogDir,
"--port", fmt.Sprintf("%d", vtworker.Port),
"--executefetch_retry_time", vtworker.ExecuteRetryTime,
"--tablet_manager_protocol", "grpc",
"--tablet_protocol", "grpc",
"--topo_implementation", vtworker.CommonArg.TopoImplementation,
"--topo_global_server_address", vtworker.CommonArg.TopoGlobalAddress,
"--topo_global_root", vtworker.CommonArg.TopoGlobalRoot,
"--service_map", vtworker.ServiceMap,
"--grpc_port", fmt.Sprintf("%d", vtworker.GrpcPort),
"--cell", cell,
"--command_display_interval", "10ms",
)
if *isCoverage {
vtworker.proc.Args = append(vtworker.proc.Args, "--test.coverprofile=vtworker.out", "--test.v")
}
vtworker.proc.Args = append(vtworker.proc.Args, vtworker.ExtraArgs...)
vtworker.proc.Stderr = os.Stderr
vtworker.proc.Stdout = os.Stdout
vtworker.proc.Env = append(vtworker.proc.Env, os.Environ()...)
log.Infof("%v", strings.Join(vtworker.proc.Args, " "))
err = vtworker.proc.Start()
if err != nil {
return
}
vtworker.exit = make(chan error)
go func() {
vtworker.exit <- vtworker.proc.Wait()
}()
timeout := time.Now().Add(60 * time.Second)
for time.Now().Before(timeout) {
if vtworker.IsHealthy() {
return nil
}
select {
case err := <-vtworker.exit:
return fmt.Errorf("process '%s' exited prematurely (err: %s)", vtworker.Name, err)
default:
time.Sleep(300 * time.Millisecond)
}
}
return fmt.Errorf("process '%s' timed out after 60s (err: %s)", vtworker.Name, <-vtworker.exit)
}
// IsHealthy function checks if vtworker process is up and running
func (vtworker *VtworkerProcess) IsHealthy() bool {
resp, err := http.Get(vtworker.VerifyURL)
if err != nil {
return false
}
if resp.StatusCode == 200 {
return true
}
return false
}
// TearDown shutdowns the running vtworker process
func (vtworker *VtworkerProcess) TearDown() error {
if vtworker.proc == nil || vtworker.exit == nil {
return nil
}
// Attempt graceful shutdown with SIGTERM first
vtworker.proc.Process.Signal(syscall.SIGTERM)
select {
case err := <-vtworker.exit:
vtworker.proc = nil
return err
case <-time.After(10 * time.Second):
vtworker.proc.Process.Kill()
vtworker.proc = nil
return <-vtworker.exit
}
}
// ExecuteCommand executes any vtworker command
func (vtworker *VtworkerProcess) ExecuteCommand(args ...string) (err error) {
args = append([]string{"--vtworker_client_protocol", "grpc",
"--server", vtworker.Server, "--log_dir", vtworker.LogDir, "--stderrthreshold", "info"}, args...)
if *isCoverage {
args = append([]string{"--test.coverprofile=" + getCoveragePath("vtworkerclient-exec-cmd.out")}, args...)
}
tmpProcess := exec.Command(
"vtworkerclient",
args...,
)
log.Info(fmt.Sprintf("Executing vtworkerclient with arguments %v", strings.Join(tmpProcess.Args, " ")))
return tmpProcess.Run()
}
func (vtworker *VtworkerProcess) ExecuteCommandInBg(args ...string) (*exec.Cmd, error) {
args = append([]string{"--vtworker_client_protocol", "grpc",
"--server", vtworker.Server, "--log_dir", vtworker.LogDir, "--stderrthreshold", "info"}, args...)
tmpProcess := exec.Command(
"vtworkerclient",
args...,
)
log.Info(fmt.Sprintf("Executing vtworkerclient with arguments %v", strings.Join(tmpProcess.Args, " ")))
return tmpProcess, tmpProcess.Start()
}
// ExecuteVtworkerCommand executes any vtworker command
func (vtworker *VtworkerProcess) ExecuteVtworkerCommand(port int, grpcPort int, args ...string) (err error) {
args = append([]string{
"--port", fmt.Sprintf("%d", port),
"--executefetch_retry_time", vtworker.ExecuteRetryTime,
"--tablet_manager_protocol", "grpc",
"--tablet_protocol", "grpc",
"--topo_implementation", vtworker.CommonArg.TopoImplementation,
"--topo_global_server_address", vtworker.CommonArg.TopoGlobalAddress,
"--topo_global_root", vtworker.CommonArg.TopoGlobalRoot,
"--service_map", vtworker.ServiceMap,
"--grpc_port", fmt.Sprintf("%d", grpcPort),
"--cell", vtworker.Cell,
"--log_dir", vtworker.LogDir, "--stderrthreshold", "1"}, args...)
if *isCoverage {
args = append([]string{"--test.coverprofile=" + getCoveragePath("vtworker-exec-cmd.out")}, args...)
}
tmpProcess := exec.Command(
"vtworker",
args...,
)
log.Info(fmt.Sprintf("Executing vtworker with arguments %v", strings.Join(tmpProcess.Args, " ")))
return tmpProcess.Run()
}
// VtworkerProcessInstance returns a vtworker handle
// configured with the given Config.
// The process must be manually started by calling Setup()
func VtworkerProcessInstance(httpPort int, grpcPort int, topoPort int, hostname string, tmpDirectory string) *VtworkerProcess {
vtctl := VtctlProcessInstance(topoPort, hostname)
vtworker := &VtworkerProcess{
Name: "vtworker",
Binary: "vtworker",
CommonArg: *vtctl,
ServiceMap: "grpc-tabletmanager,grpc-throttler,grpc-queryservice,grpc-updatestream,grpc-vtctl,grpc-vtworker,grpc-vtgateservice",
LogDir: tmpDirectory,
Port: httpPort,
GrpcPort: grpcPort,
ExecuteRetryTime: "1s",
CommandDisplayInterval: "10ms",
Directory: os.Getenv("VTDATAROOT"),
Server: fmt.Sprintf("%s:%d", hostname, grpcPort),
}
vtworker.VerifyURL = fmt.Sprintf("http://%s:%d/debug/vars", hostname, vtworker.Port)
return vtworker
}
// GetVars returns map of vars
func (vtworker *VtworkerProcess) GetVars() (map[string]any, error) {
resultMap := make(map[string]any)
resp, err := http.Get(vtworker.VerifyURL)
if err != nil {
return nil, fmt.Errorf("error getting response from %s", vtworker.VerifyURL)
}
if resp.StatusCode == 200 {
respByte, _ := io.ReadAll(resp.Body)
err := json.Unmarshal(respByte, &resultMap)
if err != nil {
return nil, fmt.Errorf("not able to parse response body")
}
return resultMap, nil
}
return nil, fmt.Errorf("unsuccessful response")
}

3
go/test/endtoend/keyspace/keyspace_test.go
Просмотреть файл

@ -120,9 +120,6 @@ func TestMain(m *testing.M) {
if err := clusterForKSTest.StartKeyspace(*keyspaceUnsharded, []string{keyspaceUnshardedName}, 1, false); err != nil {
return 1
}
if err := clusterForKSTest.VtctlclientProcess.ExecuteCommand("SetKeyspaceShardingInfo", "--", "--force", keyspaceUnshardedName, "keyspace_id", "uint64"); err != nil {
return 1
}
if err := clusterForKSTest.VtctlclientProcess.ExecuteCommand("RebuildKeyspaceGraph", keyspaceUnshardedName); err != nil {
return 1
}

549
go/test/endtoend/recovery/shardedrecovery/sharded_recovery_test.go
Просмотреть файл

@ -1,549 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package shardedrecovery
import (
"context"
"fmt"
"os/exec"
"strconv"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/recovery"
_ "vitess.io/vitess/go/vt/vtgate/grpcvtgateconn"
"vitess.io/vitess/go/vt/vtgate/vtgateconn"
)
var (
primary *cluster.Vttablet
replica1 *cluster.Vttablet
rdOnly *cluster.Vttablet
replica2 *cluster.Vttablet
replica3 *cluster.Vttablet
shard0Primary *cluster.Vttablet
shard0Replica *cluster.Vttablet
shard0RdOnly *cluster.Vttablet
shard1Primary *cluster.Vttablet
shard1Replica *cluster.Vttablet
shard1RdOnly *cluster.Vttablet
localCluster *cluster.LocalProcessCluster
cell = cluster.DefaultCell
hostname = "localhost"
keyspaceName = "test_keyspace"
shardName = "0"
shard0Name = "-80"
shard1Name = "80-"
commonTabletArg = []string{
"--vreplication_healthcheck_topology_refresh", "1s",
"--vreplication_healthcheck_retry_delay", "1s",
"--vreplication_retry_delay", "1s",
"--degraded_threshold", "5s",
"--lock_tables_timeout", "5s",
"--watch_replication_stream",
"--serving_state_grace_period", "1s"}
recoveryKS = "recovery_keyspace"
vtInsertTest = `create table vt_insert_test (
id bigint auto_increment,
msg varchar(64),
primary key (id)
) Engine=InnoDB`
vSchema = `{
"sharded": true,
"vindexes": {
"hash": {
"type": "hash"
}
},
"tables": {
"vt_insert_test": {
"column_vindexes": [
{
"column": "id",
"name": "hash"
}
]
}
}
}`
)
// Test recovery from backup flow.
// test_recovery will:
// - create a shard with primary and replica1 only
// - run InitShardPrimary
// - insert some data
// - take a backup
// - insert more data on the primary
// - perform a resharding
// - create a recovery keyspace
// - bring up tablet_replica2 in the new keyspace
// - check that new tablet does not have data created after backup
// - check that vtgate queries work correctly
func TestUnShardedRecoveryAfterSharding(t *testing.T) {
defer cluster.PanicHandler(t)
_, err := initializeCluster(t)
defer localCluster.Teardown()
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ApplySchema(keyspaceName, vtInsertTest)
require.NoError(t, err)
recovery.InsertData(t, primary, 1, keyspaceName)
cluster.VerifyRowsInTablet(t, replica1, keyspaceName, 1)
// insert more data on the primary
recovery.InsertData(t, primary, 2, keyspaceName)
// backup the replica
err = localCluster.VtctlclientProcess.ExecuteCommand("Backup", replica1.Alias)
require.NoError(t, err)
// check that the backup shows up in the listing
output, err := localCluster.ListBackups("test_keyspace/0")
require.NoError(t, err)
assert.Equal(t, 1, len(output))
assert.True(t, strings.HasSuffix(output[0], replica1.Alias))
// insert more data on the primary
recovery.InsertData(t, primary, 3, keyspaceName)
err = localCluster.VtctlclientProcess.ApplyVSchema(keyspaceName, vSchema)
require.NoError(t, err)
// create the split shards
for _, tablet := range []*cluster.Vttablet{shard0Primary, shard0Replica, shard0RdOnly, shard1Primary, shard1Replica, shard1RdOnly} {
tablet.VttabletProcess.ExtraArgs = []string{"--binlog_use_v3_resharding_mode=true"}
err = tablet.VttabletProcess.Setup()
require.NoError(t, err)
}
err = localCluster.VtctlProcess.ExecuteCommand("InitShardPrimary", "--", "--force", "test_keyspace/-80", shard0Primary.Alias)
require.NoError(t, err)
err = localCluster.VtctlProcess.ExecuteCommand("InitShardPrimary", "--", "--force", "test_keyspace/80-", shard1Primary.Alias)
require.NoError(t, err)
shardedTablets := []*cluster.Vttablet{shard0Primary, shard0Replica, shard0RdOnly, shard1Primary, shard1Replica, shard1RdOnly}
for _, tablet := range shardedTablets {
_ = tablet.VttabletProcess.WaitForTabletStatus("SERVING")
require.NoError(t, err)
}
for _, tablet := range shardedTablets {
assert.Equal(t, tablet.VttabletProcess.GetTabletStatus(), "SERVING")
}
// we need to create the schema, and the worker will do data copying
for _, keyspaceShard := range []string{"test_keyspace/-80", "test_keyspace/80-"} {
err = localCluster.VtctlclientProcess.ExecuteCommand("CopySchemaShard", "test_keyspace/0", keyspaceShard)
require.NoError(t, err)
}
err = localCluster.VtctlclientProcess.ExecuteCommand("SplitClone", "test_keyspace", "0", "-80,80-")
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "test_keyspace/0", "rdonly")
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "test_keyspace/0", "replica")
require.NoError(t, err)
// then serve primary from the split shards
err = localCluster.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "test_keyspace/0", "primary")
require.NoError(t, err)
// remove the original tablets in the original shard
removeTablets(t, []*cluster.Vttablet{primary, replica1, rdOnly})
for _, tablet := range []*cluster.Vttablet{replica1, rdOnly} {
err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteTablet", tablet.Alias)
require.NoError(t, err)
}
err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteTablet", "--", "--allow_primary", primary.Alias)
require.NoError(t, err)
// rebuild the serving graph, all mentions of the old shards should be gone
err = localCluster.VtctlclientProcess.ExecuteCommand("RebuildKeyspaceGraph", "test_keyspace")
require.NoError(t, err)
// delete the original shard
err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteShard", "test_keyspace/0")
require.NoError(t, err)
// now bring up the recovery keyspace and a tablet, letting it restore from backup.
recovery.RestoreTablet(t, localCluster, replica2, recoveryKS, "0", keyspaceName, commonTabletArg)
// check the new replica does not have the data
cluster.VerifyRowsInTablet(t, replica2, keyspaceName, 2)
vtgateInstance := localCluster.NewVtgateInstance()
vtgateInstance.TabletTypesToWait = "REPLICA"
err = vtgateInstance.Setup()
localCluster.VtgateGrpcPort = vtgateInstance.GrpcPort
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard0Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", keyspaceName, shard1Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard0Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", keyspaceName, shard1Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", recoveryKS, shardName), 1)
require.NoError(t, err)
// Build vtgate grpc connection
// check that vtgate doesn't route queries to new tablet
grpcAddress := fmt.Sprintf("%s:%d", localCluster.Hostname, localCluster.VtgateGrpcPort)
vtgateConn, err := vtgateconn.Dial(context.Background(), grpcAddress)
require.NoError(t, err)
session := vtgateConn.Session("@replica", nil)
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64(3)")
// check that new tablet is accessible by using ks.table
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from recovery_keyspace.vt_insert_test", "INT64(2)")
// check that new tablet is accessible with 'use ks'
cluster.ExecuteQueriesUsingVtgate(t, session, "use `recovery_keyspace@replica`")
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64(2)")
// check that new tablet is accessible with `use ks:shard`
cluster.ExecuteQueriesUsingVtgate(t, session, "use `recovery_keyspace:0@replica`")
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64(2)")
vtgateConn.Close()
err = vtgateInstance.TearDown()
require.NoError(t, err)
}
// Test recovery from backup flow.
// test_recovery will:
// - create a shard with primary and replica1 only
// - run InitShardPrimary
// - insert some data
// - perform a resharding
// - take a backup of both new shards
// - insert more data on the primaries of both shards
// - create a recovery keyspace
// - bring up tablet_replica2 and tablet_replica3 in the new keyspace
// - check that new tablets do not have data created after backup
// - check that vtgate queries work correctly
func TestShardedRecovery(t *testing.T) {
defer cluster.PanicHandler(t)
_, err := initializeCluster(t)
defer localCluster.Teardown()
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ApplySchema(keyspaceName, vtInsertTest)
require.NoError(t, err)
recovery.InsertData(t, primary, 1, keyspaceName)
cluster.VerifyRowsInTablet(t, replica1, keyspaceName, 1)
// insert more data on the primary
recovery.InsertData(t, primary, 4, keyspaceName)
err = localCluster.VtctlclientProcess.ApplyVSchema(keyspaceName, vSchema)
require.NoError(t, err)
// create the split shards
shardedTablets := []*cluster.Vttablet{shard0Primary, shard0Replica, shard0RdOnly, shard1Primary, shard1Replica, shard1RdOnly}
for _, tablet := range shardedTablets {
tablet.VttabletProcess.ExtraArgs = []string{"--binlog_use_v3_resharding_mode=true"}
err = tablet.VttabletProcess.Setup()
require.NoError(t, err)
}
err = localCluster.VtctlProcess.ExecuteCommand("InitShardPrimary", "--", "--force", "test_keyspace/-80", shard0Primary.Alias)
require.NoError(t, err)
err = localCluster.VtctlProcess.ExecuteCommand("InitShardPrimary", "--", "--force", "test_keyspace/80-", shard1Primary.Alias)
require.NoError(t, err)
for _, tablet := range shardedTablets {
_ = tablet.VttabletProcess.WaitForTabletStatus("SERVING")
require.NoError(t, err)
}
// we need to create the schema, and the worker will do data copying
for _, keyspaceShard := range []string{"test_keyspace/-80", "test_keyspace/80-"} {
err = localCluster.VtctlclientProcess.ExecuteCommand("CopySchemaShard", "test_keyspace/0", keyspaceShard)
require.NoError(t, err)
}
err = localCluster.VtctlclientProcess.ExecuteCommand("SplitClone", "test_keyspace", "0", "-80,80-")
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "test_keyspace/0", "rdonly")
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "test_keyspace/0", "replica")
require.NoError(t, err)
// then serve primary from the split shards
err = localCluster.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "test_keyspace/0", "primary")
require.NoError(t, err)
// remove the original tablets in the original shard
removeTablets(t, []*cluster.Vttablet{primary, replica1, rdOnly})
for _, tablet := range []*cluster.Vttablet{replica1, rdOnly} {
err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteTablet", tablet.Alias)
require.NoError(t, err)
}
err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteTablet", "--", "--allow_primary", primary.Alias)
require.NoError(t, err)
// rebuild the serving graph, all mentions of the old shards should be gone
err = localCluster.VtctlclientProcess.ExecuteCommand("RebuildKeyspaceGraph", "test_keyspace")
require.NoError(t, err)
// delete the original shard
err = localCluster.VtctlclientProcess.ExecuteCommand("DeleteShard", "test_keyspace/0")
require.NoError(t, err)
qr, err := shard0Primary.VttabletProcess.QueryTablet("select count(*) from vt_insert_test", keyspaceName, true)
require.NoError(t, err)
shard0CountStr := qr.Rows[0][0].ToString()
shard0Count, err := strconv.Atoi(shard0CountStr)
require.NoError(t, err)
var shard0TestID string
if shard0Count > 0 {
qr, err := shard0Primary.VttabletProcess.QueryTablet("select id from vt_insert_test", keyspaceName, true)
require.NoError(t, err)
shard0TestID = qr.Rows[0][0].ToString()
require.NoError(t, err)
}
qr, err = shard1Primary.VttabletProcess.QueryTablet("select count(*) from vt_insert_test", keyspaceName, true)
require.NoError(t, err)
shard1CountStr := qr.Rows[0][0].ToString()
shard1Count, err := strconv.Atoi(shard1CountStr)
require.NoError(t, err)
var shard1TestID string
if shard1Count > 0 {
qr, err := shard1Primary.VttabletProcess.QueryTablet("select id from vt_insert_test", keyspaceName, true)
require.NoError(t, err)
shard1TestID = qr.Rows[0][0].ToString()
require.NoError(t, err)
}
// backup the new shards
err = localCluster.VtctlclientProcess.ExecuteCommand("Backup", shard0Replica.Alias)
require.NoError(t, err)
err = localCluster.VtctlclientProcess.ExecuteCommand("Backup", shard1Replica.Alias)
require.NoError(t, err)
// check that the backup shows up in the listing
output, err := localCluster.ListBackups("test_keyspace/-80")
require.NoError(t, err)
assert.Equal(t, 1, len(output))
assert.True(t, strings.HasSuffix(output[0], shard0Replica.Alias))
output, err = localCluster.ListBackups("test_keyspace/80-")
require.NoError(t, err)
assert.Equal(t, 1, len(output))
assert.True(t, strings.HasSuffix(output[0], shard1Replica.Alias))
vtgateInstance := localCluster.NewVtgateInstance()
vtgateInstance.TabletTypesToWait = "PRIMARY"
err = vtgateInstance.Setup()
localCluster.VtgateGrpcPort = vtgateInstance.GrpcPort
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard0Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard1Name), 1)
require.NoError(t, err)
// Build vtgate grpc connection
// check that vtgate doesn't route queries to new tablet
grpcAddress := fmt.Sprintf("%s:%d", localCluster.Hostname, localCluster.VtgateGrpcPort)
vtgateConn, err := vtgateconn.Dial(context.Background(), grpcAddress)
require.NoError(t, err)
session := vtgateConn.Session("@primary", nil)
cluster.ExecuteQueriesUsingVtgate(t, session, "insert into vt_insert_test (id, msg) values (2,'test 2')")
cluster.ExecuteQueriesUsingVtgate(t, session, "insert into vt_insert_test (id, msg) values (3,'test 3')")
vtgateConn.Close()
err = vtgateInstance.TearDown()
require.NoError(t, err)
// now bring up the recovery keyspace and 2 tablets, letting it restore from backup.
recovery.RestoreTablet(t, localCluster, replica2, recoveryKS, "-80", keyspaceName, commonTabletArg)
recovery.RestoreTablet(t, localCluster, replica3, recoveryKS, "80-", keyspaceName, commonTabletArg)
// check the new replicas have the correct number of rows
cluster.VerifyRowsInTablet(t, replica2, keyspaceName, shard0Count)
cluster.VerifyRowsInTablet(t, replica3, keyspaceName, shard1Count)
// start vtgate
vtgateInstance = localCluster.NewVtgateInstance()
vtgateInstance.TabletTypesToWait = "REPLICA"
err = vtgateInstance.Setup()
localCluster.VtgateGrpcPort = vtgateInstance.GrpcPort
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard0Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", keyspaceName, shard1Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard0Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", keyspaceName, shard1Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", recoveryKS, shard0Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", recoveryKS, shard1Name), 1)
require.NoError(t, err)
// check that vtgate doesn't route queries to new tablet
grpcAddress = fmt.Sprintf("%s:%d", localCluster.Hostname, localCluster.VtgateGrpcPort)
vtgateConn, err = vtgateconn.Dial(context.Background(), grpcAddress)
require.NoError(t, err)
session = vtgateConn.Session("@replica", nil)
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64(4)")
// check that new keyspace is accessible by using ks.table
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from recovery_keyspace.vt_insert_test", "INT64(2)")
// check that new keyspace is accessible with 'use ks'
cluster.ExecuteQueriesUsingVtgate(t, session, "use recovery_keyspace@replica")
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64(2)")
// check that new tablet is accessible with use `ks:shard`
cluster.ExecuteQueriesUsingVtgate(t, session, "use `recovery_keyspace:-80@replica`")
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64("+shard0CountStr+")")
recovery.VerifyQueriesUsingVtgate(t, session, "select id from vt_insert_test", "INT64("+shard0TestID+")")
cluster.ExecuteQueriesUsingVtgate(t, session, "use `recovery_keyspace:80-@replica`")
recovery.VerifyQueriesUsingVtgate(t, session, "select count(*) from vt_insert_test", "INT64("+shard1CountStr+")")
recovery.VerifyQueriesUsingVtgate(t, session, "select id from vt_insert_test", "INT64("+shard1TestID+")")
vtgateConn.Close()
err = vtgateInstance.TearDown()
require.NoError(t, err)
}
func removeTablets(t *testing.T, tablets []*cluster.Vttablet) {
var mysqlProcs []*exec.Cmd
for _, tablet := range tablets {
proc, _ := tablet.MysqlctlProcess.StopProcess()
mysqlProcs = append(mysqlProcs, proc)
tablet.VttabletProcess.TearDown()
}
for _, proc := range mysqlProcs {
err := proc.Wait()
require.NoError(t, err)
}
}
func initializeCluster(t *testing.T) (int, error) {
localCluster = cluster.NewCluster(cell, hostname)
// Start topo server
err := localCluster.StartTopo()
if err != nil {
return 1, err
}
// Start keyspace
keyspace := &cluster.Keyspace{
Name: keyspaceName,
}
localCluster.Keyspaces = append(localCluster.Keyspaces, *keyspace)
// TODO : handle shards properly
shard := &cluster.Shard{
Name: shardName,
}
shard0 := &cluster.Shard{
Name: shard0Name,
}
shard1 := &cluster.Shard{
Name: shard1Name,
}
// Defining all the tablets
primary = localCluster.NewVttabletInstance("replica", 0, "")
replica1 = localCluster.NewVttabletInstance("replica", 0, "")
rdOnly = localCluster.NewVttabletInstance("rdonly", 0, "")
replica2 = localCluster.NewVttabletInstance("replica", 0, "")
replica3 = localCluster.NewVttabletInstance("replica", 0, "")
shard0Primary = localCluster.NewVttabletInstance("replica", 0, "")
shard0Replica = localCluster.NewVttabletInstance("replica", 0, "")
shard0RdOnly = localCluster.NewVttabletInstance("rdonly", 0, "")
shard1Primary = localCluster.NewVttabletInstance("replica", 0, "")
shard1Replica = localCluster.NewVttabletInstance("replica", 0, "")
shard1RdOnly = localCluster.NewVttabletInstance("rdonly", 0, "")
shard.Vttablets = []*cluster.Vttablet{primary, replica1, rdOnly, replica2, replica3}
shard0.Vttablets = []*cluster.Vttablet{shard0Primary, shard0Replica, shard0RdOnly}
shard1.Vttablets = []*cluster.Vttablet{shard1Primary, shard1Replica, shard1RdOnly}
localCluster.VtTabletExtraArgs = append(localCluster.VtTabletExtraArgs, commonTabletArg...)
localCluster.VtTabletExtraArgs = append(localCluster.VtTabletExtraArgs, "--restore_from_backup", "--enable_semi_sync")
err = localCluster.SetupCluster(keyspace, []cluster.Shard{*shard, *shard0, *shard1})
require.NoError(t, err)
vtctldClientProcess := cluster.VtctldClientProcessInstance("localhost", localCluster.VtctldProcess.GrpcPort, localCluster.TmpDirectory)
out, err := vtctldClientProcess.ExecuteCommandWithOutput("SetKeyspaceDurabilityPolicy", keyspaceName, "--durability-policy=semi_sync")
require.NoError(t, err, out)
// Start MySql
var mysqlCtlProcessList []*exec.Cmd
for _, shard := range localCluster.Keyspaces[0].Shards {
for _, tablet := range shard.Vttablets {
if proc, err := tablet.MysqlctlProcess.StartProcess(); err != nil {
t.Fatal(err)
} else {
mysqlCtlProcessList = append(mysqlCtlProcessList, proc)
}
}
}
// Wait for mysql processes to start
for _, proc := range mysqlCtlProcessList {
if err := proc.Wait(); err != nil {
t.Fatal(err)
}
}
for _, tablet := range []cluster.Vttablet{*primary, *replica1, *rdOnly} {
if err = tablet.VttabletProcess.CreateDB(keyspaceName); err != nil {
return 1, err
}
if err = tablet.VttabletProcess.Setup(); err != nil {
return 1, err
}
}
if err = localCluster.VtctlclientProcess.InitShardPrimary(keyspaceName, shard.Name, cell, primary.TabletUID); err != nil {
return 1, err
}
return 0, nil
}

5
go/test/endtoend/recovery/unshardedrecovery/recovery.go
Просмотреть файл

@ -30,7 +30,6 @@ import (
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/recovery"
"vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
"vitess.io/vitess/go/vt/log"
"vitess.io/vitess/go/vt/vtgate/vtgateconn"
)
@ -92,11 +91,11 @@ func TestMainImpl(m *testing.M) {
}
localCluster.Keyspaces = append(localCluster.Keyspaces, *keyspace)
dbCredentialFile = initialsharding.WriteDbCredentialToTmp(localCluster.TmpDirectory)
dbCredentialFile = cluster.WriteDbCredentialToTmp(localCluster.TmpDirectory)
initDb, _ := os.ReadFile(path.Join(os.Getenv("VTROOT"), "/config/init_db.sql"))
sql := string(initDb)
newInitDBFile = path.Join(localCluster.TmpDirectory, "init_db_with_passwords.sql")
sql = sql + initialsharding.GetPasswordUpdateSQL(localCluster)
sql = sql + cluster.GetPasswordUpdateSQL(localCluster)
// https://github.com/vitessio/vitess/issues/8315
oldAlterTableMode := `
SET GLOBAL old_alter_table = ON;

491
go/test/endtoend/sharding/base_sharding.go
Просмотреть файл

@ -1,491 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package sharding
import (
"context"
"fmt"
"math"
"net/http"
"path"
"reflect"
"strconv"
"strings"
"testing"
"time"
"vitess.io/vitess/go/sqltypes"
"vitess.io/vitess/go/mysql"
"vitess.io/vitess/go/vt/log"
"vitess.io/vitess/go/json2"
"vitess.io/vitess/go/test/endtoend/cluster"
querypb "vitess.io/vitess/go/vt/proto/query"
"vitess.io/vitess/go/vt/proto/topodata"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
var (
lotRange1 uint64 = 0xA000000000000000
lotRange2 uint64 = 0xE000000000000000
// InsertTabletTemplateKsID common insert format to be used for different tests
InsertTabletTemplateKsID = `insert into %s (id, msg) values (%d, '%s') /* id:%d */`
)
const (
maxRowsToFetch = 10000
)
// CheckSrvKeyspace verifies the schema with expectedPartition
func CheckSrvKeyspace(t *testing.T, cell string, ksname string, expectedPartition map[topodata.TabletType][]string, ci cluster.LocalProcessCluster) {
srvKeyspace := GetSrvKeyspace(t, cell, ksname, ci)
currentPartition := map[topodata.TabletType][]string{}
for _, partition := range srvKeyspace.Partitions {
currentPartition[partition.ServedType] = []string{}
for _, shardRef := range partition.ShardReferences {
currentPartition[partition.ServedType] = append(currentPartition[partition.ServedType], shardRef.Name)
}
}
assert.True(t, reflect.DeepEqual(currentPartition, expectedPartition))
}
// GetSrvKeyspace return the Srv Keyspace structure
func GetSrvKeyspace(t *testing.T, cell string, ksname string, ci cluster.LocalProcessCluster) *topodata.SrvKeyspace {
output, err := ci.VtctlclientProcess.ExecuteCommandWithOutput("GetSrvKeyspace", cell, ksname)
require.Nil(t, err)
var srvKeyspace topodata.SrvKeyspace
err = json2.Unmarshal([]byte(output), &srvKeyspace)
require.Nil(t, err)
return &srvKeyspace
}
// VerifyTabletHealth checks that the tablet URL is reachable.
func VerifyTabletHealth(t *testing.T, vttablet cluster.Vttablet, hostname string) {
tabletURL := fmt.Sprintf("http://%s:%d/healthz", hostname, vttablet.HTTPPort)
resp, err := http.Get(tabletURL)
require.Nil(t, err)
assert.Equal(t, resp.StatusCode, 200)
}
// CheckValues check value from sql query to table with expected values
func CheckValues(t *testing.T, vttablet cluster.Vttablet, id uint64, msg string, exists bool, tableName string, ks string, keyType querypb.Type, dbConn *mysql.Conn) bool {
query := fmt.Sprintf("select id, msg from %s where id = %d", tableName, id)
if keyType == querypb.Type_VARBINARY {
query = fmt.Sprintf("select id, msg from %s where id = '%d'", tableName, id)
}
var result *sqltypes.Result
if dbConn != nil {
r1, err := dbConn.ExecuteFetch(query, maxRowsToFetch, true)
require.Nil(t, err)
result = r1
} else {
r2, err := vttablet.VttabletProcess.QueryTablet(query, ks, true)
require.Nil(t, err)
result = r2
}
isFound := false
if exists && len(result.Rows) > 0 {
if keyType == querypb.Type_VARBINARY {
isFound = assert.Equal(t, fmt.Sprintf("%v", result.Rows), fmt.Sprintf(`[[VARBINARY("%d") VARCHAR("%s")]]`, id, msg))
} else {
isFound = assert.Equal(t, fmt.Sprintf("%v", result.Rows), fmt.Sprintf(`[[UINT64(%d) VARCHAR("%s")]]`, id, msg))
}
} else {
assert.Equal(t, len(result.Rows), 0)
}
return isFound
}
// CheckDestinationPrimary performs multiple checks on a destination primary.
func CheckDestinationPrimary(t *testing.T, vttablet cluster.Vttablet, sourceShards []string, ci cluster.LocalProcessCluster) {
_ = vttablet.VttabletProcess.WaitForBinLogPlayerCount(len(sourceShards))
CheckBinlogPlayerVars(t, vttablet, sourceShards, 0)
checkStreamHealthEqualsBinlogPlayerVars(t, vttablet, len(sourceShards), ci)
}
// CheckBinlogPlayerVars Checks the binlog player variables are correctly exported.
func CheckBinlogPlayerVars(t *testing.T, vttablet cluster.Vttablet, sourceShards []string, replicationLagSeconds int64) {
tabletVars := vttablet.VttabletProcess.GetVars()
assert.Contains(t, tabletVars, "VReplicationStreamCount")
assert.Contains(t, tabletVars, "VReplicationLagSecondsMax")
assert.Contains(t, tabletVars, "VReplicationLagSeconds")
assert.Contains(t, tabletVars, "VReplicationSource")
assert.Contains(t, tabletVars, "VReplicationSourceTablet")
streamCountStr := fmt.Sprintf("%v", reflect.ValueOf(tabletVars["VReplicationStreamCount"]))
streamCount, _ := strconv.Atoi(streamCountStr)
assert.Equal(t, streamCount, len(sourceShards))
replicationSourceObj := reflect.ValueOf(tabletVars["VReplicationSource"])
replicationSourceValue := []string{}
assert.Equal(t, len(replicationSourceObj.MapKeys()), len(reflect.ValueOf(tabletVars["VReplicationSourceTablet"]).MapKeys()))
for _, key := range replicationSourceObj.MapKeys() {
replicationSourceValue = append(replicationSourceValue,
fmt.Sprintf("%v", replicationSourceObj.MapIndex(key)))
}
for _, shard := range sourceShards {
assert.Containsf(t, replicationSourceValue, shard, "Source shard is not matched with vReplication shard value")
}
if replicationLagSeconds != 0 {
vreplicationLagMaxStr := fmt.Sprintf("%v", reflect.ValueOf(tabletVars["VReplicationLagSecondsMax"]))
vreplicationLagMax, _ := strconv.ParseFloat(vreplicationLagMaxStr, 64)
assert.True(t, vreplicationLagMax < float64(replicationLagSeconds))
replicationLagObj := reflect.ValueOf(tabletVars["VReplicationLagSeconds"])
for _, key := range replicationSourceObj.MapKeys() {
str := fmt.Sprintf("%v", replicationLagObj.MapIndex(key))
flt, _ := strconv.ParseFloat(str, 64)
assert.True(t, flt < float64(replicationLagSeconds))
}
}
}
// checkStreamHealthEqualsBinlogPlayerVars - Checks the variables exported by streaming health check match vars.
func checkStreamHealthEqualsBinlogPlayerVars(t *testing.T, vttablet cluster.Vttablet, count int, ci cluster.LocalProcessCluster) {
tabletVars := vttablet.VttabletProcess.GetVars()
streamCountStr := fmt.Sprintf("%v", reflect.ValueOf(tabletVars["VReplicationStreamCount"]))
streamCount, _ := strconv.Atoi(streamCountStr)
vreplicationLagMaxStr := fmt.Sprintf("%v", reflect.ValueOf(tabletVars["VReplicationLagSecondsMax"]))
vreplicationLagMax, _ := strconv.ParseFloat(vreplicationLagMaxStr, 64)
assert.Equal(t, streamCount, count)
// Enforce health check because it's not running by default as
// tablets may not be started with it, or may not run it in time.
_ = ci.VtctlclientProcess.ExecuteCommand("RunHealthCheck", vttablet.Alias)
streamHealth, err := ci.VtctlclientProcess.ExecuteCommandWithOutput("VtTabletStreamHealth", "--", "--count", "1", vttablet.Alias)
require.Nil(t, err)
var streamHealthResponse querypb.StreamHealthResponse
err = json2.Unmarshal([]byte(streamHealth), &streamHealthResponse)
require.Nil(t, err, "error should be Nil")
assert.Equal(t, streamHealthResponse.Serving, false)
assert.NotNil(t, streamHealthResponse.RealtimeStats)
assert.Equal(t, streamHealthResponse.RealtimeStats.HealthError, "")
assert.NotNil(t, streamHealthResponse.RealtimeStats.BinlogPlayersCount)
assert.Equal(t, streamCount, int(streamHealthResponse.RealtimeStats.BinlogPlayersCount))
assert.Equal(t, vreplicationLagMax, float64(streamHealthResponse.RealtimeStats.FilteredReplicationLagSeconds))
}
// CheckBinlogServerVars checks the binlog server variables are correctly exported.
func CheckBinlogServerVars(t *testing.T, vttablet cluster.Vttablet, minStatement int, minTxn int, isVerticalSplit bool) {
resultMap := vttablet.VttabletProcess.GetVars()
skey := "UpdateStreamKeyRangeStatements"
tkey := "UpdateStreamKeyRangeTransactions"
if isVerticalSplit {
skey = "UpdateStreamTablesStatements"
tkey = "UpdateStreamTablesTransactions"
}
assert.Contains(t, resultMap, skey)
assert.Contains(t, resultMap, tkey)
if minStatement > 0 {
value := fmt.Sprintf("%v", reflect.ValueOf(resultMap[skey]))
iValue, _ := strconv.Atoi(value)
assert.True(t, iValue >= minStatement, fmt.Sprintf("only got %d < %d statements", iValue, minStatement))
}
if minTxn > 0 {
value := fmt.Sprintf("%v", reflect.ValueOf(resultMap[tkey]))
iValue, _ := strconv.Atoi(value)
assert.True(t, iValue >= minTxn, fmt.Sprintf("only got %d < %d transactions", iValue, minTxn))
}
}
// InsertLots inserts multiple values to vttablet
func InsertLots(t *testing.T, count uint64, vttablet cluster.Vttablet, table string, ks string) {
var query1, query2 string
var i uint64
dbConn := getDBConnFromTablet(t, &vttablet, ks)
defer dbConn.Close()
for i = 0; i < count; i++ {
query1 = fmt.Sprintf(InsertTabletTemplateKsID, table, lotRange1+i, fmt.Sprintf("msg-range1-%d", 10000+i), lotRange1+i)
query2 = fmt.Sprintf(InsertTabletTemplateKsID, table, lotRange2+i, fmt.Sprintf("msg-range2-%d", 20000+i), lotRange2+i)
// insert first query
executeQueryInTransaction(t, query1, dbConn)
executeQueryInTransaction(t, query2, dbConn)
}
}
func executeQueryInTransaction(t *testing.T, query string, dbConn *mysql.Conn) {
dbConn.ExecuteFetch("begin", maxRowsToFetch, true)
_, err := dbConn.ExecuteFetch(query, maxRowsToFetch, true)
require.NoError(t, err)
dbConn.ExecuteFetch("commit", maxRowsToFetch, true)
}
// ExecuteOnTablet executes a write query on specified vttablet
// It should always be called with a primary tablet for the keyspace/shard
func ExecuteOnTablet(t *testing.T, query string, vttablet cluster.Vttablet, ks string, expectFail bool) {
_, _ = vttablet.VttabletProcess.QueryTablet("begin", ks, true)
_, err := vttablet.VttabletProcess.QueryTablet(query, ks, true)
if expectFail {
require.Error(t, err)
} else {
require.Nil(t, err)
}
_, _ = vttablet.VttabletProcess.QueryTablet("commit", ks, true)
}
// InsertMultiValues inserts a multiple values to vttablet
func InsertMultiValues(t *testing.T, tablet cluster.Vttablet, keyspaceName string, tableName string,
fixedParentID int, ids []int, msgs []string, ksIDs []uint64) {
queryStr := fmt.Sprintf("insert into %s (parent_id, id, msg, custom_ksid_col) values", tableName)
valueSQL := ""
keyspaceIds := ""
valueIds := ""
for i := range ids {
valueSQL += fmt.Sprintf(`(%d, %d, "%s", %d)`, fixedParentID, ids[i], msgs[i], ksIDs[i])
keyspaceIds += fmt.Sprintf("%d", ksIDs[i])
valueIds += fmt.Sprintf("%d", ids[i])
if i < len(ids)-1 {
valueSQL += ","
keyspaceIds += ","
valueIds += ","
}
}
queryStr += valueSQL
queryStr += fmt.Sprintf(" /* vtgate:: keyspace_id:%s */", keyspaceIds)
queryStr += fmt.Sprintf(" /* id:%s */", valueIds)
ExecuteOnTablet(t, queryStr, tablet, keyspaceName, false)
}
// CheckLotsTimeout waits till all values are inserted
func CheckLotsTimeout(t *testing.T, vttablet cluster.Vttablet, count uint64, table string, ks string, keyType querypb.Type, pctFound int) bool {
timeout := time.Now().Add(10 * time.Second)
var percentFound float64
for time.Now().Before(timeout) {
percentFound = checkLots(t, vttablet, count, table, ks, keyType)
if int(math.Round(percentFound)) == pctFound {
return true
}
time.Sleep(300 * time.Millisecond)
}
log.Infof("expected pct %d, got pct %f", pctFound, percentFound)
return false
}
func checkLots(t *testing.T, vttablet cluster.Vttablet, count uint64, table string, ks string, keyType querypb.Type) float64 {
var isFound bool
var totalFound int
var i uint64
dbConn := getDBConnFromTablet(t, &vttablet, ks)
defer dbConn.Close()
for i = 0; i < count; i++ {
isFound = CheckValues(t, vttablet,
lotRange1+i, fmt.Sprintf("msg-range1-%d", 10000+i), true, table, ks, keyType, dbConn)
if isFound {
totalFound++
}
isFound = CheckValues(t, vttablet,
lotRange2+i, fmt.Sprintf("msg-range2-%d", 20000+i), true, table, ks, keyType, dbConn)
if isFound {
totalFound++
}
}
log.Infof("Total found %d", totalFound)
return float64(float64(totalFound) * 100 / float64(count) / 2)
}
// CheckTabletQueryServices check that the query service is enabled or disabled on the specified tablets.
func CheckTabletQueryServices(t *testing.T, vttablets []cluster.Vttablet, expectedStatus string, tabletControlEnabled bool, ci cluster.LocalProcessCluster) {
for _, tablet := range vttablets {
CheckTabletQueryService(t, tablet, expectedStatus, tabletControlEnabled, ci)
}
}
// CheckTabletQueryService check that the query service is enabled or disabled on the tablet
func CheckTabletQueryService(t *testing.T, vttablet cluster.Vttablet, expectedStatus string, tabletControlEnabled bool, ci cluster.LocalProcessCluster) {
tabletStatus := vttablet.VttabletProcess.GetTabletStatus()
assert.Equal(t, tabletStatus, expectedStatus)
queryServiceDisabled := "TabletControl.DisableQueryService set"
status := vttablet.VttabletProcess.GetStatus()
if tabletControlEnabled {
assert.Contains(t, status, queryServiceDisabled)
} else {
assert.NotContains(t, status, queryServiceDisabled)
}
if vttablet.Type == "rdonly" {
// Run RunHealthCheck to be sure the tablet doesn't change its serving state.
_ = ci.VtctlclientProcess.ExecuteCommand("RunHealthCheck", vttablet.Alias)
tabletStatus = vttablet.VttabletProcess.GetTabletStatus()
assert.Equal(t, tabletStatus, expectedStatus)
}
}
// CheckShardQueryServices checks DisableQueryService for all shards
func CheckShardQueryServices(t *testing.T, ci cluster.LocalProcessCluster, shards []cluster.Shard, cell string,
keyspaceName string, tabletType topodata.TabletType, expectedState bool) {
for _, shard := range shards {
CheckShardQueryService(t, ci, cell, keyspaceName, shard.Name, tabletType, expectedState)
}
}
// CheckShardQueryService checks DisableQueryService in the shard record's TabletControlMap.
func CheckShardQueryService(t *testing.T, ci cluster.LocalProcessCluster, cell string, keyspaceName string,
shardName string, tabletType topodata.TabletType, expectedState bool) {
// We assume that query service should be enabled unless
// DisableQueryService is explicitly True
queryServiceEnabled := true
srvKeyspace := GetSrvKeyspace(t, cell, keyspaceName, ci)
for _, partition := range srvKeyspace.Partitions {
tType := partition.GetServedType()
if tabletType != tType {
continue
}
for _, shardTabletControl := range partition.GetShardTabletControls() {
if shardTabletControl.GetName() == shardName {
if shardTabletControl.GetQueryServiceDisabled() {
queryServiceEnabled = false
}
}
}
}
assert.True(t, queryServiceEnabled == expectedState,
fmt.Sprintf("shard %s does not have the correct query service state: got %t but expected %t",
shardName, queryServiceEnabled, expectedState))
}
// GetShardInfo return the Shard information
func GetShardInfo(t *testing.T, shard1Ks string, ci cluster.LocalProcessCluster) *topodata.Shard {
output, err := ci.VtctlclientProcess.ExecuteCommandWithOutput("GetShard", shard1Ks)
require.Nil(t, err)
var shard topodata.Shard
err = json2.Unmarshal([]byte(output), &shard)
require.Nil(t, err)
return &shard
}
// checkThrottlerServiceMaxRates Checks the vtctl ThrottlerMaxRates and ThrottlerSetRate commands.
func checkThrottlerServiceMaxRates(t *testing.T, server string, names []string, rate int, ci cluster.LocalProcessCluster) {
// Avoid flakes by waiting for all throttlers. (Necessary because filtered
// replication on vttablet will register the throttler asynchronously.)
var output string
var err error
startTime := time.Now()
msg := fmt.Sprintf("%d active throttler(s)", len(names))
for {
output, err = ci.VtctlclientProcess.ExecuteCommandWithOutput("ThrottlerMaxRates", "--", "--server", server)
require.Nil(t, err)
if strings.Contains(output, msg) || (time.Now().After(startTime.Add(2 * time.Minute))) {
break
}
time.Sleep(2 * time.Second)
}
assert.Contains(t, output, msg)
for _, name := range names {
str := fmt.Sprintf("| %s | %d |", name, rate)
assert.Contains(t, output, str)
}
// Check that it's possible to change the max rate on the throttler.
newRate := "unlimited"
output, err = ci.VtctlclientProcess.ExecuteCommandWithOutput("ThrottlerSetMaxRate", "--", "--server", server, newRate)
require.Nil(t, err)
assert.Contains(t, output, msg)
output, err = ci.VtctlclientProcess.ExecuteCommandWithOutput("ThrottlerMaxRates", "--", "--server", server)
require.Nil(t, err)
for _, name := range names {
str := fmt.Sprintf("| %s | %s |", name, newRate)
assert.Contains(t, output, str)
}
assert.Contains(t, output, msg)
}
// checkThrottlerServiceConfiguration checks the vtctl (Get|Update|Reset)ThrottlerConfiguration commands.
func checkThrottlerServiceConfiguration(t *testing.T, server string, names []string, ci cluster.LocalProcessCluster) {
output, err := ci.VtctlclientProcess.ExecuteCommandWithOutput(
"UpdateThrottlerConfiguration", "--", "--server", server,
"--copy_zero_values",
"target_replication_lag_sec:12345 "+
"max_replication_lag_sec:65789 "+
"initial_rate:3 max_increase:0.4 "+
"emergency_decrease:0.5 "+
"min_duration_between_increases_sec:6 "+
"max_duration_between_increases_sec:7 "+
"min_duration_between_decreases_sec:8 "+
"spread_backlog_across_sec:9 "+
"ignore_n_slowest_replicas:0 "+
"ignore_n_slowest_rdonlys:0 "+
"age_bad_rate_after_sec:12 "+
"bad_rate_increase:0.13 "+
"max_rate_approach_threshold: 0.9 ",
)
require.Nil(t, err)
msg := fmt.Sprintf("%d active throttler(s)", len(names))
assert.Contains(t, output, msg)
output, err = ci.VtctlclientProcess.ExecuteCommandWithOutput("GetThrottlerConfiguration", "--", "--server", server)
require.Nil(t, err)
for _, name := range names {
str := fmt.Sprintf("| %s | target_replication_lag_sec:12345 ", name)
assert.Contains(t, output, str)
assert.NotContains(t, output, "ignore_n_slowest_replicas")
}
assert.Contains(t, output, msg)
// Reset clears our configuration values.
output, err = ci.VtctlclientProcess.ExecuteCommandWithOutput("ResetThrottlerConfiguration", "--", "--server", server)
require.Nil(t, err)
assert.Contains(t, output, msg)
// Check that the reset configuration no longer has our values.
output, err = ci.VtctlclientProcess.ExecuteCommandWithOutput("GetThrottlerConfiguration", "--", "--server", server)
require.Nil(t, err)
assert.NotContains(t, output, "target_replication_lag_sec:12345")
assert.Contains(t, output, msg)
}
// CheckThrottlerService runs checkThrottlerServiceMaxRates and checkThrottlerServiceConfigs
func CheckThrottlerService(t *testing.T, server string, names []string, rate int, ci cluster.LocalProcessCluster) {
checkThrottlerServiceMaxRates(t, server, names, rate, ci)
checkThrottlerServiceConfiguration(t, server, names, ci)
}
func getDBConnFromTablet(t *testing.T, vttablet *cluster.Vttablet, ks string) *mysql.Conn {
dbParams := cluster.NewConnParams(vttablet.VttabletProcess.DbPort, vttablet.VttabletProcess.DbPassword, path.Join(vttablet.VttabletProcess.Directory, "mysql.sock"), ks)
dbConn, err := mysql.Connect(context.Background(), &dbParams)
require.NoError(t, err)
return dbConn
}

38
go/test/endtoend/sharding/initialsharding/bytes/initial_sharding_bytes_test.go
Просмотреть файл

@ -1,38 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""Re-runs initial_sharding_test.go with a varbinary keyspace_id."""
*/
package bytes
import (
"testing"
"vitess.io/vitess/go/test/endtoend/cluster"
sharding "vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
querypb "vitess.io/vitess/go/vt/proto/query"
)
func TestInitialShardingBytes(t *testing.T) {
defer cluster.PanicHandler(t)
code, err := sharding.ClusterWrapper(false)
if err != nil {
t.Errorf("setup failed with status code %d", code)
}
sharding.TestInitialSharding(t, &sharding.ClusterInstance.Keyspaces[0], querypb.Type_VARBINARY, false, false)
defer sharding.ClusterInstance.Teardown()
}

66
go/test/endtoend/sharding/initialsharding/multi/initial_sharding_multi_test.go
Просмотреть файл

@ -1,66 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This test simulates the first time a database has to be split
in a multi-vttablet-single-mysql environment
We have 2 keyspaces. One keyspace is in managing mode. It's vttablets
own the MySQL instances and can reparent, start/stop server, start/stop
replication etc. Other keyspace is in non-managing mode and cannot do
any of these actions. Only TabletExternallyReparented is allowed, but
resharding should still work.
For each keyspace:
- we start with a keyspace with a single shard and a single table
- we add and populate the sharding key
- we set the sharding key in the topology
- we clone into 2 instances
- we enable filtered replication
- we move all serving types
- we remove the source tablets
- we remove the original shard
*/
package multi
import (
"testing"
"vitess.io/vitess/go/vt/log"
"vitess.io/vitess/go/test/endtoend/cluster"
sharding "vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
querypb "vitess.io/vitess/go/vt/proto/query"
)
func TestInitialShardingMulti(t *testing.T) {
defer cluster.PanicHandler(t)
code, err := sharding.ClusterWrapper(true)
if err != nil {
t.Errorf("setup failed with status code %d", code)
}
sharding.AssignMysqlPortFromKs1ToKs2()
sharding.TestInitialSharding(t, &sharding.ClusterInstance.Keyspaces[0], querypb.Type_UINT64, true, false)
log.Info("-----------------------------")
log.Info("Done with 1st keyspace test")
log.Info("-----------------------------")
sharding.TestInitialSharding(t, &sharding.ClusterInstance.Keyspaces[1], querypb.Type_UINT64, true, true)
log.Info("----------Done with 2nd keyspace test----------")
sharding.KillVtgateInstances()
sharding.KillTabletsInKeyspace(&sharding.ClusterInstance.Keyspaces[0])
sharding.KillTabletsInKeyspace(&sharding.ClusterInstance.Keyspaces[1])
defer sharding.ClusterInstance.Teardown()
}

735
go/test/endtoend/sharding/initialsharding/sharding_util.go
Просмотреть файл

@ -1,735 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package initialsharding
import (
"fmt"
"os"
"os/exec"
"path"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"vitess.io/vitess/go/test/endtoend/cluster"
"vitess.io/vitess/go/test/endtoend/sharding"
querypb "vitess.io/vitess/go/vt/proto/query"
topodatapb "vitess.io/vitess/go/vt/proto/topodata"
)
var (
// ClusterInstance instance to be used for test with different params
ClusterInstance *cluster.LocalProcessCluster
hostname = "localhost"
keyspaceName1 = "ks1"
keyspaceName2 = "ks2"
dbPwd = ""
cell = "zone1"
newInitDbFile string
dbCredentialFile string
vtgateInstances []*cluster.VtgateProcess
commonTabletArg = []string{
"--vreplication_healthcheck_topology_refresh", "1s",
"--vreplication_healthcheck_retry_delay", "1s",
"--vreplication_retry_delay", "1s",
"--degraded_threshold", "5s",
"--lock_tables_timeout", "5s",
"--watch_replication_stream",
"--enable_replication_reporter",
"--serving_state_grace_period", "1s",
"--binlog_use_v3_resharding_mode=true"}
createTabletTemplate = `
create table %s(
msg varchar(64),
id bigint(20) unsigned,
primary key (id)
) Engine=InnoDB;
`
createTabletTemplateByte = `
create table %s(
msg varchar(64),
id varbinary(64),
primary key (id)
) Engine=InnoDB;
`
insertTabletTemplate = `insert into %s(id, msg) values(%d, "%s")`
tableName = "resharding1"
vSchema = `
{
"sharded": true,
"vindexes": {
"hash_index": {
"type": "hash"
}
},
"tables": {
"%s": {
"column_vindexes": [
{
"column": "%s",
"name": "hash_index"
}
]
}
}
}
`
)
// ClusterWrapper common wrapper code for cluster
func ClusterWrapper(isMulti bool) (int, error) {
ClusterInstance = nil
ClusterInstance = cluster.NewCluster(cell, hostname)
// Start topo server
if err := ClusterInstance.StartTopo(); err != nil {
return 1, err
}
if isMulti {
WriteDbCredentialToTmp(ClusterInstance.TmpDirectory)
writeInitDBFile()
dbPwd = "VtDbaPass"
}
if err := ClusterInstance.VtctlProcess.CreateKeyspace(keyspaceName1); err != nil {
return 1, err
}
ClusterInstance.Keyspaces = append(ClusterInstance.Keyspaces, cluster.Keyspace{Name: keyspaceName1})
vtctldClientProcess := cluster.VtctldClientProcessInstance("localhost", ClusterInstance.VtctldProcess.GrpcPort, ClusterInstance.TmpDirectory)
if _, err := vtctldClientProcess.ExecuteCommandWithOutput("SetKeyspaceDurabilityPolicy", keyspaceName1, "--durability-policy=semi_sync"); err != nil {
return 1, err
}
if isMulti {
if err := ClusterInstance.VtctlProcess.CreateKeyspace(keyspaceName2); err != nil {
return 1, err
}
ClusterInstance.Keyspaces = append(ClusterInstance.Keyspaces, cluster.Keyspace{Name: keyspaceName2})
if _, err := vtctldClientProcess.ExecuteCommandWithOutput("SetKeyspaceDurabilityPolicy", keyspaceName2, "--durability-policy=semi_sync"); err != nil {
return 1, err
}
}
initClusterForInitialSharding(keyspaceName1, []string{"0"}, 3, true, isMulti)
initClusterForInitialSharding(keyspaceName1, []string{"-80", "80-"}, 3, true, isMulti)
if isMulti {
initClusterForInitialSharding(keyspaceName2, []string{"0"}, 3, true, isMulti)
initClusterForInitialSharding(keyspaceName2, []string{"-80", "80-"}, 3, true, isMulti)
}
return 0, nil
}
func initClusterForInitialSharding(keyspaceName string, shardNames []string, totalTabletsRequired int, rdonly bool, isMulti bool) {
var mysqlProcesses []*exec.Cmd
var extraArgs []string
if isMulti {
extraArgs = []string{"--db-credentials-file", dbCredentialFile}
}
for _, shardName := range shardNames {
shard := &cluster.Shard{
Name: shardName,
}
for i := 0; i < totalTabletsRequired; i++ {
// instantiate vttablet object with reserved ports
var tablet *cluster.Vttablet
if i == totalTabletsRequired-1 && rdonly {
tablet = ClusterInstance.NewVttabletInstance("rdonly", 0, "")
} else if i == 0 {
tablet = ClusterInstance.NewVttabletInstance("primary", 0, "")
} else {
tablet = ClusterInstance.NewVttabletInstance("replica", 0, "")
}
// Start Mysqlctl process
tablet.MysqlctlProcess = *cluster.MysqlCtlProcessInstance(tablet.TabletUID, tablet.MySQLPort, ClusterInstance.TmpDirectory)
if isMulti {
tablet.MysqlctlProcess.InitDBFile = newInitDbFile
tablet.MysqlctlProcess.ExtraArgs = extraArgs
}
// Start Mysqlctl process, for multi keyspace we need only 1st keyspace sql procs, that is why this check is added
if keyspaceName == keyspaceName1 {
proc, err := tablet.MysqlctlProcess.StartProcess()
if err != nil {
return
}
mysqlProcesses = append(mysqlProcesses, proc)
} else { // Since we'll be using mysql procs of keyspace-1 for ks-2, resetting this to 0
tablet.MysqlctlProcess.TabletUID = 0
}
// start vttablet process
tablet.VttabletProcess = cluster.VttabletProcessInstance(
tablet.HTTPPort,
tablet.GrpcPort,
tablet.TabletUID,
ClusterInstance.Cell,
shardName,
keyspaceName,
ClusterInstance.VtctldProcess.Port,
tablet.Type,
ClusterInstance.TopoProcess.Port,
ClusterInstance.Hostname,
ClusterInstance.TmpDirectory,
ClusterInstance.VtTabletExtraArgs,
ClusterInstance.EnableSemiSync,
ClusterInstance.DefaultCharset)
tablet.Alias = tablet.VttabletProcess.TabletPath
tablet.VttabletProcess.DbPassword = dbPwd
tablet.VttabletProcess.EnableSemiSync = true
tablet.VttabletProcess.SupportsBackup = false
shard.Vttablets = append(shard.Vttablets, tablet)
}
for idx, ks := range ClusterInstance.Keyspaces {
if ks.Name == keyspaceName {
ClusterInstance.Keyspaces[idx].Shards = append(ClusterInstance.Keyspaces[idx].Shards, *shard)
}
}
}
for _, proc := range mysqlProcesses {
proc.Wait()
}
}
// AssignMysqlPortFromKs1ToKs2 assigns mysql port of all tablets of ks1 to all corresponding tablets of ks2
func AssignMysqlPortFromKs1ToKs2() {
portMap := map[string]int{}
for _, shard := range ClusterInstance.Keyspaces[0].Shards {
for _, tablet := range shard.Vttablets {
portMap[fmt.Sprintf("%s-%s", shard.Name, tablet.Type)] = tablet.MySQLPort
}
}
for _, shard := range ClusterInstance.Keyspaces[1].Shards {
for idx, tablet := range shard.Vttablets {
port := portMap[fmt.Sprintf("%s-%s", shard.Name, tablet.Type)]
shard.Vttablets[idx].MySQLPort = port
shard.Vttablets[idx].VttabletProcess.DbPort = port
}
}
}
// TestInitialSharding - main test which accepts different params for various test
func TestInitialSharding(t *testing.T, keyspace *cluster.Keyspace, keyType querypb.Type, isMulti bool, isExternal bool) {
defer cluster.PanicHandler(t)
if isExternal {
commonTabletArg = append(commonTabletArg, "--db_host", "127.0.0.1")
commonTabletArg = append(commonTabletArg, "--disable_active_reparents")
for _, shard := range keyspace.Shards {
for _, tablet := range shard.Vttablets {
tablet.VttabletProcess.ExtraArgs = append(tablet.VttabletProcess.ExtraArgs, "--db_port", fmt.Sprintf("%d", tablet.MySQLPort))
tablet.VttabletProcess.DbPassword = dbPwd
}
}
}
if isMulti {
commonTabletArg = append(commonTabletArg, "--db-credentials-file", dbCredentialFile)
}
// Start the primary and rdonly of 1st shard
shard1 := keyspace.Shards[0]
keyspaceName := keyspace.Name
shard1Ks := fmt.Sprintf("%s/%s", keyspaceName, shard1.Name)
shard1Primary := *shard1.PrimaryTablet()
if isExternal {
for _, tablet := range shard1.Vttablets {
_ = tablet.VttabletProcess.CreateDB(keyspaceName)
}
}
// primary tablet start
shard1Primary.VttabletProcess.ExtraArgs = append(shard1Primary.VttabletProcess.ExtraArgs, commonTabletArg...)
shard1.Replica().VttabletProcess.ExtraArgs = append(shard1.Replica().VttabletProcess.ExtraArgs, commonTabletArg...)
shard1.Rdonly().VttabletProcess.ExtraArgs = append(shard1.Rdonly().VttabletProcess.ExtraArgs, commonTabletArg...)
err := shard1Primary.VttabletProcess.Setup()
require.NoError(t, err)
if isExternal {
shard1.Rdonly().VttabletProcess.ServingStatus = "SERVING"
shard1.Replica().VttabletProcess.ServingStatus = "SERVING"
}
err = shard1.Rdonly().VttabletProcess.Setup()
require.NoError(t, err)
err = shard1.Replica().VttabletProcess.Setup()
require.NoError(t, err)
// reparent to make the tablets work
if !isExternal {
// reparent to make the tablets work
err = ClusterInstance.VtctlclientProcess.InitShardPrimary(keyspace.Name, shard1.Name, cell, shard1Primary.TabletUID)
require.NoError(t, err)
} else {
err = shard1.Replica().VttabletProcess.WaitForTabletStatus("SERVING")
require.NoError(t, err)
_, err = ClusterInstance.VtctlclientProcess.ExecuteCommandWithOutput("TabletExternallyReparented", shard1Primary.Alias)
require.NoError(t, err)
}
err = shard1.Replica().VttabletProcess.WaitForTabletStatus("SERVING")
require.NoError(t, err)
err = shard1.Rdonly().VttabletProcess.WaitForTabletStatus("SERVING")
require.NoError(t, err)
for _, vttablet := range shard1.Vttablets {
assert.Equal(t, vttablet.VttabletProcess.GetTabletStatus(), "SERVING")
}
// create the tables and add startup values
sqlSchemaToApply := createTabletTemplate
if keyType == querypb.Type_VARBINARY {
sqlSchemaToApply = createTabletTemplateByte
}
err = ClusterInstance.VtctlclientProcess.ApplySchema(keyspaceName, fmt.Sprintf(sqlSchemaToApply, tableName))
require.NoError(t, err)
err = ClusterInstance.VtctlclientProcess.ApplyVSchema(keyspaceName, fmt.Sprintf(vSchema, tableName, "id"))
require.NoError(t, err)
_, err = shard1Primary.VttabletProcess.QueryTablet(fmt.Sprintf(insertTabletTemplate, tableName, uint64(0x1000000000000000), "msg1"), keyspaceName, true)
require.NoError(t, err)
_, err = shard1Primary.VttabletProcess.QueryTablet(fmt.Sprintf(insertTabletTemplate, tableName, uint64(0x9000000000000000), "msg2"), keyspaceName, true)
require.NoError(t, err)
_, err = shard1Primary.VttabletProcess.QueryTablet(fmt.Sprintf(insertTabletTemplate, tableName, uint64(0xD000000000000000), "msg3"), keyspaceName, true)
require.NoError(t, err)
// reload schema on all tablets so we can query them
for _, vttablet := range shard1.Vttablets {
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("ReloadSchema", vttablet.Alias)
}
vtgateInstance := ClusterInstance.NewVtgateInstance()
vtgateInstance.MySQLServerSocketPath = path.Join(ClusterInstance.TmpDirectory, fmt.Sprintf("mysql-%s.sock", keyspaceName))
vtgateInstance.ExtraArgs = []string{"--retry-count", fmt.Sprintf("%d", 2), "--tablet_protocol", "grpc", "--normalize_queries", "--tablet_refresh_interval", "2s"}
err = vtgateInstance.Setup()
vtgateInstances = append(vtgateInstances, vtgateInstance)
require.NoError(t, err)
for _, tabletType := range []string{"primary", "replica", "rdonly"} {
if err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.%s", keyspaceName, shard1.Name, tabletType), 1); err != nil {
assert.Fail(t, err.Error())
}
}
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("RebuildKeyspaceGraph", keyspaceName)
// run a health check on source replica so it responds to discovery
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("RunHealthCheck", shard1.Replica().Alias)
require.NoError(t, err)
// create the split shards
shard21 := keyspace.Shards[1]
shard22 := keyspace.Shards[2]
for _, shard := range []cluster.Shard{shard21, shard22} {
for idx, vttablet := range shard.Vttablets {
vttablet.VttabletProcess.ExtraArgs = append(vttablet.VttabletProcess.ExtraArgs, commonTabletArg...)
if isExternal {
// We have to InitTablet upfront so that TER below can find the tablet record.
// Otherwise, there is a race where TER runs before vttablet publishes its first record.
err = ClusterInstance.VtctlclientProcess.InitTablet(vttablet, cell, keyspaceName, hostname, shard.Name)
require.Nil(t, err)
err = vttablet.VttabletProcess.CreateDB(keyspaceName)
require.NoError(t, err)
shard.Vttablets[idx].VttabletProcess.ServingStatus = ""
}
err = vttablet.VttabletProcess.Setup()
require.NoError(t, err)
}
}
if !isExternal {
_ = ClusterInstance.VtctlclientProcess.InitShardPrimary(keyspaceName, shard21.Name, cell, shard21.PrimaryTablet().TabletUID)
_ = ClusterInstance.VtctlclientProcess.InitShardPrimary(keyspaceName, shard22.Name, cell, shard22.PrimaryTablet().TabletUID)
_ = ClusterInstance.VtctlclientProcess.ApplySchema(keyspaceName, fmt.Sprintf(sqlSchemaToApply, tableName))
_ = ClusterInstance.VtctlclientProcess.ApplyVSchema(keyspaceName, fmt.Sprintf(vSchema, tableName, "id"))
for _, shard := range []cluster.Shard{shard21, shard22} {
_ = shard.Replica().VttabletProcess.WaitForTabletStatus("SERVING")
_ = shard.Rdonly().VttabletProcess.WaitForTabletStatus("SERVING")
}
for _, shard := range []cluster.Shard{shard21, shard22} {
for _, vttablet := range shard.Vttablets {
assert.Equal(t, vttablet.VttabletProcess.GetTabletStatus(), "SERVING")
}
}
} else {
_, err = ClusterInstance.VtctlclientProcess.ExecuteCommandWithOutput("TabletExternallyReparented", shard21.PrimaryTablet().Alias)
require.NoError(t, err)
_, err = ClusterInstance.VtctlclientProcess.ExecuteCommandWithOutput("TabletExternallyReparented", shard22.PrimaryTablet().Alias)
require.NoError(t, err)
}
// must restart vtgate after tablets are up, or else wait until 1min refresh
// we want cache_ttl at zero so we re-read the topology for every test query.
_ = vtgateInstance.TearDown()
_ = vtgateInstance.Setup()
// Wait for the endpoints, either local or remote.
for _, shard := range []cluster.Shard{shard1, shard21, shard22} {
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.primary", keyspaceName, shard.Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.replica", keyspaceName, shard.Name), 1)
require.NoError(t, err)
err = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.rdonly", keyspaceName, shard.Name), 1)
require.NoError(t, err)
}
// Check srv keyspace
expectedPartitions := map[topodatapb.TabletType][]string{}
expectedPartitions[topodatapb.TabletType_PRIMARY] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_REPLICA] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_RDONLY] = []string{shard1.Name}
checkSrvKeyspaceForSharding(t, keyspaceName, expectedPartitions)
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("CopySchemaShard", "--",
"--exclude_tables", "unrelated",
shard1.Rdonly().Alias, fmt.Sprintf("%s/%s", keyspaceName, shard21.Name))
require.NoError(t, err)
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("CopySchemaShard", "--",
"--exclude_tables", "unrelated",
shard1.Rdonly().Alias, fmt.Sprintf("%s/%s", keyspaceName, shard22.Name))
require.NoError(t, err)
err = ClusterInstance.StartVtworker(cell, "--use_v3_resharding_mode=true")
require.NoError(t, err)
// Initial clone (online).
_ = ClusterInstance.VtworkerProcess.ExecuteCommand("SplitClone", "--",
"--offline=false",
"--exclude_tables", "unrelated",
"--chunk_count", "10",
"--min_rows_per_chunk", "1",
"--min_healthy_rdonly_tablets", "1",
fmt.Sprintf("%s/%s", keyspaceName, shard1.Name))
// Reset vtworker such that we can run the next command.
_ = ClusterInstance.VtworkerProcess.ExecuteCommand("Reset")
// Modify the destination shard. SplitClone will revert the changes.
// Delete row 1 (provokes an insert).
_, _ = shard21.PrimaryTablet().VttabletProcess.QueryTablet(fmt.Sprintf("delete from %s where id=%d", tableName, uint64(0x1000000000000000)), keyspaceName, true)
// Delete row 2 (provokes an insert).
_, _ = shard22.PrimaryTablet().VttabletProcess.QueryTablet(fmt.Sprintf("delete from %s where id=%d", tableName, uint64(0x9000000000000000)), keyspaceName, true)
// Update row 3 (provokes an update).
_, _ = shard22.PrimaryTablet().VttabletProcess.QueryTablet(fmt.Sprintf("update %s set msg='msg-not-3' where id=%d", tableName, uint64(0xD000000000000000)), keyspaceName, true)
// Insert row 4 (provokes a delete).
var ksid uint64 = 0xD000000000000000
insertSQL := fmt.Sprintf(sharding.InsertTabletTemplateKsID, tableName, ksid, "msg4", ksid)
sharding.ExecuteOnTablet(t, insertSQL, *shard22.PrimaryTablet(), keyspaceName, true)
_ = ClusterInstance.VtworkerProcess.ExecuteCommand("SplitClone", "--",
"--exclude_tables", "unrelated",
"--chunk_count", "10",
"--min_rows_per_chunk", "1",
"--min_healthy_rdonly_tablets", "1",
fmt.Sprintf("%s/%s", keyspaceName, shard1.Name))
// check first value is in the left shard
for _, tablet := range shard21.Vttablets {
sharding.CheckValues(t, *tablet, 0x1000000000000000, "msg1", true, tableName, keyspaceName, keyType, nil)
}
for _, tablet := range shard22.Vttablets {
sharding.CheckValues(t, *tablet, 0x1000000000000000, "msg1", false, tableName, keyspaceName, keyType, nil)
}
for _, tablet := range shard21.Vttablets {
sharding.CheckValues(t, *tablet, 0x9000000000000000, "msg2", false, tableName, keyspaceName, keyType, nil)
}
for _, tablet := range shard22.Vttablets {
sharding.CheckValues(t, *tablet, 0x9000000000000000, "msg2", true, tableName, keyspaceName, keyType, nil)
}
for _, tablet := range shard21.Vttablets {
sharding.CheckValues(t, *tablet, 0xD000000000000000, "msg3", false, tableName, keyspaceName, keyType, nil)
}
for _, tablet := range shard22.Vttablets {
sharding.CheckValues(t, *tablet, 0xD000000000000000, "msg3", true, tableName, keyspaceName, keyType, nil)
}
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("ValidateSchemaKeyspace", keyspaceName)
require.NoError(t, err)
// check the binlog players are running
sharding.CheckDestinationPrimary(t, *shard21.PrimaryTablet(), []string{shard1Ks}, *ClusterInstance)
sharding.CheckDestinationPrimary(t, *shard22.PrimaryTablet(), []string{shard1Ks}, *ClusterInstance)
// check that binlog server exported the stats vars
sharding.CheckBinlogServerVars(t, *shard1.Replica(), 0, 0, false)
for _, tablet := range []cluster.Vttablet{*shard21.Rdonly(), *shard22.Rdonly()} {
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("RunHealthCheck", tablet.Alias)
require.NoError(t, err)
}
// testing filtered replication: insert a bunch of data on shard 1,
// check we get most of it after a few seconds, wait for binlog server
// timeout, check we get all of it.
sharding.InsertLots(t, 1000, shard1Primary, tableName, keyspaceName)
assert.True(t, sharding.CheckLotsTimeout(t, *shard21.Replica(), 1000, tableName, keyspaceName, keyType, 49))
assert.True(t, sharding.CheckLotsTimeout(t, *shard22.Replica(), 1000, tableName, keyspaceName, keyType, 51))
sharding.CheckDestinationPrimary(t, *shard21.PrimaryTablet(), []string{shard1Ks}, *ClusterInstance)
sharding.CheckDestinationPrimary(t, *shard22.PrimaryTablet(), []string{shard1Ks}, *ClusterInstance)
sharding.CheckBinlogServerVars(t, *shard1.Replica(), 1000, 1000, false)
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("RunHealthCheck", shard21.Rdonly().Alias)
require.NoError(t, err)
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("RunHealthCheck", shard22.Rdonly().Alias)
require.NoError(t, err)
//use vtworker to compare the data
ClusterInstance.VtworkerProcess.Cell = cell
if !isMulti {
err = ClusterInstance.VtworkerProcess.ExecuteVtworkerCommand(ClusterInstance.GetAndReservePort(),
ClusterInstance.GetAndReservePort(),
"--use_v3_resharding_mode=true",
"MultiSplitDiff",
fmt.Sprintf("%s/%s", keyspaceName, shard1.Name))
require.NoError(t, err)
for _, shard := range []string{shard21.Name, shard22.Name} {
err = ClusterInstance.VtworkerProcess.ExecuteVtworkerCommand(ClusterInstance.GetAndReservePort(),
ClusterInstance.GetAndReservePort(),
"--use_v3_resharding_mode=true",
"SplitDiff", "--",
"--min_healthy_rdonly_tablets", "1",
fmt.Sprintf("%s/%s", keyspaceName, shard))
require.NoError(t, err)
}
}
// check we can't migrate the primary just yet
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", shard1Ks, "primary")
require.Error(t, err)
// now serve rdonly from the split shards
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", shard1Ks, "rdonly")
require.NoError(t, err)
expectedPartitions = map[topodatapb.TabletType][]string{}
expectedPartitions[topodatapb.TabletType_PRIMARY] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_REPLICA] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_RDONLY] = []string{shard21.Name, shard22.Name}
checkSrvKeyspaceForSharding(t, keyspaceName, expectedPartitions)
_ = shard21.Rdonly().VttabletProcess.WaitForTabletStatus("SERVING")
_ = shard22.Rdonly().VttabletProcess.WaitForTabletStatus("SERVING")
_ = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.rdonly", keyspaceName, shard21.Name), 1)
_ = vtgateInstance.WaitForStatusOfTabletInShard(fmt.Sprintf("%s.%s.rdonly", keyspaceName, shard22.Name), 1)
//then serve replica from the split shards
sourceTablet := shard1.Replica()
destinationTablets := []cluster.Vttablet{*shard21.Replica(), *shard22.Replica()}
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", shard1Ks, "replica")
expectedPartitions = map[topodatapb.TabletType][]string{}
expectedPartitions[topodatapb.TabletType_PRIMARY] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_REPLICA] = []string{shard21.Name, shard22.Name}
expectedPartitions[topodatapb.TabletType_RDONLY] = []string{shard21.Name, shard22.Name}
checkSrvKeyspaceForSharding(t, keyspaceName, expectedPartitions)
//move replica back and forth
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", "--", "--reverse", shard1Ks, "replica")
// After a backwards migration, queryservice should be enabled on source and disabled on destinations
sharding.CheckTabletQueryService(t, *sourceTablet, "SERVING", false, *ClusterInstance)
sharding.CheckTabletQueryServices(t, destinationTablets, "NOT_SERVING", true, *ClusterInstance)
expectedPartitions = map[topodatapb.TabletType][]string{}
expectedPartitions[topodatapb.TabletType_PRIMARY] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_REPLICA] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_RDONLY] = []string{shard21.Name, shard22.Name}
checkSrvKeyspaceForSharding(t, keyspaceName, expectedPartitions)
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", shard1Ks, "replica")
// After a forwards migration, queryservice should be disabled on source and enabled on destinations
sharding.CheckTabletQueryService(t, *sourceTablet, "NOT_SERVING", true, *ClusterInstance)
sharding.CheckTabletQueryServices(t, destinationTablets, "SERVING", false, *ClusterInstance)
expectedPartitions = map[topodatapb.TabletType][]string{}
expectedPartitions[topodatapb.TabletType_PRIMARY] = []string{shard1.Name}
expectedPartitions[topodatapb.TabletType_REPLICA] = []string{shard21.Name, shard22.Name}
expectedPartitions[topodatapb.TabletType_RDONLY] = []string{shard21.Name, shard22.Name}
checkSrvKeyspaceForSharding(t, keyspaceName, expectedPartitions)
// then serve primary from the split shards
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("MigrateServedTypes", shard1Ks, "primary")
expectedPartitions = map[topodatapb.TabletType][]string{}
expectedPartitions[topodatapb.TabletType_PRIMARY] = []string{shard21.Name, shard22.Name}
expectedPartitions[topodatapb.TabletType_REPLICA] = []string{shard21.Name, shard22.Name}
expectedPartitions[topodatapb.TabletType_RDONLY] = []string{shard21.Name, shard22.Name}
checkSrvKeyspaceForSharding(t, keyspaceName, expectedPartitions)
// check the binlog players are gone now
err = shard21.PrimaryTablet().VttabletProcess.WaitForBinLogPlayerCount(0)
require.NoError(t, err)
err = shard22.PrimaryTablet().VttabletProcess.WaitForBinLogPlayerCount(0)
require.NoError(t, err)
// make sure we can't delete a shard with tablets
err = ClusterInstance.VtctlclientProcess.ExecuteCommand("DeleteShard", shard1Ks)
require.Error(t, err)
ClusterInstance.VtworkerProcess.TearDown()
if !isMulti {
KillVtgateInstances()
KillTabletsInKeyspace(keyspace)
}
}
// KillTabletsInKeyspace kill the first shard tablets in ordered way
func KillTabletsInKeyspace(keyspace *cluster.Keyspace) {
// Teardown
shard1 := keyspace.Shards[0]
var mysqlctlProcessList []*exec.Cmd
for _, tablet := range []cluster.Vttablet{*shard1.PrimaryTablet(), *shard1.Replica(), *shard1.Rdonly()} {
proc, _ := tablet.MysqlctlProcess.StopProcess()
mysqlctlProcessList = append(mysqlctlProcessList, proc)
_ = tablet.VttabletProcess.TearDown()
}
for _, proc := range mysqlctlProcessList {
proc.Wait()
}
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("DeleteTablet", shard1.Replica().Alias)
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("DeleteTablet", shard1.Rdonly().Alias)
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("DeleteTablet", "--", "--allow_primary", shard1.PrimaryTablet().Alias)
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("RebuildKeyspaceGraph", keyspace.Name)
_ = ClusterInstance.VtctlclientProcess.ExecuteCommand("DeleteShard", keyspace.Name+"/"+shard1.Name)
}
// KillVtgateInstances stops the vtgate process
func KillVtgateInstances() {
if len(vtgateInstances) > 0 {
for _, vtgateInstance := range vtgateInstances {
_ = vtgateInstance.TearDown()
}
}
}
func checkSrvKeyspaceForSharding(t *testing.T, ksName string, expectedPartitions map[topodatapb.TabletType][]string) {
sharding.CheckSrvKeyspace(t, cell, ksName, expectedPartitions, *ClusterInstance)
}
// Create a new init_db.sql file that sets up passwords for all users.
// Then we use a db-credentials-file with the passwords.
func writeInitDBFile() {
initDb, _ := os.ReadFile(path.Join(os.Getenv("VTROOT"), "/config/init_db.sql"))
sql := string(initDb)
newInitDbFile = path.Join(ClusterInstance.TmpDirectory, "init_db_with_passwords.sql")
sql = sql + GetPasswordUpdateSQL(ClusterInstance) + `
# connecting through a port requires 127.0.0.1
# --host=localhost will connect through socket
CREATE USER 'vt_dba'@'127.0.0.1' IDENTIFIED BY 'VtDbaPass';
GRANT ALL ON *.* TO 'vt_dba'@'127.0.0.1';
GRANT GRANT OPTION ON *.* TO 'vt_dba'@'127.0.0.1';
# User for app traffic, with global read-write access.
CREATE USER 'vt_app'@'127.0.0.1' IDENTIFIED BY 'VtAppPass';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE,
REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
LOCK TABLES, EXECUTE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW,
CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER
ON *.* TO 'vt_app'@'127.0.0.1';
# User for administrative operations that need to be executed as non-SUPER.
# Same permissions as vt_app here.
CREATE USER 'vt_allprivs'@'127.0.0.1' IDENTIFIED BY 'VtAllPrivsPass';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE,
REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW,
SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER
ON *.* TO 'vt_allprivs'@'127.0.0.1';
# User for Vitess VReplication (base vstreamers and vplayer).
CREATE USER 'vt_filtered'@'127.0.0.1' IDENTIFIED BY 'VtFilteredPass';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE,
REFERENCES, INDEX, ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW,
SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER
ON *.* TO 'vt_filtered'@'127.0.0.1';
FLUSH PRIVILEGES;
`
os.WriteFile(newInitDbFile, []byte(sql), 0666)
}
// WriteDbCredentialToTmp writes json format db credentials to tmp directory
func WriteDbCredentialToTmp(tmpDir string) string {
data := []byte(`{
"vt_dba": ["VtDbaPass"],
"vt_app": ["VtAppPass"],
"vt_allprivs": ["VtAllprivsPass"],
"vt_repl": ["VtReplPass"],
"vt_filtered": ["VtFilteredPass"]
}`)
dbCredentialFile = path.Join(tmpDir, "db_credentials.json")
os.WriteFile(dbCredentialFile, data, 0666)
return dbCredentialFile
}
// GetPasswordUpdateSQL returns the sql for password update
func GetPasswordUpdateSQL(localCluster *cluster.LocalProcessCluster) string {
pwdChangeCmd := `
# Set real passwords for all users.
UPDATE mysql.user SET %s = PASSWORD('RootPass')
WHERE User = 'root' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtDbaPass')
WHERE User = 'vt_dba' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtAppPass')
WHERE User = 'vt_app' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtAllprivsPass')
WHERE User = 'vt_allprivs' AND Host = 'localhost';
UPDATE mysql.user SET %s = PASSWORD('VtReplPass')
WHERE User = 'vt_repl' AND Host = '%%';
UPDATE mysql.user SET %s = PASSWORD('VtFilteredPass')
WHERE User = 'vt_filtered' AND Host = 'localhost';
FLUSH PRIVILEGES;
`
pwdCol, _ := getPasswordField(localCluster)
return fmt.Sprintf(pwdChangeCmd, pwdCol, pwdCol, pwdCol, pwdCol, pwdCol, pwdCol)
}
// getPasswordField Determines which column is used for user passwords in this MySQL version.
func getPasswordField(localCluster *cluster.LocalProcessCluster) (pwdCol string, err error) {
tablet := &cluster.Vttablet{
Type: "relpica",
TabletUID: 100,
MySQLPort: 15000,
MysqlctlProcess: *cluster.MysqlCtlProcessInstance(100, 15000, localCluster.TmpDirectory),
}
if err = tablet.MysqlctlProcess.Start(); err != nil {
return "", err
}
tablet.VttabletProcess = cluster.VttabletProcessInstance(tablet.HTTPPort, tablet.GrpcPort, tablet.TabletUID, "", "", "", 0, tablet.Type, localCluster.TopoPort, "", "", nil, false, localCluster.DefaultCharset)
result, err := tablet.VttabletProcess.QueryTablet("select password from mysql.user limit 0", "", false)
if err == nil && len(result.Rows) > 0 {
return "password", nil
}
tablet.MysqlctlProcess.Stop()
os.RemoveAll(path.Join(tablet.VttabletProcess.Directory))
return "authentication_string", nil
}

47
go/test/endtoend/sharding/initialsharding/v3/initial_sharding_test.go
Просмотреть файл

@ -1,47 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
This test simulates the first time a database has to be split.
- we start with a keyspace with a single shard and a single table
- we add and populate the sharding key
- we set the sharding key in the topology
- we clone into 2 instances
- we enable filtered replication
- we move all serving types
- we remove the source tablets
- we remove the original shard
*/
package v3
import (
"testing"
"vitess.io/vitess/go/test/endtoend/cluster"
sharding "vitess.io/vitess/go/test/endtoend/sharding/initialsharding"
querypb "vitess.io/vitess/go/vt/proto/query"
)
func TestInitialSharding(t *testing.T) {
defer cluster.PanicHandler(t)
code, err := sharding.ClusterWrapper(false)
if err != nil {
t.Errorf("setup failed with status code %d", code)
}
sharding.TestInitialSharding(t, &sharding.ClusterInstance.Keyspaces[0], querypb.Type_UINT64, false, false)
defer sharding.ClusterInstance.Teardown()
}

29
go/test/endtoend/sharding/mergesharding/int/mergesharding_int_test.go
Просмотреть файл

@ -1,29 +0,0 @@
/*
Copyright 2019 The Vitess Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package v3
import (
"testing"
sharding "vitess.io/vitess/go/test/endtoend/sharding/mergesharding"
)
// TestMergeShardingIntShardingKey - tests merge sharding using a INT column
func TestMergeShardingIntShardingKey(t *testing.T) {
sharding.TestMergesharding(t, false /* useVarbinaryShardingKeyType */)
}

Некоторые файлы не были показаны из-за слишком большого количества измененных файлов Показать больше