Partitioning: fix handling of several edge cases
This commit fixes support for several relatively rare partitioning clauses: * HASH or KEY partitioned tables with a single partition, created by omitting both the partition list and count, were previously unsupported for diff. Now handled correctly. Fixes #111. * HASH or KEY partitioned tables without individual partition comments, data directories, or nonstandard naming, were previously unsupported for diff if they were created using an explicit list of the partitions (rather than a count). Now handled correctly. * DATA DIRECTORY clauses are now properly parsed and retained. * KEY partitioned tables with ALGORITHM clause is now properly parsed and retained.
This commit is contained in:
Evan Elias
Родитель
aac8a060e5
Коммит
d2ae6af5de
|
|
@ -95,12 +95,12 @@ The following object types are completely ignored by Skeema. Their presence won'
|
|||
|
||||
Skeema can CREATE or DROP tables using these features, but cannot ALTER them. The output of `skeema diff` and `skeema push` will note that it cannot generate or run ALTER TABLE for tables using these features, so the affected table(s) will be skipped, but the rest of the operation will proceed as normal.
|
||||
|
||||
* generated/virtual columns (MySQL 5.7+ / Percona Server 5.7+ / MariaDB 5.2+)
|
||||
* spatial column types
|
||||
* sub-partitioning (two levels of partitioning in the same table)
|
||||
* CHECK constraints (MySQL 8.0.16+ / Percona Server 8.0.16+ / MariaDB 10.2+)
|
||||
* column-level compression, with or without predefined dictionary (Percona Server 5.6.33+)
|
||||
* some features of non-InnoDB storage engines
|
||||
* spatial column types
|
||||
* generated/virtual columns
|
||||
* column-level compression, with or without predefined dictionary (Percona Server 5.6.33+)
|
||||
* CHECK constraints (MySQL 8.0.16+ / Percona Server 8.0.16+ / MariaDB 10.2+)
|
||||
|
||||
You can still ALTER these tables externally from Skeema (e.g., direct invocation of `ALTER TABLE` or `pt-online-schema-change`). Afterwards, you can update your schema repo using `skeema pull`, which will work properly even on these tables.
|
||||
|
||||
|
|
@ -136,12 +136,9 @@ Skeema v1.4.0 added support for partitioned tables. The diff/push functionality
|
|||
|
||||
Skeema intentionally ignores changes to the *list of partitions* for an already-partitioned table using RANGE or LIST partitioning methods; the assumption is that an external partition management script/cron is responsible for handling this, outside of the scope of the schema repository. Meanwhile, for HASH or KEY partitioning methods, attempting to change the partition count causes an unsupported diff error, skipping the affected table. Future versions of Skeema may add additional options controlling these behaviors.
|
||||
|
||||
Whenever a partitioned table is being dropped, Skeema will generate a series of `ALTER TABLE ... DROP PARTITION` clauses to drop all but 1 partition prior to generating the `DROP TABLE`. This avoids having a single excessively-long `DROP TABLE` operation, which could be disruptive to other queries since it holds MySQL's dict_sys mutex.
|
||||
Whenever a RANGE or LIST partitioned table is being dropped, Skeema will generate a series of `ALTER TABLE ... DROP PARTITION` clauses to drop all but 1 partition prior to generating the `DROP TABLE`. This avoids having a single excessively-long `DROP TABLE` operation, which could be disruptive to other queries since it holds MySQL's dict_sys mutex.
|
||||
|
||||
Sub-partitioning (two levels of partitioning in the same table) is not supported for diff operations yet, as this feature adds complexity and is infrequently used.
|
||||
|
||||
Two known issues are planned to be patched in an upcoming release:
|
||||
|
||||
* Skeema currently ignores `DATA DIRECTORY` clauses and `ALGORITHM = 1` clauses in partitioned tables. When creating a new table via Skeema, these clauses may be unintentionally stripped from the DDL if present. The fix will retain these clauses as written.
|
||||
* When running `skeema pull` against an environment that uses `partitioning=remove`, since the environment has no partitions, the *.sql files will be rewritten such that the `CREATE TABLE` statements lose their `PARTITION BY` clauses. By design this won't interfere with other environments that use `partitioning=keep`, however it is cosmetically undesirable and potentially confusing in SCM history.
|
||||
When running `skeema pull` against an environment that uses `partitioning=remove`, please be aware that since the environment has no partitions, the *.sql files will be rewritten such that the `CREATE TABLE` statements lose their `PARTITION BY` clauses. By design this won't interfere with other environments that use `partitioning=keep`, however it is cosmetically undesirable and potentially confusing in SCM history.
|
||||
|
||||
|
|
|
|||
2
go.mod
2
go.mod
|
|
@ -12,7 +12,7 @@ require (
|
|||
github.com/opencontainers/runc v1.0.0-rc5 // indirect
|
||||
github.com/sirupsen/logrus v1.4.2
|
||||
github.com/skeema/mybase v1.0.8
|
||||
github.com/skeema/tengo v0.8.20-0.20191023033155-f3604144981e
|
||||
github.com/skeema/tengo v0.8.20-0.20191119054631-833a1bde9244
|
||||
golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4
|
||||
golang.org/x/lint v0.0.0-20190409202823-959b441ac422
|
||||
golang.org/x/sync v0.0.0-20190423024810-112230192c58
|
||||
|
|
|
|||
4
go.sum
4
go.sum
|
|
@ -68,8 +68,8 @@ github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4
|
|||
github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
|
||||
github.com/skeema/mybase v1.0.8 h1:eqtxi5FfphYhosEEeHBsYf/9oXfwjCZ8fMnpGJ+AdxI=
|
||||
github.com/skeema/mybase v1.0.8/go.mod h1:09Uz3MIoXTNCUZWBeKDeO8SUHlQNjIEocXbc1DFvEKQ=
|
||||
github.com/skeema/tengo v0.8.20-0.20191023033155-f3604144981e h1:RgNLLi6USC85MpI/mzs9u+V7wsWyQILTTEKF06aEWDY=
|
||||
github.com/skeema/tengo v0.8.20-0.20191023033155-f3604144981e/go.mod h1:7ahmzzEKjeOzHEqq0okxccPFozsDyGmZORUfF25GRYc=
|
||||
github.com/skeema/tengo v0.8.20-0.20191119054631-833a1bde9244 h1:rKjOLp/GJrNZ/h/Mtf+GSJ2TRm8ocacKSNbH/hy9EQQ=
|
||||
github.com/skeema/tengo v0.8.20-0.20191119054631-833a1bde9244/go.mod h1:7ahmzzEKjeOzHEqq0okxccPFozsDyGmZORUfF25GRYc=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
|
||||
|
|
|
|||
|
|
@ -1121,7 +1121,7 @@ func (instance *Instance) querySchemaTables(schema string) ([]*Table, error) {
|
|||
t.CreateStatement = NormalizeCreateOptions(t.CreateStatement)
|
||||
}
|
||||
if t.Partitioning != nil {
|
||||
t.CreateStatement = NormalizePartitioning(t.CreateStatement, flavor)
|
||||
fixPartitioningEdgeCases(t, flavor)
|
||||
}
|
||||
// Index order is unpredictable with new MySQL 8 data dictionary, so reorder
|
||||
// indexes based on parsing SHOW CREATE TABLE if needed
|
||||
|
|
@ -1229,6 +1229,51 @@ func fixCreateOptionsOrder(t *Table, flavor Flavor) {
|
|||
}
|
||||
}
|
||||
|
||||
// fixPartitioningEdgeCases handles situations that are reflected in SHOW CREATE
|
||||
// TABLE, but missing (or difficult to obtain) in information_schema.
|
||||
func fixPartitioningEdgeCases(t *Table, flavor Flavor) {
|
||||
// Handle edge cases for how partitions are expressed in HASH or KEY methods:
|
||||
// typically this will just be a PARTITIONS N clause, but it could also be
|
||||
// nothing at all, or an explicit list of partitions, depending on how the
|
||||
// partitioning was originally created.
|
||||
if strings.HasSuffix(t.Partitioning.Method, "HASH") || strings.HasSuffix(t.Partitioning.Method, "KEY") {
|
||||
countClause := fmt.Sprintf("\nPARTITIONS %d", len(t.Partitioning.Partitions))
|
||||
if strings.Contains(t.CreateStatement, countClause) {
|
||||
t.Partitioning.forcePartitionList = partitionListCount
|
||||
} else if strings.Contains(t.CreateStatement, "\n(PARTITION ") {
|
||||
t.Partitioning.forcePartitionList = partitionListExplicit
|
||||
} else if len(t.Partitioning.Partitions) == 1 {
|
||||
t.Partitioning.forcePartitionList = partitionListNone
|
||||
}
|
||||
}
|
||||
|
||||
// KEY methods support an optional ALGORITHM clause, which is present in SHOW
|
||||
// CREATE TABLE but not anywhere in information_schema
|
||||
if strings.HasSuffix(t.Partitioning.Method, "KEY") && strings.Contains(t.CreateStatement, "ALGORITHM") {
|
||||
re := regexp.MustCompile(fmt.Sprintf(`PARTITION BY %s ([^(]*)\(`, t.Partitioning.Method))
|
||||
if matches := re.FindStringSubmatch(t.CreateStatement); matches != nil {
|
||||
t.Partitioning.algoClause = matches[1]
|
||||
}
|
||||
}
|
||||
|
||||
// Process DATA DIRECTORY clauses, which are easier to parse from SHOW CREATE
|
||||
// TABLE instead of information_schema.innodb_sys_tablespaces.
|
||||
if (t.Partitioning.forcePartitionList == partitionListDefault || t.Partitioning.forcePartitionList == partitionListExplicit) &&
|
||||
strings.Contains(t.CreateStatement, " DATA DIRECTORY = ") {
|
||||
for _, p := range t.Partitioning.Partitions {
|
||||
name := p.Name
|
||||
if flavor.VendorMinVersion(VendorMariaDB, 10, 2) {
|
||||
name = EscapeIdentifier(name)
|
||||
}
|
||||
name = regexp.QuoteMeta(name)
|
||||
re := regexp.MustCompile(fmt.Sprintf(`PARTITION %s .*DATA DIRECTORY = '((?:\\\\|\\'|''|[^'])*)'`, name))
|
||||
if matches := re.FindStringSubmatch(t.CreateStatement); matches != nil {
|
||||
p.dataDir = matches[1]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (instance *Instance) querySchemaRoutines(schema string) ([]*Routine, error) {
|
||||
db, err := instance.Connect("information_schema", "")
|
||||
if err != nil {
|
||||
|
|
|
|||
|
|
@ -5,16 +5,29 @@ import (
|
|||
"strings"
|
||||
)
|
||||
|
||||
// partitionListMode enum values control edge-cases for how the list of
|
||||
// partitions is represented in SHOW CREATE TABLE.
|
||||
type partitionListMode int
|
||||
|
||||
const (
|
||||
partitionListDefault partitionListMode = iota
|
||||
partitionListExplicit // List each partition individually
|
||||
partitionListCount // Just use a count of partitions
|
||||
partitionListNone // Omit partition list and count, implying just 1 partition
|
||||
)
|
||||
|
||||
// TablePartitioning stores partitioning configuration for a partitioned table.
|
||||
// Note that despite subpartitioning fields being present and possibly
|
||||
// populated, the rest of this package does not fully support subpartitioning
|
||||
// yet.
|
||||
type TablePartitioning struct {
|
||||
Method string // one of "RANGE", "RANGE COLUMNS", "LIST", "LIST COLUMNS", "HASH", "LINEAR HASH", "KEY", or "LINEAR KEY"
|
||||
SubMethod string // one of "" (no sub-partitioning), "HASH", "LINEAR HASH", "KEY", or "LINEAR KEY"; not fully supported yet
|
||||
Expression string
|
||||
SubExpression string // empty string if no sub-partitioning; not fully supported yet
|
||||
Partitions []*Partition
|
||||
Method string // one of "RANGE", "RANGE COLUMNS", "LIST", "LIST COLUMNS", "HASH", "LINEAR HASH", "KEY", or "LINEAR KEY"
|
||||
SubMethod string // one of "" (no sub-partitioning), "HASH", "LINEAR HASH", "KEY", or "LINEAR KEY"; not fully supported yet
|
||||
Expression string
|
||||
SubExpression string // empty string if no sub-partitioning; not fully supported yet
|
||||
Partitions []*Partition
|
||||
forcePartitionList partitionListMode
|
||||
algoClause string // full text of optional ALGORITHM clause for KEY or LINEAR KEY
|
||||
}
|
||||
|
||||
// Definition returns the overall partitioning definition for a table.
|
||||
|
|
@ -23,35 +36,38 @@ func (tp *TablePartitioning) Definition(flavor Flavor) string {
|
|||
return ""
|
||||
}
|
||||
|
||||
var needPartitionList bool
|
||||
for n, p := range tp.Partitions {
|
||||
if p.Values != "" || p.Comment != "" || p.Name != fmt.Sprintf("p%d", n) {
|
||||
needPartitionList = true
|
||||
break
|
||||
plMode := tp.forcePartitionList
|
||||
if plMode == partitionListDefault {
|
||||
plMode = partitionListCount
|
||||
for n, p := range tp.Partitions {
|
||||
if p.Values != "" || p.Comment != "" || p.dataDir != "" || p.Name != fmt.Sprintf("p%d", n) {
|
||||
plMode = partitionListExplicit
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
var partitionsClause string
|
||||
if needPartitionList {
|
||||
if plMode == partitionListExplicit {
|
||||
pdefs := make([]string, len(tp.Partitions))
|
||||
for n, p := range tp.Partitions {
|
||||
pdefs[n] = p.Definition(flavor)
|
||||
}
|
||||
partitionsClause = fmt.Sprintf("(%s)", strings.Join(pdefs, ",\n "))
|
||||
} else {
|
||||
partitionsClause = fmt.Sprintf("PARTITIONS %d", len(tp.Partitions))
|
||||
partitionsClause = fmt.Sprintf("\n(%s)", strings.Join(pdefs, ",\n "))
|
||||
} else if plMode == partitionListCount {
|
||||
partitionsClause = fmt.Sprintf("\nPARTITIONS %d", len(tp.Partitions))
|
||||
}
|
||||
|
||||
open, close := "/*!50100", " */"
|
||||
opener, closer := "/*!50100", " */"
|
||||
if flavor.VendorMinVersion(VendorMariaDB, 10, 2) {
|
||||
// MariaDB stopped wrapping partitioning clauses in version-gated comments
|
||||
// in 10.2.
|
||||
open, close = "", ""
|
||||
opener, closer = "", ""
|
||||
} else if strings.HasSuffix(tp.Method, "COLUMNS") {
|
||||
// RANGE COLUMNS and LIST COLUMNS were introduced in 5.5
|
||||
open = "/*!50500"
|
||||
opener = "/*!50500"
|
||||
}
|
||||
|
||||
return fmt.Sprintf("\n%s PARTITION BY %s\n%s%s", open, tp.partitionBy(flavor), partitionsClause, close)
|
||||
return fmt.Sprintf("\n%s PARTITION BY %s%s%s", opener, tp.partitionBy(flavor), partitionsClause, closer)
|
||||
}
|
||||
|
||||
// partitionBy returns the partitioning method and expression, formatted to
|
||||
|
|
@ -69,7 +85,7 @@ func (tp *TablePartitioning) partitionBy(flavor Flavor) string {
|
|||
expr = strings.Replace(expr, "`", "", -1)
|
||||
}
|
||||
|
||||
return fmt.Sprintf("%s(%s)", method, expr)
|
||||
return fmt.Sprintf("%s%s(%s)", method, tp.algoClause, expr)
|
||||
}
|
||||
|
||||
// Diff returns a set of differences between this TablePartitioning and another
|
||||
|
|
@ -127,6 +143,7 @@ type Partition struct {
|
|||
Comment string
|
||||
method string
|
||||
engine string
|
||||
dataDir string
|
||||
}
|
||||
|
||||
// Definition returns this partition's definition clause, for use as part of a
|
||||
|
|
@ -146,10 +163,15 @@ func (p *Partition) Definition(flavor Flavor) string {
|
|||
values = fmt.Sprintf("VALUES IN (%s) ", p.Values)
|
||||
}
|
||||
|
||||
var dataDir string
|
||||
if p.dataDir != "" {
|
||||
dataDir = fmt.Sprintf("DATA DIRECTORY = '%s' ", p.dataDir) // any necessary escaping is already present in p.dataDir
|
||||
}
|
||||
|
||||
var comment string
|
||||
if p.Comment != "" {
|
||||
comment = fmt.Sprintf("COMMENT = '%s' ", EscapeValueForCreateTable(p.Comment))
|
||||
}
|
||||
|
||||
return fmt.Sprintf("PARTITION %s %s%sENGINE = %s", name, values, comment, p.engine)
|
||||
return fmt.Sprintf("PARTITION %s %s%s%sENGINE = %s", name, values, dataDir, comment, p.engine)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -93,8 +93,8 @@ func (t *Table) UnpartitionedCreateStatement(flavor Flavor) string {
|
|||
}
|
||||
|
||||
// If our generated partitioning clause definition isn't 100% aligned with
|
||||
// SHOW CREATE TABLE (due to unsupported features or due to adjustments made
|
||||
// in NormalizePartitioning), just search for just the beginning of the clause.
|
||||
// SHOW CREATE TABLE (e.g. due to unsupported features), just search for just
|
||||
// the beginning of the clause.
|
||||
partClause := t.Partitioning.Definition(flavor)
|
||||
if t.UnsupportedDDL || !strings.Contains(t.CreateStatement, partClause) {
|
||||
headerPos := strings.Index(partClause, " PARTITION BY ")
|
||||
|
|
|
|||
|
|
@ -139,29 +139,6 @@ func NormalizeCreateOptions(createStmt string) string {
|
|||
return createStmt
|
||||
}
|
||||
|
||||
var reDataDirectory = regexp.MustCompile(` DATA DIRECTORY = '(\\\\|\\'|''|[^'])*'`)
|
||||
|
||||
// NormalizePartitioning adjusts the supplied CREATE TABLE's partitioning clause
|
||||
// to remove any oddities that are not reflected in information_schema and/or
|
||||
// cannot be altered by this package, but are included in SHOW CREATE TABLE.
|
||||
func NormalizePartitioning(createStmt string, flavor Flavor) string {
|
||||
if flavor.Major == 5 && flavor.Minor == 5 {
|
||||
createStmt = strings.Replace(createStmt, "PARTITION BY KEY */ /*!50531 ALGORITHM = 1 */ /*!50100 ", "PARTITION BY KEY ", 1)
|
||||
} else if flavor.MySQLishMinVersion(5, 6) || flavor == FlavorMariaDB101 {
|
||||
createStmt = strings.Replace(createStmt, "PARTITION BY KEY */ /*!50611 ALGORITHM = 1 */ /*!50100 ", "PARTITION BY KEY ", 1)
|
||||
} else if flavor.VendorMinVersion(VendorMariaDB, 10, 2) {
|
||||
createStmt = strings.Replace(createStmt, "PARTITION BY KEY ALGORITHM = 1 ", "PARTITION BY KEY ", 1)
|
||||
}
|
||||
|
||||
// Ignore DATA DIRECTORY clauses for now. These only affect the partition list
|
||||
// (which this package does not diff/alter yet), so no sense in doing extra
|
||||
// queries to introspect them, which would require looking in nonstandard
|
||||
// information_schema tables innodb_sys_datafiles and innodb_sys_tables.
|
||||
createStmt = reDataDirectory.ReplaceAllString(createStmt, "")
|
||||
|
||||
return createStmt
|
||||
}
|
||||
|
||||
// baseDSN returns a DSN with the database (schema) name and params stripped.
|
||||
// Currently only supports MySQL, via go-sql-driver/mysql's DSN format.
|
||||
func baseDSN(dsn string) string {
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ github.com/pmezard/go-difflib/difflib
|
|||
github.com/sirupsen/logrus
|
||||
# github.com/skeema/mybase v1.0.8
|
||||
github.com/skeema/mybase
|
||||
# github.com/skeema/tengo v0.8.20-0.20191023033155-f3604144981e
|
||||
# github.com/skeema/tengo v0.8.20-0.20191119054631-833a1bde9244
|
||||
github.com/skeema/tengo
|
||||
# golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4
|
||||
golang.org/x/crypto/ssh/terminal
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче