зеркало из https://github.com/github/vitess-gh.git
Merge pull request #4503 from dweitzman/remove_influx
Remove influxdb as a push backend
This commit is contained in:
Sugu Sougoumarane
GitHub
Коммит
3bd115f928
|
|
@ -24,9 +24,9 @@ Vitess also includes support for push-based metrics systems via plug-ins. Each V
|
|||
|
||||
By default, the stats_emit_period is 60s, so each component will push stats to the the selected backend every minute. This is configurable via the `--stats_emit_period` flag.
|
||||
|
||||
Vitess has preliminary plug-ins to support InfluxDB and OpenTSDB as push-based metrics backends. However, there is very limited support at this time, as InfluxDB itself is going through various API breaking changes.
|
||||
Vitess has preliminary plug-ins to support OpenTSDB as a push-based metrics backend.
|
||||
|
||||
It should be fairly straightforward to write your own plug-in, if you want to support a different backend. The plug-in package simply needs to implement the `PushBackend` interface of the `stats` package. For an example, you can see the [InfluxDB plugin](https://github.com/vitessio/vitess/blob/master/go/stats/influxdbbackend/influxdb_backend.go).
|
||||
It should be fairly straightforward to write your own plug-in, if you want to support a different backend. The plug-in package simply needs to implement the `PushBackend` interface of the `stats` package. For an example, you can see the [OpenTSDB plugin](https://github.com/vitessio/vitess/blob/master/go/stats/opentsdb/opentsdb.go).
|
||||
|
||||
Once you’ve written the backend plug-in, you also need to register the plug-in from within all the relevant Vitess binaries. An example of how to do this can be seen in [this pull request](https://github.com/vitessio/vitess/pull/469).
|
||||
|
||||
|
|
@ -36,7 +36,7 @@ Connecting Vitess to a push-based metrics system can be useful if you’re alrea
|
|||
|
||||
## Monitoring with Kubernetes
|
||||
|
||||
The existing methods for integrating metrics are not supported in a Kubernetes environment by the Vitess team yet, but are on the roadmap for the future. However, it should be possible to get the InfluxDB backend working with Kubernetes, similar to how [Heapster for Kubernetes works](https://github.com/GoogleCloudPlatform/kubernetes/tree/master/cluster/addons/cluster-monitoring).
|
||||
The existing methods for integrating metrics are not supported in a Kubernetes environment by the Vitess team yet, but are on the roadmap for the future. However, it should be possible to get the Prometheus backend working with Kubernetes, similar to how [Heapster for Kubernetes works](https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/prometheus).
|
||||
|
||||
In the meantime, if you run into issues or have questions, please post on our [forum](https://groups.google.com/forum/#!forum/vitess).
|
||||
|
||||
|
|
|
|||
|
|
@ -54,7 +54,7 @@ Setting up these components directly -- for example, writing your own topology s
|
|||
* *Recommended*. Vitess has basic support for identifying or changing a master, but it doesn't aim to fully address this feature. As such, we recommend using another program, like [Orchestrator](https://github.com/github/orchestrator), to monitor the health of your servers and to change your master database when necessary. (In a sharded database, each shard has a master.)
|
||||
|
||||
|
||||
* *Recommended*. You should have a way to monitor your database topology and set up alerts as needed. Vitess components facilitate this monitoring by exporting a lot of runtime variables, like QPS over the last few minutes, error rates, and query latency. The variables are exported in JSON format, and Vitess also supports an InfluxDB plug-in.
|
||||
* *Recommended*. You should have a way to monitor your database topology and set up alerts as needed. Vitess components facilitate this monitoring by exporting a lot of runtime variables, like QPS over the last few minutes, error rates, and query latency. The variables are exported in JSON format, and Vitess also supports a Prometheus plug-in.
|
||||
|
||||
|
||||
* *Optional*. Using the Kubernetes scripts as a base, you could run Vitess components with other configuration management systems (like Puppet) or frameworks (like Mesos or AWS images).
|
||||
|
|
|
|||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package main
|
||||
|
||||
// This plugin imports influxdbbackend to register the influxdbbackend stats backend.
|
||||
|
||||
import (
|
||||
_ "vitess.io/vitess/go/stats/influxdbbackend"
|
||||
)
|
||||
|
|
@ -41,7 +41,7 @@ import (
|
|||
|
||||
var emitStats = flag.Bool("emit_stats", false, "true iff we should emit stats to push-based monitoring/stats backends")
|
||||
var statsEmitPeriod = flag.Duration("stats_emit_period", time.Duration(60*time.Second), "Interval between emitting stats to all registered backends")
|
||||
var statsBackend = flag.String("stats_backend", "influxdb", "The name of the registered push-based monitoring/stats backend to use")
|
||||
var statsBackend = flag.String("stats_backend", "", "The name of the registered push-based monitoring/stats backend to use")
|
||||
|
||||
// NewVarHook is the type of a hook to export variables in a different way
|
||||
type NewVarHook func(name string, v expvar.Var)
|
||||
|
|
|
|||
|
|
@ -1,104 +0,0 @@
|
|||
/*
|
||||
Copyright 2017 Google Inc.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
// Package influxdbbackend is useful for publishing metrics to an InfluxDB backend (tested on v0.88).
|
||||
// It requires a database to already have been created in InfluxDB, and then specified via the
|
||||
// "--influxdb_database" flag.
|
||||
//
|
||||
// It's still a work in progress, as it publishes almost all stats as key-value string pairs,
|
||||
// instead of better JSON representations. This limitation will hopefully be fixed after the
|
||||
// release of InfluxDB v0.9, as it has better support for arbitrary metadata dicts in the
|
||||
// form of tags.
|
||||
package influxdbbackend
|
||||
|
||||
import (
|
||||
"expvar"
|
||||
"flag"
|
||||
|
||||
influxClient "github.com/influxdb/influxdb/client"
|
||||
"vitess.io/vitess/go/stats"
|
||||
"vitess.io/vitess/go/vt/log"
|
||||
"vitess.io/vitess/go/vt/servenv"
|
||||
)
|
||||
|
||||
var influxDBHost = flag.String("influxdb_host", "localhost:8086", "the influxdb host (with port)")
|
||||
var influxDBDatabase = flag.String("influxdb_database", "vitess", "the name of the influxdb database")
|
||||
var influxDBUsername = flag.String("influxdb_username", "root", "influxdb username")
|
||||
var influxDBPassword = flag.String("influxdb_password", "root", "influxdb password")
|
||||
|
||||
// InfluxDBBackend implements stats.PushBackend
|
||||
type InfluxDBBackend struct {
|
||||
client *influxClient.Client
|
||||
}
|
||||
|
||||
// init attempts to create a singleton InfluxDBBackend and register it as a PushBackend.
|
||||
// If it fails to create one, this is a noop.
|
||||
func init() {
|
||||
// Needs to happen in servenv.OnRun() instead of init because it requires flag parsing and logging
|
||||
servenv.OnRun(func() {
|
||||
config := &influxClient.ClientConfig{
|
||||
Host: *influxDBHost,
|
||||
Username: *influxDBUsername,
|
||||
Password: *influxDBPassword,
|
||||
Database: *influxDBDatabase,
|
||||
}
|
||||
client, err := influxClient.NewClient(config)
|
||||
if err != nil {
|
||||
log.Errorf("Unable to create an InfluxDB client: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
stats.RegisterPushBackend("influxdb", &InfluxDBBackend{
|
||||
client: client,
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
// PushAll pushes all expvar stats to InfluxDB
|
||||
func (backend *InfluxDBBackend) PushAll() error {
|
||||
series := []*influxClient.Series{}
|
||||
expvar.Do(func(kv expvar.KeyValue) {
|
||||
series = append(series, &influxClient.Series{
|
||||
Name: "stats",
|
||||
// TODO(aaijazi): This would be much better suited to InfluxDB v0.90's tags.
|
||||
// Ideally, we'd use some of the expvars as tags, and some as values.
|
||||
// However, as of 03/11/2015, InfluxDB v0.90 hasn't proven quite stable enough to use.
|
||||
Columns: []string{"key", "value"},
|
||||
Points: [][]interface{}{
|
||||
{kv.Key, statToValue(kv.Value)},
|
||||
},
|
||||
})
|
||||
})
|
||||
err := backend.client.WriteSeries(series)
|
||||
return err
|
||||
}
|
||||
|
||||
// statToValue converts from a stats.Stat type to a JSON representable value.
|
||||
// This is preferred to just calling the String() for things like numbers, so that
|
||||
// InfluxDB can also represent the metrics as numbers.
|
||||
// TODO(aaijazi): this needs to be extended to support better serialization of other types..
|
||||
// It's probably good to do this after InfluxDB 0.9 is released, as it has has better support
|
||||
// for arbitrary dict values (as tags).
|
||||
func statToValue(v expvar.Var) interface{} {
|
||||
switch v := v.(type) {
|
||||
case *stats.Counter:
|
||||
return v.Get()
|
||||
case stats.FloatFunc:
|
||||
return v()
|
||||
default:
|
||||
return v.String()
|
||||
}
|
||||
}
|
||||
|
|
@ -618,14 +618,6 @@
|
|||
"revision": "d3a67ab21bc8a4643fa53a3633f2d951dd50c6ca",
|
||||
"revisionTime": "2016-12-07T01:17:43Z"
|
||||
},
|
||||
{
|
||||
"checksumSHA1": "fe0NspvyJjx6DhmTjIpO0zmR+kg=",
|
||||
"path": "github.com/influxdb/influxdb/client",
|
||||
"revision": "afde71eb1740fd763ab9450e1f700ba0e53c36d0",
|
||||
"revisionTime": "2014-12-28T19:15:54Z",
|
||||
"version": "=v0.8.8",
|
||||
"versionExact": "v0.8.8"
|
||||
},
|
||||
{
|
||||
"checksumSHA1": "0ZrwvB6KoGPj2PoDNSEJwxQ6Mog=",
|
||||
"path": "github.com/jmespath/go-jmespath",
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче