Граф коммитов

192 Коммитов

Автор SHA1 Сообщение Дата
Maggie Zhou e1b31443cb Support DurationFunc
Convert Nanoseconds to seconds

Call Init (handler) inside servenv.OnRun()

Add unit tests.

For counter, it doesn't make sense to have a function called ResetAll
and not one called Reset so re-rename that.

Signed-off-by: Maggie Zhou <mzhou@slack-corp.com>
2018-04-20 16:51:46 -07:00
Maggie Zhou 50e0d11406 Dedupe metric names with namespace names.
Add plugins for each of the components.

Rename prombackend => prometheusbackend
Rename publishPromMetric to publishPrometheusMetric

Rename Reset() => ResetAll()
ResetCounter => Reset()

Rename various Counters => Gauges per Mike's helpful pointers.
Rename a few help strings.
Remove the comments in go/vt/worker/worker.go metrics and move them 100% into help functions.
Add a GaugeFuncWithMultiLabels
Add a CounterFunc

Signed-off-by: Maggie Zhou <mzhou@slack-corp.com>
2018-04-20 16:51:46 -07:00
Maggie Zhou f2afcd25d5 Get rid of the pull_backend.go
Fix tests to use the new metric names.
Fix tests that expected gauges and not counters. (more to come on this
thing, but these were the tests that indicated gauges early)
Fix the `Set()` implementation for GaugesWithMultiLabels.

Signed-off-by: Maggie Zhou <mzhou@slack-corp.com>
2018-04-20 16:51:46 -07:00
Maggie Zhou 0d17a264e0 Quick followup to @demmer's IRL comments that I could push some of the
pull_backend.go stuff into prombackend/prombackend.go's Register call
and make the pull_backend.go an interface only.

Also, fix some more types from the gigantic refactor, and run
GoLint/GoVet

Signed-off-by: Maggie Zhou <mzhou@slack-corp.com>
2018-04-20 16:51:46 -07:00
Maggie Zhou 1687f8be9a * Refactor the pull backend to use the expvar hook.
* Fix the nanoseconds => ints for histogram buckets.
* Gigantic rename of metrics:

-Int => Counter
-IntGauge => Gauge
-IntFunc => GaugeFunc
-Counters => CountersWithLabels
-MultiCounters => CountersWithMultiLabels
-MultiCounterFunc => CountersFuncWithMultiLabels
-Gauges => GaugesWithLabels
-MultiGauges=> GaugesWithMultiLabels

* Add an explicit labelName for CountersWithLabels & GaugesWithLabels

Signed-off-by: Maggie Zhou <mzhou@slack-corp.com>
2018-04-20 16:51:46 -07:00
Maggie Zhou 6b5d7e1bdc WIP PR to add support for exporting metrics to Prometheus.
Signed-off-by: Maggie Zhou <mzhou@slack-corp.com>
2018-04-20 16:51:46 -07:00
Michael Berlin 246e2898d2 Revert "Factor out glog references"
Signed-off-by: Michael Berlin <mberlin@google.com>
2018-04-20 16:44:06 -07:00
David Weitzman b0cedf1b09 Factor out glog references for ease of dropping in an adapter to a different logging library.
Signed-off-by: David Weitzman <dweitzman@pinterest.com>
2018-04-17 08:56:51 -07:00
Michael Berlin fa6234bcb2 stats: Add function to convert stats variable names to kebab-case.
This is an export of the code which we use internally.

Signed-off-by: Michael Berlin <mberlin@google.com>
2018-04-09 00:40:59 -07:00
Anthony Yeh 6914580ad5
Rewrite imports to new custom URL.
This commit consists only of the changes from running the following:

```
find go -name '*.go' | xargs -n 10 -- sed -i -r -e \
  's,"github.com/youtube/vitess/go/,"vitess.io/vitess/go/,'
```
2018-02-27 12:00:33 -08:00
Matt Harden 7910a61684 fix stats.Counters.String method to generate valid JSON for map keys 2017-07-22 19:46:54 -07:00
Matt Harden 6b36e84f15 stats: escape map keys containing dot (.) 2017-07-12 17:38:08 -07:00
Sugu Sougoumarane 7af23e1dde license: BSD->Apache v2.0
Please refer to #2694 and #2670 for motivation and reasoning for
this change.

I've tried to follow best practice in inserting the copyright
headers. In other open source projects, not all files carry
the notice. For example documentation doesn't. I've followed
similar ground rules.

I did not change the php because there is a separate LICENSE
file there by Pixel Federation. We'll first need to notify
them our intent before changing anything there.

As for the presubmit check, it's going to be non-trivial
because of the number of exceptions, like file types,
directories and generated code. So, it will have to be
a separate project.
2017-05-06 00:38:56 -07:00
Alain Jobart 657f565144 Adding 'unused' go code hook.
It is from this repository: https://github.com/dominikh/go-unused

As part of this change:
- adding download of the tool to bootstrap.sh.
- re-enabling a few tests that were disabled.
- removing a bunch of unused code.
2016-08-30 07:00:20 -07:00
Anthony Yeh 27c3151954 Rename glog to log on import, for easier package swapping. 2016-06-20 11:43:27 -07:00
Matt Harden 15fa6303db move promstats to stats/promstats 2016-06-15 17:34:44 -05:00
Sugu Sougoumarane 6e896cf69b Exporting internal changes back to open-source.
NOTE: This is an automated export. Changes were already LGTM'd internally.
2016-05-20 11:46:46 -07:00
Michael Berlin e3524b301c Exporting internal changes back to open-source.
NOTE: This is an automated export. Changes were already LGTM'd internally.
2016-05-19 13:38:53 -07:00
Michael Berlin 0cb22ce7e8 stats: Fix typo. 2016-04-29 18:40:34 -07:00
Liang Guo 45fdaa44a7 Improve counters 99th latency benchmark. 2016-03-28 13:58:15 -07:00
Anthony Yeh 32d95aaf6c Add benchmarks for stats.Counters.Add(). 2016-03-22 11:49:55 -07:00
Liang Guo 42f1c32db1 Add more comments. 2016-03-22 11:40:00 -07:00
Liang Guo 2ced68ddd8 Remove unnecessary blocking in stats package.
Otherwise it imposes global locking in VTGate.
2016-03-21 16:04:14 -07:00
Sugu Sougoumarane a446350b84 remove accidental add 2016-03-03 17:24:18 -08:00
Sugu Sougoumarane a0cb8767b9 Exporting internal changes back to open-source. 2016-03-03 17:08:51 -08:00
Sugu Sougoumarane e6b4dddbd6 Exporting internal changes back to open-source.
NOTE: This is an automated export. Changes were already LGTM'd internally.
2016-03-03 17:08:12 -08:00
Sugu Sougoumarane b800d5fe00 add hook to histogram and timings
Types that collect samples need hooks for integrating with
other similar systems. This is an already approved internal change.
2016-03-02 11:46:14 -08:00
Sugu Sougoumarane b111b0d574 Exporting internal changes back to open-source. 2016-02-24 21:30:23 -08:00
Alain Jobart 3d599b118a Deflaking rates unit tests. Instead of using time.Now()
and sleeping for predictable times, and be at the mercy of
the underlying hardware performance, use a faked clock
and fixed values, and don't even sleep.
2016-02-10 12:57:21 -08:00
Michael Berlin 51b4c63686 stats: Add Rates.TotalRate() to retrieve the current total rate (across all categories). 2016-02-01 17:10:46 -08:00
Alain Jobart 89598de963 Running gofmt -s -w on the source tree. 2015-12-15 15:38:45 -08:00
Michael Berlin f2886c3d66 stats: Remove the stats variable type "States".
Track the state of each Zookeeper connection in the "ZkCachedConn" variable instead.

Remove "ZkMetaConn" variable because "ZkCachedConn" has the same information now.

Update tabletmanager.py end-to-end test.

I'm removing states.go because it caused a deadlock in the worker.py test.

Its removal is no loss because the original intent was to detect flip-flopping with that code. However, that's no longer necessary.

The observed deadlock occurred when a) somebody polls /debug/vars while b) we create a new Zookeeper connection and publish a "ZkCachedConn" variable for it.

When the connection gets created, then the "states" object is created. The same call also calls expvar.Publish() eventually and this is where the deadlock occurs. (It's a deadlock between a mutex in the "expvar" package and a mutex in the Zookeeper connection package.)
2015-11-13 16:36:47 -08:00
Sugu Sougoumarane 6e3f6f6777 tabletserver: precreate some stats labels
Precreate some of the counter labels at the beginning.
This eliminates some monitoring jitters where an absent
label is not treated as zero value. We should ideally do
this to all stats counters. This is just some low hanging
fruit for now.
2015-09-12 23:19:30 -07:00
Anthony Yeh 32c27f538b Initialize ExternalReparents timings to 0.
When a category of the timings map goes from undefined to defined,
that event will be missed by rate aggregations. For events that happen
often (like queries), missing one is fine. But many reparent events will
be the first time that tablet has become master (since the last restart)
so missing the first event skews aggregations significantly.
2015-06-15 16:42:11 -07:00
Shengzhe Yao cb99c724d6 make sure queryservice not publish stats when not required
1. queryservice should not publish stats if either flag EnablePublishStats
    is enabled or var name is empty.
2. stats.NewInt should call Publish if name is empty.
3. mysqlctl.NewMysqld publishes dba stats only if dbaName is not empty.
4. mysqlctl.NewMysqld publishes app stats only if appName is not empty.
5. QueryEngine publishes stats only if flag EnablePublishStats is enabled.
6. RowCacheInvalidator publishes stats only if flag EnablePublishStats is enabled.
2015-04-28 15:49:46 -07:00
Ammar Aijazi 7451a0ec55 Reset worker debug variables at the beginning of each worker run 2015-03-26 12:10:00 -07:00
Ammar Aijazi 5d5c488163 Move InfluxDB init into servenv.OnRun as it depends on flags and logging 2015-03-18 08:59:24 -07:00
Ammar Aijazi 1420422114 Only support emitting stats to a single backend at a time, selected with the --stats_backend flag 2015-03-17 12:20:12 -07:00
Ammar Aijazi 427f582116 Support for emitting stats push-based monitoring backends, with initial
support for InfluxDB (v0.88).
2015-03-12 18:22:21 -07:00
sougou b9de0ffd0f Merge pull request #68 from apmichaud/histograms
Fixing timings test failures introduced by previous commit.
2014-07-25 15:26:08 -07:00
Adam Michaud c1b8f70483 Fixing timings test failures introduced by previous commit. 2014-07-25 14:01:22 -07:00
sougou cd40a00258 Merge pull request #67 from apmichaud/histograms
Updating histogram labeling and formatting in debug variable output.
2014-07-25 11:08:10 -07:00
Adam Michaud 91c5ef4e6f Updating histogram labeling and formatting in debug variable output. 2014-07-24 15:57:11 -07:00
Adam Michaud d4b8c8024d Really reverting this time. 2014-07-24 09:49:30 -07:00
Adam Michaud 3ce5c7ef8d Reverting to state of main vitess repo. 2014-07-24 09:35:27 -07:00
Adam Michaud 968892bf74 Updating histogram labeling and formatting. 2014-07-23 12:29:24 -07:00
Anthony Yeh 9b4f1a8dc9 Skip long tests in "go test -short" mode. 2014-07-22 23:52:38 -07:00
Adam Michaud 2b17b17256 Change to histogram label formatting in go/stats/timings.go. 2014-07-22 16:02:27 -07:00
Alain Jobart ad95d61122 Fixing unit tests. 2014-07-14 10:20:04 -07:00
Ryszard Szopa a73625a93a stats: rename Map* to Multi* to make multidimensionality of the counters more obvious.
Conflicts:
	go/cmd/vtgate/vtgate.go
	go/vt/tabletserver/schema_info.go
	go/vt/vtgate/scatter_conn.go
2014-07-10 19:25:46 -07:00
Ryszard Szopa 4dfe120382 vtgate: add multidimensional QPS counters. 2014-07-10 19:22:59 -07:00
Ryszard Szopa 6b6919bde6 stats: Fix rates inconsistency bug. 2014-07-10 19:07:41 -07:00
Alain Jobart 897ffc361d Exporting source keyspace/shard and source tablet alias.
Had to add a StringMap stats.
2014-07-09 14:21:45 -07:00
Alain Jobart 4faf110fce Using MapCounters, retiring Matrix. 2014-06-10 11:07:03 -07:00
Alain Jobart bd94b0e948 Adding comments to some methods. 2014-06-10 10:14:38 -07:00
Alain Jobart ba52d2ce4c Adding stats.MapCounters, similar to MapTimings.
Using it in vtgate error counting.
2014-06-10 09:57:28 -07:00
Alain Jobart 291dbea3a0 Adding more granular stats, using them in vtgate.
The Timings stats collection only had one dimensional
keys, now adding one that supports multi-dimentional keys.
Using it in vtgate for incoming and outgoing stats collection.
2014-06-09 11:52:01 -07:00
David Symonds dcc88cd329 Fix dependence on map iteration order.
Go map iteration order is undefined, and the upcoming Go 1.3 release makes it more random.
2014-05-29 16:21:03 +10:00
Liang Guo 254286caf6 Remove false warnings in vttablet 2013-10-24 14:15:56 -07:00
Liang Guo f0e2e5643f Update vttablet vars 2013-09-12 17:13:43 -07:00
Liang Guo d0092b04f0 Export memcache stats, table stats and query stats through stats package. 2013-09-08 22:22:52 -07:00
Liang Guo d14606511e Export memcache stats, table level stats, and query level stats through stats package 2013-09-05 14:21:13 -07:00
Liang Guo 2ed2a83699 Export Memcache internal stats through stats package. 2013-09-03 17:00:43 -07:00
Liang Guo 637442ea0d Merge branch 'master' into varz 2013-08-28 17:20:19 -07:00
Liang Guo 2ef617cff1 Publish states through stats package 2013-08-28 16:56:40 -07:00
Sugu Sougoumarane 420b981173 switch to a more strict register scheme 2013-08-26 17:10:10 -07:00
Sugu Sougoumarane ad9c67d954 fix race condition in vars creation vs Register. 2013-08-26 16:06:31 -07:00
Liang Guo 5548c72925 Merge branch 'master' into varz
Conflicts:
	test/resharding.py
2013-08-26 14:48:23 -07:00
Sugu Sougoumarane 4e151cf21e timings bug fix for TotalTime 2013-08-25 11:33:23 -07:00
Liang Guo 1855c26fe3 States publishes its current state as standalone Var 2013-08-23 17:21:08 -07:00
Sugu Sougoumarane b89eeaefc4 Voltron expvar exported as individual vars 2013-08-12 01:47:35 -07:00
Sugu Sougoumarane 217b2e3061 stats: change states to int64 2013-08-11 18:37:28 -07:00
Sugu Sougoumarane 0fac9bf44a stats: Add support for func based export vars. 2013-08-11 17:49:24 -07:00
Sugu Sougoumarane 067f1ff9ca bug fix: timings.Histograms signature. 2013-08-11 00:45:34 -07:00
Sugu Sougoumarane 76201bacb9 stats: tweak states for easier export 2013-08-10 23:28:41 -07:00
Sugu Sougoumarane 1e588f3138 bug fix & test for histogram labels. 2013-08-10 22:13:31 -07:00
Sugu Sougoumarane 8dfe12cf37 Export histogram labels as methods. 2013-08-10 22:05:20 -07:00
Sugu Sougoumarane 9ea71f7e1d stats: bug fixes & tweaks on expvar.Publish 2013-08-10 17:36:32 -07:00
Sugu Sougoumarane 7bb3bb7ea3 Native support for Int, Float, String (add Get) 2013-08-09 16:33:00 -07:00
Alain Jobart e0debcfdb5 Fixing flaky test. 2013-08-06 12:24:51 -07:00
Sugu Sougoumarane 08edcb2456 Prep work for alternate ways to export variables, and more comments and
tests.
2013-08-05 01:23:26 -07:00
Sugu Sougoumarane 85fd9dff0f selflessness and atomic normalization 2013-04-10 15:43:10 -07:00
Mike Solomon 81e550ad3e move handy atomic macros to sync2 2013-03-07 19:18:45 -08:00
Alain Jobart 753b4af2ea Logging query engine state transitions, and using new atomic class.
LGTM Sugu.
2013-02-25 16:35:00 -08:00
Alain Jobart aca461d2af Better pattern for atomic.Int32. Using it in zkocc only for now.
LGTM Mike & Sugu.
2013-02-14 14:23:20 -08:00
Alain Jobart 0cf54deba6 Now also count state transitions in stats.States.
LGTM Mike.
2012-12-26 16:50:09 -08:00
Alain Jobart d3feb974e8 Adding a simple way to report States to debug/vars, using it for
sql engine status, zk conn in vttablet, zkocc->zk conns.

LGTM Mike.
2012-12-26 15:41:42 -08:00
Ric Szopa 02115e7048 Change the license notice to be shorter and refer to the license file. 2012-06-07 11:55:06 -07:00
Sugu Sougoumarane e2e3e0ef00 copyright deviolation 2012-04-01 21:58:56 -07:00
Sugu Sougoumarane a082e88409 updated to go's new build system and weekly.2012-03-04 2012-03-11 17:53:40 -07:00
Sugu Sougoumarane 2899c45aea tabletserver: big change. Fix all unprotected reads, standardize APIs and naming conventions. 2012-03-08 15:13:53 -08:00
Sugu Sougoumarane 72a862e6b3 initial source code check-in. 2012-02-24 23:30:03 -08:00