The first dataQueue channel doesn't serve any purpose now.
It was initially needed because the first send used to be non-blocking,
and the subsequent ones used to block. However, the non-blocking part
was later moved to the second stage.
With dataQueue removed, the code is much simpler now.
The go runtime schedules goroutines in any order. This means that
multiple sends can happen on a buffered channel before a runnable
receiver goroutine gets scheduled. This could cause dropped messages
even on an uncontended system.
This seems to happen very frequently for streamlog where the channel
size is 1. For now, I'm increasing this to match the original sender
channel size. Hopefully, this will lead to fewer dropped messages.
The writers have individual backoff, and internal drops can result in
not-insignificant losses during query storms. These are lost simply due
to noise in the Go scheduler, not due to actual backlog.
If we're more synchronous, we can be less flaky. ServiceManagement is
too heavyweight for what we're trying to accomplish in this test.
Also discovered that the ServeLogs handler doesn't return to the client
until it serves the first method (this broke a test rewrite). This has
been annoying me anyways on some low-traffic servers, so fixing that was
a nice pickup.
This value optimises for the wrong case: the case where the feature is
unused. If a feature is inefficient, the correct approach is not to
optimise for avoiding its use, but to resolve the inefficiency.
Additionally, the common case is for this feature to be used. We have
found it useful to have an always-on querylog. We have found it to be
reasonably efficient.
This "optimisation" was added with the first iteration of this feature
and likely falls into the category of premature optimisation.
TODO(plobsing): eliminate flakiness and sleepy races in the
streamlog_test.
Simplified the streamlogger code.
Extended it to allow for local non-http subscribers.
Added tests.
The non-http subscription will be used for additional
monitoring and status reporting.
Sugu Sougoumarane
Michael Berlin
Alain Jobart
Peter Lobsinger
Anthony Yeh
Ryszard Szopa