If "docker logs" was used on an offline container, the logger is leaked, leaving it up to the finalizer to close the file handle, which could block removal of the container. Further, the json file logger could leak an open handle if the logs are read without follow due to an early return without a close. This change addresses both cases.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
This change updates how we handle long lines of output from the
container. The previous logic used a bufio reader to read entire lines
of output from the container through an intermediate BytesPipe, and that
allowed the container to cause dockerd to consume an unconstrained
amount of memory as it attempted to collect a whole line of output, by
outputting data without newlines.
To avoid that, we replace the bufio reader with our own buffering scheme
that handles log lines up to 16k in length, breaking up anything longer
than that into multiple chunks. If we can dispense with noting this
detail properly at the end of output, we can switch from using
ReadBytes() to using ReadLine() instead. We add a field ("Partial") to
the log message structure to flag when we pass data to the log driver
that did not end with a newline.
The Line member of Message structures that we pass to log drivers is now
a slice into data which can be overwritten between calls to the log
driver's Log() method, so drivers which batch up Messages before
processing them need to take additional care: we add a function
(logger.CopyMessage()) that can be used to create a deep copy of a
Message structure, and modify the awslogs driver to use it.
We update the jsonfile log driver to append a "\n" to the data that it
logs to disk only when the Partial flag is false (it previously did so
unconditionally), to make its "logs" output correctly reproduce the data
as we received it.
Likewise, we modify the journald log driver to add a data field with
value CONTAINER_PARTIAL_MESSAGE=true to entries when the Partial flag is
true, and update its "logs" reader to refrain from appending a "\n" to
the data that it retrieves if it does not see this field/value pair (it
also previously did this unconditionally).
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
Add a benchmark for measuring how the logger.Copier implementation
handles logged lines of sizes ranging up from 64 bytes to 256KB.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When told to read additional attributes from logs that we've sent to the
journal, pull out all of the non-trusted, non-user fields that we didn't
hard-code ourselves. More of PR#20726 and PR#21889.
When reading entries in the journald log reader, set the time zone on
timestamps that we read to UTC, so that we send UTC values to the client
instead of values that are local to whatever timezone dockerd happens to
be running in.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
Log drivers are instantiated on a per-container basis, and passed the
container ID (along with other information) when they're initialized.
Drivers that care about that value are caching the value that they're
passed when they're initialized and using it in favor of the value
contained in Message structures that are passed to them, so the field in
Messages is unused, so we remove it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
This fix tries to fix build errors caused by updating
aws-sdk-go to v1.1.30.
This fix fixes#22961.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
This fix tries to address the issue raised in #22358 where syslog's
message tag always starts with `docker/` and can not be removed
by changing the log tag templates.
The issue is that syslog driver hardcodes `path.Base(os.Args[0])`
as the prefix, which is the binary file name of the daemon (`dockerd`).
This could be an issue for certain situations (e.g., #22358) where
user may prefer not to have a dedicated prefix in syslog messages.
There is no way to override this behavior in the current verison of
the docker.
This fix tries to address this issue without making changes in the
default behavior of the syslog driver. An additional
`{{.DaemonName}}` has been introduced in the syslog tag. This is
assigned as the `docker` when daemon starts. The default log tag
template has also been changed from
`path.Base(os.Args[0]) + "/{{.ID}}"` to `{{.DaemonName}}/{{.ID}}`.
Therefore, there is no behavior changes when log-tag is not provided.
In order to be consistent, the default log tag for fluentd has been
changed from `docker.{{.ID}}` to `{{DaemonName}}.{{.ID}}` as well.
The documentation for log-tag has been updated to reflect this change.
Additional test cases have been added to cover changes in this fix.
This fix fixes#22358.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Since 1.9, driver specific log tag options
`syslog-tag`
`gelf-tag`
`fluentd-tag`
have been deprecated in favor of the generic tag
option which is standard across different logging
drivers.
This fix removed the deprecated driver specific
log tag options of `syslog-tag`, `gelf-tag`,
`fluentd-tag` for 1.12 and updated the docs.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
The jsonlog logger currently allows specifying envs and labels that
should be propagated to the log message, however there has been no way
to read that back.
This adds a new API option to enable inserting these attrs back to the
log reader.
With timestamps, this looks like so:
```
92016-04-08T15:28:09.835913720Z foo=bar,hello=world hello
```
The extra attrs are comma separated before the log message but after
timestamps.
Without timestaps it looks like so:
```
foo=bar,hello=world hello
```
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This fix tries to add an additional syslog-format of `rfc5424micro` which follows
the same as rfc5424 except that it use microsecond resolution for timestamp. The
purpose is to solve the issue raised in #21793 where log events might lose its
ordering if happens on the same second.
The timestamp field in rfc5424 is derived from rfc3339, though the maximium
resolution is limited to "TIME-SECFRAC" which is 6 (microsecond resolution).
The appropriate documentation (`docs/admin/logging/overview.md`) has been updated
to reflect the change in this fix.
This fix adds a unit test to cover the newly introduced format.
This fix fixes#21793.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Following a journal log almost always requires a descriptor to be
allocated. In cases where we're running out of descriptors, this means
we might get stuck while attempting to start following the journal, at a
point where it's too late to report it to the client and clean up
easily. The journal reading context will cache the value once it's
allocated, so here we move the check earlier, so that we can detect a
problem when we can still report it cleanly.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
When we set up to start following a journal, if we get error results
from sd_journal_get_fd() or sd_journal_get_events() that prevent us from
following the journal, report the error instead of just mysteriously
failing.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
This cleans up some of the use of the filepoller which makes reading
significantly more robust and gives fewer changes to fallback to the
polling based watcher.
In a lot of cases, if the file was being rotated while we were adding it
to the watcher, it would return an error that the file doesn't exist and
would fallback.
In some cases this fallback could be triggered multiple times even if we
were already on the fallback/poll-based watcher.
It also fixes an open file leak caused by not closing files properly on
rotate, as well as not closing files that were read via the `tail`
function until after the log reader is completed.
Prior to the above changes, it was relatively simple to cause the log
reader to error out by having quick rotations, for example:
```
$ docker run --name test --log-opt max-size=10b --log-opt max-files=10
-d busybox sh -c 'while true; do usleep 500000; echo hello; done'
$ docker logs -f test
```
After these changes I can run this forever without error.
Another fix removes 2 `os.Stat` calls when rotating files. The stat
calls are not needed since we are just calling `os.Rename` anyway, which
will in turn also just produce the same error that `Stat` would.
These `Stat` calls were also quite expensive.
Removing these stat calls also seemed to resolve an issue causing slow
memory growth on the daemon.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
When following a journal-based log, it was possible for the worker
goroutine, which reads the journal using the journal context and sends
entry data down the message channel, to be scheduled after the function
which started it had returned. This could create problems, since the
invoking function was closing the journal context object and message
channel before it returned, which could trigger use-after-free segfaults
and write-to-closed-channel panics in the worker goroutine.
Make the cleanup in the invoking function conditional so that it's only
done when we're not following the logs, and if we are, that it's left to
the worker goroutine to close them.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
The journald log reader keeps a map of following readers so that it can
close them properly when the journald reader object itself is closed,
but it was possible for its worker goroutine to be scheduled so that the
worker attempted to remove a reader from the map before the reader had
been added to the map. This patch adds the item to the map before
starting the goroutine which is expected to eventually remove it.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com> (github: nalind)
The GCP logging driver is calling out to GCP cloud service on package
init.
This is regardless if you are using GCP logging or not.
This change makes this happen on the first invocation of a new GCP
logging driver instance instead.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
this allows user to choose the compression type (i.e. gzip/zlib/none) using
--log-opt=gelf-compression-type=none or the compression level (-1..9) using
--log-opt=gelf-compression-level=0 for gelf driver.
Signed-off-by: Daniel Dao <dqminh@cloudflare.com>
This change centralizes the template manipulation in a single package
and adds basic string functions to their execution.
Signed-off-by: David Calavera <david.calavera@gmail.com>
Previously docker used obsolete rfc3164 syslog format for syslog. rfc3164 explicitly
uses semicolon as a separator between 'TAG' and 'Content' section of the log message.
Docker uses semicolon as a separator between image name and version tag.
When {{.ImageName}} was used as a tag expression and contained ":" syslog parser mistreated
"tag" part of the image name as syslog message body, which resulted in incorrect "syslogtag" been reported by syslog
daemon.
Use of rfc5424 log format partually fixes the issue as it does not use semicolon as a separator.
However using default rfc5424 syslog format itroduces backward incompatability because rsyslog template keyword %syslogtag%
is parsed differently. In rfc3164 it uses the "TAG" part reported before the "pid" part. In rfc5424 it uses "appname" part reported
before the pid part, while tag part is introduced by %msgid% part.
For more information on rsyslog configuration properties see: http://www.rsyslog.com/doc/master/configuration/properties.html
Added two options to specify logging in either rfc5424, rfc3164 format or unix format omitting hostname in order to keep backwards compatability with
previous versions.
Signed-off-by: Solganik Alexander <solganik@gmail.com>
Sebastiaan van Stijn
Alexander Morozov
Alexander Morozov
Stefan J. Wernli
Nalin Dahyabhai
Brian Goff
Vincent Demeester
Sven Dowideit
allencloud
Akihiro Suda
Yong Tang
Samuel Karp
Shijiang Wei
Zhuoyun Wei
Alan Scherger
Pierre Carrier
Michael Crosby
Jessica Frazelle
Antonio Murdaca
Antonio Murdaca
Daniel Dao
David Calavera
Solganik Alexander
Arnaud Porterie
Tibor Vass