Coverity Scan detected a file handle leak. This was introduced in the
refactoring done in the past couple of days. Was NOT present in any
released code, so is nothing to really care about.
fixes Coverity Scan ID 188771
This plugin is used to annotate records logged by Kubernetes containers.
It will add the namespace uuid, pod uuid, pod and namespace labels and
annotations, and other metadata associated with the pod and namespace.
It will work with either log files in `/var/log/containers/*.log` or
with journald entries with `CONTAINER_NAME` and `CONTAINER_ID_FULL`.
For file logs, the filename must match this regex:
/var/log/containers/([a-z0-9]([-a-z0-9]*[a-z0-9])?(\\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_([^_]+)_(.+)-([a-z0-9]{64})\\.log$
The first match is the pod name, the second is the container hash (not
currently used), the third is the namespace name, the fourth is the
container name, and the fifth is the container id. You can specify a
different regex by using the module or action configuration parameter
`filenameregex` but keep in mind that the field positions are hardcoded
so the regex must have the same fields in the same order.
For journald logs, there must be a field `CONTAINER_NAME` which matches this:
^[^_]+_([^\\._]+)(\\.([^_]+))?_([^_]+)_([^_]+)_[^_]+_[^_]+$
The first match is the container name, the second is the container hash
(not currently used), the third is the pod name, and the fourth is the
namespace name. The record must also have the field
`CONTAINER_ID_FULL`. You can specify a different regex by using the
module or action configuration parameter `containerregex` but keep in
mind that the field positions are hardcoded so the regex must have the
same fields in the same order.
The Kubernetes metadata is added to the record in the top-level fields
`kubernetes` and `docker`. See
https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/kubernetes.yml
and
https://github.com/ViaQ/elasticsearch-templates/blob/master/namespaces/docker.yml
for more details.
*Configuration*
`kubernetesurl` - Required - URL of the Kubernetes API server e.g.
`https://localhost:8443`
`tls.cacert` - Required - full path and file name of file containing
the CA cert of the Kubernetes API server cert issuer
`tokenfile` - Required (or `token`) - the file containing the token
to use to authenticate to the Kubernetes API server
`token` - Required (or `tokenfile`) - the token to use to
authenticate to the Kubernetes API server
`annotation_match` - Optional - by default no pod or namespace annotations
will be added to the records - this parameter is an
array of patterns to match the keys of the `annotations`
field to include in the `annotations` field or the
`namespace_annotations` field.
*Example*
module(load="imfile" mode="inotify")
module(load="mmkubernetes" kubernetesurl="https://localhost:8443"
tls.cacert="/etc/rsyslog.d/mmk8s.ca.crt"
tokenfile="/etc/rsyslog.d/mmk8s.token" annotation_match=["."])
template(name="tpl" type="list") {
property(name="jsonmesg")
constant(value="\n")
}
ruleset(name="k8s") {
action(type="mmkubernetes")
action(type="omfile" file="/var/log/k8s.log" template="tpl")
}
input(type="imfile" file="/var/log/containers/*.log" tag="kubernetes" addmetadata="on" ruleset="k8s")
if ($!_SYSTEMD_UNIT == "docker.service") and (strlen($!CONTAINER_NAME) > 0) then {
call k8s
}
*Notes*
We use lognorm instead of regex to parse filenames and CONTAINER_NAME
since it is faster than regex and this parsing is in the critical path.
We cannot use ln_loadSamplesFromString with liblognorm 2.0.2, so
disallow the filenamerules and containerrules parameters for older
rsyslog versions with the older liblognorm.
Due to a limitation in mmnormalize, we cannot directly parse a filename
like this:
something_something_this-is-a-container-name-092039840293.log
That is, we cannot handle a container name with `-` in it. Instead,
parse the entire name + id into container_name_and_id, then parse into
separate container_name and container_id in the code.
*Credits*
This work is based on https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter
and has many of the same features.
This commit greatly refactors imfile internal workings. It changes the
handling of inotify, FEN, and polling modes. Mostly unchanged is the
processing of the way a file is read and state files are kept.
This is about a 50% rewrite of the module.
Polling, inotify, and FEN modes now use greatly unified code. Some
differences still exists and may be changed with further commits. The
internal handling of wildcards and file detection has been completely
re-written from scratch. For example, previously when multi-level
wildcards were used these were not reliably detected. The code also
now provides much of the same functionality in all modes, most importantly
wildcards are now also supported in polling mode.
The refactoring sets ground for further enhancements and smaller
refactorings. This commit provides the same feature set that imfile
had previously and all existing CI tests pass, as do some newly
created tests.
Some specific changes:
- bugfix: module parameter "sortfiles" ignored
This parameter only works in Solaris FEN mode, but is otherwise
ignored. Most importantly it is ignored under Linux.
fixes https://github.com/rsyslog/rsyslog/issues/2528
- bugfix: imfile did not pick up all files when not present
at startup
fixes https://github.com/rsyslog/rsyslog/issues/2241
fixes https://github.com/rsyslog/rsyslog/issues/2230
fixes https://github.com/rsyslog/rsyslog/issues/2354
- bugfix: directories only support "*" wildcard, no others
fixes https://github.com/rsyslog/rsyslog/issues/2303
- bugfix: parameter "sortfiles" did only work in FEN mode
fixes https://github.com/rsyslog/rsyslog/issues/2528
- provides the ability to dynamically add and remove files via
multi-level wildcards
see also https://github.com/rsyslog/rsyslog/issues/1280
- the state file name currently has been changed to inode number
This will further be worked on in upcoming PRs
see also https://github.com/rsyslog/rsyslog/issues/2231
- some enhancements were also done to CI tests, most importantly
they were made more compatibile with BSD
Note that most of the mentioned bug fixes cannot be applied to older
versions, as they fix design issues which are solved by the refactoring.
Thus there are not separate commits for them.
Distro maintainers: you need to decide to apply this patch as whole
or not. Believe me, it is not worth the effort to try to extract
specific patches from this commit. There is a good reason we do
not have multiple commits.
closes https://github.com/rsyslog/rsyslog/issues/2359
