This is work torward the ability to run the testbench in parallel.
Some small issues in tests have also been detected, which were not
previously seen. They have been fixed. We did not do separate
commits for this as it would clutter the commit log with not
really relevant info.
Also move some cleanup to "make clean" target
To support parallel testbench runs, we need to have dynamic work files.
So deleting work files on each test does not work, especially as we can
no longer assume they are "left-overs" from a failed test - they may
actually (and quite likely) belong to tests that run in parallel.
So the proper solution is to do via "make clean".
closes https://github.com/rsyslog/rsyslog/pull/2916
this introduces symlink detection and following as well
as monitoring changes on them. Also added test for the new
functionality and ensuring the original symlink behavior
stays as well.
by default, we now use 10s timeout for terminating inputs. On some
slow machines (CI under heavy load) the system default is not sufficient,
what can lead to memory leaks and thus valgrind failures.
In theory, this should work without the sleep, but in practice
this triggers some issues I could not yet diagnost. Reducing,
however, seems possible. This will lead to notable speedup of
testbench runs.
changes some of the test commands to use bash functions
includes some small bug fixes to tests where bugs were
previously not seen due to different plumbing.
This does not work reliably if multiple instances of rsyslog
builds run on a single machine. We need to improve, but this
commit makes conflict less likely and provides some diagnostic
info to help guide us towards a final solution.
Also a couple of changes to testbench worth mentioning:
* use cp -f to ensure files can be overwirtten in VBUILD
* fix issue of missing include test file in EXTRA_DIST
* new supressions
* testbench: try to use local system dependency cache
avoid going to Internet repos if not absolutely necessary. For
development containers, they should be pre-populated with the
important dependencies.
* do not enable libfaketime if ASAN is selected
unfortunately, libfaketime does not work in that case
see also https://github.com/rsyslog/rsyslog/issues/174
Some old tests are carried out via the nettester tool. This was
our initial shot at a testbench a couple of years ago. While it
worked back then, the testbench framework has been much enhanced.
These old tests are nowadays very hard to handle, as they miss
debug support etc. So it is time to refactor them to new style.
As a side-activity, the testbench plumbing has been enhanced to
support some operations commonly needed by these tests. Contrary
to pre-existing plumbing, these new operations are now crafted
using bash functions, which we consider superior to the current
method. So this is also the start of converting the older-style
functionality into bash functions. We just did this now because
it was required and we entangled it into the test refactoring
because it was really needed. Else we had to write old-style
operations and convert them in another commit, which would
have been a waste of time.
Special thanks to Pascal Withopf for the initial step of taking
old tests and putting config as well as test data together into
the refactored tests, on which Rainer Gerhards than could build
to create the new tests and update testbench plumbing.
for complex multi-line messages (XML in particular), the multiLine method
does not work. We now have a capability to specify via a regex when a frame
starts (and the previous thus ends).
adds imptcp input parameter "framing.delimiter.regex"
This makes the ES tests more universally available (they should now
also support running in containers). It also simplifies the tests
as fewer support files are needed.
Add support for a 'create' write operation type in addition to
the default 'index'. Using create allows specifying a unique id
for each record, and allows duplicate document detection.
Add support for checking each record returned in a bulk index
request response. Allow specifying a ruleset to send each failed
record to. Add a local variable `omes` which contains the
information in the error response, so that users can control how
to handle responses e.g. retry, or send to an error file.
Add support for response stats - count successes, duplicates, and
different types of failures.
Add testing for bulk index rejections.
This commit greatly refactors imfile internal workings. It changes the
handling of inotify, FEN, and polling modes. Mostly unchanged is the
processing of the way a file is read and state files are kept.
This is about a 50% rewrite of the module.
Polling, inotify, and FEN modes now use greatly unified code. Some
differences still exists and may be changed with further commits. The
internal handling of wildcards and file detection has been completely
re-written from scratch. For example, previously when multi-level
wildcards were used these were not reliably detected. The code also
now provides much of the same functionality in all modes, most importantly
wildcards are now also supported in polling mode.
The refactoring sets ground for further enhancements and smaller
refactorings. This commit provides the same feature set that imfile
had previously and all existing CI tests pass, as do some newly
created tests.
Some specific changes:
- bugfix: module parameter "sortfiles" ignored
This parameter only works in Solaris FEN mode, but is otherwise
ignored. Most importantly it is ignored under Linux.
fixes https://github.com/rsyslog/rsyslog/issues/2528
- bugfix: imfile did not pick up all files when not present
at startup
fixes https://github.com/rsyslog/rsyslog/issues/2241
fixes https://github.com/rsyslog/rsyslog/issues/2230
fixes https://github.com/rsyslog/rsyslog/issues/2354
- bugfix: directories only support "*" wildcard, no others
fixes https://github.com/rsyslog/rsyslog/issues/2303
- bugfix: parameter "sortfiles" did only work in FEN mode
fixes https://github.com/rsyslog/rsyslog/issues/2528
- provides the ability to dynamically add and remove files via
multi-level wildcards
see also https://github.com/rsyslog/rsyslog/issues/1280
- the state file name currently has been changed to inode number
This will further be worked on in upcoming PRs
see also https://github.com/rsyslog/rsyslog/issues/2231
- some enhancements were also done to CI tests, most importantly
they were made more compatibile with BSD
Note that most of the mentioned bug fixes cannot be applied to older
versions, as they fix design issues which are solved by the refactoring.
Thus there are not separate commits for them.
Distro maintainers: you need to decide to apply this patch as whole
or not. Believe me, it is not worth the effort to try to extract
specific patches from this commit. There is a good reason we do
not have multiple commits.
closes https://github.com/rsyslog/rsyslog/issues/2359
some tests use rsyslog.out.* file names, which were so far not cleaned up.
This can lead to all sorts of problems, including failure of
"make distcheck" due to leftover files. We now delete all of these files
on cleanup. This should not have any undesired side-effects.
Thanks to Kasumi Hanazuki for spotting this problem.
see also https://github.com/rsyslog/rsyslog/pull/2527
This enables easy JSON generation via template.
This commit also corrects an issue with the constant "jsonf"
format. That was recently added, and the implementation problem
only became visible when used inside a larger json object. No
officially released code is affected, thought - so it rellay
is just a side-note.
closes https://github.com/rsyslog/rsyslog/issues/2347
In theory, TZ should be set by the OS. Unfortuantely, this seems
to be not the case any longer on many Linux distros. We now check
it and set it appropriate if not already given.
closes https://github.com/rsyslog/rsyslog/issues/2054
some omprog tests are racy as they depend on sleep calls to "synchoronize".
We have mostly been able to change this to more reliable synchronization.
A small window of raciness remains for some tests, but this seem to be
inevitable. Tests on fully loaded systems seem to incidate that we have
solved the issue sufficiently.
If that doesn't solve the sporadic failures, we need to further think about
how to handle this.
see also https://github.com/rsyslog/rsyslog/issues/2403
When directories and files are created at the same time,
imfile may missed subdirs or file if the machine is on high load.
The handling has been enhanced to scan newly created directories ALWAYS for
matching files.
This should fix following issues:
closes https://github.com/rsyslog/rsyslog/issues/2271
However we still have a problem when a multilevel directory configurations.
Details are discussed here https://github.com/rsyslog/rsyslog/issues/2354
They may not be accessible because they (or parts of the
network) can be down or access is not permitted. In this
case, the affected tests skip themselves.
Special thanks to Thomas D. (Whissi) for providing the check
function.
closes https://github.com/rsyslog/rsyslog/issues/2339
The error message regularly occurs if no kafka/zk instance is running,
which should be the case at start of test. However, the message
tends to make think reviewers that it is a real error. So we now
hide it.
previous timeout caused very long runtime in cases where rsyslog
segfaulted on startup, which made the testbench failure harder to
diagnose than necessary.
imfile: Initial implementation of solaris FEN API (without wildcards)
FEN API is used to implement simular event based file
monitoring like with inotify.
This commit provides partial functionality. It supports event-driven
processing of files but does not yet provide wildcard functionality.
Wildscards will be provided by a later commit.
This can be committed as-is because the previous code did also
not provide wildcard support, so this is an improvement for
Solaris in any case.
see also https://github.com/rsyslog/rsyslog/issues/1954
