some omprog tests are racy as they depend on sleep calls to "synchoronize".
We have mostly been able to change this to more reliable synchronization.
A small window of raciness remains for some tests, but this seem to be
inevitable. Tests on fully loaded systems seem to incidate that we have
solved the issue sufficiently.
If that doesn't solve the sporadic failures, we need to further think about
how to handle this.
see also https://github.com/rsyslog/rsyslog/issues/2403
When directories and files are created at the same time,
imfile may missed subdirs or file if the machine is on high load.
The handling has been enhanced to scan newly created directories ALWAYS for
matching files.
This should fix following issues:
closes https://github.com/rsyslog/rsyslog/issues/2271
However we still have a problem when a multilevel directory configurations.
Details are discussed here https://github.com/rsyslog/rsyslog/issues/2354
They may not be accessible because they (or parts of the
network) can be down or access is not permitted. In this
case, the affected tests skip themselves.
Special thanks to Thomas D. (Whissi) for providing the check
function.
closes https://github.com/rsyslog/rsyslog/issues/2339
The error message regularly occurs if no kafka/zk instance is running,
which should be the case at start of test. However, the message
tends to make think reviewers that it is a real error. So we now
hide it.
previous timeout caused very long runtime in cases where rsyslog
segfaulted on startup, which made the testbench failure harder to
diagnose than necessary.
imfile: Initial implementation of solaris FEN API (without wildcards)
FEN API is used to implement simular event based file
monitoring like with inotify.
This commit provides partial functionality. It supports event-driven
processing of files but does not yet provide wildcard functionality.
Wildscards will be provided by a later commit.
This can be committed as-is because the previous code did also
not provide wildcard support, so this is an improvement for
Solaris in any case.
see also https://github.com/rsyslog/rsyslog/issues/1954
Works equal like for kafka and zookeeper. Elasticsearch is downloaded, configured and
started/stopped by the diag.sh script. Normal user access is sufficient.
Only es-basic.sh test has been adjusted with the new
method. Elasticsearch instance is limited to 128MB memory, should be sufficient for
the testbench.
If the parameter "log.file" is specified, impstats writes its own
log file. However, HUP is not handled for this file, which makes
the functionality unusable with log rotation. It is also counter-
intuitive for users.
This patch enables correct HUP processing. As a sideline, it also
introduces a generic HUP processing framework for non-action type
of loadable modules.
closes https://github.com/rsyslog/rsyslog/issues/1662
closes https://github.com/rsyslog/rsyslog/issues/1663
When enabled, failed messages are kept in memory queue and saved on
shutdown and loaded on startup. This prevents message loss of failed
messages.
The failed items are saved in a simple Linked List and automatically
being reprocessed on resume or doaction.
Load/Save logik implemented.
This should also remove the last possible message loss from issue:
https://github.com/rsyslog/rsyslog/issues/1052
This somehow elimitated a lot of failing messages which were not
processed on shutdown (flush) before. The old callback API was
depreceated.
This should also elimitate most of the message loss when
kafka queue is full from issue:
https://github.com/rsyslog/rsyslog/issues/1052
The new code was tested with
queue.buffering.max.messages=100 and 2000 injected messages.
Testbench: Try starting kafka server second time if failed on
first try. Sometimes the broker is still registered in zookeeper.
Also fixed tryresume and doAction handling when action is
in suspend state.
Calling for Callbacks more often.
Adjusted kafka fail test settings.
diag.sh script detects of kafka server comes up proberly
Added liblz4 to dependencies for omkafka/imkafka needed
for static linking.
This addresses the omkafka queue problem from issue:
https://github.com/rsyslog/rsyslog/issues/1052
Yet it does not fully solve the message problem yet,
but I am working on this.
Basic features need to be added now.
testbench: Added kafka test with multi kafka instances
All kafka instances are connected to one zookeeper instance.
Diag script has been adjusted to propper start/stop single
kafka instances.
autotools: Added version check for librdkafka to be higher than 0.9.1
The current value (5 minutes) seemed excessevily wrong and should
probably indicate an error in any case. The new value of 2 minutes
should be fairly safe as well.
