imfile: Initial implementation of solaris FEN API (without wildcards)
FEN API is used to implement simular event based file
monitoring like with inotify.
This commit provides partial functionality. It supports event-driven
processing of files but does not yet provide wildcard functionality.
Wildscards will be provided by a later commit.
This can be committed as-is because the previous code did also
not provide wildcard support, so this is an improvement for
Solaris in any case.
see also https://github.com/rsyslog/rsyslog/issues/1954
Works equal like for kafka and zookeeper. Elasticsearch is downloaded, configured and
started/stopped by the diag.sh script. Normal user access is sufficient.
Only es-basic.sh test has been adjusted with the new
method. Elasticsearch instance is limited to 128MB memory, should be sufficient for
the testbench.
If the parameter "log.file" is specified, impstats writes its own
log file. However, HUP is not handled for this file, which makes
the functionality unusable with log rotation. It is also counter-
intuitive for users.
This patch enables correct HUP processing. As a sideline, it also
introduces a generic HUP processing framework for non-action type
of loadable modules.
closes https://github.com/rsyslog/rsyslog/issues/1662
closes https://github.com/rsyslog/rsyslog/issues/1663
When enabled, failed messages are kept in memory queue and saved on
shutdown and loaded on startup. This prevents message loss of failed
messages.
The failed items are saved in a simple Linked List and automatically
being reprocessed on resume or doaction.
Load/Save logik implemented.
This should also remove the last possible message loss from issue:
https://github.com/rsyslog/rsyslog/issues/1052
This somehow elimitated a lot of failing messages which were not
processed on shutdown (flush) before. The old callback API was
depreceated.
This should also elimitate most of the message loss when
kafka queue is full from issue:
https://github.com/rsyslog/rsyslog/issues/1052
The new code was tested with
queue.buffering.max.messages=100 and 2000 injected messages.
Testbench: Try starting kafka server second time if failed on
first try. Sometimes the broker is still registered in zookeeper.
Also fixed tryresume and doAction handling when action is
in suspend state.
Calling for Callbacks more often.
Adjusted kafka fail test settings.
diag.sh script detects of kafka server comes up proberly
Added liblz4 to dependencies for omkafka/imkafka needed
for static linking.
This addresses the omkafka queue problem from issue:
https://github.com/rsyslog/rsyslog/issues/1052
Yet it does not fully solve the message problem yet,
but I am working on this.
Basic features need to be added now.
testbench: Added kafka test with multi kafka instances
All kafka instances are connected to one zookeeper instance.
Diag script has been adjusted to propper start/stop single
kafka instances.
autotools: Added version check for librdkafka to be higher than 0.9.1
The current value (5 minutes) seemed excessevily wrong and should
probably indicate an error in any case. The new value of 2 minutes
should be fairly safe as well.
most importantly, include line numbers, so that rsyslog startup
error messages can be traced to the right line. Due to the framework,
the line number info is otherwise not visible.
- now omprog kills the program by issuing TERM signal, but it can wait indefinitely when TERM is ignored (will fix this in a separate commit)
- omkafka was complaining of deprecated-fn-use in 0.9.x versions of librdkafka, which is now fixed
- fixed an incorrect comment in imdiag
