Add support for a 'create' write operation type in addition to
the default 'index'. Using create allows specifying a unique id
for each record, and allows duplicate document detection.
Add support for checking each record returned in a bulk index
request response. Allow specifying a ruleset to send each failed
record to. Add a local variable `omes` which contains the
information in the error response, so that users can control how
to handle responses e.g. retry, or send to an error file.
Add support for response stats - count successes, duplicates, and
different types of failures.
Add testing for bulk index rejections.
Triggering condition:
- action queue in disk mode (or DA)
- batch is being processed by failed action in retry mode
- rsyslog is shut down without resuming action
In these cases messages may be lost by not properly writing them
back to the disk queue.
closes https://github.com/rsyslog/rsyslog/issues/2760
Added them after intensive checking that these are indeed false positives.
Among other, checks via TSAN as well as intensive manual debugging of that
issue. Also, no other platform shows the issues. So I conclude that it is
safe to suppress it.
when writing disk queue files during shutdown, access to freed
memory can occur under these circumstances:
- action A is processing data, but could not complete it
most importantly, the current in-process batch needs not to
be totally completed. Most probable cause for this scenario
is a suspended action in retry mode.
- action A is called from a ruleset RA which
- does not have a queue assigned
- where RA is called from a ruleset RO which is bound
to the input from which the message originated
- RO must be defined before RA inside the expanded config
- Disk queues (or the disk part of a DA queue) must be utilized
by A
When re-injecting the unprocessed messages from A into the disk queue, the
name of ruleset RO is accessed (for persisting to disk). However, RO is
already destructed at this point in time.
The patch changes the shutdown processing of rulesets, so that all
shutdown processing is done before any ruleset data is destructed. This
ensures that all data items which potentially need to be accessed
remain valid as long as some part may potentially try to access them.
This follows a the approach used in
https://github.com/rsyslog/rsyslog/pull/1857
where obviously that part of the problem was not noticed.
see also https://github.com/rsyslog/rsyslog/issues/1122
closes https://github.com/rsyslog/rsyslog/issues/2742
Make the skeleton easier to understand by removing transaction support.
Also, transaction failures did not work as explained in the skeleton,
because of issue #2420. In the future, a 'plugin-with-transactions.py'
example can be added, ideally once the issue is solved.
When authmode is configured, it is checked if the value is
valid, before using it.
If the value is invalid an error message is put out.
closes https://github.com/rsyslog/rsyslog/issues/2743
https://github.com/rsyslog/rsyslog/issues/2721
Was not working on python3 - must use `encode()` to convert the
string to a `bytes` object.
Also run the server with python -u to make sure we get the log
output from the kubernetes test server.
Based on discussion, the error level is being adjusted to reflect
that the action (reloading the lookup table) is normal/expected
and not an actual error condition.
refs rsyslog/rsyslog#2713
