In OpenSSL 1.1.0 and higher, SSLv23_method causes some errors
in TLS handshake from time to time. As this method is depreceated
since 1.1.0, I have replaced it with the follow up method
TLS_method which is the most generic one.
It fixes the random test failures in tests like
- sndrcv_tls_ossl_anon_rebind.sh
Also added some debug output in OpenSSL error handling, which is
useful when analysing debug files.
closes: ./sndrcv_tls_ossl_anon_rebind.sh
Not all imfile tests have state file directories or a global working
directory defined. This results in usage of the default location.
While state file names should be sufficiently different, there is still
some riks of using the same name in different tests. That becomes
problematic if tests are run in parallel (and they are run in
parallel inside the regular CI).
Note: tcpflood is a testbench tool. This bug could lead to testbench
false positives. No way it can affect production deployments.
The tcpflood tool did improperly assume that a TCP sendto() call
would send messages of any size in a single shot. This is not the
case. It has now been corrected to proper behavior.
As a side-activity, some int variables which acutally needed to be
size_t have been fixed as well.
The previous approach was more or less delay based. We have now
changed the code to enable imdiag to detect if HUP is underway
and wait until it is completed. The new method still employs some
kind of timeout, but is now quite reliable. Most importantly,
it works great with long-running HUP processing, which can happen
e.g. when querying the system name takes long or some actions need
longer time to persist their HUP processing.
The new approach will most likely reduce CI flakes and also speed
up testbench runs. The speedup happens from not having to wait a
full delay in cases where we detect HUP is completed (plus reduced
timeout when we cannot clearly detect this - see code comments why
the new method is still considered more reliable than the old one).
Code note: we needed to slightly re-structure the way actual HUP
processing and the "HUP mutex" is handled. After best analysis,
this does not affect the reliability or speed in production
settings.
closes https://github.com/rsyslog/rsyslog/issues/5192
- Add TLS CRL support tp GnuTLS driver using gnutls_certificate_set_x509_crl_file.
- Add code in OpenSSL driver that works with OpenSSL 1.0.2 and higher.
Disable feature on older features with error message.
- Some cosmetic changes
- testbench: Add revoked certificate for testing (Including CRL PEM and other files)
- testbench: Add testcase for gtls and ossl testing revoked certificates
The output module uses Apache "Qpid Proton C API" which is a solid
AMQP protocol library implementation that can be integrated
very well into the rsyslog dev environment.
- Implemented Delivery with submitted and accepted state checking
- impstatscounter used in testcases
- saving of failed messages in a failed list with support of saving
and restoring.
- Add testcases (requires ENV variables) to testbench
- Using application/octect-stream (binary) to send messages based on
Microsoft Code Sample:
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-c-getstarted-send
* Note original Microsoft Samplecode is not working anymore, we are using
* QPID Proton Proactor based on
https://github.com/apache/qpid-proton/blob/main/c/examples/send.c
- requires QPID-PROTON Version 0.13 or higher because of the proactor API
- Add EventProperties configuration parameters
- using internal array instead of linkedlist for better performance
- using single byte helper to store message status (ubsubmit/submitted...)
- trigger wake up if messages need to be resubmitted (rejected)
- Slow down when sender credit reaches zero (10ns).
- Add enhanced performance stress test omazureeventhubs-stress.sh
- Add support for static library linking of qpid-proton
This is needed to build the module from source and remove
library package dependencies.
- adjusted valgrind suppressions
FileCreateMode allows to set the default file mode bits
when creating new files. As of now, it has only impact on the state file.
Add test suite as well.
Minor indentation fix in run_journal.yml
### ADDED
- [OMHIREDIS] module is now able to insert entries to a Redis Stream
- [OMHIREDIS] in 'stream' mode, module can insert the message to a custom field in the entry ('msg' by default)
- [OMHIREDIS] in 'stream' mode, module can acknowledge an entry coming from imhiredis (if entry was claimed but not ACK'ed)
- [OMHIREDIS] in 'stream' mode, acknowledgements can be made from dynamic templates or static values
- [OMHIREDIS] in 'stream' mode, module can approximately cap the size of the output stream
- [OMHIREDIS] in 'stream' mode, module can delete an entry while inserting its message (useful to remove entry coming from another stream with imhiredis)
- [OMHIREDIS] new tests for 'stream' mode
### FIXED
- [CONFIGURE.AC] Missing line to give omhiredis compilation status
- [IMHIREDIS] Add support for simple XREADs from Redis Streams (Redis >= 5.0 required)
- [IMHIREDIS] Add support for XREADGROUP from Redis Streams, allowing for user to define workers to dequeue logs in a stream
- [IMHIREDIS] stream mode can select fields to extract and insert in custom keys
- [IMHIREDIS] Add tests for the new 'stream' modes
- changed diag.sh to be able to start/stop/clean a redis server
- added helper functions in diag.sh to be able to query a redis server instance
- added new tests for imhiredis module to check
- that the queue mode works, with both lpop and rpop
- that the module is capable of handling a redis server going down
- that the module is capable of handling a redis server that appears afterwards
- that the subscribe mode works
This will easily permit to drop the last n characters from a property
without the need to know the exact length of the string. This is
especially useful as the exact length is most often not known
beforehand.
Now, length can have a negative value -n to denote that the
substring should be build between startpos and the character
-n chars from the end. This is a shortcut for stripping charactes
on "both ends" of the string.
Also, some hardening against invalid startpos and length has
been added.
Add new action parameters
- rotation.sizeLimit
- rotation.sizeLimitCommand
provide automatic output file rotation functionality feature-wise
equivalent to legacy $outchannel. This finally permits to use
this feature set in rscript.
In some client-server test cases, messages are supposed to be injected into
the instance 2(client), but they are actually injected into instance 1(server),
which may lead to false negative results. This patch fixed it by replacing
'injectmsg' with 'injectmsg2', and dealt with some minor issues.
Many "$InputTCPServer..." config directives did no longer work
and were completely ignored (e.g. "$InputTCPServerStreamDriverMode").
This was a regression from a08591be5d9 (May, 5th 2021).
closes https://github.com/rsyslog/rsyslog/issues/5021
In some cases, the logrotate is not fast enough rotating the
logfile after sighub has been send to inputfilegen process. It appears
to happen more often on github CI machines, when it happens,
inputfilegen reopens the NOT yet rotated file and keeps writing
into it. However most messages written until the next sighub are lost
after the logfile finally is rotated.
In order to avoid this, I have added an extra sleep between file
CLOSE and OPEN after sighub with a default of 5ms which appears to work
very well. This should give logrotate enough time to perform the
actuzal rotate.
Also added another parameter to inputfilegen to keep the process
open after finish. Which helps syncing with the testcases.
closes: https://github.com/rsyslog/rsyslog/issues/5000
Due the patch in PR https://github.com/rsyslog/rsyslog/pull/4895
state files are deleted with a 5 second delay in order to fix
missing or duplicated messages. However in INOTIFY mode, we need
an INOTIFY event to trigger a poll_tree that triggers the
delayed deletion. The testcase imfile-statefile-delete.sh will now
create empty dummy files after 6 seconds delay in order to trigger
INOTIFY events.
This fixes & closes https://github.com/rsyslog/rsyslog/issues/4958
rsyslog.conf may affect the host's local name. These changes were
so far only activated after the first HUP. This patch now ensures
that the configured local host name is applied correctly throughout
all processing, including early startup.
This patch causes a slight change of behaviour. However, the behaviour
was inconsitent before. Now it is consistent and according to the config.
Please note: this patch also exposes a global entry point via "regular"
dynamic loading as this makes things much easier to do. This is in-line
with ongoing simplification effort.
Finally, we also remove a CI test that we do no longer need because
the problem covered is now addressed differently and the original issue
can no longer occur.
closes https://github.com/rsyslog/rsyslog/issues/4975
- These tests will help find race conditions hopefully
- fix diag.sh issues running second instance in valgrind mode only
- Add check for minimum valgrind version for new relp tests
Add this both as module an input parameter. Complements already-existing
config param notifyonconnectionclose and mirrors the similar feature from
imptcp.
The module parameter acts as default, similarly to notifyonconnectionclose.
Note that in contrast to imptcp, we emit IP addresses and not host
names. This sticks with the traditional semantics of imtcp.
Note that we also fixed a mislading error message in the case when a
disallowed sender tried to connect.
Thanks to John Chivian for suggesting the addition.
When a to-be-monitored file is being rotated, some messages may be lost or
duplicated. In case of duplication, many file lines may be duplicated
depending on actual timing. The whole bug was primarily timing depenedent
in general. It most often was visible in practice when the monitored
file was very frequently rotated (we had some report with every few
seconds).
Note that while we try hard to not lose any messages, input file
rotation always has some loss potential. This is inevitable if
the monitored file is being truncated.
Also note that this bugfix affects imfile, only. It has nothing to do
and no relation to rsyslog output files being rotated on HUP.
closes: https://github.com/rsyslog/rsyslog/issues/4797
- Fixed an issue with numbers above int64 in syntax_ipv4.
Numbers that were up to 256 above the max of an int64
could incorrectly be detected as valid ipv4 digit.
- Simplified the IPv4 digit detection function and renamed
to isPosByte.
- added testcasse for malformed IPvc4 addresses
closes: https://github.com/rsyslog/rsyslog/issues/4940
