Updated the imtcp test suite to set the "SocketBacklog" parameter to a higher
value to accommodate tests with a large number of concurrent connections.
This change addresses potential CI flakes caused by SYN queue overflows during
test runs and significantly reduces test runtime by avoiding delays during
session setup.
The adjusted SocketBacklog parameter ensures smoother handling of high
connection counts, improving test reliability and performance.
... which went unnoticed during regular make check runs, but lead to
"left-over files after distclean" after make distcheck.
This could also potentially be responsible for some flakes that were
not fully explainable.
This patch implements a simple round-robin load balancer
for omfwd. It provides equal distribution of load to a pool
of target servers.
The code currently has no different modes and no special tuning
for the load balancer. However, it works very well in the most
common use cases. Furthermore, it provides a solid base on which
more elaborate functionality can be build if there is need to.
The new functionality is fully backwards compatible with previous
configuration settings.
New action() config params:
* pool.resumeinterval
New/"changed" rstats counters
Each target receives its own set of pstats counters. Most
importantly this is the case for byte counts. That counter retains
the same naming, but there may now be multiple of these counters,
one for each target ip, port tuple.
New pstats message count to target
Among others, this can be used for checking that the load balancer
works as intended. The so-far byte count emitted does not provide
a clear indication of how many messages the targets had actually
processed.
For obvious reasons, this message count makes most sense in
advanced load balancing scenarios, but also provides additional
insight into round-robin. Non-matches indicate that targets
went offline, and we can now evaluate the impact this had
on processing.
- re-design rebind functionality
This now works at the transaction level. It causes a rebind of all
pool members. Previous code did not work 100% correct since for a
couple of years now (after output batching integration).
As cleanup, rebindInterval support has been removed from tcpClt,
because omfwd is the only user. This permits a cleaner code path.
We also noticed a bug with rebindInterval: it caused some mild
message duplication for quite some time. This went unnoticed.
To address that efficiently, rebindInterval in the future will
be considered once per batch. That means up to (maxBatchSize - 1)
messages may be transmitted more than the rebindinterval is.
That's the cleanest mode of operation and should not make any
difference for real deployments.
Some additional work done in this commit:
netstream: harden component against upper-layer logic errors
network subsystem: better handle API errors and provide more info
omfwd: add new parameter "iobuffer.maxsize"
add new global parameter debug.abortoninternalerror and use it
This parameter permits to make test runs fail when an internal error
is detected and gracefully handled by rsyslog. While it is great to
have it gracefully handled in practice, we should not accept this
during testing. The new parameter permits to abort in this case and
emits the related error message beforehand. It is turned on by
default in our regular tests.
add dedicated error code for "hard" program errors
omfwd: some cleanup + error message fix + new debug level messages
imptcp: improve error messages
add omfwd option to NOT do extended connection check
also output wrkr id in some omfwd messages (primarily debugging aid)
better debug info via LogMsg() interface
improve messages regarding imptcp and omfwd suspension / thread IDs
refactor and enchance minitcpsrvr for mimicing died servers
new global (debugging) option, correction of an informational msg
add global option allmessagestostderr
add new tests
- output all loaded ciphers and engines.
- Add new global option "defaultopensslengine" to customize the
default openssl engine. If not defined, openssl will handle the
default engine automatically.
- Add simple openssl performance test with defaultopensslengine
set to rdrand (Intel).
- removed unneeded testcase files in runtime folder.
- corrected whitelist settings for debug.files in TLS testcases
- Extracted basic OpenSSL helper functions into own module net_ossl.h/net_ossl.c
Both are compiled into lmnsd_ossl.
- Cleanup of OpenSSL code, fixed minor compiler and linking issues.
- Added DTLS Sender option DTLS into tcpflood for testbench.
- Add initial implementation of imdtls input module. Added to configure and makefile
- Add initial implementation of omdtls output module. Added to configure and makefile
- Add multiple basic tests for imdtls receiving data by using tcpflood.
- Add multiple send-receive test for imdtls and omdtls based on existing tls tests.
- Add timeout and sessionbreak tests for imdtls stress testing.
closes: https://github.com/rsyslog/rsyslog/issues/5211
The previous test did not always detect an abort of rsyslog/omusrmsg.
The detection method has now been improved, so it is far more
probable that an abort is detected.
While doing this, we noticed that the omusrmsg-noabort-legacy test was
now a 100% duplicate. There is no need any longer to check pure legacy
syntax, and so that test has been removed.
We also added a valgrind-based test ofr omusrmsg-noabort, which furthers
strengthens bug detection. Most importantly, it helps us to detect
potentially new memory leaks on all CI platforms (in case the lib
behaves differently depending on os/distro).
see also https://github.com/rsyslog/rsyslog/issues/5294
- Added test for connection interrupts (requires root)
- Corrected handling of PN_TRANSPORT_CLOSED.
- Make sure Connection is being reestablished trough tryResume
- Enhanced Debug log output
closes: https://github.com/rsyslog/rsyslog/issues/5269
This provides richer and easier to process logs for error and warning
cases. One goal is to enable automatic operations without the need
to consolidate multiple message to a single information.
This improves one situation in gtls driver and provides a more
generic approach in ossl driver for OpenSSL error reporting.
There is probably still room for improvement, however this patch
is at least a good starting point for further work. Please
provide feedback if you need more!
closes https://github.com/rsyslog/rsyslog/issues/5244
- Add TLS CRL support tp GnuTLS driver using gnutls_certificate_set_x509_crl_file.
- Add code in OpenSSL driver that works with OpenSSL 1.0.2 and higher.
Disable feature on older features with error message.
- Some cosmetic changes
- testbench: Add revoked certificate for testing (Including CRL PEM and other files)
- testbench: Add testcase for gtls and ossl testing revoked certificates
The output module uses Apache "Qpid Proton C API" which is a solid
AMQP protocol library implementation that can be integrated
very well into the rsyslog dev environment.
- Implemented Delivery with submitted and accepted state checking
- impstatscounter used in testcases
- saving of failed messages in a failed list with support of saving
and restoring.
- Add testcases (requires ENV variables) to testbench
- Using application/octect-stream (binary) to send messages based on
Microsoft Code Sample:
https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-c-getstarted-send
* Note original Microsoft Samplecode is not working anymore, we are using
* QPID Proton Proactor based on
https://github.com/apache/qpid-proton/blob/main/c/examples/send.c
- requires QPID-PROTON Version 0.13 or higher because of the proactor API
- Add EventProperties configuration parameters
- using internal array instead of linkedlist for better performance
- using single byte helper to store message status (ubsubmit/submitted...)
- trigger wake up if messages need to be resubmitted (rejected)
- Slow down when sender credit reaches zero (10ns).
- Add enhanced performance stress test omazureeventhubs-stress.sh
- Add support for static library linking of qpid-proton
This is needed to build the module from source and remove
library package dependencies.
- adjusted valgrind suppressions
### ADDED
- [OMHIREDIS] module is now able to insert entries to a Redis Stream
- [OMHIREDIS] in 'stream' mode, module can insert the message to a custom field in the entry ('msg' by default)
- [OMHIREDIS] in 'stream' mode, module can acknowledge an entry coming from imhiredis (if entry was claimed but not ACK'ed)
- [OMHIREDIS] in 'stream' mode, acknowledgements can be made from dynamic templates or static values
- [OMHIREDIS] in 'stream' mode, module can approximately cap the size of the output stream
- [OMHIREDIS] in 'stream' mode, module can delete an entry while inserting its message (useful to remove entry coming from another stream with imhiredis)
- [OMHIREDIS] new tests for 'stream' mode
### FIXED
- [CONFIGURE.AC] Missing line to give omhiredis compilation status
- [IMHIREDIS] Add support for simple XREADs from Redis Streams (Redis >= 5.0 required)
- [IMHIREDIS] Add support for XREADGROUP from Redis Streams, allowing for user to define workers to dequeue logs in a stream
- [IMHIREDIS] stream mode can select fields to extract and insert in custom keys
- [IMHIREDIS] Add tests for the new 'stream' modes
- changed diag.sh to be able to start/stop/clean a redis server
- added helper functions in diag.sh to be able to query a redis server instance
- added new tests for imhiredis module to check
- that the queue mode works, with both lpop and rpop
- that the module is capable of handling a redis server going down
- that the module is capable of handling a redis server that appears afterwards
- that the subscribe mode works
This will easily permit to drop the last n characters from a property
without the need to know the exact length of the string. This is
especially useful as the exact length is most often not known
beforehand.
Now, length can have a negative value -n to denote that the
substring should be build between startpos and the character
-n chars from the end. This is a shortcut for stripping charactes
on "both ends" of the string.
Also, some hardening against invalid startpos and length has
been added.
Add new action parameters
- rotation.sizeLimit
- rotation.sizeLimitCommand
provide automatic output file rotation functionality feature-wise
equivalent to legacy $outchannel. This finally permits to use
this feature set in rscript.
rsyslog.conf may affect the host's local name. These changes were
so far only activated after the first HUP. This patch now ensures
that the configured local host name is applied correctly throughout
all processing, including early startup.
This patch causes a slight change of behaviour. However, the behaviour
was inconsitent before. Now it is consistent and according to the config.
Please note: this patch also exposes a global entry point via "regular"
dynamic loading as this makes things much easier to do. This is in-line
with ongoing simplification effort.
Finally, we also remove a CI test that we do no longer need because
the problem covered is now addressed differently and the original issue
can no longer occur.
closes https://github.com/rsyslog/rsyslog/issues/4975
- These tests will help find race conditions hopefully
- fix diag.sh issues running second instance in valgrind mode only
- Add check for minimum valgrind version for new relp tests
- Fixed an issue with numbers above int64 in syntax_ipv4.
Numbers that were up to 256 above the max of an int64
could incorrectly be detected as valid ipv4 digit.
- Simplified the IPv4 digit detection function and renamed
to isPosByte.
- added testcasse for malformed IPvc4 addresses
closes: https://github.com/rsyslog/rsyslog/issues/4940
The zstd library provides better and faster compression than zlib.
This patch integrates zstd as a dynamically-loadable functionality.
As such, no further dependencies need to be added to the rsyslog
base package.
Due to the increased performance, usage of zstd is highly recommended
for high-volume use cases.
This patch also refactor zlib compression in order to unify handling
in both compression cases.
This change allows to include extra CA files
so that no "unable to get issuer certificates" issue
is obtained when using chained cert files.
Proposed new parameter name is "NetstreamDriverCAExtraFiles"
fixes#4851
Signed-off-by: Sergio Arroutbi <sarroutb@redhat.com>
We disable ElasticSearch strict security in testbench, as
we do not need it for test runs, and it complicates thing.s
Note: this does NOT introduce a security weakness, because we use
only temporary testing ES instances which are always immediately
discarded after the test AND are run in our own test env with
user permissions.
When action.errorfile.maxsize configuration
option is enabled and error file already has a
certain size smaller than max size configured,
it is increasing higher than configured max
size as the error file is considered
to be zero in code.
This fix reads current error file size and
limits the size to the maximum size configured
fixes#4821
Signed-off-by: Sergio Arroutbi <sarroutb@redhat.com>
action.errorfile.maxsize has been added to enable
option to limit the amount of bytes dumped to
configured errorfile
fixes#4733
Signed-off-by: Sergio Arroutbi <sarroutb@redhat.com>
The order of execution was changed to a less optimal (more startups,
thus slower) order to work-around a testbench issue. This has been
fixed and so we can restore the original order.
