This commit performs a broad modernization of widely used rsyslog
macros to align with modern C practices and support automated
formatting tools like clang-format. The changes focus on improving
syntactic regularity, readability, and tooling compatibility — without
altering behavior.
Macros refactored in this commit now follow a consistent,
statement-like form with explicit trailing semicolons. Where
applicable, macro blocks that define module interfaces (`queryEtryPt`)
have been updated to use simple `if` statements instead of `else if`
chains. While this slightly increases evaluation time, the affected
functions are only called once per module during load time to register
supported interfaces — making the performance cost irrelevant in
practice.
These improvements serve multiple purposes:
- Enable reliable clang-format usage without mangling macro logic
- Simplify reasoning about macro-expanded code for human readers
- Reduce style drift and merge conflicts
- Facilitate development for contributors using assistive tools
- Support future formatting pipelines using:
1. `clang-format`
2. a post-fixup normalization script
Refactored macros:
- MODULE_TYPE_NOKEEP
- MODULE_TYPE_KEEP
- MODULE_TYPE_INPUT
- MODULE_TYPE_OUTPUT
- MODULE_TYPE_FUNCTION
- MODULE_TYPE_PARSER
- MODULE_TYPE_LIB
- DEF_IMOD_STATIC_DATA
- DEF_OMOD_STATIC_DATA
- DEF_PMOD_STATIC_DATA
- DEF_FMOD_STATIC_DATA
- DEFobjStaticHelpers
- SIMP_PROP(...)
And all `queryEtryPt()` dispatch macros:
- CODEqueryEtryPt_STD_MOD_QUERIES
- CODEqueryEtryPt_STD_OMOD_QUERIES
- CODEqueryEtryPt_STD_OMODTX_QUERIES
- CODEqueryEtryPt_STD_OMOD8_QUERIES
- CODEqueryEtryPt_TXIF_OMOD_QUERIES
- CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES
- CODEqueryEtryPt_STD_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_QUERIES
- CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES
- CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES
- CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES
- CODEqueryEtryPt_STD_PMOD_QUERIES
- CODEqueryEtryPt_STD_PMOD2_QUERIES
- CODEqueryEtryPt_STD_FMOD_QUERIES
- CODEqueryEtryPt_STD_SMOD_QUERIES
- CODEqueryEtryPt_doHUPWrkr
- CODEqueryEtryPt_doHUP
This general modernization reduces macro misuse, improves DX, and
lays the foundation for a robust, automated style normalization
system.
See also: https://github.com/rsyslog/rsyslog/issues/5747
This may be an interim solution, but definitely a useful one
to address style inconsistencies.
Style inconsistencies are problematic as they tend to pollute
git history by causing pseudo-changes.
This commit also contains a set of fixes to existing files, so
that we get to a clean state.
This Implements suffix comparison similar to startswith. Note that we
do intentionally not use libestr functions in order to speed up adaption.
It would otherwise probably take years for distros to upgrade libestr.
With some help of the Codex and Gemini AI Agents.
closes https://github.com/rsyslog/rsyslog/issues/1255
* Add note for indentation cleanup
* Converted tabs to spaces in various Python scripts to match PEP 8 indentation.
With the help of AI-Agent: Codex, Google Gemini Code review
* testbench: more robust method to build testing tools
The previous method failed depending on configure settings and
available support libraries. This is now handled by make check.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Testbench tools like tcpflood are now auto-generated at test startup
if missing, reducing errors and confusion during local test runs.
Updated AGENTS.md with precise instructions for running tests
in both human and AI agent contexts, improving onboarding
for new developers and AI-based workflows.
Network restrictions can prevent fetching Kafka or Zookeeper packages. When both download attempts fail, skip the test by exiting with code 77 instead of reporting an error.
AI-Agent: Codex 2025-06
Commit ffaf6dc4620da added proper variable sync, but dropped the check
if active count had actually changed. As such, the output was always
generated, which could pollute the log heavily.
also fixes some codestyle issues introduced by earlier commits.
With the help of AI-Agent: Codex 2025-06
The omsendertrack module is designed to track and report statistics for
message senders across all inputs in Rsyslog. It periodically outputs a
JSON file containing information about each sender.
Note: This commit provides minimalistic basic functionality as a PoC.
We will check it's usefulness in practice and expect follow-up PRs
to enhance functionality and include feedback from early testing.
However, this module is solid, just feature limited.
see also: https://github.com/rsyslog/rsyslog/issues/5599
As a result, it did not show up in pstats. Alos, we now use the input name
in worker thread to easily identify where they belong to. As thread names
have very limited length, the thread naming now is
"w<worker-number>/<input-name>".
This patch introduces a counter to track the number of discarded messages.
Additionally, it fixes a bug where the submitted message count was incremented
even when the message was discarded due to exceeding the allowed message rate
within a given interval.
when neither GTLS or OSSL TLS drivers are enabled, this test cannot pass.
We now check for this condition and execute it only it at least one of
the two is configured.
closes https://github.com/rsyslog/rsyslog/issues/5592
This commit significantly enhances imtcp by introducing a fully
functional worker thread pool, enabling true multi-threaded
processing for better scalability under high loads. This is
particularly beneficial when using TLS connections.
Notable changes:
- Implemented a complete worker pool for imtcp.
- Introduced the `workerthreads` config parameter for tuning
concurrency.
- Improved epoll efficiency by enabling edge-triggered mode.
- Added starvation handling via `starvationProtection.maxReads`.
- Refactored session accept logic and optimized network object
handling.
- Removed an obsolete network driver layer for event notification.
- Fixed multiple issues related to message timing, EPOLLERR
handling, and tests.
- Improved performance in poll() mode by reducing redundant
allocations.
- Introduced new CI tests for imtcp without epoll.
- Allowed disabling imtcp tests via a new configure switch.
- Added new impstats counters for worker thread pool statistics.
Details:
- The worker pool replaces an outdated experimental
implementation.
- If `workerthreads=1`, no worker pool is created to minimize
context switches.
- Moves worker pool variables inside `tcpsrv` instance to
prevent conflicts.
- Extracts session `accept()` logic into a dedicated function
for clarity.
- Fixes message ordering inconsistencies in multi-threaded
scenarios.
- Properly handles `EPOLLERR` notifications to improve error
resilience.
- Optimizes poll() mode by avoiding unnecessary reallocation
of file descriptors.
- Replaces the old network driver layer for event notification
with a streamlined solution.
- Now uses **conditional compilation** to select the best
method (epoll or poll) at build time.
- This significantly reduces code complexity, improves
maintainability, and boosts performance.
- The previous "thread pool" was a rough experiment that did
not perform significantly better than single-threaded mode.
- The **new implementation** allows multiple worker threads
on platforms with `epoll`.
- On non-epoll systems, an optimized **poll() based
single-threaded approach** is used, which is expected to
perform better than the old "thread pool."
- Adds `pthread_setname_np` only when available to improve
portability.
- Fixes test cases that assumed strict message timing, which
was unreliable.
- Reduces test parallelism for TSAN CI runs to prevent
resource exhaustion.
- Moves a test case to `imdiag` to ensure stable execution.
- Provides a new CI environment to verify `imtcp` behavior
without epoll.
- Introduces `--enable-imtcp-tests` configure switch for test
flexibility.
- Improves debug logging and adds better error handling for
worker pool startup.
New configuration parameters:
- `workerthreads`: Defines the number of worker threads for
imtcp. If set to 1, no worker pool is created.
- `starvationProtection.maxReads`: Defines the maximum number
of consecutive reads a worker can perform before being
interrupted to allow other sessions to be processed.
New impstats counters (emitted only when `workerthreads > 1`):
- `runs`: Number of times the worker thread has been invoked.
- `read`: Number of read calls performed by the worker.
For TLS, this includes read/write calls.
- `accept`: Number of `accept()` calls handled by the worker.
- `starvation_protect`: Number of times a socket was sent
back to the queue due to reaching the maximum number of
consecutive requests, ensuring fair scheduling of sessions.
These changes significantly enhance rsyslog’s TCP handling
performance and stability, particularly in high-volume
environments.
Closes#5529, #5532, #5578, #5580.
This changes two scripts supporting solaris CI builds to use two
instead of one threads when building needed dependecies. This
should result in a light speedup without overloading the
Solaris builders.
Note: OpenCSW provides the builders, and the rsyslog team
is thankful for that.
If openssl is compiled without support for openssl engines,
do not include engine selection. The global option
"defaultopensslengine" will not work in this case. It's support
is depreciated in newer OpenSSL versions
OpenSSL Engine Support feature relies on an outdated version of OpenSSL and is strictly experimental. No support or guarantees are provided. Use at your own risk.
closes: https://github.com/rsyslog/rsyslog/issues/5429
While this could lead to some inefficiency, it should not have caused
any real harm. But with data races it is never sure if more severe
issues occur. However, here only very strange use cases can be
envisioned where this might be the case.
In any case, the issue is now solved.
This also fixes some TSAN CI "flakes".
When the snmpreceiver is fully started and listening, the file is created.
diag.sh checks for the .started file before proceeding with the textbench
see also: https://github.com/rsyslog/rsyslog/pull/5558
Note: we temporarily disable omamqp1 testing while we search for
the missing dependency.
We also add a special setting that permits us to control if
known-(very)-flaky test shall be run.
closes https://github.com/rsyslog/rsyslog/issues/5546
The system packages on Ubuntu 24 appear to be broken for python3-pysnmp4.
And when we update the package using
pip install pyasn1 pysnmp --break-system-packages --upgrade
We need adapt the code, so a new version snmptrapreceiverv2.py
has been added which will automatically been chosen if Python is >3.10
closes: https://github.com/rsyslog/rsyslog/issues/5554
The test results were not deterministic. Sometimes it passed, sometimes
it failed. The omhttp-validate-response.py script tried to parse the
json input line by line, but sometimes the message and response fields
are on separate lines, thus it failed to parse. This is the case
for both files used to control the checking.
This patch updates the check program so that it reads in complete
lines when parsing fails, thus removing this condition for flakiness.
HOWEVER, after doing so the test still sometimes fails. This may
point into a problem with the contributed omhttp module. Unfortunately,
I do not have enough insight into that module (nor time) to try to
address the root cause.
closes https://github.com/rsyslog/rsyslog/issues/5439
Connections were originally opened sequentially. This commit adds the
capability to open them via multiple threads in parallel.
This is needed at least for cases where we need to test multiple
pending connection requests, which can be accepted with an
"accept() loop".
It can also be useful for speeding up parts of the testbench.
However, the additional concurrency makes it potentially harder
to debug if bugs are detected. So tests may use multiple threads
for connect during regular operations but should also offer the
option (by editing) to revert to a single thread for easier
debugging.
closes https://github.com/rsyslog/rsyslog/issues/5535
Updated the imtcp test suite to set the "SocketBacklog" parameter to a higher
value to accommodate tests with a large number of concurrent connections.
This change addresses potential CI flakes caused by SYN queue overflows during
test runs and significantly reduces test runtime by avoiding delays during
session setup.
The adjusted SocketBacklog parameter ensures smoother handling of high
connection counts, improving test reliability and performance.
Updated the imptcp test suite to set the "SocketBacklog" parameter to a higher
value to accommodate tests with a large number of concurrent connections.
This change addresses potential CI flakes caused by SYN queue overflows during
test runs and significantly reduces test runtime by avoiding delays during
session setup.
The adjusted SocketBacklog parameter ensures smoother handling of high
connection counts, improving test reliability and performance.
Note: The upcoming gnu23 C standard is overdoing it with type-safety. Inside
rsyslog, we historically have method tables for generic calls, which
keeps the code small and easy to understand. This would not decently be
possible with the new type-safety requirements.
So this commit works around these warning in a way that pretends to
provide more type safety. We have done this in the least intrusive
way to reduce the risk for regressions in code that works well in
since decades. Also note that the code already does parameter
validation.
There would have been more elaborate ways to make gnu23 compile happy,
e.g. by using a union of structs to provide the data element. Some folks
consider this type safe. In reality, it is not a bit better than
traditional C without types at all. Because the caller still needs to
ensure it picks the right struct from the union. As this approach
would also have larger regeression potential, we have not used it.
Right now, we have suppressed some of the new warnings, as working
around them would have required an even larger time budget and
potentially larger regression potential. In the long term we may
want to look into enabling them, as they would potentially be
beneficial for new code not involving method tables.
Some nits, however, were detected and have been fixed.
This patch also "fixes" some false positive test failures, mostly
be disabling some test functionality after confirmation these are
flakes.
see also https://github.com/rsyslog/rsyslog/issues/5507
also bump default development container to Ubuntu 24.04
This commit also contains some nitfixes that were detected
by the newer CI components. If it is just nitfixes, we do not
create separate commits for them.
... which went unnoticed during regular make check runs, but lead to
"left-over files after distclean" after make distcheck.
This could also potentially be responsible for some flakes that were
not fully explainable.
Solaris 10 does not support SO_REUSEPORT, which we need for some
tests. It is used in minitcpsrvr.c, and will now not be used if
not available.
Note that tests requiring that option will also need to be disabled
for that platform.
commit which introduced regression: 1c0f9bb
The option name in setsockopts is not a bitmask, so SO_REUSEADDR and
SO_REUSEPORT can't be ORed together. Instead apply the options via
separate calls.
Fixes: #5456
Thanks: Chris Hofstaedtler <zeha@debian.org>
This patch implements a simple round-robin load balancer
for omfwd. It provides equal distribution of load to a pool
of target servers.
The code currently has no different modes and no special tuning
for the load balancer. However, it works very well in the most
common use cases. Furthermore, it provides a solid base on which
more elaborate functionality can be build if there is need to.
The new functionality is fully backwards compatible with previous
configuration settings.
New action() config params:
* pool.resumeinterval
New/"changed" rstats counters
Each target receives its own set of pstats counters. Most
importantly this is the case for byte counts. That counter retains
the same naming, but there may now be multiple of these counters,
one for each target ip, port tuple.
New pstats message count to target
Among others, this can be used for checking that the load balancer
works as intended. The so-far byte count emitted does not provide
a clear indication of how many messages the targets had actually
processed.
For obvious reasons, this message count makes most sense in
advanced load balancing scenarios, but also provides additional
insight into round-robin. Non-matches indicate that targets
went offline, and we can now evaluate the impact this had
on processing.
- re-design rebind functionality
This now works at the transaction level. It causes a rebind of all
pool members. Previous code did not work 100% correct since for a
couple of years now (after output batching integration).
As cleanup, rebindInterval support has been removed from tcpClt,
because omfwd is the only user. This permits a cleaner code path.
We also noticed a bug with rebindInterval: it caused some mild
message duplication for quite some time. This went unnoticed.
To address that efficiently, rebindInterval in the future will
be considered once per batch. That means up to (maxBatchSize - 1)
messages may be transmitted more than the rebindinterval is.
That's the cleanest mode of operation and should not make any
difference for real deployments.
Some additional work done in this commit:
netstream: harden component against upper-layer logic errors
network subsystem: better handle API errors and provide more info
omfwd: add new parameter "iobuffer.maxsize"
add new global parameter debug.abortoninternalerror and use it
This parameter permits to make test runs fail when an internal error
is detected and gracefully handled by rsyslog. While it is great to
have it gracefully handled in practice, we should not accept this
during testing. The new parameter permits to abort in this case and
emits the related error message beforehand. It is turned on by
default in our regular tests.
add dedicated error code for "hard" program errors
omfwd: some cleanup + error message fix + new debug level messages
imptcp: improve error messages
add omfwd option to NOT do extended connection check
also output wrkr id in some omfwd messages (primarily debugging aid)
better debug info via LogMsg() interface
improve messages regarding imptcp and omfwd suspension / thread IDs
refactor and enchance minitcpsrvr for mimicing died servers
new global (debugging) option, correction of an informational msg
add global option allmessagestostderr
add new tests
- output all loaded ciphers and engines.
- Add new global option "defaultopensslengine" to customize the
default openssl engine. If not defined, openssl will handle the
default engine automatically.
- Add simple openssl performance test with defaultopensslengine
set to rdrand (Intel).
- removed unneeded testcase files in runtime folder.
- corrected whitelist settings for debug.files in TLS testcases
