Why:
Older platforms need consistent formatted string allocation, and the
remaining copy helpers kept triggering review noise around classic C
string APIs.
A major motivation is to avoid very common AI review false positives:
those tools often do not understand the actual scope and safety checks,
and then mechanically flag strcpy-style APIs despite the surrounding
bounds and initialization logic being correct.
Impact: string allocation and bounded copy paths are now explicit and
portable across the tree.
Before/After: ad hoc unsafe string helpers remained; now allocation and
bounded copies follow one portable pattern.
Technical Overview:
Add a complete asprintf and vasprintf compatibility layer with shared
prototypes so older libc variants build without local wrappers.
Replace repo-wide strcpy, strcat, strncat, sprintf, and direct strncpy
uses with explicit memcpy-based bounded copies or exact-width byte
copies as appropriate for each destination.
Add rsCStrAppendParts() for incremental string assembly so callers can
build pre-sized buffers without repeated snprintf return handling.
Update the unicode helper copy routine so existing ustrncpy() call sites
no longer route to libc strncpy semantics.
This also removes a broad class of review distractions from automated AI
reviewers that key off banned function names without understanding the
actual copy contract at the call site.
Extend the stringbuf unit coverage for the new append helper and the
formatted-allocation compatibility path.
With the help of AI-Agents: Codex
Why
Native post-quantum TLS support should be usable and testable on newer
distro baselines without adding provider-mode compatibility work for
older platforms.
Impact
Rsyslog now has native-PQ smoke tests, clearer TLS diagnostics, updated
CI baselines and helper images, and a new post-quantum tutorial for
supported distros.
Before/After
Before: Fedora CI still targeted Fedora 41, PQ-capable TLS settings had
no dedicated rsyslog tests or user-facing tutorial, and stricter clang
builds could fail on warning-group handling.
After: CI targets Fedora 43, native PQ usage is documented and smoke-
tested, helper images include the required tools, and the branch builds
and tests cleanly with the newer compiler/container combinations.
Technical Overview
The CI matrix now replaces the Fedora 41 lane with Fedora 43 and adds a
matching Fedora 43 development image.
The Debian 13 and Fedora 43 development containers now install the
GnuTLS CLI utilities needed for native PQ capability checks.
The OpenSSL TLS config path logs clearer messages when a command or
value is unavailable on the native OpenSSL build.
The GnuTLS TLS config path reports unsupported priority-string options
more explicitly.
Two new shell tests add native PQ smoke coverage for OpenSSL and GnuTLS
using the existing gnutlsPriorityString control surface.
Those tests self-skip unless the local native TLS libraries expose the
required hybrid group support.
The imtcp parameter docs and omfwd docs now explain the native-only PQ
support policy and include example configurations.
A new tutorial documents native PQ usage for OpenSSL and GnuTLS on
supported newer distro versions.
The shared runtime warning policy in rsyslog.h now tolerates clang
handling of unknown warning groups so older and newer clang lanes remain
warning-free under the existing finalize_it error-handling pattern.
Testbench follow-ups harden omfwd-lb-susp with isolated retry attempts,
skip rcvr_fail_restore on ARM where it is timing-flaky, and keep local
SC2181 suppressions where if-exec rewrites would reduce shell-script
usability.
The Fedora 43 Dockerfile now cleans the dnf cache after install and
locally suppresses the non-useful DL3041 package-version pinning warning.
Older distro versions remain intentionally unsupported for PQ in this
phase because we expect users to move to newer baselines first.
If there is demand later, older-version support can be considered in a
separate effort.
With the help of AI-Agents: Codex
Why:
Literal string copies into fixed-size local buffers can appear on hot paths
and are not always good candidates for formatting APIs. Some modern AI
assistants replace strcpy with snprintf by default; that can be safer in
many cases, but it may also add overhead when the source is a constant
literal and the destination is a known-sufficient fixed buffer.
Impact: No behavior change; compile-time size checks added for literals.
Before/After:
Before, constant fallback strings used strcpy directly at call sites.
After, constant fallback strings use a shared checked helper macro.
Technical Overview:
Add RS_STATIC_ASSERT in runtime/rsyslog.h with a C11 _Static_assert path
and a pre-C11 typedef fallback for older platforms.
Add RS_COPY_LITERAL(dst, lit), implemented as size-checked memcpy of the
literal including its trailing NUL byte, with Doxygen comments.
Use RS_COPY_LITERAL in runtime/nsd_ptcp.c and extend the same pattern to
other fixed-buffer literal strcpy sites in runtime/, tools/, and plugins/.
Keep non-matching strcpy sites unchanged where source or destination is not
a compile-time literal-to-fixed-buffer copy.
This keeps constant-copy operations lean on hot code paths while making
the fixed-buffer precondition explicit in shared code.
The helper centralizes intent so human and AI contributors are less likely
to apply high-overhead snprintf substitutions where formatting is not
required.
Even snprintf is not "set and forget": callers must still handle return
edge cases (ret < 0, truncation via ret >= size, and related pitfalls).
With the help of AI-Agents: Codex
Why:
Keep strict warning builds useful without relying on broad
diagnostic suppressions.
Impact: removes redundant-decls noise while keeping
missing-prototypes at zero in full check builds.
Before/After:
Before mixed macro/local/header declarations; after a
layered prototype model with no warning class regressions.
Technical Overview:
Introduce PROTOTYPEobjConstruct() and standardize new
declarations on PROTOTYPE* macros in obj-types.h.
Split object prototype groups into PROTOTYPEObjBasic() and
PROTOTYPEObjFull(), keeping PROTOTYPEObj() as a compatibility
alias to Basic and using Full where typed ctor/dtor visibility
is valid.
Remove redundant local macro predeclarations for
QueryInterface/Destruct/Construct/DebugPrint/Set methods and
add header-owned prototypes where required.
Add BEGINmodInitNoPredecl() and apply it to built-in tool
modules that already publish modInit prototypes in headers.
Drop the obsolete -Wredundant-decls suppression pragma.
Closes https://github.com/rsyslog/rsyslog/issues/5700
With the help of AI-Agents: Codex
Admins often report "gibberish" when a TLS-enabled sender connects to a
plain imtcp port. Making the mismatch explicit reduces operator confusion
and support churn, and points directly to remediation.
Impact: logs one explicit error per mismatched connection; no change to
parsing or transport on plain listeners.
Before/After: before, ClientHello bytes were ingested as binary with no
hint; after, imtcp detects a TLS ClientHello on ptcp and logs a clear
message with a troubleshooting URL.
Technically, we add a small per-session probe in tcps_sess_t and sample
the first 5 bytes of new sessions. If the record header matches a TLS
handshake (type 0x16, version 0x03.00–0x04, length 40–16384) and the
listener is plain TCP (streamDriver.mode=0), we emit a single error and
disable further probing for that session. The probe is called from
DataRcvd() and returns RS_RET_SERVER_NO_TLS when triggered; the session
is otherwise left untouched.
Runtime: introduce RS_RET_SERVER_NO_TLS (-2465) to tag the condition.
Docs: add imtcp troubleshooting section and a dedicated FAQ page.
Tests/tools: add test imtcp-tls-gibberish.sh and extend tcpflood with
-H to send only a ClientHello (OpenSSL and GnuTLS paths tolerate early
termination and non-blocking I/O for this mode).
Harden default endpoints for cloud-native use: make health/metrics
scrapes proxy-friendly and allow locking them down with Basic Auth.
This aligns imhttp with common Kubernetes/Prometheus patterns and
supports metrics-only deployments.
Impact: /metrics now exports full rsyslog stats with Content-Length;
health and metrics can be gated via htpasswd; unified 500 on failures.
Technical details:
- Add module params: healthCheckBasicAuthFile and metricsBasicAuthFile.
When set, attach a Basic Auth handler that reads an htpasswd file;
reuse the same handler for per-input endpoints by passing the file via
cbdata.
- Rework Prometheus handler to collect data through statsobj in
Prometheus format. Accumulate lines into a growable buffer with
overflow checks, append an imhttp_up gauge, then reply with an
explicit Content-Length and close the connection.
- Fix metrics buffer termination to use a single NUL byte; prevent a
leak when buffer growth fails; consolidate error paths so the buffer is
freed and a single HTTP 500 is emitted.
- Docs: describe new auth options, clarify default paths, document that
metrics responses carry Content-Length, and add examples (including
metrics-only setups).
Before/After: metrics previously exposed a minimal body without auth;
now they export full rsyslog stats with optional Basic Auth and a
Content-Length header.
Improve maintainability and robustness of the TCP server by clarifying
locking/ownership, tightening invariants, and simplifying queueing.
Also fix a long-standing pragma macro typo across the tree.
Impact: Internal behavior only. EPOLL re-arm now occurs while holding
pSess->mut; starvation cap counts only successful reads.
Before/After:
Before: EPOLL re-arm happened after leaving the critical section; read
starvation cap counted loop iterations; closeSess() sometimes unlocked;
select_* helpers used on non-epoll path; enqueueWork() returned status.
After: EPOLLONESHOT is re-armed before unlocking; starvation cap counts
only RS_RET_OK reads; closeSess() never unlocks; poll_* helpers replace
select_*; enqueueWork() is void (best-effort).
Technical details:
- Replace notifyReArm() with rearmIoEvent() (EPOLL_CTL_MOD with
EPOLLONESHOT|EPOLLET; asserts efd/sock; logs on failure).
- doReceive(): explicit state machine; would-block path re-arms before
unlock; close path unlocks then calls closeSess(); starvation handoff
enqueues without re-arming.
- Initialize ioDirection for listener and session descriptors; add
assert(sock >= 0) and widespread ATTR_NONNULL annotations.
- startWrkrPool(): single finalize rollback (cancel/join partial
threads; destroy cond/mutex); stopWrkrPool(): destroy cond/mutex.
- enqueueWork(): FIFO append under lock and cond signal; returns void.
- Cleanup hardening on construct failure: free ppLstn, ppLstnPort,
ppioDescrPtr; free fromHostIP on SessAccept() error.
- Non-epoll: rename select_Add/Poll/IsReady -> poll_*; RunPoll() uses
poll_* and sets sane ioDirection defaults.
- Typo fix: standardize PRAGMA_IGNORE_Wswitch_enum in header and all
users (action.c, rainerscript.c, template.c, tcpsrv.c).
This commit applies the new canonical formatting style using `clang-format` with custom settings (notably 4-space indentation), as part of our shift toward automated formatting normalization.
⚠️ No functional changes are included — only whitespace and layout modifications as produced by `clang-format`.
This change is part of the formatting modernization strategy discussed in:
https://github.com/rsyslog/rsyslog/issues/5747
Key context:
- Formatting is now treated as a disposable view, normalized via tooling.
- The `.clang-format` file defines the canonical style.
- A fixup script (`devtools/format-code.sh`) handles remaining edge cases.
- Formatting commits are added to `.git-blame-ignore-revs` to reduce noise.
- Developers remain free to format code however they prefer locally.
Reorganized the “Config Settings” section in rsyslog.h to improve
readability and documentation:
- Added a banner header for the Config Settings block
- Consolidated general MAXSIZE/BUFSIZE guidelines into a single Doxygen
comment
- Provided individual `@brief` comments for each `#define`
- Aligned macro definitions for consistent formatting
No functional changes were made; this is purely a documentation and
style cleanup.
Undefine common syslog severity and facility macros before
redefining them in rsyslog.h. As rsyslog, we need consistent
definitions. This resolves "redefined" warnings when other
system headers are included. Improves build cleanliness.
among others, remove some warning suppressions by "fixing" the
respective constructs with work-arounds (root cause is compilers
do not handle enums in switch well).
The omsendertrack module is designed to track and report statistics for
message senders across all inputs in Rsyslog. It periodically outputs a
JSON file containing information about each sender.
Note: This commit provides minimalistic basic functionality as a PoC.
We will check it's usefulness in practice and expect follow-up PRs
to enhance functionality and include feedback from early testing.
However, this module is solid, just feature limited.
see also: https://github.com/rsyslog/rsyslog/issues/5599
This commit significantly enhances imtcp by introducing a fully
functional worker thread pool, enabling true multi-threaded
processing for better scalability under high loads. This is
particularly beneficial when using TLS connections.
Notable changes:
- Implemented a complete worker pool for imtcp.
- Introduced the `workerthreads` config parameter for tuning
concurrency.
- Improved epoll efficiency by enabling edge-triggered mode.
- Added starvation handling via `starvationProtection.maxReads`.
- Refactored session accept logic and optimized network object
handling.
- Removed an obsolete network driver layer for event notification.
- Fixed multiple issues related to message timing, EPOLLERR
handling, and tests.
- Improved performance in poll() mode by reducing redundant
allocations.
- Introduced new CI tests for imtcp without epoll.
- Allowed disabling imtcp tests via a new configure switch.
- Added new impstats counters for worker thread pool statistics.
Details:
- The worker pool replaces an outdated experimental
implementation.
- If `workerthreads=1`, no worker pool is created to minimize
context switches.
- Moves worker pool variables inside `tcpsrv` instance to
prevent conflicts.
- Extracts session `accept()` logic into a dedicated function
for clarity.
- Fixes message ordering inconsistencies in multi-threaded
scenarios.
- Properly handles `EPOLLERR` notifications to improve error
resilience.
- Optimizes poll() mode by avoiding unnecessary reallocation
of file descriptors.
- Replaces the old network driver layer for event notification
with a streamlined solution.
- Now uses **conditional compilation** to select the best
method (epoll or poll) at build time.
- This significantly reduces code complexity, improves
maintainability, and boosts performance.
- The previous "thread pool" was a rough experiment that did
not perform significantly better than single-threaded mode.
- The **new implementation** allows multiple worker threads
on platforms with `epoll`.
- On non-epoll systems, an optimized **poll() based
single-threaded approach** is used, which is expected to
perform better than the old "thread pool."
- Adds `pthread_setname_np` only when available to improve
portability.
- Fixes test cases that assumed strict message timing, which
was unreliable.
- Reduces test parallelism for TSAN CI runs to prevent
resource exhaustion.
- Moves a test case to `imdiag` to ensure stable execution.
- Provides a new CI environment to verify `imtcp` behavior
without epoll.
- Introduces `--enable-imtcp-tests` configure switch for test
flexibility.
- Improves debug logging and adds better error handling for
worker pool startup.
New configuration parameters:
- `workerthreads`: Defines the number of worker threads for
imtcp. If set to 1, no worker pool is created.
- `starvationProtection.maxReads`: Defines the maximum number
of consecutive reads a worker can perform before being
interrupted to allow other sessions to be processed.
New impstats counters (emitted only when `workerthreads > 1`):
- `runs`: Number of times the worker thread has been invoked.
- `read`: Number of read calls performed by the worker.
For TLS, this includes read/write calls.
- `accept`: Number of `accept()` calls handled by the worker.
- `starvation_protect`: Number of times a socket was sent
back to the queue due to reaching the maximum number of
consecutive requests, ensuring fair scheduling of sessions.
These changes significantly enhance rsyslog’s TCP handling
performance and stability, particularly in high-volume
environments.
Closes#5529, #5532, #5578, #5580.
Note: The upcoming gnu23 C standard is overdoing it with type-safety. Inside
rsyslog, we historically have method tables for generic calls, which
keeps the code small and easy to understand. This would not decently be
possible with the new type-safety requirements.
So this commit works around these warning in a way that pretends to
provide more type safety. We have done this in the least intrusive
way to reduce the risk for regressions in code that works well in
since decades. Also note that the code already does parameter
validation.
There would have been more elaborate ways to make gnu23 compile happy,
e.g. by using a union of structs to provide the data element. Some folks
consider this type safe. In reality, it is not a bit better than
traditional C without types at all. Because the caller still needs to
ensure it picks the right struct from the union. As this approach
would also have larger regeression potential, we have not used it.
Right now, we have suppressed some of the new warnings, as working
around them would have required an even larger time budget and
potentially larger regression potential. In the long term we may
want to look into enabling them, as they would potentially be
beneficial for new code not involving method tables.
Some nits, however, were detected and have been fixed.
This patch also "fixes" some false positive test failures, mostly
be disabling some test functionality after confirmation these are
flakes.
see also https://github.com/rsyslog/rsyslog/issues/5507
This patch implements a simple round-robin load balancer
for omfwd. It provides equal distribution of load to a pool
of target servers.
The code currently has no different modes and no special tuning
for the load balancer. However, it works very well in the most
common use cases. Furthermore, it provides a solid base on which
more elaborate functionality can be build if there is need to.
The new functionality is fully backwards compatible with previous
configuration settings.
New action() config params:
* pool.resumeinterval
New/"changed" rstats counters
Each target receives its own set of pstats counters. Most
importantly this is the case for byte counts. That counter retains
the same naming, but there may now be multiple of these counters,
one for each target ip, port tuple.
New pstats message count to target
Among others, this can be used for checking that the load balancer
works as intended. The so-far byte count emitted does not provide
a clear indication of how many messages the targets had actually
processed.
For obvious reasons, this message count makes most sense in
advanced load balancing scenarios, but also provides additional
insight into round-robin. Non-matches indicate that targets
went offline, and we can now evaluate the impact this had
on processing.
- re-design rebind functionality
This now works at the transaction level. It causes a rebind of all
pool members. Previous code did not work 100% correct since for a
couple of years now (after output batching integration).
As cleanup, rebindInterval support has been removed from tcpClt,
because omfwd is the only user. This permits a cleaner code path.
We also noticed a bug with rebindInterval: it caused some mild
message duplication for quite some time. This went unnoticed.
To address that efficiently, rebindInterval in the future will
be considered once per batch. That means up to (maxBatchSize - 1)
messages may be transmitted more than the rebindinterval is.
That's the cleanest mode of operation and should not make any
difference for real deployments.
Some additional work done in this commit:
netstream: harden component against upper-layer logic errors
network subsystem: better handle API errors and provide more info
omfwd: add new parameter "iobuffer.maxsize"
add new global parameter debug.abortoninternalerror and use it
This parameter permits to make test runs fail when an internal error
is detected and gracefully handled by rsyslog. While it is great to
have it gracefully handled in practice, we should not accept this
during testing. The new parameter permits to abort in this case and
emits the related error message beforehand. It is turned on by
default in our regular tests.
add dedicated error code for "hard" program errors
omfwd: some cleanup + error message fix + new debug level messages
imptcp: improve error messages
add omfwd option to NOT do extended connection check
also output wrkr id in some omfwd messages (primarily debugging aid)
better debug info via LogMsg() interface
improve messages regarding imptcp and omfwd suspension / thread IDs
refactor and enchance minitcpsrvr for mimicing died servers
new global (debugging) option, correction of an informational msg
add global option allmessagestostderr
add new tests
The clang Pragma()s were not correctly enabled due to the new
gcc version check. Also, gcc 4.2 upwards was not processed
correctly. This broke rsyslog build on the affected platforms.
Note: checking the exact version of clang is more complex than
worth the effort. Even very old versions (down to v 3) support
the macros. So we just check if we compile via clang.
Some old compilers, eg. GCC 4.4.7 on el6 do not support #pragma GCC
diagnostic in functions. Rsyslog has PRAGMA_* macros to handle those
cases, so let's use them.
Even if hostname FQDN is configured, rsyslog internal messages generated
after rsyslog startup and before the first HUP will not necessarily have
FQDN but instead only the shortname of the local host. This commit
fixes the situation.
Special thanks to github user eciii for doing a great bug analysis
and helping us considerably to fix the issue.
closes https://github.com/rsyslog/rsyslog/issues/5218
- Add TLS CRL support tp GnuTLS driver using gnutls_certificate_set_x509_crl_file.
- Add code in OpenSSL driver that works with OpenSSL 1.0.2 and higher.
Disable feature on older features with error message.
- Some cosmetic changes
- testbench: Add revoked certificate for testing (Including CRL PEM and other files)
- testbench: Add testcase for gtls and ossl testing revoked certificates
The option allows to drop the capabilities to only
the necessary set, to minimize security exposure in
case there was ever a mistake in a networking
plugin or some other input resource. Moreover, it adds
ability to change uid and gid while retaining the
previously specified capabilities. Resolves#4986
Add ability to change uid and gid while retaining the capabilities previously specified
rsyslog.conf may affect the host's local name. These changes were
so far only activated after the first HUP. This patch now ensures
that the configured local host name is applied correctly throughout
all processing, including early startup.
This patch causes a slight change of behaviour. However, the behaviour
was inconsitent before. Now it is consistent and according to the config.
Please note: this patch also exposes a global entry point via "regular"
dynamic loading as this makes things much easier to do. This is in-line
with ongoing simplification effort.
Finally, we also remove a CI test that we do no longer need because
the problem covered is now addressed differently and the original issue
can no longer occur.
closes https://github.com/rsyslog/rsyslog/issues/4975
Commit 829a6f6 "CI: improve use of github action matrix feature;
optimization" has introduced use of #pragma GCC diagnostic in function
doNameLine() of runtime/conf.c. This is not supported by old compilers
like GCC 4.4.7 on el6.
Matrix feature was initially barely usable because github did not permit
to restart just failed jobs. This has changed. So we now migrate back to
the use of matrix feature where it is useful.
We also reduce the number of plain compile tests to newest and oldest
compiler version only. This saves CI ressources. The important other
ones (distro default!) are used during other CI tests anyways.
We also fix some minor coding errors detected by new compiler builds.
None of them seems to be related to any real issues.
This is a fine-tuning option which permits to control whether or not
rsyslog shall alays take the flow control setting from the message. If
so, non-primary queues may also block when reaching high water mark.
This permits to add some synchronous processing to rsyslog core engine.
However, it is dangerous, as improper use may make the core engine
stall. As such, enabling this option requires very careful planning
of the rsyslog configuration and deep understanding of the consequences.
Note that the option is applied to individual queues, so a configuration
with a large number of queues can (and must if use) be fine-tuned to
the exact use case.
The rsyslog team strongly recommends to let the option turned off,
which is the default setting.
see also https://github.com/rsyslog/rsyslog/issues/3941
If the same name is specified for multiple queues, the queue files
will become corrupted. This commit adds a check during config parsing.
If duplicate names are detected the config parser errors out and the
related object is not created.
Note: this may look to a change-of-behaviour to some users. However,
this never worked and it was pure luck that these users did not run
into big problems (e.g. DA queues were never going to disk at the
same time). So it is acceptable to error out in this hard error case.
closes https://github.com/rsyslog/rsyslog/issues/1385
When the work directory has not been set or is invalid, state files
are created in the root of the file system. This is neither expected
nor desirable. We now complain loudly about this fact. For backwards
compatibility reasons, we still need to support running imfile in
this case.
closes https://github.com/rsyslog/rsyslog/issues/1296
When loading CA, Cert or Keyfile, the OpenSSL error stack will be output into LogError.
Also added 3 tests with invalid certificates/keyfiles for testbench.
