Why:
TCP inputs can detect oversized frames before the core submit path sees a raw message larger than maxMessageSize. That left oversizemsg.errorfile unwritten for truncated TCP input.
Impact:
Configured oversize error logs now receive a JSON record for imtcp and imptcp frames that are truncated before core submit.
Before/After:
Before, pre-truncated TCP frames only emitted the normal warning; after, they also honor oversizemsg.errorfile.
Technical Overview:
Add a per-session flag that marks the current TCP frame as oversized when imtcp or imptcp detects the condition before submit. When the truncated message object is created, write the configured oversize JSON record before handing it to the ratelimit/submit path. Reset the flag when the message is submitted or a new frame starts. Add an imptcp regression test for oversizemsg.input.mode=truncate with oversizemsg.errorfile.
Closes: https://github.com/rsyslog/rsyslog/issues/5228
With the help of AI-Agents: Codex
Why
The previous change kept TCP batch processing alive on ratelimit helper
errors, but it left message ownership with the caller in those error
paths. Continuing without cleanup leaks the message object.
Impact
Unexpected ratelimit helper errors now drop and free the current message
before processing continues.
Before/After
Before, non-discard helper errors could leak `pMsg` in imptcp and the
generic TCP session path. After, both paths destroy the unqueued message
before returning success to the caller.
Technical Overview
`ratelimitAddMsg()` and `ratelimitAddMsgPerSource()` only take ownership
of a message when they enqueue or explicitly discard it internally.
When they return an unexpected error, ownership remains with the caller.
Add `msgDestruct(&pMsg)` in the non-discard error branch for `imptcp`
and `tcps_sess`, and update the debug text to reflect that the message is
being dropped before batch processing continues.
This preserves the non-fatal caller behavior while closing the memory
leak identified by gemini-code-assist and cubic.
With the help of AI-Agents: Codex
Why
TCP inputs treat rate limiting as a normal control path. Propagating
helper return codes into the submit path makes batch processing more
fragile than necessary and overstates submitted-message accounting.
Impact
TCP inputs now keep processing after ratelimit helper drops or internal
helper errors, while submit counters only advance for accepted messages.
Before/After
Before, callers ignored helper results for accounting and could be wired
to abort on helper returns. After, they treat drops and helper errors as
non-fatal and log them for diagnosis.
Technical Overview
Capture the return code from ratelimitAddMsg() and
ratelimitAddMsgPerSource() in the imptcp and generic TCP submit paths.
Only increment the submit counter when the helper accepts the message.
Treat RS_RET_DISCARDMSG as an expected outcome and continue processing.
For other helper errors, emit DBGPRINTF diagnostics and continue the
batch instead of returning failure to the caller.
This keeps ratelimit behavior aligned with normal message shedding while
preserving observability for unexpected helper problems.
With the help of AI-Agents: Codex
Why
Guard obvious zero-size allocation edges and remove a couple of
stale or misleading notes in the imtcp/tcpsrv path so future
reviews see the intended behavior directly in code.
Impact
Empty key and lookup-table files now fail before zero-size
allocations; OpenSSL accepted sessions use per-session callback
context.
Before/After
Before, empty files could flow into malloc(0)-style paths and the
OpenSSL accept path stored listener-owned callback context.
After, those runtime paths fail early and accepted OpenSSL sessions
carry their own callback context.
Technical Overview
Reject empty key files in the libgcry and OpenSSL runtime helpers
before allocating buffers from sb.st_size.
Reject empty lookup-table files before allocating and parsing the
JSON payload.
Update accepted OpenSSL sessions to store pNew-owned callback state
in SSL ex-data instead of the listener object's pointers.
Remove a stale TODO in the imtcp octet-count parser because the
zero-count case already falls into the framing-error path.
Add a short comment that maxReads==0 is intentional and documented
behavior that disables starvation protection.
With the help of AI-Agents: Codex
Why
Prevent malformed or oversized runtime inputs from reaching unsafe
allocation and framing paths in the main TCP runtime components.
Impact
Rejects overflow-sized framing configurations and invalid internal
severity values with focused regression coverage.
Before/After
Before, several runtime framing paths could overflow size arithmetic;
after, they fail closed before allocation or copy.
Technical Overview
The outbound TCP client framing path now validates octet-stuffing and
octet-counting size arithmetic before building framed buffers.
TCP session construction now uses size_t-based allocation for the base
receive buffer and rejects regex framing configurations whose
maxMessageSize would overflow the doubled session buffers.
The listener setup adds the same regex framing guard early so invalid
configurations fail during setup instead of later in session creation.
Facility and severity decoding now NUL-terminates the symbolic lookup
buffer and uses checked strtol parsing for numeric values.
Two shell tests cover the regex framing overflow case and invalid long
internalmsg severity values, and distcheck includes both successfully.
With the help of AI-Agents: Codex
This improves code readability and maintainability by correcting
typos in comments. While non-functional, clear comments reduce
confusion for contributors and support long-term project quality.
Before: comments contained minor typos and inconsistencies.
After: comments use corrected spelling and clearer wording.
Impact: none (no functional or behavioral changes).
The patch updates comment text only, without modifying logic,
interfaces, or runtime behavior. No changes to queue semantics,
transactions, or module interactions are introduced.
This aligns with ongoing maintenance efforts to keep the codebase
clean and easier to understand for contributors and reviewers.
Fixes: https://github.com/rsyslog/rsyslog/issues/6023
AI-Agent: Copilot 2026-03
Non-technical: centralize and reuse rate-limit definitions so admins
can apply consistent policies across listeners. This is part of an
ongoing series to improve rate limiting and its manageability.
Before: inputs set per-listener interval/burst ad hoc.
After: inputs can reference a named ratelimit() policy shared across
listeners; per-listener values remain as fallback.
Impact: New ratelimit() object and RateLimit.Name param for imtcp/imptcp.
If a policy file is configured but libyaml is unavailable, config fails.
Technical details:
- Add top-level ratelimit() Rainerscript object. Parsed in rsconf and
stored in a central registry (hashtable + rwlock) on rsconf.
- New runtime API: ratelimitAddConfig(), ratelimitNewFromConfig(), plus
cfgs init/destruct on rsconf lifecycle.
- imtcp/imptcp accept RateLimit.Name; when set, tcpsrv/imptcp build the
ratelimiter from the named policy; otherwise legacy interval/burst is
used. Thread-safety retained via ratelimitSetThreadSafe().
- tcpsrv gains ownership helpers for listener params and frees them on
errors; imtcp explicitly transfers ownership and nulls the pointer.
- Optional libyaml: detected at configure; runtime parser loads simple
key/value policy files (interval, burst, severity).
- Docs: new ratelimit object page; imtcp/imptcp parameter references and
module docs updated; design-decisions note added for libyaml.
- Tests: add ratelimit_name.sh (guarded for imtcp+imptcp) to validate
named policy application and observable throttling.
Refs: https://github.com/rsyslog/rsyslog/issues/6201
With the help of AI-Agents: Antigravity
imudp: add ratelimit.name support
This commit adds the `ratelimit.name` parameter to imudp, allowing listeners
to utilize the global rate limit registry (shared state).
Features:
- New `ratelimit.name` string parameter.
- Integration with `ratelimitNewFromConfig`.
- Strict mutual exclusivity: specifying `ratelimit.name` prohibits the use
of legacy per-listener parameters (`ratelimit.burst`, `ratelimit.interval`).
If a conflict occurs, an error is logged and the named rate limit takes precedence.
- Updated documentation.
- New regression test `tests/imudp_ratelimit_name.sh`.
With the help of AI Agent: Google Antigravity
Non-technical: improves operator ergonomics and closes a feature gap
with imptcp. Enables regex-based start-of-frame detection and optional
multi-line message handling on TCP inputs.
Impact: New config params; defaults keep existing behavior unchanged.
Before: imtcp framed messages via octet-counting or LF delimiter only.
After: imtcp can treat lines not starting a new frame as continuations
(MultiLine) and can split frames on a regex start pattern.
Technical:
- Adds imtcp params: MultiLine (bool) and framing.delimiter.regex (string).
Regex compilation happens in tcpsrv on listener creation; errors if
regex is set without FEATURE_REGEXP.
- tcps_sess adds a regex-aware path that tracks current-line offset,
runs the compiled regex on line starts, and uses a second buffer to
handle split packets cleanly. On >2x max-line without a match, we
submit and reset to avoid unbounded growth.
- Introduces input state eInMsgCheckMultiLine and LF lookahead to decide
continuation vs new frame; when at buffer end, defers the decision to
the next packet.
- Updates processDataRcvd signature to accept a movable cursor and
buffer length for lookahead; DataRcvd passes these and advances the
pointer accordingly.
- Wires regexp object usage in tcpsrv/tcps_sess init/exit; frees compiled
patterns on listener teardown and error paths. Tests cover both new
code paths (regex framing and multi-line).
Closes https://github.com/rsyslog/rsyslog/issues/5637
* imtcp: include peer ports in runtime log messages
Ensure tcpsrv and session diagnostics report the sender's source port alongside the existing host/IP fields so administrators can pinpoint connections precisely. This updates imtcp connection notices, close/error paths, and framing warnings to use a consistent "host (ip:port)" style already present in TLS mismatch logs.
With the help of AI-Agent: Codex
Admins often report "gibberish" when a TLS-enabled sender connects to a
plain imtcp port. Making the mismatch explicit reduces operator confusion
and support churn, and points directly to remediation.
Impact: logs one explicit error per mismatched connection; no change to
parsing or transport on plain listeners.
Before/After: before, ClientHello bytes were ingested as binary with no
hint; after, imtcp detects a TLS ClientHello on ptcp and logs a clear
message with a troubleshooting URL.
Technically, we add a small per-session probe in tcps_sess_t and sample
the first 5 bytes of new sessions. If the record header matches a TLS
handshake (type 0x16, version 0x03.00–0x04, length 40–16384) and the
listener is plain TCP (streamDriver.mode=0), we emit a single error and
disable further probing for that session. The probe is called from
DataRcvd() and returns RS_RET_SERVER_NO_TLS when triggered; the session
is otherwise left untouched.
Runtime: introduce RS_RET_SERVER_NO_TLS (-2465) to tag the condition.
Docs: add imtcp troubleshooting section and a dedicated FAQ page.
Tests/tools: add test imtcp-tls-gibberish.sh and extend tcpflood with
-H to send only a ClientHello (OpenSSL and GnuTLS paths tolerate early
termination and non-blocking I/O for this mode).
Some deployments need to disambiguate multiple senders sharing an IP,
for example autossh or similar tunnel setups. Exposing the source port
improves observability and lets pipelines key on a stable tuple.
Impact: new property/JSON field; tcps_sess IF v4; out-of-tree modules
must rebuild.
Before: messages exposed fromhost and fromhost-ip only.
After: messages also expose fromhost-port and jsonmesg includes it.
Introduce PROP_FROMHOST_PORT and wire it through msg.{h,c}. For TCP,
capture the remote port on accept, store it in tcps_sess, and attach it
to the msg on submit. For other inputs, resolveDNS derives the port from
the sockaddr when available; local inputs return an empty string. Add a
getter, duplication and destructor handling, and name<->ID mapping. Add
the field to jsonmesg output. Update docs, lexer keywords, and the
external plugin interface doc (property is modifiable). Bump
tcps_sessCURR_IF_VERSION to 4 and add SetHostPort() to the interface.
Include a focused test (fromhost-port.sh) that verifies the property.
Non-technical rationale: allow identification by (fromhost-ip,
fromhost-port) where IP alone is shared across systems (e.g., autossh).
With help from AI-Agents: ChatGPT
While at the time of merge I was confident it would fix a data race,
the root cause has now surfaced to be a simple state advancement
(single-threaded) bug. See commit c5fd73499 for details. As such
I revert this patch. it caused no harm, but complicates code, adds
a bit of computation and is no longer needed.
I cannot 100% outrule it might have addressed some edge cases.
I know an environment where we can verify this within the next
month or so. If, unexpectedly, this shows regressions, we can
re-enable the patch. But I am 99.99% sure it is not needed.
see also: https://github.com/rsyslog/rsyslog/pull/5806
see also: https://github.com/rsyslog/rsyslog/pull/5993
This commit fixes a race condition that could occur when two threads tried to close the same TCP session simultaneously. This could lead to an "epoll_ctl failed: Bad file descriptor" error message.
The fix introduces an atomic flag `being_closed` to the `tcps_sess_t` struct. This flag is used to ensure that the session-closing logic is executed only once per session.
This commit applies the new canonical formatting style using `clang-format` with custom settings (notably 4-space indentation), as part of our shift toward automated formatting normalization.
⚠️ No functional changes are included — only whitespace and layout modifications as produced by `clang-format`.
This change is part of the formatting modernization strategy discussed in:
https://github.com/rsyslog/rsyslog/issues/5747
Key context:
- Formatting is now treated as a disposable view, normalized via tooling.
- The `.clang-format` file defines the canonical style.
- A fixup script (`devtools/format-code.sh`) handles remaining edge cases.
- Formatting commits are added to `.git-blame-ignore-revs` to reduce noise.
- Developers remain free to format code however they prefer locally.
This commit performs a broad modernization of widely used rsyslog
macros to align with modern C practices and support automated
formatting tools like clang-format. The changes focus on improving
syntactic regularity, readability, and tooling compatibility — without
altering behavior.
Macros refactored in this commit now follow a consistent,
statement-like form with explicit trailing semicolons. Where
applicable, macro blocks that define module interfaces (`queryEtryPt`)
have been updated to use simple `if` statements instead of `else if`
chains. While this slightly increases evaluation time, the affected
functions are only called once per module during load time to register
supported interfaces — making the performance cost irrelevant in
practice.
These improvements serve multiple purposes:
- Enable reliable clang-format usage without mangling macro logic
- Simplify reasoning about macro-expanded code for human readers
- Reduce style drift and merge conflicts
- Facilitate development for contributors using assistive tools
- Support future formatting pipelines using:
1. `clang-format`
2. a post-fixup normalization script
Refactored macros:
- MODULE_TYPE_NOKEEP
- MODULE_TYPE_KEEP
- MODULE_TYPE_INPUT
- MODULE_TYPE_OUTPUT
- MODULE_TYPE_FUNCTION
- MODULE_TYPE_PARSER
- MODULE_TYPE_LIB
- DEF_IMOD_STATIC_DATA
- DEF_OMOD_STATIC_DATA
- DEF_PMOD_STATIC_DATA
- DEF_FMOD_STATIC_DATA
- DEFobjStaticHelpers
- SIMP_PROP(...)
And all `queryEtryPt()` dispatch macros:
- CODEqueryEtryPt_STD_MOD_QUERIES
- CODEqueryEtryPt_STD_OMOD_QUERIES
- CODEqueryEtryPt_STD_OMODTX_QUERIES
- CODEqueryEtryPt_STD_OMOD8_QUERIES
- CODEqueryEtryPt_TXIF_OMOD_QUERIES
- CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES
- CODEqueryEtryPt_STD_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_QUERIES
- CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES
- CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES
- CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES
- CODEqueryEtryPt_STD_PMOD_QUERIES
- CODEqueryEtryPt_STD_PMOD2_QUERIES
- CODEqueryEtryPt_STD_FMOD_QUERIES
- CODEqueryEtryPt_STD_SMOD_QUERIES
- CODEqueryEtryPt_doHUPWrkr
- CODEqueryEtryPt_doHUP
This general modernization reduces macro misuse, improves DX, and
lays the foundation for a robust, automated style normalization
system.
See also: https://github.com/rsyslog/rsyslog/issues/5747
This commit significantly enhances imtcp by introducing a fully
functional worker thread pool, enabling true multi-threaded
processing for better scalability under high loads. This is
particularly beneficial when using TLS connections.
Notable changes:
- Implemented a complete worker pool for imtcp.
- Introduced the `workerthreads` config parameter for tuning
concurrency.
- Improved epoll efficiency by enabling edge-triggered mode.
- Added starvation handling via `starvationProtection.maxReads`.
- Refactored session accept logic and optimized network object
handling.
- Removed an obsolete network driver layer for event notification.
- Fixed multiple issues related to message timing, EPOLLERR
handling, and tests.
- Improved performance in poll() mode by reducing redundant
allocations.
- Introduced new CI tests for imtcp without epoll.
- Allowed disabling imtcp tests via a new configure switch.
- Added new impstats counters for worker thread pool statistics.
Details:
- The worker pool replaces an outdated experimental
implementation.
- If `workerthreads=1`, no worker pool is created to minimize
context switches.
- Moves worker pool variables inside `tcpsrv` instance to
prevent conflicts.
- Extracts session `accept()` logic into a dedicated function
for clarity.
- Fixes message ordering inconsistencies in multi-threaded
scenarios.
- Properly handles `EPOLLERR` notifications to improve error
resilience.
- Optimizes poll() mode by avoiding unnecessary reallocation
of file descriptors.
- Replaces the old network driver layer for event notification
with a streamlined solution.
- Now uses **conditional compilation** to select the best
method (epoll or poll) at build time.
- This significantly reduces code complexity, improves
maintainability, and boosts performance.
- The previous "thread pool" was a rough experiment that did
not perform significantly better than single-threaded mode.
- The **new implementation** allows multiple worker threads
on platforms with `epoll`.
- On non-epoll systems, an optimized **poll() based
single-threaded approach** is used, which is expected to
perform better than the old "thread pool."
- Adds `pthread_setname_np` only when available to improve
portability.
- Fixes test cases that assumed strict message timing, which
was unreliable.
- Reduces test parallelism for TSAN CI runs to prevent
resource exhaustion.
- Moves a test case to `imdiag` to ensure stable execution.
- Provides a new CI environment to verify `imtcp` behavior
without epoll.
- Introduces `--enable-imtcp-tests` configure switch for test
flexibility.
- Improves debug logging and adds better error handling for
worker pool startup.
New configuration parameters:
- `workerthreads`: Defines the number of worker threads for
imtcp. If set to 1, no worker pool is created.
- `starvationProtection.maxReads`: Defines the maximum number
of consecutive reads a worker can perform before being
interrupted to allow other sessions to be processed.
New impstats counters (emitted only when `workerthreads > 1`):
- `runs`: Number of times the worker thread has been invoked.
- `read`: Number of read calls performed by the worker.
For TLS, this includes read/write calls.
- `accept`: Number of `accept()` calls handled by the worker.
- `starvation_protect`: Number of times a socket was sent
back to the queue due to reaching the maximum number of
consecutive requests, ensuring fair scheduling of sessions.
These changes significantly enhance rsyslog’s TCP handling
performance and stability, particularly in high-volume
environments.
Closes#5529, #5532, #5578, #5580.
This patch slightly improves performance for tcpsrv-based servers.
This affects imtcp and imgssapi as well as some helpers.
No other functional change is included in this patch.
The permittedPeers settig was actually forgotten during the refactoring
of TLS input() level settings. This functionality is now added.
closes: https://github.com/rsyslog/rsyslog/issues/4706
When "discardTruncatedMsg" is enabled in imtcp, messages were incorrectly
skipped if the last character before the truncation was the LFdelimiter.
Add two tests for octet stuffing and framing.
closes: https://github.com/rsyslog/rsyslog/issues/4281
The actual problem is in the tcpserver component. However, the prime user
is imtcp and so users will likely experience this as imtcp problem.
When a too-long message is truncated, the byte after the truncation
position becomes the first byte of the next message. This will garble
the next messages and in almost all cases render it is syslog-incompliant.
The same problem does NOT occur when the message is split.
This commit fixes the issue. It also includes a testbench fix.
Unfortunately the test for exactly this feature was not properly
crafted and so could not detect the problem.
closes https://github.com/rsyslog/rsyslog/issues/3580
Commit b0e8ce6c added a new variable to track processing of a single
message split into multiple packets. Several bits of code weren't
ported to the new variable and could cause mishandling.
Messages not terminated by a NL were discarded upon connection
termination.
Remove the use sizeof(char) or sizeof(uchar) in calculations for memory
allocation or string length. There are no known platforms for which
sizeof(char) or sizeof(uchar) is not 1, and c99 has defined sizeof(char)
to be 1 (section 6.5.3.4 of c99).
