Why
YAML-backed ratelimit policies were only reloaded on HUP, which made
policy tuning slower and harder to automate.
Impact
policyWatch and policyWatchDebounce now reload watched ratelimit
policies automatically when inotify is available, and otherwise fall
back cleanly to HUP-only behavior.
Before/After
Before: external ratelimit policy changes required HUP to take effect.
After: watched policy files reload automatically after a debounce
interval, while unsupported builds keep the previous behavior.
Technical Overview
Add policyWatch and policyWatchDebounce to ratelimit() config parsing
and store the settings in shared ratelimit state.
Introduce an inotify-backed watcher path in runtime/ratelimit.c that
watches parent directories, coalesces rapid file events with a debounce
window, and reloads both policy and per-source policy files by reusing
existing HUP-era parse-and-swap logic.
Keep unsupported builds working by compiling the watcher code behind
inotify feature checks and downgrading watch requests to warning-only
HUP semantics.
Document the new options and add integration coverage for direct file
updates, debounce behavior, rename-based replacement, and YAML frontend
parity via tests guarded by inotify availability.
Closes https://github.com/rsyslog/rsyslog/issues/6599
With the help of AI-Agents: Codex
Why:
Enable CI validation on ARM architectures so platform-specific regressions
are caught before merge. arm64 uses native GitHub runners; armhf uses QEMU
because GitHub does not offer 32-bit ARM runners. Network namespace tests may
fail under QEMU where the mount syscall is not properly emulated.
Impact:
- New arm_CI job runs on armhf and arm64 when relevant files change.
- arm64 uses native ubuntu-24.04-arm runner; armhf uses QEMU on x64.
- Netns tests skip gracefully (exit 77) if ip netns add fails.
Before:
- No ARM CI; netns tests could fail with cryptic errors under QEMU.
After:
- arm_CI: armhf (QEMU, reduced test set) and arm64 (native, expanded tests, ASan).
- require_netns_capable() in diag.sh; netns tests call it and skip when unavailable.
- ratelimit double-free fixed in ratelimit.c.
- skip_ASAN() in diag.sh; empty-hostname, omfile-read-only* skip when ASan enabled
(LD_PRELOAD/read-only behavior conflicts).
Technical Overview:
- Add arm_CI job to run_checks.yml: matrix over armhf/arm64. armhf:
runs-on ubuntu-24.04, QEMU + Docker Buildx, reduced configure (disable-default-tests,
many modules disabled). arm64: runs-on ubuntu-24.04-arm (native), expanded
configure (default tests, gnutls, relp, imfile, etc.). Conditional QEMU
setup only for armhf.
- Add devtools/ci/Dockerfile.arm: Ubuntu 24.04 with build tools, gnutls,
libestr, libfastjson, zlib, iproute2, libgcrypt, librelp, uuid, libyaml
(for arm64 expanded build).
- Add require_netns_capable() to diag.sh; use in imtcp-netns.sh,
uxsock_multiple_netns.sh, tcp_forwarding_ns_tpl.sh.
- Add skip_ASAN() to diag.sh; use in empty-hostname.sh, omfile-read-only.sh,
omfile-read-only-errmsg.sh.
- Add devtools/ci/Dockerfile.arm to arm_CI changed-files filter.
- Quote $GITHUB_OUTPUT and $GITHUB_STEP_SUMMARY in clang static analyzer steps.
- Fix double-free in ratelimit.c: shared->name is the hashtable key, freed by
hashtable_destroy; remove redundant free(shared->name) in ratelimitFreeShared.
Non-technical: centralize and reuse rate-limit definitions so admins
can apply consistent policies across listeners. This is part of an
ongoing series to improve rate limiting and its manageability.
Before: inputs set per-listener interval/burst ad hoc.
After: inputs can reference a named ratelimit() policy shared across
listeners; per-listener values remain as fallback.
Impact: New ratelimit() object and RateLimit.Name param for imtcp/imptcp.
If a policy file is configured but libyaml is unavailable, config fails.
Technical details:
- Add top-level ratelimit() Rainerscript object. Parsed in rsconf and
stored in a central registry (hashtable + rwlock) on rsconf.
- New runtime API: ratelimitAddConfig(), ratelimitNewFromConfig(), plus
cfgs init/destruct on rsconf lifecycle.
- imtcp/imptcp accept RateLimit.Name; when set, tcpsrv/imptcp build the
ratelimiter from the named policy; otherwise legacy interval/burst is
used. Thread-safety retained via ratelimitSetThreadSafe().
- tcpsrv gains ownership helpers for listener params and frees them on
errors; imtcp explicitly transfers ownership and nulls the pointer.
- Optional libyaml: detected at configure; runtime parser loads simple
key/value policy files (interval, burst, severity).
- Docs: new ratelimit object page; imtcp/imptcp parameter references and
module docs updated; design-decisions note added for libyaml.
- Tests: add ratelimit_name.sh (guarded for imtcp+imptcp) to validate
named policy application and observable throttling.
Refs: https://github.com/rsyslog/rsyslog/issues/6201
With the help of AI-Agents: Antigravity
imudp: add ratelimit.name support
This commit adds the `ratelimit.name` parameter to imudp, allowing listeners
to utilize the global rate limit registry (shared state).
Features:
- New `ratelimit.name` string parameter.
- Integration with `ratelimitNewFromConfig`.
- Strict mutual exclusivity: specifying `ratelimit.name` prohibits the use
of legacy per-listener parameters (`ratelimit.burst`, `ratelimit.interval`).
If a conflict occurs, an error is logged and the named rate limit takes precedence.
- Updated documentation.
- New regression test `tests/imudp_ratelimit_name.sh`.
With the help of AI Agent: Google Antigravity
Improve code clarity and static analysis around ratelimit helpers.
This clarifies the expected non-null contract for callers.
Before: ratelimit helpers assumed non-null without annotation.
After: helpers assert and declare non-null parameters.
Impact: static analysis may warn on null callers.
Add ATTR_NONNULL annotations to ratelimitMsg and ratelimitAddMsg
in headers and definitions to make the contract explicit.
Add debug asserts so unexpected nulls fail fast in debug builds.
This keeps runtime behavior unchanged in release builds while
making ownership and expectations clearer to readers and tools.
With the help of AI-Agents: Codex CLI
This commit applies the new canonical formatting style using `clang-format` with custom settings (notably 4-space indentation), as part of our shift toward automated formatting normalization.
⚠️ No functional changes are included — only whitespace and layout modifications as produced by `clang-format`.
This change is part of the formatting modernization strategy discussed in:
https://github.com/rsyslog/rsyslog/issues/5747
Key context:
- Formatting is now treated as a disposable view, normalized via tooling.
- The `.clang-format` file defines the canonical style.
- A fixup script (`devtools/format-code.sh`) handles remaining edge cases.
- Formatting commits are added to `.git-blame-ignore-revs` to reduce noise.
- Developers remain free to format code however they prefer locally.
This commit performs a broad modernization of widely used rsyslog
macros to align with modern C practices and support automated
formatting tools like clang-format. The changes focus on improving
syntactic regularity, readability, and tooling compatibility — without
altering behavior.
Macros refactored in this commit now follow a consistent,
statement-like form with explicit trailing semicolons. Where
applicable, macro blocks that define module interfaces (`queryEtryPt`)
have been updated to use simple `if` statements instead of `else if`
chains. While this slightly increases evaluation time, the affected
functions are only called once per module during load time to register
supported interfaces — making the performance cost irrelevant in
practice.
These improvements serve multiple purposes:
- Enable reliable clang-format usage without mangling macro logic
- Simplify reasoning about macro-expanded code for human readers
- Reduce style drift and merge conflicts
- Facilitate development for contributors using assistive tools
- Support future formatting pipelines using:
1. `clang-format`
2. a post-fixup normalization script
Refactored macros:
- MODULE_TYPE_NOKEEP
- MODULE_TYPE_KEEP
- MODULE_TYPE_INPUT
- MODULE_TYPE_OUTPUT
- MODULE_TYPE_FUNCTION
- MODULE_TYPE_PARSER
- MODULE_TYPE_LIB
- DEF_IMOD_STATIC_DATA
- DEF_OMOD_STATIC_DATA
- DEF_PMOD_STATIC_DATA
- DEF_FMOD_STATIC_DATA
- DEFobjStaticHelpers
- SIMP_PROP(...)
And all `queryEtryPt()` dispatch macros:
- CODEqueryEtryPt_STD_MOD_QUERIES
- CODEqueryEtryPt_STD_OMOD_QUERIES
- CODEqueryEtryPt_STD_OMODTX_QUERIES
- CODEqueryEtryPt_STD_OMOD8_QUERIES
- CODEqueryEtryPt_TXIF_OMOD_QUERIES
- CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES
- CODEqueryEtryPt_STD_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_QUERIES
- CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES
- CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES
- CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES
- CODEqueryEtryPt_STD_PMOD_QUERIES
- CODEqueryEtryPt_STD_PMOD2_QUERIES
- CODEqueryEtryPt_STD_FMOD_QUERIES
- CODEqueryEtryPt_STD_SMOD_QUERIES
- CODEqueryEtryPt_doHUPWrkr
- CODEqueryEtryPt_doHUP
This general modernization reduces macro misuse, improves DX, and
lays the foundation for a robust, automated style normalization
system.
See also: https://github.com/rsyslog/rsyslog/issues/5747
Rate limiting may not have worked if the considered message had already
been parsed (not having NEEDS_PARSING in msgFlags).
This affects also imuxsock in its default configuration
(useSpecialParser="true" and ratelimit.severity="1")
A test for imuxsock is included which requires
./configure --enable-liblogging-stdlog
Adding new rate limit option to omfwd for rate limiting
syslog messages sent to the remote server
ratelimit.interval:
Specifies the rate-limiting interval in seconds.
Default value is 0, which turns off rate limiting.
ratelimit.burst
Specifies the rate-limiting burst in number of messages.
fixes#4423https://github.com/rsyslog/rsyslog/issues/4423
In theory, rsyslog should call parsers on the queue worker threads whenever
possible. This enables the parsers to be executed in parallel. There are
some cases where parsers needs to be called earlier, namely when parsed
data is needed for rate-limiting.
The logic to do this previously did not work correctly and was fixed six
years ago (!) by b51dd22. Unfortunately, b51dd22 was overly agressive:
it actually makes the early parser call now mandatory, effectively moving
parsing to the input side where there is no to little concurrency.
We still do not need to call the parser when all messages, regardless of
severity, need to be rate-limited. This is the default and very frequent
case. This patch introduces support for this and as such makes parsers
able to run in parallel in the frequent case again.
closes https://github.com/rsyslog/rsyslog/issues/4187
The burst parameter in the ratelimit was increased to an unsigned int
but the interval remained an unsigned short. While it may be unusual,
there is possibly a chance to need to represent an interval longer than
about 3/4 of a day.
While here, go through and normalize all the various incarnations of
rate limiting to be explicitly unsigned int for the burst and interval.
In the community we frequently discuss handling of oversize messages.
David Lang rightfully suggested to create a central capability inside
rsyslog core to handle them.
We need to make a distinction between input and output messages. Also,
input messages frequently need to have some size restrictions done at
a lower layer (e.g. protocol layer) for security reasons. Nevertheless,
we should have a central capability
* for cases where it need not be handled at a lower level
* as a safeguard when a module invalidly emits it (imfile is an example,
see https://github.com/rsyslog/rsyslog/pull/2632 for a try to fix it
on the module level - we will replace that with the new capability
described here).
The central capability works on message submission, and so cannot be
circumvented. It has these capabilities:
* overisze message handling modes:
- truncate message
- split message
this is of questionable use, but also often requested. In that mode,
the oversize message content is split into multiple messages. Usually,
this ends up with message segments where all but the first is lost
anyhow as the regular filter rules do not match the other fragments.
As it is requested, we still implemented it.
- accept message as is, even if oversize
This may be required for some cases. Most importantly, it makes
quite some sense when writing messages to file, where oversize
does not matter (accept from a DoS PoV).
* report message to a special "oversize message log file" (not via the
regular engine, as that would obviously cause another oversize message)
This commit, as the title says, handles oversize INPUT messages.
see also https://github.com/rsyslog/rsyslog/issues/2190
closes https://github.com/rsyslog/rsyslog/issues/2681
closes https://github.com/rsyslog/rsyslog/issues/498
Note: this commit adds global parameters:
* "oversizemsg.errorfile",
is used to specify the location of the oversize message log file.
* "oversizemsg.report",
is used to control if an error shall be reported when an oversize
message is seen. The default it "on".
* add global parameter "oversizemsg.input.mode"
is used to specify the mode with which oversized messages will
be handled.
rsyslog uses unsigned short for configuration setting
SystemLogRateLimitBurst. Being just 16 bits, unsigned short cannot
hold values bigger than 65535. On our LDAP server rsyslog misbehaved with
SystemLogRateLimitBurst being bigger than 65535. Make it unsigned int to
accept bigger values.
access to the Linux-like ratelimiter was not properly synchronized, and
the system rate-limiter was not using it in any case.
This could lead to the rate-limit settings not being properly
respected, but no "hard error".
(for example, it did not work with imdup). The reason was that
the PRI value was used, but that needed parsing of the message, which
was done too late.
and make imjournal use it.
The new mode is needed, as imjournal uses journal's timestamp
as message generation time (which otherwise is when the message
entered the system, and the ratelimiter uses this as current
timestamp in order to save performance).
It is debatable if imjournal is doing the right thing here. But
it doesn't feel totally wrong. So let's safe that debate for
later ;)
The message that reports how many messages were lost due to
ratelimiting was sent before reseting the state that led to it. If it
itself got ratelimited, this could lead to an endless loop.
If messages are being dropped because of ratelimiting, an internal
message is generated to inform about this fact. This should happen
only uppon the firs occurance but the counter that tracks the number
of dropped messages was incremented only after sending the message. If
the message itself gets ratelimited, an endless loop spins out of
control. Thanks to Jerry James for notifying about this.
Move the severity-check logic inside the ratelimiter and add a new
function ratelimitSetSeverity() to manipulate the treshold.
Currently only utilized by the imuxsock module.
