Why:
Enabling imfile coverage on ARM exposed two arm64 follow-up problems. The
impstats crash in imfile-statistics was caused by a generic stringbuf off-by-one
when a finalized cstr lands exactly on the allocation boundary, and the
rename-while-stopped test could shut rsyslog down before startup processing had
made the renamed-file backlog observable.
Impact:
The ARM CI change remains in place, while the newly exposed arm64 failures are
addressed instead of being masked off.
Before:
- ARM CI enabled imfile coverage, but arm64 could crash in stats reporting when
cstrFinalize() wrote the trailing NUL past the end of a just-filled buffer.
- The rename-while-stopped test could rely on a timing race between imfile's
startup scan and the testbench queue-empty shutdown helper.
After:
- rsCStrAppendStrWithLen() keeps one spare byte for cstrFinalize(), preventing
the impstats JSON formatting overflow seen in imfile-statistics on arm64.
- imfile-rename-while-stopped.sh waits for the expected backlog to be emitted
before calling shutdown_when_empty in each phase.
- The armhf/arm64 imfile coverage enablement in CI is preserved.
Technical Overview:
- Keep the existing ARM workflow enablement from this PR.
- Fix the stringbuf append/finalize contract so appended payloads reserve space
for the trailing NUL written during finalization.
- Make the rename-while-stopped test wait for the expected output count before
invoking the queue-empty shutdown helper, so the test checks rename/persist
behavior instead of a startup/shutdown timing race.
See also https://github.com/rsyslog/rsyslog/issues/6576
With the help of AI-Agents: Codex
Why:
ASan found a real leak in yaml-basic-yamlonly.sh. The translation tests
had been masking the same issue by disabling leak detection, which is
not acceptable for a real parser/loader bug.
Impact: YAML script-buffer loading now shuts down cleanly under ASan,
and the translation tests run without detect_leaks=0 overrides.
Before/After:
Before, YAML ruleset script loading leaked the top-level synthetic lexer
buffer name and tests suppressed leak detection. After, the leak is
freed in the lexer teardown path and the related tests run cleanly with
ASan leak detection enabled.
Technical Overview:
The YAML loader can push synthetic RainerScript buffers into the lexer
stack via cnfAddConfigBuffer().
For top-level synthetic buffers, yamlconf restores cnfcurrfn before the
lexer pop path runs, so the old top-level-name exception in popfile()
left the duplicated buffer name unreachable.
Adjust popfile() to free the top-level buffer name when cnfcurrfn no
longer points at that allocation, while preserving the normal top-level
file path ownership model.
Remove the temporary detect_leaks=0 overrides from the config-translate
shell tests now that the underlying leak is fixed.
Validate with yaml-basic-yamlonly.sh and the translation test set under
ASan.
With the help of AI-Agents: Codex
Why:
Common distro defaults still use traditional selector/action lines.
Translating those into native YAML makes migration output easier to
review and edit than falling back to script text.
Impact: Common legacy file and omusrmsg actions now emit structured
YAML, with docs and tests covering the supported subset.
Before/After:
Before, lines like user.* -/var/log/user.log fell back to warnings and
script output. After, they become structured YAML filter/actions or
YAML-native statements for repeated top-level selector lines.
Technical Overview:
Teach the translation layer to recognize a limited set of legacy action
forms and convert them into normal action parameter lists.
Support /path and -/path as omfile actions and :omusrmsg:... as
omusrmsg actions.
Promote repeated top-level selector lines into YAML-native statements
under RSYSLOG_DefaultRuleset instead of overwriting earlier entries.
Prefer prifilt('...') for readable constant selector expressions, with a
fallback to double quotes if the selector contains a single quote.
Add focused shell tests for single-line shorthand and a Debian-style
legacy default snippet.
Document the new legacy translation support and clearly state that it is
limited and does not cover all legacy constructs.
With the help of AI-Agents: Codex
Why
The config translation feature should emit native YAML when a ruleset body is
simple enough to map directly, and the documentation and tests should reflect
that actual behavior.
Impact
Improves YAML translation for simple rulesets and updates docs and tests to
match.
Before/After
Before: simple translated rulesets always fell back to YAML script blocks.
After: simple action-only and selector-plus-action rulesets emit structured
YAML with actions or filter plus actions, while legacy shorthand remains
covered by explicit warning-based tests.
Technical Overview
Extend the translator to retain a small structured model for simple ruleset
bodies while keeping the existing script fallback for complex logic. This lets
YAML output use native actions and filter plus actions when the captured
statement tree is directly representable.
Update the translation tutorial and YAML reference to show the new structured
output and explain that complex rulesets still fall back to script blocks.
Refresh the exact-output tests, add coverage for selector-to-filter/actions
translation, use an omfile action with a file parameter in the basic
RainerScript-to-YAML examples, and add a regression test for the common distro
legacy shorthand `user.* -/var/log/user.log` path.
With the help of AI-Agents: Codex
Why: provide a migration path between supported rsyslog config
formats without requiring external conversion tooling.
Impact: rsyslogd can now emit canonical YAML or RainerScript
from supported configs during -N1 validation, with warning comments
for legacy or potentially misunderstood constructs.
Before/After: config validation could inspect syntax only; it can
now also write a translated canonical config file.
Technical Overview:
Add a retained translation document that captures config objects and
ruleset script during config load, before transient parser data is
consumed or destroyed.
Wire rsyslogd to accept a translation target format and output path,
then emit canonical YAML or modern RainerScript from the retained
representation.
Preserve module, input, queue, ruleset, and other shared config
objects, normalize top-level script into an explicit default
ruleset, and add in-output warning comments for legacy or lossy
translations.
Guard syntax cloning so translation-only capture does not add normal
runtime overhead, and report item-scoped fatal translation warnings
through the standard error path.
Add focused testbench coverage for RainerScript-to-YAML,
YAML-to-RainerScript, and legacy-warning output.
With the help of AI-Agents: Codex
Why
Rsyslog's test suite and documentation used a mix of the bash-specific
"source" command and the portable POSIX "." command. This inconsistency
caused issues with repo-policy checks and potential portability
concerns.
Impact
Testing and documentation. All test wrappers in tests/ and guidance
in READMEs and skills now consistently use the POSIX standard.
Before/After
Before: 200+ files used "source", others used ".".
After: "source" is replaced by "." for script inclusion across the
repository. Repo-policy checks are tightened to enforce this.
Closes: https://github.com/rsyslog/rsyslog/issues/6635
Why:
The YAML policy file should cover the full transformation behavior that
mmjsontransform exposes, so operators can update policy-driven JSON
normalization in one place and reload it on HUP.
Impact: Policy files can now control mode as well as rename/drop rules,
and HUP reload swaps the effective policy atomically.
Before/After:
Before: policy files could only rename or drop keys and the action
parameter mode stayed fixed until restart.
After: policy files can also select flatten or unflatten mode, and HUP
reload updates that mode together with the other policy rules.
Technical Overview:
The action parameter mode remains the fallback default, while the active
policy can now carry its own mode override inside the reloadable policy
state.
The YAML loader validates a top-level mode key, swaps it together with
the rename/drop structures, and keeps the previous in-memory policy when
reload fails.
Action execution resolves the effective mode under the policy lock, so a
single HUP can switch both preprocessing rules and the final transform.
The focused policy test now covers startup flatten mode, HUP reload to
unflatten mode, and invalid-mode reload rejection with retention of the
previous policy.
The docs and ChangeLog were updated to describe the expanded policy
schema and the HUP-reload behavior.
Validation:
- ./devtools/format-code.sh --git-changed
- make -j$(nproc)
- ./tests/mmjsontransform-policy-basic.sh
With the help of AI-Agents: Codex
Why:
JSON normalization often needs a small preprocessing step before
flattening or unflattening so deployments can drop noisy fields and
rename source keys without extra ruleset glue.
Impact: mmjsontransform can now load a YAML policy, reload it on HUP,
and apply rename/drop rules before transformation.
Before/After:
Before: mmjsontransform only transformed the structure that it was given.
After: it can preprocess input keys via a YAML policy and then run the
normal flatten or unflatten logic on the adjusted JSON.
Technical Overview:
This adds an optional action parameter, policy, which loads a YAML file
with map.rename and map.drop directives.
Policy state is shared per action instance, guarded by a mutex, loaded
at startup, and rechecked on HUP so mtime changes trigger reload while a
failed reload keeps the previous in-memory policy.
Drop rules are compiled into a lookup object, rename rules are compiled
into a mapping object, and both are applied before the main transform.
Nested rename handling keeps targets relative to the current recursion
context, and the loader cleans up partial allocations correctly on error
paths.
The patch also documents the new parameter, registers its reference page
for distribution, and adds a focused test that covers initial policy
application plus HUP-driven reload.
Validation:
- ./tests/mmjsontransform-policy-basic.sh
With the help of AI-Agents: Codex
wait_startup() was checking the elapsed time against TB_STARTTEST (set
once when the test begins), so any test that calls startup() multiple
times with message-processing phases in between could exhaust the 60 s
startup budget before the last startup even gets a chance to wait.
diskqueue-oncorruption-missing-segment.sh is the canonical example: it
runs three corruption modes each with two rsyslog invocations; the first
invocation of each mode injects 15,000 messages (~20 s on arm64/ASAN),
so by the time the sixth startup is attempted the global clock has
already ticked past TB_STARTTEST + 60.
Fix: record a per-call deadline at the top of wait_startup() and check
against that instead of the global test-start time.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- imdiag: add 5 module-scope config params to control key queue timeouts
(mainmsgqueuetimeoutshutdown, mainmsgqueuetimeoutenqueue,
inputshutdowntimeout, defaultactionqueuetimeoutshutdown,
defaultactionqueuetimeoutenqueue); these replace per-test legacy
$MainMsg* directives and eliminate races on slow machines (Solaris)
- imdiag: add assert(0) for unhandled params in setModCnf/newInpInst
- imdiag: modExit sets abortTimeout=-1 after joining timeoutGuard thread
to prevent double-cancel if resetConfigVariables runs afterwards
- diag.sh: generate_conf --yaml-only flag for YAML-only test mode
- diag.sh: yaml preamble uses new imdiag module params
- diag.sh: add RSTB_MAIN_Q_TO_ENQUEUE separate from action queue var
- diag.sh: restore .started with instance-ID suffix for content_check
- diag.sh: wait_startup() uses imdiag port file for startup detection
- tests: add yaml-basic-yamlonly.sh smoke test
- doc: 6 new reference pages for new imdiag params (with meta blocks)
- doc: imdiag.rst updated with new module-scope params table
- doc: dev_testbench.rst corrected .started vs port-file description
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Why:
Tests that exercise the YAML configuration loader need to run without
any RainerScript preamble. The existing generate_conf function always
writes a .conf file using legacy and RainerScript directives, making it
impossible to test pure-YAML startup paths.
Technical Overview:
- imdiag: replace EMPTY_STRUCT with real modConfData_s fields; add
BEGINsetModCnf (listenportfilename, aborttimeout), BEGINendCnfLoad,
BEGINnewInpInst (port), and the three STD_CONF2 queryEtryPt macros.
Legacy $IMDiag* directives are preserved unchanged. Fix a NULL
deref in addTCPListener when pszLstnPortFileName is not set.
- diag.sh: generate_conf gains a --yaml-only flag that writes a pure
YAML preamble (version/global/mainqueue/modules/inputs) instead of a
.conf file. net.ipprotocol is resolved before the preamble is written
to avoid a duplicate global: key. add_yaml_conf() mirrors add_conf()
for the yaml path. startup_common selects .yaml when
RSYSLOG_YAML_ONLY=1. wait_startup comment documents that .started is
absent in yaml-only mode and that the OR logic handles it.
- tests/yaml-basic-yamlonly.sh: new test exercising the yaml-only path
end-to-end (imtcp, 100 messages, seq_check).
- tests/Makefile.am: register yaml-basic-yamlonly.sh under TESTS_LIBYAML.
- tests/AGENTS.md: document the yaml-only mode, its limitations, and
the expected naming/registration conventions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce a dedicated rsyslog_config agent skill that captures the
dual-frontend config architecture (RainerScript grammar + yamlconf.c)
and the parity rules that must be observed whenever the config layer
is extended.
Key additions:
- .agent/skills/rsyslog_config/SKILL.md: full skill covering the shared
backend model, a parity table for each change type, key file index,
YAML test conventions, and documentation requirements.
- AGENTS.md: register rsyslog_config in the skills table.
- runtime/AGENTS.md: add yamlconf.c/h to key components with an
explicit cross-reference to the parity rule.
- tests/AGENTS.md: add config format coverage rule and YAML test
naming/registration conventions.
- plugins/AGENTS.md: extend the parameter documentation rule to include
YAML examples, and note that new statement/block types require
yamlconf.c changes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
grep -q exits 2 on a missing file, which is indistinguishable from
exit 1 (no match) in a plain if/then check. Add an explicit -f test
so the test fails loudly if the unconditional scratch write never ran.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Why
The PR baseline still left a few regressions in omhttp retry handling,
header setup, and cleanup paths.
Impact
Retries keep their previous behavior, compressed requests carry the
correct headers, and worker/server state cleanup is consistent.
Before/After
Before, health-check throttling was still worker-local and compressed
POSTs could reuse a non-gzip header set; after, shared health state is
tracked globally and headers follow the actual payload encoding.
Technical Overview
Add shared per-server last-check timestamps in instance state so workers
observe the same health-check throttle window.
Separate send-failure suspension from failed health-check backoff so the
default healthchecktimedelay=-1 still allows immediate retry probing.
Rebuild per-server curl headers whenever gzip usage changes so
Content-Encoding matches the request body, which restores compressed
batch compatibility such as the VictoriaLogs path.
Tighten worker/server cleanup by zero-initializing the server pointer
array, centralizing server-data destruction, and fixing dynamic
rest-path replacement leaks.
With the help of AI-Agents: Codex
runtime/yamlconf.c:
- Convert file header to @file Doxygen format (COMMENTING_STYLE.md §1)
- Add Doxygen @brief to YAMLCONF_MAX_DEPTH macro (COMMENTING_STYLE.md §4)
- Extract yaml_value_to_nvlst_node() helper: the ~30-line scalar/sequence
→ nvlst-node conversion pattern was copy-pasted across 4 functions
(build_nvlst_from_mapping, parse_template_sequence, parse_ruleset_sequence,
build_one_stmt_rs); consolidates ~120 lines into one well-documented place
- Add focused Doxygen on yaml_value_to_nvlst_node() and yamlconf_load()
(public API, mandatory per COMMENTING_STYLE.md §1)
tests/yaml-template-list.sh:
- grep -q checked only that *some* lines contain 'msgnum:'; replace with
grep -c comparison to NUMMESSAGES so every line must match
tests/yaml-statements-unset.sh:
- unset: was exercised but never verified; add verifyfmt template that
captures %.scratch% after unset, then assert the pre-unset marker value
is absent from the verify file
tests/yaml-statements-call.sh:
- $.tag was set but never read; add $.tag == "seen" to rs2's filter so
seq_check fails if set: didn't actually store the value
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Why
The testbench can declare rsyslog startup complete as soon as the
started marker or imdiag port file appears. On slower runners, the pid
file may still be missing for a short period after that.
Impact: avoids spurious testbench failures in lookup-table reload tests
and other helpers that signal or stop rsyslog by pid.
Before/After: pid-based helpers could race missing, empty, or malformed
pid files; they now wait for a valid pid before reading or signaling it.
Technical Overview
Add a dedicated helper that waits for a non-empty pid file containing a
numeric pid for a specific rsyslog instance.
Call that helper from getpid(), issue_HUP(), shutdown_when_empty(),
and shutdown_immediate() before they read or signal the process pid.
Keep the helper silent on success so command substitutions like
pid=\$(getpid) remain valid.
This keeps the relaxed startup diagnostics in wait_startup() while
making pid-driven actions robust on slower systems such as s390 Debian.
With the help of AI-Agents: Codex
rsyslog now accepts .yaml / .yml files as a full alternative to
RainerScript configuration. Whenever a .yaml or .yml file is
referenced (e.g. via $IncludeConfig or as the main config), it is
routed through the new yamlconf loader instead of the bison/flex
parser. RainerScript and YAML configurations may coexist.
Supported top-level sections
global: global settings
main_queue: main queue settings
modules: module loading (load:, params:)
inputs: input objects templates: string, subtree, plugin, and list templates
(including all property/constant modifiers)
rulesets: rulesets with optional script: block (raw
RainerScript), filter/action shortcut (if:/then:),
and the full statements: block:
action:, set:, unset:, call:, call_indirect:,
foreach:, if:, stop:, continue:,
reload_lookup_table:
outputs: output objects (top-level convenience alias)
include: recursive includes (.yaml processed immediately;
.conf files pushed to LIFO stack in reverse order
for correct document-order processing)
Implementation notes
- Thin front-end over existing cnfobj/nvlst/cnfDoObj() machinery;
all parameter validation happens downstream unchanged.
- Script blocks synthesised as RainerScript strings and injected via
cnfAddConfigBuffer() (requires two trailing NUL bytes).
- YAML -> YAML includes are synchronous recursive calls;
YAML -> .conf includes use the flex buffer stack.
- Mixed include lists (.yaml + .conf) emit a warning because strict
document order cannot be preserved across the two mechanisms.
- Built conditionally: --with-libyaml / HAVE_LIBYAML.
Tests
19 new shell tests cover: basic config, modules, inputs, templates
(string/subtree/list with modifiers), rulesets with script/filter/
statements blocks, stop/continue, set/unset, foreach, call,
call_indirect, reload_lookup_table, includes, error handling.
Documentation
doc/source/configuration/yaml_config.rst (new reference page)
Inline comments in yamlconf.c explain all invariants and non-obvious
design choices (es_getBufAddr non-null-termination, LIFO stack
ordering, cnfAddConfigBuffer double-NUL requirement, etc.)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Allow YAML rulesets to use structured filter: + actions: keys as
an alternative to inline script: blocks. This covers the most
common use-case (route by syslog priority or property filter to
one or more outputs) without any RainerScript in the file.
- filter: accepts syslog PRI selectors (e.g. *.info) or property
filters (starting with ':'). Omitting filter: routes all messages.
- actions: is a YAML sequence; each item maps to a module action().
- filter:/actions: and script: are mutually exclusive per ruleset.
Backed by cnfstmtNewPRIFILT / cnfstmtNewPROPFILT / cnfstmtNewAct;
no new grammar rules required.
Add yaml-filter-actions.sh testbench test (all 5 YAML tests pass).
Update yaml_config.rst with Structured Filter Shortcut section and
revised complete example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rsyslog now accepts .yaml/.yml files as configuration files
in addition to RainerScript (.conf). The format is triggered
automatically based on file extension, both for the main config
file (-f flag) and for include() directives.
Architecture:
- New runtime/yamlconf.c (libyaml event-based parser) builds the
same cnfobj+nvlst structures that the bison parser produces and
calls cnfDoObj() for each block.
- Ruleset script: keys are synthesised into a complete RainerScript
ruleset(name=...) { ... } block and pushed onto the flex buffer
stack so the existing lex/bison pipeline handles all filter
expressions and statements unchanged.
- routing in rsconf.c::load() and cnfDoInclude() detects .yaml/.yml
by extension and delegates to yamlconf_load().
- cnfHasPendingBuffers() (new) lets rsconf.c flush queued script
buffers after YAML-as-main-config loading.
- Guarded by #ifdef HAVE_LIBYAML throughout; graceful error when
libyaml is absent.
Schema (top-level YAML keys):
global, modules, inputs, templates, rulesets, mainqueue,
include, parser, lookup_table, dyn_stats, ratelimit, timezone
Parameter names are identical to RainerScript; all type coercion
and validation is reused via nvlstGetParams() unchanged.
Tests: yaml-basic, yaml-include, yaml-ruleset-script, yaml-error
Documentation: doc/source/configuration/yaml_config.rst
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
GnuTLS silently ignores expired CRLs -- it loads them into credentials
without checking the validity period. This means rsyslog kept accepting
TLS connections even when the configured CRL file had expired, unlike
the OpenSSL driver which rejects them via X509_V_ERR_CRL_HAS_EXPIRED
in the verify callback.
Fix this by manually checking the CRL thisUpdate/nextUpdate timestamps
when loading the CRL file. The file is loaded once, validated, then
passed to credentials via gnutls_certificate_set_x509_crl_mem(). The
nextUpdate timestamp is also cached on the connection object so that
each TLS handshake can cheaply detect if the CRL has expired since
startup, matching OpenSSL's per-connection behavior.
Add CRL tests for both GnuTLS and OpenSSL drivers, each with two
phases: valid CRL (communication succeeds) and expired CRL (connection
rejected, no messages received). Includes a pre-generated expired CRL
test fixture.
add initial tests
omazuredce: add Azure Monitor output module
This adds a documented Azure Monitor output path so rsyslog can
ship events to Azure-hosted observability workflows. It also makes
the module easier to review and iterate on as a project feature.
Impact: adds a new output module, config parameters, docs, and tests.
Before: rsyslog had no native Azure Monitor DCE output module.
After: rsyslog can batch and send records to Azure Monitor via
omazuredce.
The change adds a new omazuredce plugin and wires it into the
build system and module map. The module obtains Azure tokens,
renders payloads from templates, batches records by size, and
flushes them through the existing action queue model.
The patch also documents the module and its parameters so the
feature is usable without external notes. Supporting config and
build integration are included so the module can be compiled and
packaged consistently with the rest of the tree.
AI-Agent: Codex 2025-06
With the help of AI-Agents: Codex
add tests and address issues
address issues
more issues
more issues
more issues
issues
issues
fix issues
fixfix
fixy
fix
fix fixy fix pants
fix fixy fix pants2
redo the doc
Why
Native post-quantum TLS support should be usable and testable on newer
distro baselines without adding provider-mode compatibility work for
older platforms.
Impact
Rsyslog now has native-PQ smoke tests, clearer TLS diagnostics, updated
CI baselines and helper images, and a new post-quantum tutorial for
supported distros.
Before/After
Before: Fedora CI still targeted Fedora 41, PQ-capable TLS settings had
no dedicated rsyslog tests or user-facing tutorial, and stricter clang
builds could fail on warning-group handling.
After: CI targets Fedora 43, native PQ usage is documented and smoke-
tested, helper images include the required tools, and the branch builds
and tests cleanly with the newer compiler/container combinations.
Technical Overview
The CI matrix now replaces the Fedora 41 lane with Fedora 43 and adds a
matching Fedora 43 development image.
The Debian 13 and Fedora 43 development containers now install the
GnuTLS CLI utilities needed for native PQ capability checks.
The OpenSSL TLS config path logs clearer messages when a command or
value is unavailable on the native OpenSSL build.
The GnuTLS TLS config path reports unsupported priority-string options
more explicitly.
Two new shell tests add native PQ smoke coverage for OpenSSL and GnuTLS
using the existing gnutlsPriorityString control surface.
Those tests self-skip unless the local native TLS libraries expose the
required hybrid group support.
The imtcp parameter docs and omfwd docs now explain the native-only PQ
support policy and include example configurations.
A new tutorial documents native PQ usage for OpenSSL and GnuTLS on
supported newer distro versions.
The shared runtime warning policy in rsyslog.h now tolerates clang
handling of unknown warning groups so older and newer clang lanes remain
warning-free under the existing finalize_it error-handling pattern.
Testbench follow-ups harden omfwd-lb-susp with isolated retry attempts,
skip rcvr_fail_restore on ARM where it is timing-flaky, and keep local
SC2181 suppressions where if-exec rewrites would reduce shell-script
usability.
The Fedora 43 Dockerfile now cleans the dnf cache after install and
locally suppresses the non-useful DL3041 package-version pinning warning.
Older distro versions remain intentionally unsupported for PQ in this
phase because we expect users to move to newer baselines first.
If there is demand later, older-version support can be considered in a
separate effort.
With the help of AI-Agents: Codex
Add ratelimit.interval, ratelimit.burst, and ratelimit.name parameters
to omusrmsg. This prevents DoS scenarios where emergency message floods
can overwhelm user terminals.
The implementation follows the established output module pattern:
- Sentinel-based (-1) defaults for mutual-exclusivity detection
- ratelimit.name references a shared ratelimit() configuration object
- Per-action ratelimiters via ratelimitNew()/ratelimitSetLinuxLike()
- ratelimitMsgCount() gate in doAction
Also adds module-level config (modConfData_s) with beginCnfLoad,
endCnfLoad, checkCnf, activateCnf, freeCnf entry points, needed to
stash rsconf_t* for ratelimitNewFromConfig().
Includes documentation updates and a config-validation test.
Closes: https://github.com/rsyslog/rsyslog/issues/4547
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire ratelimit.interval, ratelimit.burst, and ratelimit.name into imrelp,
following the established pattern from imtcp/imptcp/imudp. When
ratelimit.name is set, the listener uses a shared named rate limiter
(with per-source support). Local interval/burst and named rate limiters
are mutually exclusive.
Includes parameter documentation, module doc update, and integration test.
Closes: #6597
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Why: Commit 3ee31a8d0 added shared named ratelimit support
(ratelimit.name + ratelimitNewFromConfig) but only wired it
into imtcp, imptcp, and imudp. Seven modules with existing
ratelimit.interval/ratelimit.burst support were left without
the ability to reference shared ratelimit() objects.
Impact: Adds ratelimit.name parameter to omfwd,
omelasticsearch, omhttp, imhttp, imjournal, imklog, and
imuxsock. Existing configurations remain unchanged.
Before: These seven modules could only use inline
ratelimit.interval/ratelimit.burst parameters.
After: All modules can now reference shared ratelimit()
configuration objects via ratelimit.name.
Technical Overview:
For each module, the same pattern from the imudp reference
implementation is applied:
- Add uchar *pszRatelimitName to config struct
- Add ratelimit.name to cnfparamdescr
- Change interval/burst defaults to -1 (sentinel)
- Parse ratelimit.name in config handler
- Add mutual-exclusivity check (name vs interval/burst)
- Branch ratelimiter creation on pszRatelimitName:
if set, call ratelimitNewFromConfig(); else legacy path
- Free pszRatelimitName in destructor
imuxsock is the most complex: it supports both per-socket
ratelimit.name and syssock.ratelimit.name, with the
per-source hashtable ratelimiter path also updated.
imklog uses legacy parameter names (ratelimitinterval,
ratelimitburst) but the new parameter uses the dotted
form (ratelimit.name) for consistency.
Each module receives a parameter doc page, module doc
update, and a config-validation test.
With the help of AI-Agents: GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Why:
logrotate 3.8+ refuses to rotate files in world/group-writable parent
directories unless a su directive is present. su root root causes
"Operation not permitted" in CI running as non-root (uid 1000).
Impact:
All five imfile-logrotate* tests now pass with logrotate 3.8+ in root
and non-root environments.
Before/After:
Before: Tests timeout or fail under non-root CI with logrotate 3.8+.
After: Dynamic su uses current user/group; rotation succeeds in CI.
Technical Overview:
Add su '"$(id -un)"' '"$(id -gn)"' to each test's logrotate config block.
This satisfies logrotate's security check without switching to root.
Affected: imfile-logrotate.sh, imfile-logrotate-async.sh,
imfile-logrotate-copytruncate.sh, imfile-logrotate-multiple.sh,
imfile-logrotate-nocopytruncate.sh.
Why: The omotel module only supported JSON encoding for OTLP/HTTP
exports. Native protobuf encoding produces smaller payloads with
lower serialization overhead, which is the preferred transport for
high-volume OpenTelemetry deployments.
Impact: Users can now set protocol="http/protobuf" to use binary
protobuf encoding. The default remains "http/json". A pre-existing
bug in JSON timestamp encoding (timeUnixNano/observedTimeUnixNano
emitted as integers instead of strings) is also fixed.
Before: omotel only supported http/json. The timeUnixNano fields
were encoded as JSON integers, violating the proto3 JSON mapping
for fixed64 which requires string representation.
After: omotel supports both http/json and http/protobuf. The JSON
encoder emits timestamps as strings per the OTLP specification.
protobuf-c is a required build dependency when --enable-omotel is
set.
Technical Overview:
Vendors three upstream OTLP .proto files (common, resource, logs)
in plugins/omotel/proto/ and adds protoc-c build rules to
Makefile.am that generate C bindings at build time. The generated
files are handled as nodist sources with BUILT_SOURCES and
CLEANFILES entries.
Implements otlp_protobuf.c (~650 lines) with a
omotel_protobuf_build_export() entry point that constructs a full
ExportLogsServiceRequest protobuf tree from rsyslog log records.
The tree is assembled incrementally — container nodes are linked
to the request immediately upon creation so that
free_export_request() can walk and free a partially-built tree on
any error path. KV attribute arrays use NULL-after-transfer
ownership tracking with kv_array_free() cleanup.
OOM from make_*_kv() helpers is detected and propagated
(kv_array_add treats NULL as OOM error; callers guard empty values
explicitly). TraceId/SpanId are decoded from hex strings to raw
bytes as required by the binary protobuf wire format.
The flush path in omotel.c selects between JSON and protobuf
based on the protocol config parameter. omotel_http_client_config
gains a content_type field so the HTTP layer sends the correct
Content-Type header for each encoding. Gzip compression operates
on the selected buffer regardless of encoding.
configure.ac requires protobuf-c >= 1.0.0 and protoc-c when
--enable-omotel is set. Documentation, MODULE_METADATA.yaml, and
an integration test (omotel-protobuf-basic.sh) are included.
With the help of AI-Agents: GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Rainer Gerhards <rgerhards@adiscon.com>
Why:
This branch combines two related hardening steps for disk queue reliability:
- robust corruption detection/recovery handling in disk queue state/file validation
- worker startup cancellation-race closure that could lead to shutdown wait loops
Impact:
- disk queue scan now rejects out-of-range segment sequence numbers early and
reports corruption deterministically.
- worker startup no longer exposes a cancellation window before cleanup
registration.
- test/CI diagnostics preserve timeout backtraces (gdb) in ARM jobs and print
them to stdout for post-mortem debugging.
- test script cleanup removes redundant operations and uses a macOS-friendlier
segment enumeration path.
Technical Overview:
- runtime/queue.c:
- add out-of-range sequence-number rejection during spool scan
- keep orphan-loop range check as defensive fallback
- runtime/wtp.c:
- disable cancellation and register cleanup before publishing RUNNING
- document startup/cancellation invariant inline
- runtime/wti.c:
- add concise cancellation-contract comment
- devtools/ci/Dockerfile.arm:
- install gdb for CI timeout diagnostics
- tests/diskqueue-oncorruption-missing-segment.sh:
- emit timeout gdb backtraces to stdout
- drop redundant STARTED_LOG truncate
- avoid GNU find -printf/mapfile dependency in segment listing
* build: default-enable impstats-push and align CI containers
Enable impstats-push by default and keep configure strict when dependencies are missing.
Update CI/container definitions for distro differences (CentOS/OpenEuler/Ubuntu and workflow overrides), add explicit --disable-impstats-push where impstats is disabled, and fix impstats protobuf generation for distcheck/VPATH builds.
Why:
Enable CI validation on ARM architectures so platform-specific regressions
are caught before merge. arm64 uses native GitHub runners; armhf uses QEMU
because GitHub does not offer 32-bit ARM runners. Network namespace tests may
fail under QEMU where the mount syscall is not properly emulated.
Impact:
- New arm_CI job runs on armhf and arm64 when relevant files change.
- arm64 uses native ubuntu-24.04-arm runner; armhf uses QEMU on x64.
- Netns tests skip gracefully (exit 77) if ip netns add fails.
Before:
- No ARM CI; netns tests could fail with cryptic errors under QEMU.
After:
- arm_CI: armhf (QEMU, reduced test set) and arm64 (native, expanded tests, ASan).
- require_netns_capable() in diag.sh; netns tests call it and skip when unavailable.
- ratelimit double-free fixed in ratelimit.c.
- skip_ASAN() in diag.sh; empty-hostname, omfile-read-only* skip when ASan enabled
(LD_PRELOAD/read-only behavior conflicts).
Technical Overview:
- Add arm_CI job to run_checks.yml: matrix over armhf/arm64. armhf:
runs-on ubuntu-24.04, QEMU + Docker Buildx, reduced configure (disable-default-tests,
many modules disabled). arm64: runs-on ubuntu-24.04-arm (native), expanded
configure (default tests, gnutls, relp, imfile, etc.). Conditional QEMU
setup only for armhf.
- Add devtools/ci/Dockerfile.arm: Ubuntu 24.04 with build tools, gnutls,
libestr, libfastjson, zlib, iproute2, libgcrypt, librelp, uuid, libyaml
(for arm64 expanded build).
- Add require_netns_capable() to diag.sh; use in imtcp-netns.sh,
uxsock_multiple_netns.sh, tcp_forwarding_ns_tpl.sh.
- Add skip_ASAN() to diag.sh; use in empty-hostname.sh, omfile-read-only.sh,
omfile-read-only-errmsg.sh.
- Add devtools/ci/Dockerfile.arm to arm_CI changed-files filter.
- Quote $GITHUB_OUTPUT and $GITHUB_STEP_SUMMARY in clang static analyzer steps.
- Fix double-free in ratelimit.c: shared->name is the hashtable key, freed by
hashtable_destroy; remove redundant free(shared->name) in ratelimitFreeShared.
Harden mysqld-start testbench helper by ensuring the MySQL datadir is initialized before starting mysqld.
This makes MySQL-backed tests resilient across distro/container variants where mysql-server may be installed but system tables are not pre-created.
Use mysqld --initialize-insecure when available, with mysql_install_db as a fallback.
Why
VictoriaLogs jsonline is a target deployment path for omhttp users and
we need a direct integration signal in PR CI.
Impact
Adds a real-container omhttp->VictoriaLogs validation path and a scoped
CI job for relevant PRs.
Before/After
Before: no CI test validated omhttp against VictoriaLogs jsonline.
After: PRs touching omhttp or this test run a minimal live integration
check.
Technical Overview
Add tests/omhttp-victorialogs-jsonline.sh to send batched newline JSONL
payloads with omhttp to /insert/jsonline and verify indexed results via
/select/logsql/query.
Use jsonf list templating and a per-run marker to isolate records during
query validation. Keep transport on plain HTTP for CI simplicity.
Register the test in tests/Makefile.am under TESTS_OMHTTP so it is part
of testbench distribution and invocable as a single .log target.
Add a new run_checks.yml job named victorialogs_CI that starts a
VictoriaLogs service container, runs only
omhttp-victorialogs-jsonline.log, and gates execution with changed-files
filters for the test, omhttp components, and the workflow itself.
With the help of AI-Agents: Codex (GPT-5)
Why:
Improve reliability of the imjournal basic test under CI timing
variance and journal churn, reducing non-actionable failures.
Impact: imjournal basic tests become more resilient and may skip
when host journal load is excessive.
Before/After: test could fail due to startup/noise races; now it
waits for readiness, retries under noise, and reports cleaner errors.
Technical Overview:
- Added a readiness probe phase in imjournal-basic before the real
assertion message to avoid IgnorePreviousMessages startup races.
- Added short retry logic for high recent journal volume and skip
only after retries are exhausted.
- Added tunables for noise window, threshold, retries, and sleep.
- Updated journal_print to write newline-terminated records and
explicitly fflush/fclose so writes are durably handed to journald.
- Updated content_check_with_count to treat missing output file as
not-yet-done until the test hard deadline, then fail.
- Improved failure diagnostics for missing output file to avoid
secondary shell errors that obscure the primary failure.
With the help of AI-Agents: Codex
Adds first-class integration with VictoriaMetrics to simplify ops
dashboards and move toward project-supported telemetry without
sidecar collectors.
Impact: New optional feature (off by default). No behavior change
unless configured via push.* parameters.
Before: impstats could only log locally or emit text formats.
After: impstats can push counters to Prometheus-compatible endpoints.
Technical: implement a native Prometheus Remote Write path in
impstats, encoding counters to protobuf and compressing with snappy
over HTTP via libcurl. Replace interim text parsing with a new
statsobj v14 API (GetAllCounters) that iterates raw uint64 counters,
keeps atomic reads for IntCtr and best-effort reads for Int. Add
metric builder with Prometheus-compliant sanitization and the naming
pattern <origin>_<name>_<counter>_total. Provide TLS knobs (CA, mTLS,
insecureSkipVerify), static/dynamic labels, timeout, and optional
batching by bytes/series. Build is gated behind
--enable-impstats-push with protobuf-c/snappy/curl checks. Ship docs,
basic/VM integration tests, and a GitHub Actions workflow using a
VictoriaMetrics service; TSAN jobs disable impstats-push.
Configuration: push.url, push.labels, push.timeout.ms,
push.label.{instance,job,origin,name}, push.tls.{cafile,certfile,
keyfile,insecureSkipVerify}, push.batch.{maxBytes,maxSeries}.
With the Help of AI Agents: ChatGPT codex 5.2
When `re_extract` executes a regex that matches an empty string (length 0), `regexec` returns `rm_eo = 0`. The loop logic `iOffs += pmatch[0].rm_eo` results in `iOffs` not advancing, causing the next iteration to find the same empty match at the same position. This leads to finding the same match repeatedly (up to `matchnbr`), which is incorrect behavior (subsequent matches should be distinct).
This commit fixes the issue by detecting zero-length matches and forcing an advance of `iOffs` by 1, while ensuring we do not overrun the string buffer.
Fixes issue #230 (potential infinite loop scenario).
Signed-off-by: Rainer Gerhards <rgerhards@adiscon.com>
Co-authored-by: Jules Agent
Co-authored-by: rgerhards <1482123+rgerhards@users.noreply.github.com>
