Motivation: code coverage reports were incomplete. This lays a better
base for consistent reporting via GitHub Actions, with room for follow-ups.
It also removes a test flake source in Kafka jobs.
Impact: CI/tests only; no runtime behavior or ABI changes expected.
Before: Coverage uploads were inconsistent; Kafka tests could hang while
reading from /dev/urandom to generate topic names.
After: Coverage is collected with lcov and uploaded via a dedicated GH
Action; Kafka topics use fast $RANDOM-based hex, avoiding early-boot
entropy stalls.
Technical details:
- Add two workflows: "codecov base" and "codecov kafka" on Ubuntu 24.04.
Use lcov capture with unexecuted blocks and prune common noise; upload
with token for same-repo PRs and tokenless for forks.
- Update .codecov.yml: add path fixes for container (/rsyslog) and
runner layouts; explicitly set comment: false and patch: false.
- Bump actions/checkout to v4 in existing workflows; add an actionlint
job to catch YAML problems early.
- Switch codecov jobs in container matrix to 24.04 images.
- Improve run-ci.sh lcov invocation to be more tolerant of line/macro
mismatches.
- Testbench: replace /dev/urandom topic generation with 8-char hex from
$RANDOM; adjust diag.sh path/quoting for zookeeper helper.
Add a RainerScript function to convert strings to uppercase. The
implementation mirrors tolower() but operates per-byte using
toupper(). Tests demonstrate the new function, documentation and
ChangeLog entries added.
closes: https://github.com/rsyslog/rsyslog/issues/3666
With the help of AI-Agent: ChatGPT
kafkacat has been renamed to kcat, as it looks for trademark issues.
Kafka module tests depend on that utility and are skipped if command
is not available. This was now always the case for newer development
containers.
Name was now adjusted in all tests.
Adds -w option to tcpflood to write its local port to a file.
The option accepts a filename and writes the port number followed
by a trailing newline. tcpflood now rejects -w if more than one
connection would be opened.
With help of AI-Agent: ChatGPT
random() emitted a warning only to debug output when its maximum
exceeded RAND_MAX. Log the message to normal output as well so that
operators see the misconfiguration without enabling debugging.
Add a regression test verifying that both debug and standard logs
contain the warning.
AI-Agent: ChatGPT
Work towards full completion of the module’s interface as it exits PoC.
The new name senderid (formerly: template) matches intent (identify a
sender, not an output format) and avoids confusion. Also make state
persistence safer via atomic writes.
Merged a bit pre-completion in order to get a static analyzer fix
into the main code base. Some additional cleanup PR will follow.
State is much cleaner now than in PoC.
Impact: Configs must use senderid=; statefile is now required. Tests
and docs updated.
Before/After: action(... template="name") -> action(... senderid="name").
Technical details:
- Replace the config parameter and instance field; drop legacy template
parsing in parseSelectorAct. The module now reads only senderid=.
- Enforce required statefile (descriptor + runtime) and cache a temp
path "<statefile>.tmp" to ensure rename() is atomic. Provide a
one-time fallback build if the cache is missing (e.g., reload flow).
- Add built-in template "StdOmSenderTrack-senderid" yielding
"%fromhost-ip%" and use it as the default senderid template.
- Improve memory ownership (free cached temp path/default template) and
keep existing locking/queue semantics unchanged.
- Update tests to use senderid= and prune PoC doc note accordingly.
see also: https://github.com/rsyslog/rsyslog/issues/5599
This change introduces profile-based configuration for omhttp. Profiles
simplify setup for common backends such as Grafana Loki and Splunk HEC by
applying recommended defaults for batching, paths, formats, and retry codes.
This improves usability and avoids error-prone manual tuning.
Impact: HTTP error handling semantics are changed; retry behavior for
3xx and 4xx differs from previous releases.
Technically, HTTP status handling is refactored: transport errors and 5xx
remain retriable, while 3xx (redirection) and 4xx (client errors) are now
treated as permanent failures (`RS_RET_DATAFAIL`). Retry ruleset logic in
batch mode is clarified: if a retry ruleset is configured, messages are
queued there; otherwise, core retry is used. In single-message mode, core
retry is always used.
The new `profile` parameter applies bundled settings. The `loki` profile
enables batching, lokirest formatting, compression, and a default path of
`loki/api/v1/push`. The `hec:splunk` profile sets HEC-specific defaults
including newline batching and size limits. A new test validates the Loki
profile end-to-end. Documentation typo in imptcp parameter reference is
also fixed.
Refs: https://github.com/rsyslog/rsyslog/issues/5957
Refs: https://github.com/rsyslog/rsyslog/pull/5972
Multiple intermediate CAs were not honored because strtok() modified
the original config value at the first comma. This led to only the
first file being considered. This change preserves the original value
and validates files using a duplicate buffer.
Impact: Users can reliably specify multiple CA intermediates via
NetstreamDriverCAExtraFiles; TLS chains that require intermediates
now validate as expected. Tests added.
Before: parsing mutated the stored string, effectively truncating at
the first comma and ignoring subsequent CA files.
After: we strdup() for validation, leave the original string intact,
and set it only if all files are accessible. Error handling follows
existing RS_RET patterns; temporary buffer is freed on all paths.
Additionally, add a test (OpenSSL backend) that generates a root,
two intermediates, and leaf certs on both client and server. The test
verifies chains with openssl, aligns CN/PermittedPeer, and exercises
a full send/receive path to guard against regressions. Makefile is
updated to include the new test in TESTS and EXTRA_DIST.
Fixes: https://github.com/rsyslog/rsyslog/issues/5207
this test loses messages because of tcp unreliability, so this is
expected. We are still investigating ways to improve test performance
even in spite of this problem. As an interim solution against flakes,
we at least temporarily increase permitted loss again.
Subtree-type templates failed to set the generated string length.
Modules relying on lenStr, such as omfwd, forwarded empty frames.
Set lenStr to the subtree length and add a regression test.
AI-Agent: ChatGPT
Modernize our dev/CI matrix to match current distro lines and reduce
toolchain drift. This keeps project-provided images aligned with what
contributors run locally and moves us toward “project-supported” bases.
Impact: CI matrix refresh only; no runtime behavior change intended.
Before: CI used Fedora 35/36 and Debian 10/11 dev-base images.
After: CI uses Fedora 41/42 and Debian 11/13 dev-base images.
Update run_checks.yml to use new container tags and pass consistent
configure extras (incl. enabling dtls in relevant jobs). Add Dockerfiles
and helper scripts for debian:13 and fedora:{41,42}. Bump the cached
ZooKeeper tarball to 3.9.3 across legacy images. Debian 13 adds a small
libgcrypt-config wrapper that defers to pkg-config to keep builds working.
Fedora images refresh toolchains (clang18 et al.) and re-sync module flags.
No API/ABI changes; this is build/CI plumbing only.
closes: https://github.com/rsyslog/rsyslog/issues/5989
This modernizes contributor experience by simplifying PR templates and
removing the lengthy GDPR disclaimer that often discouraged or confused
new contributors. The update encourages more participation and shows how
responsible AI can be used to improve open source workflows.
Impact: none on runtime behavior; contributor workflow improved.
Before: PR template included long GDPR block; commit message rules were
scattered and partly implicit.
After: PR template is concise, GDPR text removed, and commit assistant
usage is documented across README, CONTRIBUTING, and AGENTS.md.
Technical changes include:
- PR template: drop GDPR notice, add commit-assistant references.
- CONTRIBUTING.md: add explicit commit rules and workflow guidance.
- AGENTS.md: require canonical base prompt and commit-first workflow.
- README.md: point to assistant and updated guidance.
- base_prompt.txt: enforce "Findings:" colon format.
- Minor formatting corrections in comments.
Why: Modernize omhttp so HTTP forwarding is viable and maintainable in
current deployments (incl. Docker). Align the module with core-supported
interfaces, reduce maintenance risk, and pave the way to promote it from
contrib to an officially supported module.
Technical overview (conceptual):
- Switch plugin API from legacy TXIF (doAction) to OMODTX
(commitTransaction). Update query entry points accordingly.
- Keep beginTransaction for batch init; fold the final flush from
endTransaction into commitTransaction per OMODTX.
- Preserve external behavior: batching thresholds (maxBatchSize and
maxBatchBytes), dynRestPath handling, statistics, and HUP behavior.
- In commitTransaction, process all records in one pass and submit
batches when thresholds are crossed; non-batch mode posts per record.
- Retry test adjusted to tolerate duplicates during migration
(omhttp-retry.sh), reflecting current OMODTX replay characteristics.
Addresses from #5957:
- Core modernization of omhttp to the project’s current module
interface, suitable for containerized use cases.
- First step toward “project-supported” status by aligning with the
supported core API and updating tests.
Planned follow-ups (separate commits):
- Fix dynRestPath after a batch flush (set new path right after
initializeBatch).
- Revisit commit/replay semantics to minimize duplicates under retry, or
document dup-tolerant behavior and test both modes.
- Verify and free batch.restPath and headerBuf; add focused tests (path
change mid-batch, maxBatchBytes edges, TLS/gzip, HUP reinit).
- Documentation and packaging steps required for “project-supported”
status (module docs, release notes, CI/packaging policy).
- Evaluate if a compatibility query hook is needed for older cores.
see also: https://github.com/rsyslog/rsyslog/issues/5957
tcpflood opens TLS connections in parallel. OpenSSL updates internal X509 structures during the handshake, which leads ThreadSanitizer to report races when multiple handshakes run concurrently. Protect the handshake with a mutex and guard the error counter with its own lock.
AI-Agent: ChatGPT
This implements brace-style env vars inside backticks with `echo` and
fixes termination of unbraced `$VAR`. Most importantly, variables can
now be adjacent to static text (e.g., `foo${ENV}bar`), which previously
did not work and caused confusion. This aligns behavior with common
bash expectations.
Why
- Users expect `${VAR}` and `$VAR!` to expand while keeping punctuation.
- Concatenations like `foo${ENV}bar` are common and should be valid.
What
- Add support for braced variables `${VAR}` with proper `}` handling.
- For unbraced `$VAR`, stop the name at the first non `[A-Za-z0-9_]`
char and emit that char (e.g., `!`, `.`) as literal output.
- Improve error handling for overlong env var names.
- Keep other shell features (e.g., `$(pwd)`) unsupported by design.
Docs
- Update `constant_strings.rst` to document `${VAR}`, the new
termination rules, and examples including `foo${ENV}bar` and `$VAR!`.
Tests
- Add `rscript_backticks_braces_envvar.sh` for `${VAR}` support.
- Add `rscript_backticks_static_text.sh` for `$VAR!` and adjacency.
- Convert `rscript_backticks_empty_envvar-vg.sh` to non-VG variant and
adjust `tests/Makefile.am`.
Compatibility
- Backward compatible. Changes affect only previously broken edge cases.
Misc
- Refresh lexer copyright years.
- action.c: Add iSuspended flag to prevent infinite loops when transactions
are suspended multiple times. Retry on first suspension
and abort with RS_RET_SUSPENDED on subsequent suspensions.
- tests/omprog-transactions-failed-messages.sh: Remove TODO comment and
workaround code related to issue #2420 (deferred messages within
transactions not being retried), as the underlying issue appears to
be resolved.
- tests/omprog-feedback-timeout.sh: Update expected output to reflect
improved transaction handling behavior. The test now expects additional
message processing cycles and proper timeout handling when the omprog
action is suspended and restarted.
- tests/omprog-feedback.sh: Make robust against timing variations from
new action.c retry logic by replacing exact sequence matching with
pattern-based validation to ensure cross-system compatibility.
- tests/omhttp-retry-timeout.sh: Optimize test parameters for better
reliability by reducing message count from 10000 to 5000, adding
sequence check options, and reducing queue batch size from 2048 to 500
to prevent test timeouts and improve stability.
- omhttp-batch-fail-with-400.sh test: resolve queue growth issue with
HTTP 400 errors. The test was experiencing a queue growth issue where
the queue size was increasing. This was caused by the omhttp module
incorrectly treating HTTP 400 errors as retriable when they should be
treated as permanent failures.
FIX: Added httpretrycodes=["500", "502", "503", "504"] configuration.
This explicitly specifies that only 5xx server errors should be retried.
HTTP 400 errors are now properly treated as permanent failures.
Some tests needed to be adapted, because they expected an "exactly once"
paradigm, which the fixed bug seemed to provide in some cases (but not
reliably). Actually, rsyslog guarantees "at least once", so duplicates
can occur and are typical if transaction-like logic is used with
non-transactional outputs.
This addresses the transaction suspension edge case and cleans up
temporary workaround code that is no longer needed. The test updates
ensure that the improved transaction handling behavior is properly
validated across different scenarios and that tests correctly reflect
rsyslog's actual delivery semantics.
closes https://github.com/rsyslog/rsyslog/issues/2420
If the srcdir variable was not defined, all tests initialize it
to ".". While this is fine, if srcdrv was not previously exported
the newly set variable was local to the execution shell and did
not promote to rsyslog environment.
However, srcdir is granted to be in environment, many processes
in testbench depend on this. We now ensure this by doing an
explicit export of srcdir during script init phase.
It looks like at some point a change purely intend for local testing
went into git, rendering the test incorrect. Still needs to be finally
confirmed.
potentially fixes https://github.com/rsyslog/rsyslog/issues/5880
This patch adds a robust, optional mechanism for handling "headerless" input—
log messages that do not conform to traditional syslog formatting.
- **Headerless detection (opt-in)**
- Controlled by the new `detect.headerless` boolean (default: off)
- Detects messages with **no PRI** and **no valid timestamp**
- Excludes structured inputs (e.g. JSON starting with `{` or `[`) as
before
- Injects default `hostname` and `tag` values
- Flags message internally as `HEADERLESS_MSG` for further processing
- **Fallback processing options**
- `headerless.ruleset`: route headerless messages to a dedicated ruleset
- `headerless.errorfile`: optionally store raw input to a file
- `headerless.drop`: discard headerless messages early if desired
- **Thread-safe HUP signal handling**
- New `doHUPParser` entry point allows safe log rotation for error file
- Follows standard reopen-on-write pattern post-HUP
- **Testing & Maintenance**
- Adds two test cases: `pmrfc3164-headerless.sh` and `pmrfc3164-drop.sh`
- Extends documentation for all new parameters
- Cleans up code formatting, includes, and bumps copyright
Some environments produce mixed or malformed input streams. This patch enables
early, lightweight detection of non-syslog input, with customizable recovery
and routing strategies. It avoids unnecessary parsing work and gives operators
better tools to isolate or discard garbage input—without breaking legacy behavior.
This patch improves the GitHub Actions PR validation workflow:
- Added comprehensive inline documentation for clarity.
- Corrected branch reference from 'master' to 'main'
in `tests/CI/PR_validation.sh`.
These changes enhance CI/CD clarity and maintainability.
With support from AI Agents: Gemini
This commit applies the new canonical formatting style using `clang-format` with custom settings (notably 4-space indentation), as part of our shift toward automated formatting normalization.
⚠️ No functional changes are included — only whitespace and layout modifications as produced by `clang-format`.
This change is part of the formatting modernization strategy discussed in:
https://github.com/rsyslog/rsyslog/issues/5747
Key context:
- Formatting is now treated as a disposable view, normalized via tooling.
- The `.clang-format` file defines the canonical style.
- A fixup script (`devtools/format-code.sh`) handles remaining edge cases.
- Formatting commits are added to `.git-blame-ignore-revs` to reduce noise.
- Developers remain free to format code however they prefer locally.
This commit adds the initial implementation of the `mmaitag` module,
a pluggable AI-based message classification processor.
The module supports two providers:
- `gemini`: uses the Gemini API via libcurl and JSON-C
- `gemini_mock`: a deterministic in-memory mock used for testing
Each log message is classified individually and tagged with a label
stored in a configurable message variable (default: `$.aitag`).
Included:
- Provider abstraction layer (`ai_provider.h`)
- Gemini provider with API key support (inline or via file)
- Prompt customization and input property selection
- Two regression tests: basic and invalid-key scenarios
- Sphinx documentation under `doc/configuration/modules`
Note: mmaitag is in its infancy and primarily a PoC. Future work
will improve batching, performance, and add more providers.
Refs:
- https://github.com/rsyslog/rsyslog/issues/5747 (AI-first strategy)
With the help of AI Agents: Codex, Gemini
This also includes a basic Prometheus scrape entry point, which
currently can only be used for health checking.
We plan futher enhancement in this direction.
Also some "fixes" in regard to rsyslog code style and portability
methods.
This commit extends the impstats input module to support a new
"prometheus" format for exporting statistics. The output is
generated via statsobj.c, which already provides Prometheus-
compatible formatting.
To enable this, a new "format" config option value ("prometheus")
was added and recognized by impstats.c. When selected, impstats
invokes the appropriate statsobj function to generate output in
Prometheus exposition format.
The default behavior remains unchanged. Existing formats such as
"json" and "legacy" are not modified.
A new test has been added under tests/ to validate correct
Prometheus output generation. The test verifies that the format
header and at least one representative metric line conform to the
Prometheus text format specification.
This enhancement enables rsyslog to integrate more easily with
Prometheus-based monitoring stacks and paves the way for native
metrics scraping without external translation layers.
When enabled, state files are deleted when the watched file is moved or
rotated away. This prevents accumulation of stale state files from
rotated logs. By default the state file is kept. Does not change
default behavior for existing inputs. The config option is available as
per-input and per-module parameter.
