This modernizes contributor experience by simplifying PR templates and
removing the lengthy GDPR disclaimer that often discouraged or confused
new contributors. The update encourages more participation and shows how
responsible AI can be used to improve open source workflows.
Impact: none on runtime behavior; contributor workflow improved.
Before: PR template included long GDPR block; commit message rules were
scattered and partly implicit.
After: PR template is concise, GDPR text removed, and commit assistant
usage is documented across README, CONTRIBUTING, and AGENTS.md.
Technical changes include:
- PR template: drop GDPR notice, add commit-assistant references.
- CONTRIBUTING.md: add explicit commit rules and workflow guidance.
- AGENTS.md: require canonical base prompt and commit-first workflow.
- README.md: point to assistant and updated guidance.
- base_prompt.txt: enforce "Findings:" colon format.
- Minor formatting corrections in comments.
Why: Modernize omhttp so HTTP forwarding is viable and maintainable in
current deployments (incl. Docker). Align the module with core-supported
interfaces, reduce maintenance risk, and pave the way to promote it from
contrib to an officially supported module.
Technical overview (conceptual):
- Switch plugin API from legacy TXIF (doAction) to OMODTX
(commitTransaction). Update query entry points accordingly.
- Keep beginTransaction for batch init; fold the final flush from
endTransaction into commitTransaction per OMODTX.
- Preserve external behavior: batching thresholds (maxBatchSize and
maxBatchBytes), dynRestPath handling, statistics, and HUP behavior.
- In commitTransaction, process all records in one pass and submit
batches when thresholds are crossed; non-batch mode posts per record.
- Retry test adjusted to tolerate duplicates during migration
(omhttp-retry.sh), reflecting current OMODTX replay characteristics.
Addresses from #5957:
- Core modernization of omhttp to the project’s current module
interface, suitable for containerized use cases.
- First step toward “project-supported” status by aligning with the
supported core API and updating tests.
Planned follow-ups (separate commits):
- Fix dynRestPath after a batch flush (set new path right after
initializeBatch).
- Revisit commit/replay semantics to minimize duplicates under retry, or
document dup-tolerant behavior and test both modes.
- Verify and free batch.restPath and headerBuf; add focused tests (path
change mid-batch, maxBatchBytes edges, TLS/gzip, HUP reinit).
- Documentation and packaging steps required for “project-supported”
status (module docs, release notes, CI/packaging policy).
- Evaluate if a compatibility query hook is needed for older cores.
see also: https://github.com/rsyslog/rsyslog/issues/5957
tcpflood opens TLS connections in parallel. OpenSSL updates internal X509 structures during the handshake, which leads ThreadSanitizer to report races when multiple handshakes run concurrently. Protect the handshake with a mutex and guard the error counter with its own lock.
AI-Agent: ChatGPT
This implements brace-style env vars inside backticks with `echo` and
fixes termination of unbraced `$VAR`. Most importantly, variables can
now be adjacent to static text (e.g., `foo${ENV}bar`), which previously
did not work and caused confusion. This aligns behavior with common
bash expectations.
Why
- Users expect `${VAR}` and `$VAR!` to expand while keeping punctuation.
- Concatenations like `foo${ENV}bar` are common and should be valid.
What
- Add support for braced variables `${VAR}` with proper `}` handling.
- For unbraced `$VAR`, stop the name at the first non `[A-Za-z0-9_]`
char and emit that char (e.g., `!`, `.`) as literal output.
- Improve error handling for overlong env var names.
- Keep other shell features (e.g., `$(pwd)`) unsupported by design.
Docs
- Update `constant_strings.rst` to document `${VAR}`, the new
termination rules, and examples including `foo${ENV}bar` and `$VAR!`.
Tests
- Add `rscript_backticks_braces_envvar.sh` for `${VAR}` support.
- Add `rscript_backticks_static_text.sh` for `$VAR!` and adjacency.
- Convert `rscript_backticks_empty_envvar-vg.sh` to non-VG variant and
adjust `tests/Makefile.am`.
Compatibility
- Backward compatible. Changes affect only previously broken edge cases.
Misc
- Refresh lexer copyright years.
- action.c: Add iSuspended flag to prevent infinite loops when transactions
are suspended multiple times. Retry on first suspension
and abort with RS_RET_SUSPENDED on subsequent suspensions.
- tests/omprog-transactions-failed-messages.sh: Remove TODO comment and
workaround code related to issue #2420 (deferred messages within
transactions not being retried), as the underlying issue appears to
be resolved.
- tests/omprog-feedback-timeout.sh: Update expected output to reflect
improved transaction handling behavior. The test now expects additional
message processing cycles and proper timeout handling when the omprog
action is suspended and restarted.
- tests/omprog-feedback.sh: Make robust against timing variations from
new action.c retry logic by replacing exact sequence matching with
pattern-based validation to ensure cross-system compatibility.
- tests/omhttp-retry-timeout.sh: Optimize test parameters for better
reliability by reducing message count from 10000 to 5000, adding
sequence check options, and reducing queue batch size from 2048 to 500
to prevent test timeouts and improve stability.
- omhttp-batch-fail-with-400.sh test: resolve queue growth issue with
HTTP 400 errors. The test was experiencing a queue growth issue where
the queue size was increasing. This was caused by the omhttp module
incorrectly treating HTTP 400 errors as retriable when they should be
treated as permanent failures.
FIX: Added httpretrycodes=["500", "502", "503", "504"] configuration.
This explicitly specifies that only 5xx server errors should be retried.
HTTP 400 errors are now properly treated as permanent failures.
Some tests needed to be adapted, because they expected an "exactly once"
paradigm, which the fixed bug seemed to provide in some cases (but not
reliably). Actually, rsyslog guarantees "at least once", so duplicates
can occur and are typical if transaction-like logic is used with
non-transactional outputs.
This addresses the transaction suspension edge case and cleans up
temporary workaround code that is no longer needed. The test updates
ensure that the improved transaction handling behavior is properly
validated across different scenarios and that tests correctly reflect
rsyslog's actual delivery semantics.
closes https://github.com/rsyslog/rsyslog/issues/2420
If the srcdir variable was not defined, all tests initialize it
to ".". While this is fine, if srcdrv was not previously exported
the newly set variable was local to the execution shell and did
not promote to rsyslog environment.
However, srcdir is granted to be in environment, many processes
in testbench depend on this. We now ensure this by doing an
explicit export of srcdir during script init phase.
It looks like at some point a change purely intend for local testing
went into git, rendering the test incorrect. Still needs to be finally
confirmed.
potentially fixes https://github.com/rsyslog/rsyslog/issues/5880
This patch adds a robust, optional mechanism for handling "headerless" input—
log messages that do not conform to traditional syslog formatting.
- **Headerless detection (opt-in)**
- Controlled by the new `detect.headerless` boolean (default: off)
- Detects messages with **no PRI** and **no valid timestamp**
- Excludes structured inputs (e.g. JSON starting with `{` or `[`) as
before
- Injects default `hostname` and `tag` values
- Flags message internally as `HEADERLESS_MSG` for further processing
- **Fallback processing options**
- `headerless.ruleset`: route headerless messages to a dedicated ruleset
- `headerless.errorfile`: optionally store raw input to a file
- `headerless.drop`: discard headerless messages early if desired
- **Thread-safe HUP signal handling**
- New `doHUPParser` entry point allows safe log rotation for error file
- Follows standard reopen-on-write pattern post-HUP
- **Testing & Maintenance**
- Adds two test cases: `pmrfc3164-headerless.sh` and `pmrfc3164-drop.sh`
- Extends documentation for all new parameters
- Cleans up code formatting, includes, and bumps copyright
Some environments produce mixed or malformed input streams. This patch enables
early, lightweight detection of non-syslog input, with customizable recovery
and routing strategies. It avoids unnecessary parsing work and gives operators
better tools to isolate or discard garbage input—without breaking legacy behavior.
This patch improves the GitHub Actions PR validation workflow:
- Added comprehensive inline documentation for clarity.
- Corrected branch reference from 'master' to 'main'
in `tests/CI/PR_validation.sh`.
These changes enhance CI/CD clarity and maintainability.
With support from AI Agents: Gemini
This commit applies the new canonical formatting style using `clang-format` with custom settings (notably 4-space indentation), as part of our shift toward automated formatting normalization.
⚠️ No functional changes are included — only whitespace and layout modifications as produced by `clang-format`.
This change is part of the formatting modernization strategy discussed in:
https://github.com/rsyslog/rsyslog/issues/5747
Key context:
- Formatting is now treated as a disposable view, normalized via tooling.
- The `.clang-format` file defines the canonical style.
- A fixup script (`devtools/format-code.sh`) handles remaining edge cases.
- Formatting commits are added to `.git-blame-ignore-revs` to reduce noise.
- Developers remain free to format code however they prefer locally.
This commit adds the initial implementation of the `mmaitag` module,
a pluggable AI-based message classification processor.
The module supports two providers:
- `gemini`: uses the Gemini API via libcurl and JSON-C
- `gemini_mock`: a deterministic in-memory mock used for testing
Each log message is classified individually and tagged with a label
stored in a configurable message variable (default: `$.aitag`).
Included:
- Provider abstraction layer (`ai_provider.h`)
- Gemini provider with API key support (inline or via file)
- Prompt customization and input property selection
- Two regression tests: basic and invalid-key scenarios
- Sphinx documentation under `doc/configuration/modules`
Note: mmaitag is in its infancy and primarily a PoC. Future work
will improve batching, performance, and add more providers.
Refs:
- https://github.com/rsyslog/rsyslog/issues/5747 (AI-first strategy)
With the help of AI Agents: Codex, Gemini
This also includes a basic Prometheus scrape entry point, which
currently can only be used for health checking.
We plan futher enhancement in this direction.
Also some "fixes" in regard to rsyslog code style and portability
methods.
This commit extends the impstats input module to support a new
"prometheus" format for exporting statistics. The output is
generated via statsobj.c, which already provides Prometheus-
compatible formatting.
To enable this, a new "format" config option value ("prometheus")
was added and recognized by impstats.c. When selected, impstats
invokes the appropriate statsobj function to generate output in
Prometheus exposition format.
The default behavior remains unchanged. Existing formats such as
"json" and "legacy" are not modified.
A new test has been added under tests/ to validate correct
Prometheus output generation. The test verifies that the format
header and at least one representative metric line conform to the
Prometheus text format specification.
This enhancement enables rsyslog to integrate more easily with
Prometheus-based monitoring stacks and paves the way for native
metrics scraping without external translation layers.
When enabled, state files are deleted when the watched file is moved or
rotated away. This prevents accumulation of stale state files from
rotated logs. By default the state file is kept. Does not change
default behavior for existing inputs. The config option is available as
per-input and per-module parameter.
This commit performs a broad modernization of widely used rsyslog
macros to align with modern C practices and support automated
formatting tools like clang-format. The changes focus on improving
syntactic regularity, readability, and tooling compatibility — without
altering behavior.
Macros refactored in this commit now follow a consistent,
statement-like form with explicit trailing semicolons. Where
applicable, macro blocks that define module interfaces (`queryEtryPt`)
have been updated to use simple `if` statements instead of `else if`
chains. While this slightly increases evaluation time, the affected
functions are only called once per module during load time to register
supported interfaces — making the performance cost irrelevant in
practice.
These improvements serve multiple purposes:
- Enable reliable clang-format usage without mangling macro logic
- Simplify reasoning about macro-expanded code for human readers
- Reduce style drift and merge conflicts
- Facilitate development for contributors using assistive tools
- Support future formatting pipelines using:
1. `clang-format`
2. a post-fixup normalization script
Refactored macros:
- MODULE_TYPE_NOKEEP
- MODULE_TYPE_KEEP
- MODULE_TYPE_INPUT
- MODULE_TYPE_OUTPUT
- MODULE_TYPE_FUNCTION
- MODULE_TYPE_PARSER
- MODULE_TYPE_LIB
- DEF_IMOD_STATIC_DATA
- DEF_OMOD_STATIC_DATA
- DEF_PMOD_STATIC_DATA
- DEF_FMOD_STATIC_DATA
- DEFobjStaticHelpers
- SIMP_PROP(...)
And all `queryEtryPt()` dispatch macros:
- CODEqueryEtryPt_STD_MOD_QUERIES
- CODEqueryEtryPt_STD_OMOD_QUERIES
- CODEqueryEtryPt_STD_OMODTX_QUERIES
- CODEqueryEtryPt_STD_OMOD8_QUERIES
- CODEqueryEtryPt_TXIF_OMOD_QUERIES
- CODEqueryEtryPt_IsCompatibleWithFeature_IF_OMOD_QUERIES
- CODEqueryEtryPt_STD_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_QUERIES
- CODEqueryEtryPt_STD_CONF2_setModCnf_QUERIES
- CODEqueryEtryPt_STD_CONF2_OMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_IMOD_QUERIES
- CODEqueryEtryPt_STD_CONF2_PREPRIVDROP_QUERIES
- CODEqueryEtryPt_STD_CONF2_CNFNAME_QUERIES
- CODEqueryEtryPt_STD_PMOD_QUERIES
- CODEqueryEtryPt_STD_PMOD2_QUERIES
- CODEqueryEtryPt_STD_FMOD_QUERIES
- CODEqueryEtryPt_STD_SMOD_QUERIES
- CODEqueryEtryPt_doHUPWrkr
- CODEqueryEtryPt_doHUP
This general modernization reduces macro misuse, improves DX, and
lays the foundation for a robust, automated style normalization
system.
See also: https://github.com/rsyslog/rsyslog/issues/5747
This may be an interim solution, but definitely a useful one
to address style inconsistencies.
Style inconsistencies are problematic as they tend to pollute
git history by causing pseudo-changes.
This commit also contains a set of fixes to existing files, so
that we get to a clean state.
This Implements suffix comparison similar to startswith. Note that we
do intentionally not use libestr functions in order to speed up adaption.
It would otherwise probably take years for distros to upgrade libestr.
With some help of the Codex and Gemini AI Agents.
closes https://github.com/rsyslog/rsyslog/issues/1255
* Add note for indentation cleanup
* Converted tabs to spaces in various Python scripts to match PEP 8 indentation.
With the help of AI-Agent: Codex, Google Gemini Code review
* testbench: more robust method to build testing tools
The previous method failed depending on configure settings and
available support libraries. This is now handled by make check.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Testbench tools like tcpflood are now auto-generated at test startup
if missing, reducing errors and confusion during local test runs.
Updated AGENTS.md with precise instructions for running tests
in both human and AI agent contexts, improving onboarding
for new developers and AI-based workflows.
Network restrictions can prevent fetching Kafka or Zookeeper packages. When both download attempts fail, skip the test by exiting with code 77 instead of reporting an error.
AI-Agent: Codex 2025-06
Commit ffaf6dc4620da added proper variable sync, but dropped the check
if active count had actually changed. As such, the output was always
generated, which could pollute the log heavily.
also fixes some codestyle issues introduced by earlier commits.
With the help of AI-Agent: Codex 2025-06
The omsendertrack module is designed to track and report statistics for
message senders across all inputs in Rsyslog. It periodically outputs a
JSON file containing information about each sender.
Note: This commit provides minimalistic basic functionality as a PoC.
We will check it's usefulness in practice and expect follow-up PRs
to enhance functionality and include feedback from early testing.
However, this module is solid, just feature limited.
see also: https://github.com/rsyslog/rsyslog/issues/5599
As a result, it did not show up in pstats. Alos, we now use the input name
in worker thread to easily identify where they belong to. As thread names
have very limited length, the thread naming now is
"w<worker-number>/<input-name>".
This patch introduces a counter to track the number of discarded messages.
Additionally, it fixes a bug where the submitted message count was incremented
even when the message was discarded due to exceeding the allowed message rate
within a given interval.
when neither GTLS or OSSL TLS drivers are enabled, this test cannot pass.
We now check for this condition and execute it only it at least one of
the two is configured.
closes https://github.com/rsyslog/rsyslog/issues/5592
This commit significantly enhances imtcp by introducing a fully
functional worker thread pool, enabling true multi-threaded
processing for better scalability under high loads. This is
particularly beneficial when using TLS connections.
Notable changes:
- Implemented a complete worker pool for imtcp.
- Introduced the `workerthreads` config parameter for tuning
concurrency.
- Improved epoll efficiency by enabling edge-triggered mode.
- Added starvation handling via `starvationProtection.maxReads`.
- Refactored session accept logic and optimized network object
handling.
- Removed an obsolete network driver layer for event notification.
- Fixed multiple issues related to message timing, EPOLLERR
handling, and tests.
- Improved performance in poll() mode by reducing redundant
allocations.
- Introduced new CI tests for imtcp without epoll.
- Allowed disabling imtcp tests via a new configure switch.
- Added new impstats counters for worker thread pool statistics.
Details:
- The worker pool replaces an outdated experimental
implementation.
- If `workerthreads=1`, no worker pool is created to minimize
context switches.
- Moves worker pool variables inside `tcpsrv` instance to
prevent conflicts.
- Extracts session `accept()` logic into a dedicated function
for clarity.
- Fixes message ordering inconsistencies in multi-threaded
scenarios.
- Properly handles `EPOLLERR` notifications to improve error
resilience.
- Optimizes poll() mode by avoiding unnecessary reallocation
of file descriptors.
- Replaces the old network driver layer for event notification
with a streamlined solution.
- Now uses **conditional compilation** to select the best
method (epoll or poll) at build time.
- This significantly reduces code complexity, improves
maintainability, and boosts performance.
- The previous "thread pool" was a rough experiment that did
not perform significantly better than single-threaded mode.
- The **new implementation** allows multiple worker threads
on platforms with `epoll`.
- On non-epoll systems, an optimized **poll() based
single-threaded approach** is used, which is expected to
perform better than the old "thread pool."
- Adds `pthread_setname_np` only when available to improve
portability.
- Fixes test cases that assumed strict message timing, which
was unreliable.
- Reduces test parallelism for TSAN CI runs to prevent
resource exhaustion.
- Moves a test case to `imdiag` to ensure stable execution.
- Provides a new CI environment to verify `imtcp` behavior
without epoll.
- Introduces `--enable-imtcp-tests` configure switch for test
flexibility.
- Improves debug logging and adds better error handling for
worker pool startup.
New configuration parameters:
- `workerthreads`: Defines the number of worker threads for
imtcp. If set to 1, no worker pool is created.
- `starvationProtection.maxReads`: Defines the maximum number
of consecutive reads a worker can perform before being
interrupted to allow other sessions to be processed.
New impstats counters (emitted only when `workerthreads > 1`):
- `runs`: Number of times the worker thread has been invoked.
- `read`: Number of read calls performed by the worker.
For TLS, this includes read/write calls.
- `accept`: Number of `accept()` calls handled by the worker.
- `starvation_protect`: Number of times a socket was sent
back to the queue due to reaching the maximum number of
consecutive requests, ensuring fair scheduling of sessions.
These changes significantly enhance rsyslog’s TCP handling
performance and stability, particularly in high-volume
environments.
Closes#5529, #5532, #5578, #5580.
This changes two scripts supporting solaris CI builds to use two
instead of one threads when building needed dependecies. This
should result in a light speedup without overloading the
Solaris builders.
Note: OpenCSW provides the builders, and the rsyslog team
is thankful for that.
If openssl is compiled without support for openssl engines,
do not include engine selection. The global option
"defaultopensslengine" will not work in this case. It's support
is depreciated in newer OpenSSL versions
OpenSSL Engine Support feature relies on an outdated version of OpenSSL and is strictly experimental. No support or guarantees are provided. Use at your own risk.
closes: https://github.com/rsyslog/rsyslog/issues/5429
While this could lead to some inefficiency, it should not have caused
any real harm. But with data races it is never sure if more severe
issues occur. However, here only very strange use cases can be
envisioned where this might be the case.
In any case, the issue is now solved.
This also fixes some TSAN CI "flakes".
When the snmpreceiver is fully started and listening, the file is created.
diag.sh checks for the .started file before proceeding with the textbench
see also: https://github.com/rsyslog/rsyslog/pull/5558
Note: we temporarily disable omamqp1 testing while we search for
the missing dependency.
We also add a special setting that permits us to control if
known-(very)-flaky test shall be run.
closes https://github.com/rsyslog/rsyslog/issues/5546
The system packages on Ubuntu 24 appear to be broken for python3-pysnmp4.
And when we update the package using
pip install pyasn1 pysnmp --break-system-packages --upgrade
We need adapt the code, so a new version snmptrapreceiverv2.py
has been added which will automatically been chosen if Python is >3.10
closes: https://github.com/rsyslog/rsyslog/issues/5554
