Reduce Makefile clutter and make the test harness easier to reason
about for humans and machines. This also aims to lower CI flakes by
making the dist artifact complete and predictable.
Impact: build/test harness only; no runtime change. New
TEST_RUN_TYPE=MOCK-OK opt-in fast path in diag.sh.
Before: many scattered conditional TESTS entries; some scripts were
only in EXTRA_DIST when corresponding features were enabled, leading
to missing files in "make dist*" tarballs. Duplicates existed.
After: conditional test lists are grouped into variables (e.g.,
TESTS_ELASTICSEARCH_MINIMAL) and appended to TESTS under the same
conditionals; all lists are always added to EXTRA_DIST. Duplicate
entries removed. diag.sh recognizes TEST_RUN_TYPE=MOCK-OK for mock
distchecks and exits success without executing.
Technically, this extracts per-feature test groups into variables,
reuses them in both TESTS (within feature guards) and EXTRA_DIST (un-
conditionally), and keeps existing .log chaining to serialize suites.
The change also keeps check_PROGRAMS and environment wiring within the
ENABLE_TESTBENCH guard. The new MOCK-OK path in diag.sh is isolated to
special runs and does not affect normal testing.
With the help of AI Agents: Google Antigravity
Operators want dynstats to survive restarts for consistent metrics and
smoother observability in containers and rolling deploys.
Before: dynstats buckets were ephemeral; restarts reset counters.
After: optional on-disk persistence restores counters; worker thread is
started on demand and torn down with the owning rsconf.
Impact: New state files under WorkDirectory (or statefile.directory)
when enabled; slight I/O overhead on configured thresholds. Defaults
preserve previous behavior (persistence off).
This adds two thresholds to trigger persistence:
- persistStateInterval (count-based) and persistStateTimeInterval
(time-based), both default 0 (disabled). A new statefile.directory
can override WorkDirectory for dynstats files.
On bucket creation, existing JSON state ("dynstats-state:<bucket>")
is loaded to rehydrate counters. Updates may enqueue async writes to a
lazily-started file-write worker; teardown performs a final sync flush
without holding the bucket lock to avoid I/O-induced deadlocks.
Worker lifecycle is tied to rsconf: init in dynstats_initCnf(),
start on first persistent bucket, stop in dynstats_destroyAllBuckets().
The latter now takes rsconf_t* and is invoked from rsconf destruct,
avoiding prior hangs when loadConf/runConf differed. Per-bucket stats
track flushed bytes/counts/errors; a "file-write-worker" group reports
queue size/enqueues. Docs updated; tests add dynstats-persist(+vg) to
verify restore-after-restart and clean shutdown.
With the help of AI Agents: GitHub Copilot, cubic-dev-ai, ChatGPT codex
Co-authored-by: Rainer Gerhards <rgerhards@adiscon.com>
This implements certificate revocation checking using OCSP (RFC 6960)
for the OpenSSL network stream driver. The feature is disabled by
default and can be enabled via the new StreamDriver.TlsRevocationCheck
configuration parameter.
This is a rebased and refactored version of the original implementation
by Daniel Gollub from June 2020, updated to work with the current main
branch and enhanced with proper plumbing, security hardening, tests,
and documentation.
OCSP Implementation:
- Implements OCSP (RFC 6960) for certificate revocation checking
- Supports OCSP over HTTP transport protocol (HTTPS not implemented)
- Supports Nonce extension for replay protection
- Uses "strict" revocation policy (any OCSP error fails verification)
- Does not support TLS OCSP stapling
- CRL-only certificates are not supported
Configuration Parameter:
- New parameter: StreamDriver.TlsRevocationCheck (binary, default: off)
- Can be set at module or input level
- Disabled by default for backward compatibility and to avoid
unexpected blocking I/O in existing configurations
- Only applies to OpenSSL driver (not available for GnuTLS/mbedTLS)
Usage:
module(load="imtcp" StreamDriver.Name="ossl"
StreamDriver.Mode="1"
StreamDriver.AuthMode="x509/name"
StreamDriver.TlsRevocationCheck="on")
Full Plumbing Through Network Stack:
- imtcp: Added iStrmTlsRevocationCheck parameter parsing and config
- tcpsrv: Added DrvrTlsRevocationCheck field and SetDrvrTlsRevocationCheck()
- netstrms: Added Set/Get functions for revocation check configuration
- netstrm: Added SetDrvrTlsRevocationCheck() pass-through
- nsd interface: Bumped version 18 -> 19, added SetTlsRevocationCheck()
- nsd_ossl: Implemented SetTlsRevocationCheck(), stores flag in SSL ex_data
- nsd_gtls: Added stub returning RS_RET_VALUE_NOT_SUPPORTED
- nsd_mbedtls: Added stub returning RS_RET_VALUE_NOT_SUPPORTED
- nsd_ptcp: Added stub returning RS_RET_VALUE_NOT_SUPPORTED
Security Hardening:
- Fixed OCSP_basic_verify() to not use OCSP_TRUSTOTHER flag (prevents
forged OCSP responses from rogue responder certificates)
- Added Content-Length validation (1MB limit) to prevent memory
exhaustion attacks from malicious OCSP responders
- Changed SSL ex_data index from 2 to 3 to avoid collision with imdtls
- Added proper struct field initialization and copying in AcceptConnReq
- Added socket read/write timeouts (SO_RCVTIMEO/SO_SNDTIMEO) to prevent
indefinite blocking during OCSP response I/O (BIO_gets, BIO_write,
d2i_OCSP_RESPONSE_bio operations now bound by OCSP_TIMEOUT)
Compatibility:
- Added OpenSSL 1.0.2 compatibility (CentOS 7 support)
- Disabled OCSP for WolfSSL builds (API not available)
- Fixed variable shadowing warnings
Known Limitations (documented in code and user documentation):
- OCSP checks perform blocking network I/O (DNS + socket operations)
during TLS handshake, which can cause latency of up to 5 seconds
per OCSP responder
- Potential DoS vector: malicious certificates with multiple slow/
unresponsive OCSP responder URLs can block worker threads
- No async OCSP support or response caching (future enhancement)
Tests:
- imtcp-tls-ossl-revocationcheck-off.sh: Verifies parameter can be
set to "off" and normal TLS operation works
- imtcp-tls-gtls-revocationcheck-error.sh: Verifies error message
when attempting to enable OCSP with unsupported GnuTLS driver
Documentation:
- Created comprehensive parameter reference page
- Added EXPERIMENTAL FEATURE warning about blocking I/O and DoS risks
- Integrated into imtcp module documentation
- Documented usage examples and important considerations
Changes from original implementation by Daniel Gollub:
- Moved OCSP functions from nsd_ossl.c to net_ossl.c (new location
for SSL helper functions in current codebase)
- Updated to use SSL_CTX directly instead of separate trusted_issuers
- Added full parameter plumbing through all network stack layers
- Added StreamDriver.TlsRevocationCheck configuration parameter
- Added security hardening (OCSP_TRUSTOTHER fix, Content-Length
validation, ex_data index collision fix, socket timeout fix)
- Added OpenSSL 1.0.2 and WolfSSL compatibility
- Added support for all NSD drivers (stub implementations)
- Added comprehensive tests and documentation
- Fixed variable shadowing and compiler warnings
- Adapted to current code structure and formatting standards
Original-Author: Daniel Gollub <dgollub@att.com>
Co-authored-by: Daniel Gollub <dgollub@att.com>
With the help of AI-Agents: GitHub Copilot CLI
Better observability: expose per-file ingestion metrics so operators can
see if a specific file is active and how much data it contributes over
time.
BEFORE: impstats had no per-file imfile metrics.
AFTER: impstats reports per-file bytes.processed and lines.processed.
Impact: New impstats objects per watched file; minor per-line overhead.
This change introduces a stats object per active imfile file. The object
is named with the file path and marked with origin "imfile". Two new
resettable counters are registered: bytes.processed (offset delta per
read) and lines.processed (incremented on each submitted line). Counters
use atomic helpers to remain thread-safe. Objects are constructed when a
file is opened and destructed when it is closed; associated counter
mutexes are released to avoid leaks. The module now acquires/releases
the statsobj interface during init/exit. A new test
(imfile-statistics.sh) validates single- and multi-file cases and checks
that impstats outputs the expected counters. Build glue is updated to
include and run the new test.
Non-technical: users want SNI support so outbound TLS can target
virtual hosts and interoperate with common TLS gateways and CDNs.
Impact: user-visible behavior change and new config knob; ABI of
internal netstream interfaces incremented (modules must rebuild).
Before/After: previously SNI was never set; now SNI is set to the
target hostname (not for literal IPs), or to a configured value.
This change plumbs a new "remote SNI" through the netstream stack and
omfwd. New API hooks SetRemoteSNI are added to nsd, netstrm, and
netstrms, with IF versions bumped. nsd_ossl and nsd_gtls honor an
explicit remoteSNI first; otherwise they auto-set SNI when the target
is a hostname (skip for IPv4/IPv6 literals). nsd_ptcp rejects SNI with
RS_RET_VALUE_NOT_SUPPORTED. omfwd gains
- StreamDriverRemoteSNI / StreamDriver.RemoteSNI (aliases),
and passes it during TCPSendInitTarget. Destructors in gtls/ossl and
netstrms free the new remoteSNI field.
Tests add helper SNI servers (OpenSSL and GnuTLS) and cover three
cases per TLS lib: no SNI for IP targets, auto SNI for hostnames, and
configured SNI override. Build glue and diag helpers are included.
In 2026 rebase and some fixup with the help of AI Agents:
ChatGPT Codex
Co-authored-by: Rainer Gerhards <rgerhards@adiscon.com>
Signed-off-by: Rainer Gerhards <rgerhards@adiscon.com>
Users need to parse delimited strings (CSV, tags, paths) into arrays
for iteration or JSON output without external processing.
Impact: New RainerScript function available to all users.
Before: No native way to split strings into arrays in RainerScript.
After: split(string, separator) returns a JSON array of substrings.
Technical overview:
Implements doFunct_split() in grammar/rainerscript.c
Registers "split" in scriptFunct table with 2 required args
Adds CNFFUNC_SPLIT enum in rainerscript.h
Uses unified strstr-based iteration for all separator lengths
Handles edge cases: empty input, leading/trailing/consecutive delimiters
Includes error handling for json-c memory allocation failures
Returns empty JSON array on null/empty input or separator
Includes documentation (rs-split.rst) and test scripts
Improve usability by providing a simple way to check if an IP is inside
a CIDR subnet directly in RainerScript. This reduces awkward workarounds
and makes common filtering and routing tasks easier to express.
Impact: New function; existing configurations are unaffected.
Before: No built-in to test membership of an IP in a CIDR subnet.
After: is_in_subnet(ip, cidr) returns 1 if ip is in cidr, else 0.
Add is_in_subnet() as a built-in taking two args (IP string and CIDR).
Both IPv4 and IPv6 are supported. Inputs are parsed with inet_pton; the
CIDR mask is validated for range (0..32 / 0..128). Matching is done by
masking both the address and the network and comparing results. Invalid
inputs and family mismatches yield 0. The function returns a numeric
value. It is registered in the functions[] table and documented. Tests
cover IPv4/IPv6 basics, /0 and host masks, mismatches, and invalid
inputs. No HUP/state or OMODTX semantics are involved.
closes: https://github.com/rsyslog/rsyslog/issues/1391
With the help of AI Agents: Google Jules, Gemini (CLI),
ChatGPT Codex (CLI)
Among others, this patch includes a test for a hypothetical data
pipeline which ingests qradar json, transforms it, and ships to the
final destination.
The mmsnareparse tests fail on Launchpad builds because three
required test data files are not included in the distribution
tarball when make dist is run.
The files sample-windows2022-security.data,
sample-windows2025-security.data, and sample-events.data were
missing from EXTRA_DIST in tests/Makefile.am, causing test
failures with "No such file or directory" errors.
This patch adds all three missing files to EXTRA_DIST so they
are properly included in distribution packages.
Fixes: https://github.com/rsyslog/rsyslog/issues/6360
Documenting how to use sparseArray with ipv42num() for efficient
IPv4 subnet matching.
Added a regression test to verify this functionality.
see also: https://github.com/rsyslog/rsyslog/issues/4906
Simplify large-scale configs by auto-discovering receivers via DNS SRV
records. This reduces per-host configuration and helps enterprise and
container setups where target pools change over time.
Impact: new param `targetSrv`; config now errors on conflicts or empty
SRV answers; feature depends on resolver support.
Before: omfwd required a static host/port list via `target`/`port`.
After: `targetSrv` resolves `_syslog._udp|_tcp.<domain>` to build the
target pool, honoring RFC 2782 priority/weight and reusing existing
pool/load-balance logic.
Technically, add action param `targetSrv` (mutually exclusive with
`target`). During action init, perform SRV query via resolver
(`res_nquery`, `ns_initparse`) and translate answers into host/port
pairs. Preserve priority; randomly order same-priority entries using
weights. If explicit ports were set, warn and ignore when `targetSrv`
is used. Link rsyslogd with libresolv when available; configure checks
for headers and `ns_initparse`. Provide clear error paths (config check
fails) for missing support or empty SRV response. Docs cover usage and
env overrides `RSYSLOG_DNS_SERVER`/`RSYSLOG_DNS_PORT`. Tests add a
minimal UDP DNS server and cases for TCP/UDP success and error paths.
Fixes: https://github.com/rsyslog/rsyslog/issues/6314
With the help of AI Agent: ChatGPT Codex
Users need to parse delimited strings (CSV, tags, paths) into arrays
for iteration or JSON output without external processing.
Impact: New RainerScript function available to all users.
Before: No native way to split strings into arrays in RainerScript.
After: split(string, separator) returns a JSON array of substrings.
Technical overview:
Implements doFunct_split() in grammar/rainerscript.c
Registers "split" in scriptFunct table with 2 required args
Adds CNFFUNC_SPLIT enum in rainerscript.h
Uses unified strstr-based iteration for all separator lengths
Handles edge cases: empty input, leading/trailing/consecutive delimiters
Includes error handling for json-c memory allocation failures
Returns empty JSON array on null/empty input or separator
Includes documentation (rs-split.rst) and test scripts
This fix ensures that parse_json() only succeeds if the entire input
string is a valid JSON value. This prevents false positives when a
non-JSON string happens to start with a valid JSON value, like a number.
Documentation is updated to reflect this stricter validation.
Impact: Corrects false-success in parse_json() for malformed input.
Modified doFunc_parse_json in grammar/rainerscript.c to check if the
json-c tokener consumed the entire provided string. After parsing, the
remainder of the string is scanned for any non-whitespace characters.
If trailing garbage is found, the function now returns RS_SCRIPT_EINVAL
instead of RS_SCRIPT_EOK. Updated rs-parse_json.rst to document the
requirement for a complete JSON object/value. Added a regression test and
updated the testbench Makefile.am to include the new validation scenario.
Fixes: https://github.com/rsyslog/rsyslog/issues/4970
AI-Agent: Antigravity
This change adds the capability to overwrite the statistics log file
instead of appending to it. This is particularly useful for
observability tools like Prometheus scraping sidecars or node exporter,
which expect a consistent and complete set of metrics in a single file.
The implementation ensures atomicity by writing the statistics to a
temporary file and then renaming it to the final destination. This
prevents reader processes from seeing partial or inconsistent data
during the emission process.
This commit includes:
- The implementation in impstats.c.
- New test cases in the testbench.
- User-facing documentation for the new parameter.
Impact: Users can now enable atomic overwrites using
log.file.overwrite="on". Default behavior remains append.
Refs: no issue
AI-Agent: Antigravity
OpenTelemetry adoption: provide a first-party path to ship rsyslog
logs directly to OTLP collectors, enabling cleaner OTel pipelines and
container-friendly deployments without sidecars.
Impact: new output module; user-visible config surface; off by default
unless explicitly enabled at build and in config.
Before: no native OpenTelemetry (OTLP) exporter in rsyslog.
After: new "omotlp" action streams logs via OTLP/HTTP JSON with
configurable batching (count/bytes/timeout), optional gzip, retry/
backoff, TLS/mTLS, custom headers, and proxy support.
Technically, the action holds immutable config while each worker owns
an HTTP client and a batch buffer guarded by a mutex. A small flush
thread handles timeout-based flushes; batches also flush immediately on
thresholds and when the action queue transaction completes. HTTP 2xx
acknowledges and clears the batch; 4xx drops it; 5xx maps to
RS_RET_SUSPENDED for retry by the action queue. Stats per instance track
batches submitted/success/dropped/retried, HTTP 4xx/5xx, records.sent,
and cumulative request latency. Parameters honor OTEL_* env var
fallbacks when not explicitly set. Build is gated behind
--enable-omotlp; docs and tests cover batching, compression, TLS/mTLS,
proxy, and trace-correlation flows.
* New functionality for omuxsock
This builds on "PR#6121 net: Add NetworkNamespace APIS"
to add Network Namespace support for omuxsock. In
addition, new functionality is added to support
abstract socket names, as well as connected sockets.
Changes are isolated to omuxsock to provide the new
functionality:
a. Abstract unix socket names (including network namespaces)
b. Connected socket support (SOCK_STREAM, SOCK_SEQPACKET)
c. Support for load, and action configuration
d. Bug fix related to closing/reopening sockets
This bug would cause the socket to be closed and
reopened for every message sent.
New tests are added as follows:
a. uxsock_multiple.sh tests basic functionality with
multiple output instances.
b. uxsock_multiple_netns.sh is similar but using
multiple namespaces.
c. uxsock_simple_abstract.sh mirrors the existing
uxsock_simple.sh but tests with abstract socket
names.
The uxsockrcvr test program was modified to support
these new tests.
Signed-off-by: Billie Alsup <balsup@cisco.com>
Add ignoreTrailingPattern.regex parameter to support POSIX extended
regular expressions for dynamic trailing data removal. This enables
matching patterns with variable prefixes (e.g., numeric prefixes) that
cannot be handled by the static ignoreTrailingPattern parameter.
The new parameter is mutually exclusive with ignoreTrailingPattern.
When a regex pattern is provided, it is compiled during action instance
creation using regcomp with REG_EXTENDED. Compilation errors are reported
via regerror and cause configuration load to abort. The compiled regex
is freed in freeInstance using regfree.
The detect_and_truncate_trailing_extradata function now checks
ignoreTrailingPattern_isRegex to determine whether to use strstr
(static) or regexec (regex) for pattern matching. For regex matches,
the entire last token (including the matched prefix) is truncated and
stored in $!extradata_section, consistent with static pattern behavior.
Documentation updated to clarify truncation behavior and provide regex
examples. Test suite extended with mmsnareparse-trailing-extradata-regex.sh
covering various numeric prefix scenarios. All test cases updated to use
"custom_section" terminology for consistency.
Impact:
- Backward compatible: existing ignoreTrailingPattern continues to work
- New functionality: regex support for dynamic pattern matching
- Configuration validation: mutual exclusivity enforced at load time
Tests:
- mmsnareparse-trailing-extradata.sh (regression test)
- mmsnareparse-trailing-extradata-regex.sh (new regex test)
Co-authored-by: alorbach <alorbach@adiscon.com>
... for trailing extra-data removal.
Add configurable mechanism to detect and remove trailing extra-data sections
from messages before parsing. This addresses cases where third-party enrichers
append non-standard data (e.g., "enrichment_section: fromhost-ip=...") that
can interfere with Snare event parsing.
The ignoreTrailingPattern parameter can be set at both module and action
levels, with action-level values overriding module defaults. When configured,
the parser searches for the pattern in trailing positions (after the last
tab-separated token). If found, the message is truncated at the start of the
last token, removing the entire trailing section including any preceding
content in that token (e.g., dynamic numeric prefixes).
The truncated extra-data section is optionally exposed as a !extradata_section
message property, allowing downstream processing to access the removed content
if needed (e.g., for extracting sender IP addresses).
Implementation details:
- Pattern matching is literal string-based (not regex)
- Truncation only occurs when pattern appears in valid trailing positions
- Conservative detection for non-tab messages (last 20% or 200 chars)
- Proper memory management for pattern strings and extra-data sections
- No changes to existing behavior when parameter is not set
Added test case mmsnareparse-trailing-extradata.sh with anonymized sample
data validating Event ID 13 parsing with trailing enrichment section.
Updated documentation in doc/source/configuration/modules/mmsnareparse.rst
with parameter description and usage notes.
docs: enhance AGENTS.md with WSL build/test instructions
Added complete dependency installation, module-specific configure examples,
test execution patterns, and debugging workflow based
on actual development sessions.
Co-authored-by: alorbach <alorbach@adiscon.com>
Add support for Microsoft Sysinternals Sysmon events to the mmsnareparse
plugin using an external JSON definition file (sysmon_definitions.json).
This enables generic parsing of Sysmon events without hardcoding
event-specific logic, making it extensible for other event channels.
Key changes:
- Enhanced locate_snare_payload() to detect Sysmon events when MSWinEventLog
is in syslog tag (RFC3164 parsing scenario)
- Updated populate_event_metadata() to extract Channel from raw message
when version/channel fields are removed by syslog parser
- Fixed key-value parsing to handle single-space-separated pairs in
Sysmon descriptions (e.g., "User: CORP\NETWORK SERVICE")
- Improved pattern selection to prefer EventData section patterns when
sectionName is NULL, fixing User field storage location
- Added sysmon_definitions.json with event type mappings and field patterns
- Added test case mmsnareparse-sysmon.sh validating Event IDs 1, 3, and 5
The implementation is generic and extensible - other event channels can
be added by creating additional JSON definition files following the same
structure.
Co-authored-by: alorbach <alorbach@adiscon.com>
Rename the Snare Windows Security parser module from "mmsnarewinsec"
to "mmsnareparse" for clearer naming and consistency with other parser
modules. Update code identifiers, build system, docs, tests, CI flags,
and paths accordingly.
What changed
- MODULE_CNFNAME set to "mmsnareparse"; default macro renamed
- Log/error/debug tags updated to "mmsnareparse"
- plugins/mmsnarewinsec/* moved to plugins/mmsnareparse/*
- configure.ac:
- add --enable-mmsnareparse
- AM_CONDITIONAL(ENABLE_MMSNAREPARSE)
- AC_CONFIG_FILES now includes plugins/mmsnareparse/Makefile
- Makefile.am subdir switch to plugins/mmsnareparse
- Tests renamed and updated (scripts and testsuites directory)
- Docs page renamed and examples updated
- CI workflow uses --enable-mmsnareparse
- Rebase to main; resolved configure.ac conflict
Impact
- Backwards-incompatible module name and configure flag changes.
Migration
- Config: module(load="mmsnareparse"), action(type="mmsnareparse")
- Build: use --enable-mmsnareparse
Co-authored-by: alorbach <alorbach@adiscon.com>
This tool requires rsyslog to build with TLS support. If not present,
the test will always fail. This is solved by executing it only if gnutls
is enabled. As this is a fequently tested environment, this does not
reduce test coverage. It is easier to do then checking for both gnutls
and openssl.
Many thanks to Michael Biebl for bringing this to our attention.
closes https://github.com/rsyslog/rsyslog/issues/6224
Plain JSON embedded in text is common in production logs. This change
lets users parse such logs without cookies, improving ease of use and
lowering onboarding friction while keeping legacy behavior intact.
Before/After: cookie-only JSON -> find-json parses first top-level {}.
Impact: Default behavior unchanged. New mode and counters are opt-in.
Technical details:
- Add action parameter `mode` with `cookie` (default) and `find-json`.
The new mode scans for the first `{` and uses json_tokener to validate
a complete top-level object; quotes/escapes are respected.
- Add `max_scan_bytes` (default 65536) to bound scanning work and
`allow_trailing` (default on) to accept or reject non-whitespace data
after the parsed object. On reject/fail we return RS_RET_NO_CEE_MSG and
fall back to {"msg":"..."} while preserving parsesuccess semantics.
- Expose per-worker scan counters via statsobj/impstats and rsyslogctl:
scan.attempted, scan.found, scan.failed, scan.truncated. Counters are
active only in find-json mode and are resettable.
- Use length-aware cookie parsing (getMSG/getMSGLen) and keep legacy
RS_RET codes. Cookie mode behavior remains unchanged.
- Update docs: module overview, parameter references, statistics section
(impstats usage), and examples incl. mixed-mode routing. Add developer
engine overview page.
- Add tests for basic scanning, trailing control, scan limit, invalid
JSON, invalid mode, and parser validation edge cases.
With the help of AI Agent: Copilot
Real-world need: unflatten dotted JSON into nested objects, and optionally
flatten back for downstream tools. This introduces a general transformer
with a dedicated output tree. Interface is intentionally unstable.
Impact: New module behind --enable-mmjsontransform; no default behavior
changes. New tests and docs added. Parameters and behavior may change.
Add mmjsontransform, a message modification module that rewrites dotted
JSON keys. By default it "unflattens" an input object to nested containers
and stores the result in a configured output property. A mode parameter
also supports "flatten" to collapse nested trees into dotted keys. The
action refuses to overwrite an existing destination, validates that input
is a JSON object, and reports conflicts with precise key paths. Per-action
config is immutable; workers hold pointers only, so no extra locking. Docs
(Sphinx + parameter refs) and doxygen coverage included, plus a regression
test exercising nested arrays/objects. Build system and CI scripts gain
--enable-mmjsontransform and a basic test hook. An experimental companion
mmjsonrewrite module is wired similarly for dotted-key expansion.
Before/After: Previously no built-in JSON un/flatten; now an action can
unflatten (default) or flatten JSON into a separate message property.
With the help of AI Agents: ChatGPT codex, gemini
Implement complete NXLog Snare-formatted Windows Security event parser
with multi-format support (RFC5424/RFC3164), 100+ field patterns, and
advanced features including GUID/IP/timestamp type detection, runtime
configuration, enhanced validation modes, and comprehensive test suite.
Features:
- Parse major Windows security event types (4624, 4625, 4634, etc.)
- Extract structured data into configurable JSON containers (!win default)
- Handle modern Windows telemetry (LAPS, TLS, WDAC, WUFB, Kerberos)
- Type-aware parsing with validation and fallback handling
- Runtime configuration support for custom field patterns
- Thread-safe design with no shared mutable state
- 9 comprehensive test scripts covering all functionality
Impact: Enables structured analysis of Windows Security events for
SIEM integration, threat detection, and compliance reporting while
preserving original payloads for forensic investigation.
Files: contrib/mmsnarewinsec/, tests/mmsnarewinsec-*.sh,
doc/source/configuration/modules/mmsnarewinsec.rst
Subtree templates copied data into the worker buffer but left lenStr at
zero. Output modules that respect lenStr (omfwd, omfile, others) therefore
emitted empty payloads even though the buffer held valid JSON.
Set lenStr to the subtree length immediately after the memcpy. This aligns
the subtree branch with the existing regular/jsonftree/strgen paths and
restores correct forwarding behaviour for all modules.
Add regression coverage:
* retain omfwd-subtree-tpl.sh to prove network forwarding now delivers the
subtree payload
* add omfile-subtree-jsonf.sh to exercise subtree data consumed via
exec_template() and rendered through an option.jsonf list template
Before: subtree templates built the JSON text but omfwd saw lenStr=0 and
sent empty frames or files.
After: lenStr matches the copied bytes, so modules transmit the expected
JSON content.
Closes: https://github.com/rsyslog/rsyslog/issues/6206
Newer Elasticsearch versions reject typed APIs. This adapts the module
so shipping works out of the box with modern clusters and keeps the
codebase aligned with typeless ES conventions.
Impact: bulk metadata no longer includes _type unless explicitly set.
Old setups that relied on a default "events" type (ES < 8) may need
explicit configuration.
Technically, the default searchType is now NULL
(OMES_SEARCHTYPE_DEFAULT). setPostURL continues to route requests to
the typeless /_doc endpoint, but bulk metadata is generated without a
_type field when searchType is unset. The legacy default "events" for
ES < 8 is removed. Tests are updated to stop passing searchType, and
the searchType-empty test is dropped to reflect the new default. No
OMODTX or action-queue semantics change.
Closes: https://github.com/rsyslog/rsyslog/issues/5060
We want easy nested JSON to match common schemas (e.g., Elastic ECS)
without external processors. This introduces an opt-in mode so existing
jsonf users keep exact behavior while enabling structured output when
requested.
Impact: No change unless option.jsonftree is enabled. With jsonftree,
dotted outnames render as nested objects; empty containers are skipped.
On name collisions (object vs value), we fall back to flat rendering.
Before: jsonf always emitted flat name/value pairs, even for dotted
outnames. After: jsonf remains flat by default; enabling jsonftree makes
"host.hostname" and "host.ip" render as {"host":{"hostname":...,"ip":...}}.
Technically, we add option.jsonftree to templates. When set, we lazily
build a per-template JSON tree (tplJsonNode) from dotted segments and
render it in one pass, reusing existing jsonf formatting for leaves.
The tree state is tracked on the template and freed on template delete.
Config parsing enforces mutual exclusivity among sql, stdsql, json,
jsonf, and jsonftree. Constants record bJSONf to reuse serialized
fragments. Tests cover nested output and pure-json cases using
option.jsonftree.
This parses the LEEF message (if it is) and creates a JSON subtree.
The current implementation is PoC and will be provided to gather
early review.
Changes, including breaking changes, may happen in future versions of
this module.
Admins often report "gibberish" when a TLS-enabled sender connects to a
plain imtcp port. Making the mismatch explicit reduces operator confusion
and support churn, and points directly to remediation.
Impact: logs one explicit error per mismatched connection; no change to
parsing or transport on plain listeners.
Before/After: before, ClientHello bytes were ingested as binary with no
hint; after, imtcp detects a TLS ClientHello on ptcp and logs a clear
message with a troubleshooting URL.
Technically, we add a small per-session probe in tcps_sess_t and sample
the first 5 bytes of new sessions. If the record header matches a TLS
handshake (type 0x16, version 0x03.00–0x04, length 40–16384) and the
listener is plain TCP (streamDriver.mode=0), we emit a single error and
disable further probing for that session. The probe is called from
DataRcvd() and returns RS_RET_SERVER_NO_TLS when triggered; the session
is otherwise left untouched.
Runtime: introduce RS_RET_SERVER_NO_TLS (-2465) to tag the condition.
Docs: add imtcp troubleshooting section and a dedicated FAQ page.
Tests/tools: add test imtcp-tls-gibberish.sh and extend tcpflood with
-H to send only a ClientHello (OpenSSL and GnuTLS paths tolerate early
termination and non-blocking I/O for this mode).
Add kafkaHeader parameter to define key/value pairs
that are attached as headers to every produced message.
Require librdkafka v0.11 for header support. Update
configure checks, docs and add a regression test.
closes: https://github.com/rsyslog/rsyslog/issues/5185
With help of AI-Agent: OpenAI ChatGPT
This builds on "PR#6121 net: Add NetworkNamespace APIS"
to add Network Namespace support to imtcp module. This
extends imtcp to support a wider range of Unix/Linux
environments (or any environment supporting network
namespaces).
The imtcp module is enhanced to accept a NetworkNamespace
parameter, both as a default at the module level, and
on a per-instance basis.
The tcpsrv module is enhanced to allow the NetworkNamespace
to be applied to a listener's configuration parameters.
Finally, the netstrm module is enhanced to switch namespaces
before invoking the downstream (driver specific) LstnInit
function.
A new test imtcp-netns (and associated imtcp-netns-vg) is
added to test this functionality. This must be run as root
(technically it must be run by a user with CAP_SYS_ADMIN
capabilities, as network namespace creating/change is
required).
A slight change to diag.sh is made to allow passing $RS_REDIR
to valgrind (as $RS_REDIR is used in the imtcp-netns.sh
test for some negative cases).
Signed-off-by: Billie Alsup <balsup@cisco.com>
Some deployments need to disambiguate multiple senders sharing an IP,
for example autossh or similar tunnel setups. Exposing the source port
improves observability and lets pipelines key on a stable tuple.
Impact: new property/JSON field; tcps_sess IF v4; out-of-tree modules
must rebuild.
Before: messages exposed fromhost and fromhost-ip only.
After: messages also expose fromhost-port and jsonmesg includes it.
Introduce PROP_FROMHOST_PORT and wire it through msg.{h,c}. For TCP,
capture the remote port on accept, store it in tcps_sess, and attach it
to the msg on submit. For other inputs, resolveDNS derives the port from
the sockaddr when available; local inputs return an empty string. Add a
getter, duplication and destructor handling, and name<->ID mapping. Add
the field to jsonmesg output. Update docs, lexer keywords, and the
external plugin interface doc (property is modifiable). Bump
tcps_sessCURR_IF_VERSION to 4 and add SetHostPort() to the interface.
Include a focused test (fromhost-port.sh) that verifies the property.
Non-technical rationale: allow identification by (fromhost-ip,
fromhost-port) where IP alone is shared across systems (e.g., autossh).
With help from AI-Agents: ChatGPT
Add a RainerScript function to convert strings to uppercase. The
implementation mirrors tolower() but operates per-byte using
toupper(). Tests demonstrate the new function, documentation and
ChangeLog entries added.
closes: https://github.com/rsyslog/rsyslog/issues/3666
With the help of AI-Agent: ChatGPT
