Rename the Snare Windows Security parser module from "mmsnarewinsec"
to "mmsnareparse" for clearer naming and consistency with other parser
modules. Update code identifiers, build system, docs, tests, CI flags,
and paths accordingly.
What changed
- MODULE_CNFNAME set to "mmsnareparse"; default macro renamed
- Log/error/debug tags updated to "mmsnareparse"
- plugins/mmsnarewinsec/* moved to plugins/mmsnareparse/*
- configure.ac:
- add --enable-mmsnareparse
- AM_CONDITIONAL(ENABLE_MMSNAREPARSE)
- AC_CONFIG_FILES now includes plugins/mmsnareparse/Makefile
- Makefile.am subdir switch to plugins/mmsnareparse
- Tests renamed and updated (scripts and testsuites directory)
- Docs page renamed and examples updated
- CI workflow uses --enable-mmsnareparse
- Rebase to main; resolved configure.ac conflict
Impact
- Backwards-incompatible module name and configure flag changes.
Migration
- Config: module(load="mmsnareparse"), action(type="mmsnareparse")
- Build: use --enable-mmsnareparse
Co-authored-by: alorbach <alorbach@adiscon.com>
- replace placeholder hostnames with example.net values and show a realistic TCP port value\n- expand the surrounding explanation so readers understand when omfwd falls back to its default port and how independent queues behave\n\nAI-Agent: ChatGPT
This tool requires rsyslog to build with TLS support. If not present,
the test will always fail. This is solved by executing it only if gnutls
is enabled. As this is a fequently tested environment, this does not
reduce test coverage. It is easier to do then checking for both gnutls
and openssl.
Many thanks to Michael Biebl for bringing this to our attention.
closes https://github.com/rsyslog/rsyslog/issues/6224
Switch the database tutorial to reference list-based templates and document libdbi as the generic plugin example. Update the priority recording tutorial to demonstrate list templates and reiterate how property() and constant() map to property replacer behavior.
AI-Agent: ChatGPT
Replace the retry-count loop with a deadline-based poll so the helper\nfails after a sensible timeout instead of hanging when impstats stops\nemitting metrics. Use a longer default window for the Valgrind variant\nso it tolerates the slower runtime.\n\nAI-Agent: ChatGPT
Plain JSON embedded in text is common in production logs. This change
lets users parse such logs without cookies, improving ease of use and
lowering onboarding friction while keeping legacy behavior intact.
Before/After: cookie-only JSON -> find-json parses first top-level {}.
Impact: Default behavior unchanged. New mode and counters are opt-in.
Technical details:
- Add action parameter `mode` with `cookie` (default) and `find-json`.
The new mode scans for the first `{` and uses json_tokener to validate
a complete top-level object; quotes/escapes are respected.
- Add `max_scan_bytes` (default 65536) to bound scanning work and
`allow_trailing` (default on) to accept or reject non-whitespace data
after the parsed object. On reject/fail we return RS_RET_NO_CEE_MSG and
fall back to {"msg":"..."} while preserving parsesuccess semantics.
- Expose per-worker scan counters via statsobj/impstats and rsyslogctl:
scan.attempted, scan.found, scan.failed, scan.truncated. Counters are
active only in find-json mode and are resettable.
- Use length-aware cookie parsing (getMSG/getMSGLen) and keep legacy
RS_RET codes. Cookie mode behavior remains unchanged.
- Update docs: module overview, parameter references, statistics section
(impstats usage), and examples incl. mixed-mode routing. Add developer
engine overview page.
- Add tests for basic scanning, trailing control, scan limit, invalid
JSON, invalid mode, and parser validation edge cases.
With the help of AI Agent: Copilot
* imdtls: split parameter docs into reference pages; add summary list-tables; fix anchors
- Split imdtls input parameters into dedicated reference files with canonical anchors, scope usage, and usage examples
- Replace inline parameter tables in the module page with list-tables that include summary includes and a hidden toctree
- Updated imdtls documentation to emphasize camelCase parameter names and preserve all descriptive details
Wtiht the help of AI-Agent: ChatGPT
- wrap omfwd-lb-1target-retry-full_buf.sh around the shared skeleton so the scenario is retried once before reporting failure, easing TCP buffer timing races
- document follow-on stabilisation ideas for minitcpsrvr coordination and omfwd diagnostics
AI-Agent: Codex
Real-world tarballs must be self-contained for doc builds. This change
verifies release completeness by building the docs from the tarball,
not the checkout.
Impact: Release tarballs now include additional doc assets; CI fails
early if a doc-required file is missing.
Before: CI built Sphinx docs directly from the repo tree. Missing files
could be masked by in-tree paths. After: CI creates a 'make dist'
tarball, unpacks it, and builds docs from that tree to catch omissions.
Technically, the doc workflow installs build deps, runs autoreconf and
a minimal './configure', issues 'make dist', unpacks into
'doc-builder/', and runs 'make html' under 'doc-builder/doc'. The
artifact path is updated accordingly.
For distribution, 'configure.ac' now composes DOC_FILES from
'doc/source' (*.rst, *.conf, *.jpg, *.png) and explicitly adds
'tutorials/cert-script.tar.gz', 'doc/Makefile', and
'doc/ai/module_map.yaml'. The duplicate listing of
'doc/ai/module_map.yaml' is removed from 'Makefile.am' EXTRA_DIST to
avoid drift; DOC_FILES remains the single source of truth for doc
payloads.
* ci/tests: add Elasticsearch 8 workflow
- allow the testbench to honour RSYSLOG_TESTBENCH_EXTERNAL_ES_URL
so externally managed Elasticsearch services can be reused without local
start/stop logic
- provision Elasticsearch 8 via GitHub Actions and run the omelasticsearch
suite inside the development container
With the help of AI-Agent: ChatGPT
* im3195: split parameter docs into reference pages
- split Input3195ListenPort docs into a dedicated reference entry with legacy directive mapping
- replace inline parameter table with summary list-table and hidden toctree
- reiterate camelCase guidance for parameter casing clarity
With the help of AI-Agent: ChatGPT
Real-world need: unflatten dotted JSON into nested objects, and optionally
flatten back for downstream tools. This introduces a general transformer
with a dedicated output tree. Interface is intentionally unstable.
Impact: New module behind --enable-mmjsontransform; no default behavior
changes. New tests and docs added. Parameters and behavior may change.
Add mmjsontransform, a message modification module that rewrites dotted
JSON keys. By default it "unflattens" an input object to nested containers
and stores the result in a configured output property. A mode parameter
also supports "flatten" to collapse nested trees into dotted keys. The
action refuses to overwrite an existing destination, validates that input
is a JSON object, and reports conflicts with precise key paths. Per-action
config is immutable; workers hold pointers only, so no extra locking. Docs
(Sphinx + parameter refs) and doxygen coverage included, plus a regression
test exercising nested arrays/objects. Build system and CI scripts gain
--enable-mmjsontransform and a basic test hook. An experimental companion
mmjsonrewrite module is wired similarly for dotted-key expansion.
Before/After: Previously no built-in JSON un/flatten; now an action can
unflatten (default) or flatten JSON into a separate message property.
With the help of AI Agents: ChatGPT codex, gemini
Parsing QRadar-formatted JSON is common in real-world pipelines and
migration projects. This change adds CI coverage to ensure rsyslog
can ingest and re-emit such events without semantic drift.
Impact: CI-only; no runtime behavior change.
Add testsuites/qradar_json with two representative QRadar events. A
new Python validator compares input vs. emitted JSON after removing
the CEE cookie, ignoring whitespace and key order for semantic
equality. A shell fallback uses jq for normalized compares and a
basic field presence check when jq is unavailable. Increase the
tests/diagtalker input buffer from 2 KiB to 10 KiB so large QRadar
payloads are not truncated during test runs.
BEFORE: No semantic check for QRadar JSON; large messages could
truncate in diagtalker.
AFTER: CI asserts input/output JSON equivalence; diagtalker handles
~10 KiB lines without truncation.
Implement complete NXLog Snare-formatted Windows Security event parser
with multi-format support (RFC5424/RFC3164), 100+ field patterns, and
advanced features including GUID/IP/timestamp type detection, runtime
configuration, enhanced validation modes, and comprehensive test suite.
Features:
- Parse major Windows security event types (4624, 4625, 4634, etc.)
- Extract structured data into configurable JSON containers (!win default)
- Handle modern Windows telemetry (LAPS, TLS, WDAC, WUFB, Kerberos)
- Type-aware parsing with validation and fallback handling
- Runtime configuration support for custom field patterns
- Thread-safe design with no shared mutable state
- 9 comprehensive test scripts covering all functionality
Impact: Enables structured analysis of Windows Security events for
SIEM integration, threat detection, and compliance reporting while
preserving original payloads for forensic investigation.
Files: contrib/mmsnarewinsec/, tests/mmsnarewinsec-*.sh,
doc/source/configuration/modules/mmsnarewinsec.rst
Subtree templates copied data into the worker buffer but left lenStr at
zero. Output modules that respect lenStr (omfwd, omfile, others) therefore
emitted empty payloads even though the buffer held valid JSON.
Set lenStr to the subtree length immediately after the memcpy. This aligns
the subtree branch with the existing regular/jsonftree/strgen paths and
restores correct forwarding behaviour for all modules.
Add regression coverage:
* retain omfwd-subtree-tpl.sh to prove network forwarding now delivers the
subtree payload
* add omfile-subtree-jsonf.sh to exercise subtree data consumed via
exec_template() and rendered through an option.jsonf list template
Before: subtree templates built the JSON text but omfwd saw lenStr=0 and
sent empty frames or files.
After: lenStr matches the copied bytes, so modules transmit the expected
JSON content.
Closes: https://github.com/rsyslog/rsyslog/issues/6206
omkafka: restore producev fallback for older librdkafka
Reason: keep omkafka building across librdkafka versions, including
newer distro toolchains and legacy environments, to reduce friction
for users and CI/container images.
Before: builds failed or used unsupported code paths with some
librdkafka versions; header config could silently miscompile.
After: feature-detect produceva/headers, fall back to legacy produce,
and reject header config when not supported.
Impact: possible configuration-time error if kafka headers are set
but the installed librdkafka lacks header support.
This change introduces compile-time feature detection. When
produceva is available, we use the vector API; otherwise we fall back
to the classic produce path and adjust error handling accordingly.
Header support is compiled only when the library provides it; the
instance struct and cleanup are guarded to match. A missing
RD_KAFKA_VERSION is defined to avoid preprocessor failures. No queue
or OMODTX semantics are changed; HUP and stats remain unaffected. No
public API/ABI is exposed from omkafka, so impact is limited to build
and configuration behavior.
Add comprehensive macOS CI support with two new GitHub Actions
workflows:
- run_macos.yml: PR-triggered CI with matrix strategy covering macOS
13–15, x64/arm64 architectures, and sanitizer combinations (none,
ASAN, TSAN)
- run_macos_weekly.yml: Scheduled weekly testing with full matrix
coverage and automated failure reporting via GitHub issues
- Set sin_len in tests/diagtalker.c on macOS to fix connect() EINVAL,
unblocking TLS certvalid tests.
Root cause and fix details (macOS testbench)
- Why tests failed
On macOS 14 the imdiag control listener often ended up IPv6-only.
The plain TCP listener creates an IPv6 socket and sets IPV6_V6ONLY;
the companion IPv4 bind can fail on macOS when sharing an ephemeral
port, leaving only the IPv6 listener active. Our injector
tests/diagtalker.c was IPv4-only (AF_INET to 127.0.0.1), so it could
not reach the imdiag port, causing repeated connect retries and
timeouts. CI logs showed “cannot connect to 127.0.0.1:<port> …
Connection refused” alongside benign OpenSSL anon-mode warnings.
- What we changed
1) Made the injector dual-stack by switching diagtalker to
getaddrinfo(AF_UNSPEC) and trying both IPv6 and IPv4 (with fallback
to 127.0.0.1 and ::1). This removes the hard dependency on IPv4
reachability when the listener is IPv6-only on macOS.
2) Added an opt-in testbench knob to enforce IPv4 where appropriate:
generate_conf() now honors RSTB_FORCE_IPV4=1 (or
RSTB_NET_IPPROTO=ipv4-only) to inject
global(net.ipprotocol="ipv4-only"). We enable this only in the IPv4
test variant so the IPv6 wrapper remains pure IPv6.
- Impact
The injector/listener address-family mismatch is eliminated, resolving
the macOS connect() failures and unblocking the TLS “certvalid” and
anonymous tests on macOS runners.
Refs: https://github.com/rsyslog/rsyslog/issues/5635
Refs: https://github.com/Homebrew/homebrew-core/pull/221869
Refs: https://github.com/Homebrew/homebrew-core/pull/226378
- detect librdkafka's produce helpers via the exported RD_KAFKA_V_* macros so feature checks succeed on modern headers instead of always disabling the new paths
- add a rd_kafka_producev branch that preserves timestamps and optional record headers when librdkafka lacks rd_kafka_produceva but still ships the variadic helper macros, falling back to rd_kafka_produce only for pre-0.9.4 builds
- only enable kafkaheader configuration when a usable producev/a helper is available so unsupported libraries fail configuration rather than silently dropping headers
- ./devtools/format-code.sh
* AI: align AGENTS.md and README; expand harness tips
This refines testbench docs for both AI agents and human operators.
Goal: clearer authoring guidance and a smoother quickstart so agents
produce higher-quality tests and contributors run them reliably.
Before/After: scattered guidance across files -> cross-linked docs with
explicit helper usage, Valgrind wrapper patterns, and quickstart.
Non-technical rationale:
This improves maintainability and AI readiness by giving agents concrete
patterns and reducing duplicate boilerplate. It also aligns operator docs
with the harness so CI and local runs behave consistently.
With the help of AI Agents: ChatGPT, gemini
* imtcp: include peer ports in runtime log messages
Ensure tcpsrv and session diagnostics report the sender's source port alongside the existing host/IP fields so administrators can pinpoint connections precisely. This updates imtcp connection notices, close/error paths, and framing warnings to use a consistent "host (ip:port)" style already present in TLS mismatch logs.
With the help of AI-Agent: Codex
* omelasticsearch: detect target platform at startup
Add a curl-based probe during configuration validation to query the configured Elasticsearch/OpenSearch cluster and record its platform and version. Store the detected metadata in the instance configuration so it can be used for tuning and validation, and log the result during startup.
With the help of AI-Agent: Codex
