omlibdi: improvide doc by parameter split
splitting the monolithic omlibdbi doc into smaller chunks for better
human and AI understanding. Follows overall rsyslog doc strategy.
With the help of AI Agent: codex
Removes a data race on the action instance so anonymization stays
reliable under concurrent workers. This improves stability and makes
mappings deterministic.
Impact: Previously rare nondeterminism or crashes removed; slight extra
contention possible at very high event rates.
Before/After:
BEFORE: concurrent updates to IPv4 trie and IPv6 caches raced.
AFTER: caches are mutex-protected; mappings remain consistent.
Technical:
- Add ipv4Mutex for the IPv4 consistency trie; ipv6Mutex for IPv6 and
embedded-IPv4 hash tables.
- Initialize both in createInstance; destroy in freeInstance.
- Lock around lookup/insert paths in findip() and findIPv6(); ensure
unlock on all ABORT/finalize paths.
- No change to algorithms, ABI, or randConsis semantics.
With the help of AI agent: codex
Add ignoreTrailingPattern.regex parameter to support POSIX extended
regular expressions for dynamic trailing data removal. This enables
matching patterns with variable prefixes (e.g., numeric prefixes) that
cannot be handled by the static ignoreTrailingPattern parameter.
The new parameter is mutually exclusive with ignoreTrailingPattern.
When a regex pattern is provided, it is compiled during action instance
creation using regcomp with REG_EXTENDED. Compilation errors are reported
via regerror and cause configuration load to abort. The compiled regex
is freed in freeInstance using regfree.
The detect_and_truncate_trailing_extradata function now checks
ignoreTrailingPattern_isRegex to determine whether to use strstr
(static) or regexec (regex) for pattern matching. For regex matches,
the entire last token (including the matched prefix) is truncated and
stored in $!extradata_section, consistent with static pattern behavior.
Documentation updated to clarify truncation behavior and provide regex
examples. Test suite extended with mmsnareparse-trailing-extradata-regex.sh
covering various numeric prefix scenarios. All test cases updated to use
"custom_section" terminology for consistency.
Impact:
- Backward compatible: existing ignoreTrailingPattern continues to work
- New functionality: regex support for dynamic pattern matching
- Configuration validation: mutual exclusivity enforced at load time
Tests:
- mmsnareparse-trailing-extradata.sh (regression test)
- mmsnareparse-trailing-extradata-regex.sh (new regex test)
Co-authored-by: alorbach <alorbach@adiscon.com>
... for trailing extra-data removal.
Add configurable mechanism to detect and remove trailing extra-data sections
from messages before parsing. This addresses cases where third-party enrichers
append non-standard data (e.g., "enrichment_section: fromhost-ip=...") that
can interfere with Snare event parsing.
The ignoreTrailingPattern parameter can be set at both module and action
levels, with action-level values overriding module defaults. When configured,
the parser searches for the pattern in trailing positions (after the last
tab-separated token). If found, the message is truncated at the start of the
last token, removing the entire trailing section including any preceding
content in that token (e.g., dynamic numeric prefixes).
The truncated extra-data section is optionally exposed as a !extradata_section
message property, allowing downstream processing to access the removed content
if needed (e.g., for extracting sender IP addresses).
Implementation details:
- Pattern matching is literal string-based (not regex)
- Truncation only occurs when pattern appears in valid trailing positions
- Conservative detection for non-tab messages (last 20% or 200 chars)
- Proper memory management for pattern strings and extra-data sections
- No changes to existing behavior when parameter is not set
Added test case mmsnareparse-trailing-extradata.sh with anonymized sample
data validating Event ID 13 parsing with trailing enrichment section.
Updated documentation in doc/source/configuration/modules/mmsnareparse.rst
with parameter description and usage notes.
docs: enhance AGENTS.md with WSL build/test instructions
Added complete dependency installation, module-specific configure examples,
test execution patterns, and debugging workflow based
on actual development sessions.
Co-authored-by: alorbach <alorbach@adiscon.com>
* omdtls: split parameter docs into reference pages; add summary list-tables; fix anchors
- Split Module/Input parameters into per-parameter files in reference/parameters
- Replace inline tables with list-tables + included summaries
- Add hidden toctree with new parameter pages
- Fix anchors and add usage examples for all parameters
With the help of AI-Agent: ChatGPT
* add concise conceptual model sections to queue, parser, and ruleset docs for CS readers\n* extend log pipeline pattern and stage pages with abstract semantics\n* summarize reliability and syslog whitepapers with core models for ingestion\n\nAI-Agent: ChatGPT
Add support for Microsoft Sysinternals Sysmon events to the mmsnareparse
plugin using an external JSON definition file (sysmon_definitions.json).
This enables generic parsing of Sysmon events without hardcoding
event-specific logic, making it extensible for other event channels.
Key changes:
- Enhanced locate_snare_payload() to detect Sysmon events when MSWinEventLog
is in syslog tag (RFC3164 parsing scenario)
- Updated populate_event_metadata() to extract Channel from raw message
when version/channel fields are removed by syslog parser
- Fixed key-value parsing to handle single-space-separated pairs in
Sysmon descriptions (e.g., "User: CORP\NETWORK SERVICE")
- Improved pattern selection to prefer EventData section patterns when
sectionName is NULL, fixing User field storage location
- Added sysmon_definitions.json with event type mappings and field patterns
- Added test case mmsnareparse-sysmon.sh validating Event IDs 1, 3, and 5
The implementation is generic and extensible - other event channels can
be added by creating additional JSON definition files following the same
structure.
Co-authored-by: alorbach <alorbach@adiscon.com>
- Split legacy history content into two focused documents:
* about/history.rst – rsyslog history (2004–present, reverse-chronological)
* about/origins.rst – syslog heritage (1983–2003)
- Added missing cross-references to existing docs (Beginner’s Guide,
omelasticsearch, modules index, containers).
- Strengthened 1983 BSD syslogd heritage statement.
- Created redirect stub for old doc/history.rst.
- Updated about/index.rst to include new origins page.
- Polished timelines, wording, and structure for clarity and long-term maintainability.
With the help of AI-Agent: ryslog doc assistant (openai based)
* doc: add json-ld to doc set (base schema)
This adds basic json-ld to the doc set so that articles are properly flagged as techArticle or FAQ.
This also adds a toggle switch and user doc on how to disable json-ld generation. That is expected to be done when building versions for distro packages for offline consumption.
Note that while this method improves semantic understanding, we do not expect a notable improvement in large-scale RAG systems, as the chunking/indexing usually works well enough. This is not yet graph-RAG json-ld - we need to work more on the ontology.
With the help of AI-Agent: ChatGPT
This was introduced in 78541ff, which I accidentally merged with the
subproject (I should have rejected merge unless this was fixed.).
Now that I notice, this is the clean-up. Sorry for any confusion this
might have caused.
Thanks for Juha Kallioinen for alerting me.
see also: https://github.com/rsyslog/rsyslog/discussions/6301
* omhdfs: split parameter docs into reference pages
- Split module parameter documentation into dedicated reference pages for OMHDFSFileName, OMHDFSHost, OMHDFSPort, and OMHDFSDefaultTemplate
- Replace inline parameter tables with summary list-table includes and add hidden toctree plus casing note on module page
- Document legacy directive names and provide usage examples with camelCase configuration snippets
- Verified new pages with docutils to ensure markup parses cleanly
With the help of AI-Agent: ChatGPT
* omgssapi: split parameter docs into reference pages
- Introduce dedicated parameter reference files for GssForwardServiceName, GssMode, and ActionGSSForwardDefaultTemplate
- Replace inline module parameter description with list-table summaries and add hidden toctree on module page
- Link imgssapi documentation to the omgssapi module reference without duplicating toctree entries
- AI-Agent: ChatGPT
* omgssapi: correct parameter default documentation
Fix parameter reference defaults to omit erroneous module= prefixes.
With the help of AI-Agent: ChatGPT
* omclickhouse: split parameter docs into reference pages
- move each action parameter into a dedicated reference file with proper anchors
- replace inline CSV tables with summary list-table and hidden toctree includes
- carry over descriptions, defaults, and usage examples for all module parameters
With the help of AI-Agent: ChatGPT
* imkafka: enhance stats support, including zabbix format
Adapted omkafka pstats counters to the consumer imkafka modules for pstats tracking. Implements both global (module-level) and local (topic+cg-level) metrics. Implements librdkafka metrics (rtt_avg_usec, throttle_avg_msec, int_latency_avg_usec) variables, global categorized (ctrKafkaRespTimedOut, ctrKafkaRespTransport, ctrKafkaRespBrokerDown, ctrKafkaRespAuth, ctrKafkaRespSSL, ctrKafkaRespOther) as well as the normal kafka stats (KafkaFail, Submitted, Received, EOF, PollEmpty, MaxLag).
MaxLag is a consumer metric only.
JSON helpers implemented to handle librdkafka's JSON emissions for time-window stats. Uses the traditional libfastjson library which is already needed for compilation. (parse librdkafka emission -> output as Ctr)
Updated docs to highlight changes around the new format "zabbix". Listed notice that log.syslog is not recommended/log.file is recommended, along with suggestions if log.syslog is necessary.
Updated impstats.c to include new formatting compatible with Zabbix LLD protocols. Implemented arrayed JSON support under the format="zabbix".
Docs should reflect that large pstats records in an arrayed-json object (single object per pstats interval) should not be used in conjunction with log.syslog="on" due to truncation issues. Small emissions are usually OK.
log.file is recommended as it bypasses the traditional $MaxMsgSize limitations.
* doc: clarify release tracks
Highlight that the daily and scheduled channels are both production-ready and note the long-running CI and review process backing the daily stable stream.
With the help of AI-Agent: ChatGPT
* omazureeventhubs: split parameter docs into reference pages; add summary list-tables; fix anchors
- Split action parameter documentation into dedicated reference pages with scope-correct anchors, summaries, and usage examples.
- Replace the module page's inline parameter blocks with a list-table that includes the new summaries and a hidden toctree.
- Preserve existing narrative content while aligning case-sensitivity guidance with camelCase recommendations.
With the help of AI-Agent: ChatGPT
* omhttp: split parameter docs into reference pages; add summary list-tables; fix anchors
- Split module parameters into per-parameter reference pages with summary metadata
- Replace inline parameter section with list-table and hidden toctree on the module page
- Update module overview references and usage examples to the new anchors
- Preserve existing descriptions, notes, and examples for all parameters
With the help of AI-Agent: ChatGPT
* docs: clarify calendar-based release cadence
- record the bi-monthly Tuesday schedule and why December ships early\n- explain that v8 stays the major digit unless the numbering scheme itself changes\n- relate the 8.yymm.0 format to other calendar-based projects and update the newcomer guide note
AI-Agent: ChatGPT
* Update doc/source/about/release_versioning.rst
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
---------
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* imsolaris: split parameter docs into reference pages; add summary list-tables; fix anchors
- Split IMSolarisLogSocketName docs into reference/parameters entry with scoped anchors
- Replace inline module parameter text with list-table summary and hidden toctree
- Document legacy directive usage and provide module example
With the help of AI-Agent: ChatGPT
* imrelp: split parameter docs into reference pages
- Split Module/Input parameters into per-parameter reference pages with scoped anchors and summaries
- Replace inline parameter descriptions with list-tables that include summary includes and add hidden toctree
- Preserve TLS guidance, legacy aliases, and usage examples while aligning casing and anchor conventions
With the help of AI-Agent: ChatGPT
* omsendertrack: split parameter docs into reference pages
- replace inline parameter tables with summary list-table includes and add hidden toctree
- create per-parameter reference pages with preserved descriptions and usage examples
- update internal references to new scoped anchors for senderid, statefile, interval, and cmdfile
With the help of AI-Agent: ChatGPT
* Revise README for clarity and formatting improvements
Updated formatting and improved clarity in README content, including section titles and descriptions.
Test imrelp-tls-cfgcmd.sh seems to fail but still report success, which
can create a false impression.
Suspected root cause is in librelp. It may also be that the test
actually succeeds, where it just looks like it failed because of abort
of tcpflood testing tool (due to librelp bug).
The whole point of the test is that an error is generated, and this may
very well happen. And only tcpflood aborts because of the librelp bug.
Core file detection than jumps in, and invalidly treats the tcpflood
core file as a test failure.
While this is investigated, the test will be skipped. Highly like that
this needs to be forwarded either to librelp or is a native tcpflood
bug.
Thanks to Chris Hofstaedtler for reporting the bug and Michael Biebl for
forwarding it upstream.
see also: https://github.com/rsyslog/rsyslog/issues/6267
* immark: split parameter docs into reference pages
- add an Interval parameter reference page with metadata, usage,
and the legacy alias
- replace the module parameter table with a summary list-table
that includes the new page
- add a hidden toctree entry and update the casing guidance to
match the parameter ref split
With the help of AI-Agent: ChatGPT
impstats: split parameter docs into reference pageS
- Split impstats module parameters into dedicated reference pages with scoped anchors and usage examples
- Replace inline parameter content with summary list-table and hidden toctree on the module page
- Preserve defaults, notes, and formatting while aligning example casing per policy
With the help of AI-Agent: ChatGPT
