mirror of
https://github.com/rsyslog/rsyslog.git
synced 2025-12-15 19:50:40 +01:00
7960b7f03e
Plain JSON embedded in text is common in production logs. This change
lets users parse such logs without cookies, improving ease of use and
lowering onboarding friction while keeping legacy behavior intact.
Before/After: cookie-only JSON -> find-json parses first top-level {}.
Impact: Default behavior unchanged. New mode and counters are opt-in.
Technical details:
- Add action parameter `mode` with `cookie` (default) and `find-json`.
The new mode scans for the first `{` and uses json_tokener to validate
a complete top-level object; quotes/escapes are respected.
- Add `max_scan_bytes` (default 65536) to bound scanning work and
`allow_trailing` (default on) to accept or reject non-whitespace data
after the parsed object. On reject/fail we return RS_RET_NO_CEE_MSG and
fall back to {"msg":"..."} while preserving parsesuccess semantics.
- Expose per-worker scan counters via statsobj/impstats and rsyslogctl:
scan.attempted, scan.found, scan.failed, scan.truncated. Counters are
active only in find-json mode and are resettable.
- Use length-aware cookie parsing (getMSG/getMSGLen) and keep legacy
RS_RET codes. Cookie mode behavior remains unchanged.
- Update docs: module overview, parameter references, statistics section
(impstats usage), and examples incl. mixed-mode routing. Add developer
engine overview page.
- Add tests for basic scanning, trailing control, scan limit, invalid
JSON, invalid mode, and parser validation edge cases.
With the help of AI Agent: Copilot
35 lines
1.3 KiB
Bash
Executable File
35 lines
1.3 KiB
Bash
Executable File
#!/bin/bash
|
|
# Test mmjsonparse find-json mode with trailing data and allow_trailing parameter
|
|
# This file is part of the rsyslog project, released under ASL 2.0
|
|
|
|
. ${srcdir:=.}/diag.sh init
|
|
generate_conf
|
|
add_conf '
|
|
module(load="../plugins/mmjsonparse/.libs/mmjsonparse")
|
|
|
|
template(name="outfmt" type="string" string="%msg% parsesuccess=%parsesuccess% json=%$!%\n")
|
|
|
|
# Test allow_trailing=on (default)
|
|
if $msg contains "TRAILING_ON" then {
|
|
action(type="mmjsonparse" mode="find-json" allow_trailing="on")
|
|
action(type="omfile" file=`echo $RSYSLOG_OUT_LOG` template="outfmt")
|
|
stop
|
|
}
|
|
|
|
# Test allow_trailing=off
|
|
if $msg contains "TRAILING_OFF" then {
|
|
action(type="mmjsonparse" mode="find-json" allow_trailing="off")
|
|
action(type="omfile" file=`echo $RSYSLOG_OUT_LOG` template="outfmt")
|
|
stop
|
|
}
|
|
'
|
|
startup
|
|
injectmsg_literal '<167>Jan 16 16:57:54 host.example.net TAG: TRAILING_ON {"test":1} garbage after'
|
|
injectmsg_literal '<167>Jan 16 16:57:54 host.example.net TAG: TRAILING_OFF {"test":2} garbage after'
|
|
shutdown_when_empty
|
|
wait_shutdown
|
|
|
|
export EXPECTED=' TRAILING_ON {"test":1} garbage after parsesuccess=OK json={ "test": 1 }
|
|
TRAILING_OFF {"test":2} garbage after parsesuccess=FAIL json={ "msg": " TRAILING_OFF {\"test\":2} garbage after" }'
|
|
cmp_exact
|
|
exit_test |