Real-world need: unflatten dotted JSON into nested objects, and optionally
flatten back for downstream tools. This introduces a general transformer
with a dedicated output tree. Interface is intentionally unstable.
Impact: New module behind --enable-mmjsontransform; no default behavior
changes. New tests and docs added. Parameters and behavior may change.
Add mmjsontransform, a message modification module that rewrites dotted
JSON keys. By default it "unflattens" an input object to nested containers
and stores the result in a configured output property. A mode parameter
also supports "flatten" to collapse nested trees into dotted keys. The
action refuses to overwrite an existing destination, validates that input
is a JSON object, and reports conflicts with precise key paths. Per-action
config is immutable; workers hold pointers only, so no extra locking. Docs
(Sphinx + parameter refs) and doxygen coverage included, plus a regression
test exercising nested arrays/objects. Build system and CI scripts gain
--enable-mmjsontransform and a basic test hook. An experimental companion
mmjsonrewrite module is wired similarly for dotted-key expansion.
Before/After: Previously no built-in JSON un/flatten; now an action can
unflatten (default) or flatten JSON into a separate message property.
With the help of AI Agents: ChatGPT codex, gemini
kafkacat has been renamed to kcat, as it looks for trademark issues.
Kafka module tests depend on that utility and are skipped if command
is not available. This was now always the case for newer development
containers.
Name was now adjusted in all tests.
- uses http library to provide http input.
user would need to configure an 'endpoint' as input, along
with a ruleset, defining how the input should be routed in
rsyslog.
bugfix, free dynamic buf if created, don't echo to client data.
do data framing using newline character.
more input options support:
- input name
- flowcontrol
add support for gzip content support, and other options
- gzip content
- parse linefeeds by default, option to ignore linefeeds
add support for gzip content support, and other options
- gzip content
- parse linefeeds by default, option to ignore linefeeds
WIP - misc edits enable port, documentroot module parameter
fix overloaded stack issue
update tests to use available port instead of hard-coded one.
Add 'octet counted framing' support
- option is "SupportOctetCountedFraming", currently 'off' by
default.
update imhttp-getrequest-file.sh test to use $srcdir
imhttp - support multi-threaded connection contexts.
- tests for large data posts
- Add header data as metadata option
move mg_start into activatecnf instead of in runinput
- plugin will get (docker) container logs from a host as well as filling out some
basic container metadata as id, name, image, labels.
- requirements: curl-7.40.0+, for unix_domain_socket option (Docker API listens on a
local socket). Additionally, plugin will handle the issue of log lines larger than
16KB begin split by Docker.
- include imdocker-unittests in testbench
- enable imdocker tests in osx travis tests
- use curl master branch for imdocker-tests
- fix cflags for pthread
- enforce minimum curl version during configuration - to 7.40.0 for imdocker
- container polls after the initial one are automatically filtered utilizing the 'since' option as per docker api
- imdocker tests refactored.
- add escapeLF config option
Test bench test added:
- imdocker-basic: checks for completeness.
- imdocker-basic-vg: same as basic with valgrind enabled.
- imdocker-long-logline: checks imdocker can handle 16K+ log lines.
- imdocker-long-logline-vg: same as long-logline test with valgrind enabled.
- add option 'retrieveNewLogsFromStart' which will cause imdocker to ignore the 'tail'
option when retrieving container logs for newly activated containers. Containers that
were already active when imdocker first starts still uses the tail option.
- imdocker-new-logs-from-start: checks that new containers will ignore the tail option
- imdocker-new-logs-from-start-vg: valgrind enabled
- multi-line support via docker label: imdocker.startregex
added multi-line tests for testbench
Travis test bench related history and notes:
- Travis tests, enable imdocker
- imdocker travis tests not enabled in precise or trusty ubuntu versions of travis-test.
Assuming travis tests will run in ubuntu 16
- tests: harden container name against strange file system pathes
RSYSLOG_DYNNAME contains the file system path, which is NOT guarnateed
to be suitable for things other than file names.
- tests: bugfix and guard against false negative
- content_check_with count was done before rsyslog was terminated, so
it was uncertain how many messages were written at that spot
(especially on slow machines)
- so far, only the number of lines was checked, not their content. So
any problems with the actual messages would not have been detected.
- improved support for listcontainer options, fix racy imdocker tests
The configure/Makefile checks were not correct, leading to the
build of journal components when not necessary, even if not
supported by the platform. Thus lead to invald build and test
failures. This commit fixes that.
This often causes trouble when the packages are rebuild by the 0mq project
(which happens frequently). We already do intensive testing of the 0mq
components in the buildbot infrastructure, where we use dedicated containers.
This is reliable, as the containers already contain everything needed and so
do not need to reach out to the 0mq package archives. In the light of this,
let's save us the trouble of Travis failures. The only downside is that
users cannot pre-test with their local Travis when modifying 0mq modules,
which is quite acceptable.
* add an overall timeout value for tests - if running longer,
testbench framework tries to FAIL and end test. Note that
this is not bullet-proof and not intended to be so.
* guard against hanging rsyslog instances via a new imdiag
feature to abort after n number of seconds; among others,
this guards as against timeout-cancel in CI, which is always
pretty hard to diagnose - now we see these errors in test-suite.log
* fix a bug in tcp zip test, which actually did not use zip mode
* experimnentalls add debug output to better understand
shutdown_when_empty operation; goal is to improve understanding
and then remove that code again.
* improve shutdown predicate for a couple of tests
* made travis run make check with two parallel threads, for which
we seem ready now. Nevertheless, it's still experimental and we
may roll this back if required.
Fixed some issues with sndrcv kafka tests.
Generating kafka topics dynamically now it kafka tests.
Limited messagecount in some tests to 50000 for now.
We already know they are currently unstable, so there is not point
in messing up CI result each time. The test will still be carried out
in nightly checks, so the problems won't be forgotten.
We run only some distcheck test that we cannot yet run inside
the containers on travis. This reduces the amount of redundant
work done, speeding up Travis runtime.
Any missing checks are still detected by buildbot part of CI.
Also a couple of changes to testbench worth mentioning:
* use cp -f to ensure files can be overwirtten in VBUILD
* fix issue of missing include test file in EXTRA_DIST
* new supressions
* testbench: try to use local system dependency cache
avoid going to Internet repos if not absolutely necessary. For
development containers, they should be pre-populated with the
important dependencies.
* do not enable libfaketime if ASAN is selected
unfortunately, libfaketime does not work in that case
see also https://github.com/rsyslog/rsyslog/issues/174
This makes the ES tests more universally available (they should now
also support running in containers). It also simplifies the tests
as fewer support files are needed.
This is necessary as we hit the Travis max runtime limit per VM,
so we need to duplicate the tests. This is done via
--enable-testbench1 and --enable-testbench2 which we than use
in different VMs.
This PR also includes updates to the Travis scripts so that we
use the new capability. We have only duplicated Travis VMs where
acutally necessary -- we may need to do more of this in the future.
closes https://github.com/rsyslog/rsyslog/issues/2196
The PostgreSQL output module was woefully out-of-date the following
list is changes made to update the module to current Rsyslog standards.
- allow for v6 configuration syntax
- configurable ports
- support transactional interface
- push db connection into workers (libpq is threadsafe)
- enable module testing on travis
- ensure configuration syntax backwards compatibility
- formatting around postgres core templating
- use new test conventions
- add new configuration syntax test
- add valgrind tests for new and old syntax
- add threading tests
- add action queue long running tests
- add action queue valgrind test
according to Brian Knox they require an outdated version of the client
lib. So we do not bother any longer about them. After all, they are
scheduled to be removed in 2018-01.
the very new version introduced new warning compiler flags, which require
some work to fix build issues. We do not want to hold the rest of the
useful work in this commit set just for that. So we temporarily disable
the new package and will work in parallel to make things build cleanly
again.
We execute different code pathes under cron jobs. It doesn't make sense
to redo what we already did before a merge. Right now, cron builds a
Coverity scan submission and uploads it. This should be set to run
once a day.
closes https://github.com/rsyslog/rsyslog/issues/2037
